I'm using the excellent Redis based Bull.js as a job queue on Kubernetes.
It's configured as a cluster:
When Kubernetes restarts upon deployments, I run into this following error:
BRPOPLPUSH { ReplyError: MOVED 2651 <IP_ADDRESS>:6379
at parseError (/usr/src/app/node_modules/ioredis/node_modules/redis-parser/lib/parser.js:179:12)
at parseType (/usr/src/app/node_modules/ioredis/node_modules/redis-parser/lib/parser.js:302:14)
command:
{ name: 'brpoplpush',
args:
[ '{slack}:slack notifications:wait',
'{slack}:slack notifications:active',
'5' ] } }
Where <IP_ADDRESS>
is, I think the cluster IP? I didn't configure this, but I'm trying to debug this. I want to know if I need to enable cluster mode for Bull.js or if this is a configuration issue outside of the Bull.js project?
Or is it a networking issue with K8s?
Would enabling: https://github.com/OptimalBits/bull#cluster-support be the solution? Is this the right approach?
Here is my code:
import Queue from 'bull';
import config from 'config';
import { run as slackRun } from './tasks/send-slack-message';
import { run as emailRun } from './tasks/send-email';
const redisConfig = {
redis: {
host: config.redis.host,
port: config.redis.port
}
};
const slackQueue = new Queue('slack notifications', { ...redisConfig, ...{ prefix: '{slack}' } });
const emailQueue = new Queue('email notifications', { ...redisConfig, ...{ prefix: '{email}' } });
slackQueue.process(slackRun);
emailQueue.process(emailRun);
emailQueue.on('completed', (job, result) => {
job.remove();
});
export { emailQueue, slackQueue };
import { emailQueue, slackQueue } from 'worker/worker';
const queueOptions = {
attempts: 2,
removeOnComplete: true,
backoff: {
type: 'exponential',
delay: 60 * 1000
}
};
emailQueue.add(
{
params: {
from: email,
fromname: name,
text: body
}
},
queueOptions
);
slackQueue.add(
{
channelId: SLACK_CHANNELS.FEEDBACK,
attachments: [
{
text: req.body.body
}
]
},
queueOptions
);
This is the configmap:
Name: redis-cluster-config
Namespace: default
Labels: <none>
Annotations: <none>
Data
====
update-node.sh:
----
#!/bin/sh
REDIS_NODES="/data/nodes.conf"
sed -i -e "/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${POD_IP}/" ${REDIS_NODES}
exec "$@"
redis.conf:
----
cluster-enabled yes
cluster-require-full-coverage no
cluster-node-timeout 15000
cluster-config-file nodes.conf
cluster-migration-barrier 1
appendonly yes
# Other cluster members need to be able to connect
protected-mode no
Events: <none>
Hitobat is right though.
If this doesn't help:
const Redis = require('ioredis');
...
const ioCluster = new Redis.Cluster([redisConfig.redis]);
const slackQueue = new Queue('slack notifications', {
prefix: '{slack}' ,
createClient: () => ioCluster
});
const emailQueue = new Queue('email notifications', {
prefix: '{email}' ,
createClient: () => ioCluster
});
I would go without ioredis or try to downgrade redis engine to 4.x.