I have created a Storm topology which connects to Redis-cluster using Jedis library. Storm component always expects that Redis is up and running and only then it connects to Redis and subscribes the events.Currently we use pub-sub strategy of Redis.
Below is the code sample that explains my Jedis Connectivity inside Storm to for Redis.
try { jedis.psubscribe(listener, pattern); } catch(Exception ex) { //catch statement here. } finally { pool.returnResource(jedis); } .... pool = new JedisPool(new JedisPoolConfig(), host, port); //redis host port ListenerThread listener = new ListenerThread(queue, pool, pattern); listener.start(); EXPECTED BEHAVIOUR
Once Redis dies and comes back online, Storm is expected to identify the status of Redis. It must not need a restart in case when Redis die and come online.
ACTUAL BEHAVIOUR
Once Redis restarts due to any reason, I always have to restart the Storm topology as well and only then it starts listening back to Redis.
QUESTION
How can I make Storm listen and reconnect to Redis again after Redis is restarted? any guidance would be appreciated, viz. docs, forum answer.
1 Answers
Answers 1
This is a common issue with apache-storm where connection thread is alivein stale condition, although the source from where you are consuming is down/restarted. Ideally it should retry to create new connection thread instead reusing the existing one. Hence the Idea is to have it automated it by by detecting the Exception (e.g. JMSConnectionError in case of JMS).
refer this Failover Consumer Example which will give you brief idea what to do in such cases.(P.S this is JMS which would be JMS in redis your case.)
The Steps would be something like this.
- Catch Exception in case of ERROR or connection lost.
- Init connection (if not voluntarily closed by program) from catch.
- If Exception got to step 1.
0 comments:
Post a Comment