Andrei Lukovenko
2014-05-28 08:32:07 UTC
Hi,
Consider the following configuration:
* master A (slaveof no one)
* slave B (slaveof master-A-ip master-A-port)
* slave C (slaveof master-A-ip master-A-port)
* Sentinel (quorum=1), considers A as the master
If master A fails, Sentinel promotes one of the slaves, and change
configs accordingly. So the configuration becomes:
* master A (down, slaveof no one)
* master B (slaveof no one)
* slave C (slaveof master-B-ip master-B-port)
* Sentinel (quorum=1), considers B as the master
Now for some reason Sentinel goes down and restarts. As sentinel.conf has
not been rewritten, Sentinel still thinks about host A being the master. As
host A is down, the system becomes unresponsive, and our system
administrator recovers host A:
* master A (slaveof no one)
* master B (slaveof no one)
* slave C (slaveof master-B-ip master-B-port)
* Sentinel (quorum=1), considers A as the master
It is a very possible case of split brain, and currently there is no
straight way do avoid it. I have considered the following workarounds:
a) Restoring redis.conf to the original state before restarting an
instance. Not good, as we lose all the benefits of rewriting it.
b) Manually resolving these conflicts.
In my opinion, it would've been better if either:
a) We could explicitly describe our network configuration, including slaves
in sentinel.conf. Then after restarting a sentinel it would turn B and C to
slaves of A.
b) Sentinel would rewrite sentinel.conf after changing configuration. In
this example, after promoting B to master and changing B's and C's config,
it would also rewrite it's own config to consider B as the master.
What do you think?
Consider the following configuration:
* master A (slaveof no one)
* slave B (slaveof master-A-ip master-A-port)
* slave C (slaveof master-A-ip master-A-port)
* Sentinel (quorum=1), considers A as the master
If master A fails, Sentinel promotes one of the slaves, and change
configs accordingly. So the configuration becomes:
* master A (down, slaveof no one)
* master B (slaveof no one)
* slave C (slaveof master-B-ip master-B-port)
* Sentinel (quorum=1), considers B as the master
Now for some reason Sentinel goes down and restarts. As sentinel.conf has
not been rewritten, Sentinel still thinks about host A being the master. As
host A is down, the system becomes unresponsive, and our system
administrator recovers host A:
* master A (slaveof no one)
* master B (slaveof no one)
* slave C (slaveof master-B-ip master-B-port)
* Sentinel (quorum=1), considers A as the master
It is a very possible case of split brain, and currently there is no
straight way do avoid it. I have considered the following workarounds:
a) Restoring redis.conf to the original state before restarting an
instance. Not good, as we lose all the benefits of rewriting it.
b) Manually resolving these conflicts.
In my opinion, it would've been better if either:
a) We could explicitly describe our network configuration, including slaves
in sentinel.conf. Then after restarting a sentinel it would turn B and C to
slaves of A.
b) Sentinel would rewrite sentinel.conf after changing configuration. In
this example, after promoting B to master and changing B's and C's config,
it would also rewrite it's own config to consider B as the master.
What do you think?
--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To post to this group, send email to redis-db-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To post to this group, send email to redis-db-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.