[Bucardo-general] Best practice for failover

[Greg Sabino Mullane]

Looks like this may have been partially answered on IRC, but 
for the archives I'll attempt as well:

> I am currently investigating a replication scenario for a client. This
> scenario involves two master servers (m1, m2) and n slaves (currently
> only one, maybe 2 or 3 later on). The bucardo database is also hosted on m1.
> 
> At the moment, there is a swap sync between m1 and m2 and a pushdelta
> between m1 and s1 and another one between m2 and s1. The whole setup is
> done via a shell script I wrote, so all information is available outside
> of the bucardo database as well.

Why are there two pushdelta syncs? If m1 and m2 are swaps, it should be 
sufficient to have a single pushdelta from m1 or m2.

> If m2 goes down, most scenarios are clear, there is no impact, as the
> bucardo database is on m1 and bucardo runs on m1 as well. When m2 comes
> up again, it should catch up when I kick the sync(s).

Generally correct. Bucardo will complain loudly about m2 not being reachable, 
but should be happy once it comes back.

> If m2 goes down permanently, I'll remove all syncs, databases, herds on
> it, set up a new m2 with the empty database structures (this is fast and
> already working). Then I set up a new replication and afterwards I
> a) either sync via bucardo doing a onetimecopy?
> b) sync via pg_dump|psql?
> 
> What is the common procedure here? What are you doing?

Create a swap sync from m1 to m2, but don't turn it on (but make sure the 
triggers, etc. are created). This will make sure all the triggers are in 
place. Change the type from swap to pushdelta, and set onetimecopy. 
Restart bucardo, kick the sync. Switch it back to swap, restart Bucardo, 
and you should be good.

> It gets more complicated when m1 goes down. What happens to the
> replication between m2 and the slaves while m1 is down? What happens
> when m1 comes up again?

There will be no replication between m2 and slaves, as the Bucardo daemon 
won't be around to do the work.

> Does it make sense to replicate the bucardo database as well? Or should
> I set it up via our scripts (done in a few seconds) to keep m2 and the
> slaves in sync.

You can replicate the Bucardo database, but it generally changes its 
core information so infrequently and it's so very very small I recommend 
simply making a pg_dump of it. Thus, if m1 goes down, you do the same steps 
as above, but restore the bucardo database first on m1 or m2 (or anywhere, 
technically).

-- 
Greg Sabino Mullane greg at endpoint.com
End Point Corporation
PGP Key: 0x14964AC8