[Bucardo-general] Automatic Cascading Replication

[Greg Sabino Mullane]

Just a heads up, there is a lot of conversation at times on 
the #bucardo IRC channel on freenode, so make sure you are 
visiting there for some interesting (but low volume) traffic!

> First, I'm not sure that tracking should be on a dbgroup-basis, 
> but on a db-basis. This would allow one to have two multi-master 
> replication setups on a table and yet have the data in question 
> replicated by only one of those syncs. (Or might that create a 
> race-condition such that they would probably both copy the data anyway?)

I don't think this will work out. If we did that, we'd have some problems:

* Our main 'get deltas' query would no longer work, but would have to be a 
handful of queries, per database. Additionally, it would add a lot of 
complexity to push_rows. Because each DB might demand a different select 
of distinct rows from bucardo_delta, we would also have to scan the 
source database(s) multiple times, worse case scenario, once per 
target DB. Ugh.

* Since the data should already be in a logically self-contained set, 
it should never be the case that we save anything by letting some other 
sync do the work for us, because it's either going to violate our 
existing assumptions of which tables should get replicated as a group, 
or be the same as us in which case we don't win anything.

* If we really want to avoid over-copying, there are probably better 
approaches, but I am not convinced this is much of a problem, as 
makedelta is 99% of the time used to replicate in a non-overlapping 
target fashion.

> As an aside: Ideally, when a table is being replicated with makedelta enabled, 
> the deltas would be created, but the autokick, if there is one, is *not* fired. 
> Any other sync autokicks would be.

Not sure that this means - do you mean not firing the autokick of the current 
sync? Right now the kid ignores its own messages, but ideally we would train 
the MCP to ignore such notices if they come from one of its (grandkids') PIDs.
Certainly this is a TODO. We could probably simply have the KID send the 
MCP a quick NOTIFY on startup. Could also go the other way, and have the information 
about what PID did the kick bubble down to the CTL and then the KID, but 
that would be trickier.

> * Replace "makedelta" with "cascade" and default to "auto"
> * Let the triggers do the work as much as possible

I'm okay with the automatic searching, although we would have to be careful to 
check that the sync is active, as well as some other edge cases. However, 
I've been thinking hard about the "always" triggers mentioned mostly 
on channel (the idea being we set delta to 'always' and thus we can simply 
rely on the trigger populating things rather than doing manual inserts 
in the code for makedeltas). I really don't think that will work, as 'always' 
triggers are just too dangerous. I'd much rather put the onus on Bucardo itself 
to emulate the trigger action, rather than violating the principle of least 
surprise by having 'SET session_replication_role = replica' NOT disable the 
Bucardo triggers. Even if we added some trickery to have them not do the inserts 
via a GUC setting or the like, just having them get called adds a lot of 
overhead for bulk loading, one of the main reasons people "turn off" the 
Bucardo delta triggers. If it was 'always', the only way to truly avoid 
them would be to do an ALTER TABLE which we really, really want to avoid 
as that is some heavy locking.

It's a shame that session_replication_role is as limited as it is, as having 
a little more discretion would be nice (e.g. setting a trigger to always fire 
unless session_replication_role X is enabled, but not on normal 'replica'). 
The current origin/local settings are too weak and advisory to really matter 
here. Something to perhaps iron out and suggest to -hackers someday.

-- 
Greg Sabino Mullane greg at endpoint.com
End Point Corporation
PGP Key: 0x14964AC8
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 163 bytes
Desc: not available
URL: <https://mail.endcrypt.com/pipermail/bucardo-general/attachments/20121213/68ed82f3/attachment.sig>