[Bucardo-general] feedback from testing Bucardo for enterprise use
nick at farrell.name
Sat Mar 27 07:17:00 UTC 2010
I'm resending this email, as the mailing list didn't like the HTML
tags in my earlier version:
I've done some more experimenting with Bucardo, and am quite happy
with it, but I've found some limitations. I'd be happy for people to
correct me if any of these things are just due to my not understanding
the documentation, or other silly mistakes.
It's also worth mentioning my configuration: I have one central
cluster, and multiple remote clusters. I have a 'content' table which
is being replicated outwards to each remote cluster, and I have a
'member' table which is replicated back from each remote cluster to
the centre, ensuring that the primary keys of the members do not
conflict. The python code to generate this (on Ubuntu 9.04, with
postgresql 8.3) is here. This creates multiple clusters on a single
Problem 1: When a remote cluster becomes unavailable, other syncs fail.
If I create a single sync which contains all the remote databases, and
one database becomes unavailable, none of the remaining databases
receive any more data. While disappointing, I understand that this
might be the price I need to pay for grouping all my remote databases
into one dbgroup, and only needing to run one trigger per row.
The bigger problem is that I still get a similar error, even when I
create a separate sync between the central database and each remote
database. I would have expected that a failure to process one sync
would not prevent other syncs from running, but this is indeed the
case. I end up seeing something like this:
MCP Warning: Killed (line 44): DBI
connect('dbname=replication;port=5436','bucardo',...) failed: could
not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket
/usr/local/lib/perl/5.10.0/Bucardo.pm line 266
[Fri Mar 26 16:13:55 2010] MCP Database problem, will respawn after a
short sleep: 15
which repeats until I turn the offending cluster back on.
This is a deal-breaker for me, and I imagine any other serious users
of this software. If Bucardo.pm could handle these errors more
gracefully, and when one sync failed move on to the next, it would
make a big difference.
Problem 2: The more syncs a table is involved in, the longer any
transactions on it take.
Every sync adds an additional trigger to each table it's watching. If
I have a dozen remote databases, these 12 triggers add noticable
overhead to the times required to modify each record. Could the delta
table simply be touched once for each record? This would speed up all
updates on these tables, reduce the delta table size, and speed up
delta table lookups, done during the syncs.
Problem 3: Large sync operations take a lot of memory
If a remote database is unavailable for extended periods, the amount
of data to transfer will get pretty big. It appears that Bucardo
doesn't pass along the data in chunks, but buffers it all in memory.
For example, when I disable one of two clusters, write ~1.5m rows to
my central database, let the delta triggers settle, then re-enable a
remote cluster, I see:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1468 root 20 0 815m 634m 1172 S 3.0 31.9 1:03.66 bucardo_ctl
1466 root 20 0 815m 634m 1172 S 5.0 31.9 1:03.57 bucardo_ctl
This doesn't make much sense to me, and again prevents enterprise use.
Looking at the DBD::Pg man pages, pg_getcopydata and pg_putcopydata
work a line at a time, so there shouldn't be any problem with COPYing
the records across in multiple passes. Much less peak memory use.
I'd be happy to hear from the devs, as to whether these are known
problems, and how realistic it would be to expect future releases to
address these bugs. While I could dabble in the code myself, I'm not a
big perl person and wouldn't do the job as cleanly.
More information about the Bucardo-general