[Bucardo-general] Bucardo design question

[Karl Denninger]

Michelle Sullivan wrote:
> Greg,
>
> I'm really running into problems getting this to catch up - the time to
> gather the data for the sync is increasing to unworkable levels.  The
> problems are not due to Bucardo but my knowledge about the cleanup crons
> , and a couple of hardware issues.  These have been solved and I need
> for this to catchup really quick.  Can you explain (very quickly) how
> Bucardo knows about transactions....
>
> Specifically I have writers in my app that will 'BEGIN' then insert or
> update several tables before 'COMMIT'ing...  Does Bucardo see the update
> of all the tables as a single transaction?
>
> Does Bucardo know about the foreign keys so that it only inserts rows in
> the correct order?
>
> How does Bucardo choose what to sync from each table, point in time (ie
> now) or to the last transaction?  I'm happy to trawl the code if you
> don't have time, but would it be possible to limit the number of
> transactions in the sync (of course my previous questions could answer
> that question for me ;-) ).... I just need to know whether I'd be
> wasting my time as I have 4 days to get this system into production -
> which means bucardo has to catchup.  I have already found (and solved)
> the reason it got out of sync in the first place, but it's taking around
> 1-2 days to start writing transactions to the other side of the swap
> sync due to the huge number and the fact the sync process uses all
> available ram and an extra 16G of swap...! ;-)  I suspect limiting
> (temporarily) to updating 100k transactions at a time would speed this
> process up immensely.
>
>
> Thanks in advance,
>
> Michelle
Slony can handle this by forcing a transaction separator - you can stick
that into the CRON every 20 minutes or so, which keeps this problem from
coming up.

That being "missing" on Bucardo is a BIG problem IMHO, for exactly the
reason you're running into.

-- Karl
-------------- next part --------------
A non-text attachment was scrubbed...
Name: karl.vcf
Type: text/x-vcard
Size: 124 bytes
Desc: not available
Url : https://mail.endcrypt.com/pipermail/bucardo-general/attachments/20100803/b5b7783e/attachment.vcf