[Bucardo-general] Swap replication latency
Alex Balashov
abalashov at evaristesys.com
Mon May 9 03:15:36 UTC 2011
Anyone have any insights here?
I know this is an open-source project alongside a commercial endeavour, and thus do not wish to presume either upon your time or generosity in the least.
However, I really am at a dead end here. I don't know what else to try. Whenever statement volume goes beyond 2-3 inserts or updates/sec (average), Bucardo can't keep up.
--
Alex Balashov - Principal
Evariste Systems LLC
260 Peachtree Street NW
Suite 2200
Atlanta, GA 30303
Tel: +1-678-954-0670
Fax: +1-404-961-1892
Web: http://www.evaristesys.com/
On May 5, 2011, at 9:30 PM, Alex Balashov <abalashov at evaristesys.com> wrote:
> P.S. I think the reason it can't catch up at a normal pace is because
> it takes me about ~5 minutes to add the table to the sync, run the 'add
> sync' command (which entails a validate_sync()), and then start it.
>
> During that time, as many as several thousand net deltas happen, and so
> in order to catch up Bucardo would have to operate at a transactional
> throughput that is fast enough to both catch up to the changes
> accumulated during the sync reinitialisation + stay on top of new ones
> at the same pace. Evidently it cannot do that, at least as tuned by
> default.
>
> So, is there a way I can turbocharge it?
>
> On 05/05/2011 09:27 PM, Alex Balashov wrote:
>
>> I've troubleshot this some more on our side by process of elimination of
>> synced tables, and what I found is that the sync time trends back toward
>> a normal 5-6 sec when all tables that are modified at a high volume are
>> excluded.
>>
>> As a test, I added one of the high-volume tables, which was doing
>> roughly ~2000 UPDATEs every 10 minutes, and maybe 300 INSERTs during the
>> same time frame. Bucardo fell way behind to a 11+ minute sync run time,
>> and the delta table started blowing up.
>>
>> These statements do not take any appreciably long time to run. Yeah,
>> they're not the fastest thing ever, but the point is that both database
>> servers can easily handle the load. So the bottleneck really is
>> Bucardo, not the target database server; the hardware is identical, and
>> the target server's load average constantly hangs around ~0.01, and the
>> source database server isn't exactly breaking a sweat either. We have
>> Postgres set to log queries over 15 ms and are getting hardly any
>> complaints.
>>
>> Thus, I can only come to the conclusion that Bucardo is not spinning
>> nearly hard enough.
>>
>> So the question is, what--if any--parameters can I tweak to make
>> Bucardo's transactional throughput faster? Preferably several times
>> faster. Is it even possible, or is this basically a low-speed
>> replication system and there is not much that can be done here?
>>
>> Thanks,
>>
>
>
> --
> Alex Balashov - Principal
> Evariste Systems LLC
> 260 Peachtree Street NW
> Suite 2200
> Atlanta, GA 30303
> Tel: +1-678-954-0670
> Fax: +1-404-961-1892
> Web: http://www.evaristesys.com/
> _______________________________________________
> Bucardo-general mailing list
> Bucardo-general at bucardo.org
> https://mail.endcrypt.com/mailman/listinfo/bucardo-general
More information about the Bucardo-general
mailing list