[Bucardo-general] Swap replication latency

Alex Balashov abalashov at evaristesys.com
Mon May 9 03:15:36 UTC 2011


Anyone have any insights here?  

I know this is an open-source project alongside a commercial endeavour, and thus do not wish to presume either upon your time or generosity in the least.

However, I really am at a dead end here.  I don't know what else to try.  Whenever statement volume goes beyond 2-3 inserts or updates/sec (average), Bucardo can't keep up.

--
Alex Balashov - Principal
Evariste Systems LLC
260 Peachtree Street NW
Suite 2200
Atlanta, GA 30303
Tel: +1-678-954-0670
Fax: +1-404-961-1892
Web: http://www.evaristesys.com/

On May 5, 2011, at 9:30 PM, Alex Balashov <abalashov at evaristesys.com> wrote:

> P.S.   I think the reason it can't catch up at a normal pace is because 
> it takes me about ~5 minutes to add the table to the sync, run the 'add 
> sync' command (which entails a validate_sync()), and then start it.
> 
> During that time, as many as several thousand net deltas happen, and so 
> in order to catch up Bucardo would have to operate at a transactional 
> throughput that is fast enough to both catch up to the changes 
> accumulated during the sync reinitialisation + stay on top of new ones 
> at the same pace.  Evidently it cannot do that, at least as tuned by 
> default.
> 
> So, is there a way I can turbocharge it?
> 
> On 05/05/2011 09:27 PM, Alex Balashov wrote:
> 
>> I've troubleshot this some more on our side by process of elimination of
>> synced tables, and what I found is that the sync time trends back toward
>> a normal 5-6 sec when all tables that are modified at a high volume are
>> excluded.
>> 
>> As a test, I added one of the high-volume tables, which was doing
>> roughly ~2000 UPDATEs every 10 minutes, and maybe 300 INSERTs during the
>> same time frame.  Bucardo fell way behind to a 11+ minute sync run time,
>> and the delta table started blowing up.
>> 
>> These statements do not take any appreciably long time to run.  Yeah,
>> they're not the fastest thing ever, but the point is that both database
>> servers can easily handle the load.  So the bottleneck really is
>> Bucardo, not the target database server;  the hardware is identical, and
>> the target server's load average constantly hangs around ~0.01, and the
>> source database server isn't exactly breaking a sweat either.  We have
>> Postgres set to log queries over 15 ms and are getting hardly any
>> complaints.
>> 
>> Thus, I can only come to the conclusion that Bucardo is not spinning
>> nearly hard enough.
>> 
>> So the question is, what--if any--parameters can I tweak to make
>> Bucardo's transactional throughput faster?  Preferably several times
>> faster.  Is it even possible, or is this basically a low-speed
>> replication system and there is not much that can be done here?
>> 
>> Thanks,
>> 
> 
> 
> -- 
> Alex Balashov - Principal
> Evariste Systems LLC
> 260 Peachtree Street NW
> Suite 2200
> Atlanta, GA 30303
> Tel: +1-678-954-0670
> Fax: +1-404-961-1892
> Web: http://www.evaristesys.com/
> _______________________________________________
> Bucardo-general mailing list
> Bucardo-general at bucardo.org
> https://mail.endcrypt.com/mailman/listinfo/bucardo-general


More information about the Bucardo-general mailing list