[Bucardo-general] please help with this.

Mon Feb 1 07:37:09 UTC 2010

Hi folks

Anyone got more for this for me ? This is crucial not just for me but for anyone who has significant numbers of unique constraint violations that need to be dealt with. You can see from the log snippet below that we get into a loop of deleting 180000 (approx) records (based on 10,000 per chunk), copying 170000 + some, discovering a conflict, running the custom code to resolve the conflict and then starting the entire set  all over again, only to find another conflict and so starting again all over.

Is this a design flaw ? All of the records are being prevented from replication by individual events. Surely the majority of the records should replicate. This would happen correctly if each chunk was deleted and copied as a transaction. some chunks would fail and need correction but it would not mean failing every chunk and would not mean copying the entire record set again.

please please please look at this.

regards
Grant

What I am seeing all day in the log is

Feb  1 19:21:21 console Bucardo[16368]: KID Deleting chunk 4 
Feb  1 19:21:21 console Bucardo[16368]: KID Deleting chunk 5 
Feb  1 19:21:22 console Bucardo[16368]: KID Deleting chunk 6 
Feb  1 19:21:22 console Bucardo[16368]: KID Deleting chunk 7 
Feb  1 19:21:22 console Bucardo[16368]: KID Deleting chunk 8 
Feb  1 19:21:22 console Bucardo[16368]: KID Deleting chunk 9 
Feb  1 19:21:22 console Bucardo[16368]: KID Deleting chunk 10 
Feb  1 19:21:22 console Bucardo[16368]: KID Deleting chunk 11 
Feb  1 19:21:22 console Bucardo[16368]: KID Deleting chunk 12 
Feb  1 19:21:22 console Bucardo[16368]: KID Deleting chunk 13 
Feb  1 19:21:23 console Bucardo[16368]: KID Deleting chunk 14 
Feb  1 19:21:23 console Bucardo[16368]: KID Deleting chunk 15 
Feb  1 19:21:23 console Bucardo[16368]: KID Deleting chunk 16 
Feb  1 19:21:23 console Bucardo[16368]: KID Deleting chunk 17 
Feb  1 19:21:23 console Bucardo[16368]: KID Deleting chunk 18 
Feb  1 19:21:23 console Bucardo[16368]: KID Begin COPY to public.tblmailaddresslist 
Feb  1 19:21:24 console Bucardo[16368]: KID Copying chunk 1 from t2 to messagestore 
Feb  1 19:21:28 console Bucardo[16368]: KID Copying chunk 2 from t2 to messagestore 
Feb  1 19:21:32 console Bucardo[16368]: KID Copying chunk 3 from t2 to messagestore 
Feb  1 19:21:37 console Bucardo[16368]: KID Copying chunk 4 from t2 to messagestore 
Feb  1 19:21:42 console Bucardo[16368]: KID Copying chunk 5 from t2 to messagestore 
.......
Feb  1 19:22:37 console Bucardo[16368]: KID Copying chunk 18 from t2 to messagestore 
Feb  1 19:22:38 console Bucardo[16368]: KID Exception caught: DBD::Pg::db pg_putcopyend failed: ERROR:  duplicate key value violates unique constraint "tblmailaddresslist_emailaddress_key"\nCONTEXT:  COPY tblmailaddresslist, line 2: "1281808^Ikellyvenhuizen at hotmail.com^I2009-12-03 18:04:35.927361+11^If^I4^I0^I0^I0" at /usr/local/share/perl/5.10.0/Bucardo.pm line 4925. 
Feb  1 19:22:38 console Bucardo[16368]: KID Rolling back to target savepoint, due to database error: DBD::Pg::db pg_putcopyend failed: ERROR:  duplicate key value violates unique constraint "tblmailaddresslist_emailaddress_key"\nCONTEXT:  COPY tblmailaddresslist, line 2: "1281808^Ikellyvenhuizen at hotmail.com^I2009-12-03 18:04:35.927361+11^If^I4^I0^I0^I0" at /usr/local/share/perl/5.10.0/Bucardo.pm line 4925. 
Feb  1 19:22:38 console Bucardo[16368]: KID ELS mod set rowinfo pkey to 1281808 
Feb  1 19:22:38 console Bucardo[16368]: KID Trying exception code 1: mailaddress_in 
Feb  1 19:22:38 console Bucardo[16368]: KID Running exception custom code 1: mailaddress_in 
Feb  1 19:22:38 console Bucardo[16368]: KID Finished custom code 1 
Feb  1 19:22:38 console Bucardo[16368]: KID Message from exception code 1: ELS updating from db: t2 pkmailid 1281808 to 1533518 for addresskellyvenhuizen at hotmail.com 
Feb  1 19:22:38 console Bucardo[16368]: KID Warning! Code exception 1: UPDATE affected 1 rows 
Feb  1 19:22:38 console Bucardo[16368]: KID Exception handler thinks we can try again 
Feb  1 19:22:47 console Bucardo[16368]: KID Deleting rows from public.tblmailaddresslist 
Feb  1 19:22:48 console Bucardo[16368]: KID Deleting chunk 1 
Feb  1 19:22:48 console Bucardo[16368]: KID Deleting chunk 2 
Feb  1 19:22:48 console Bucardo[16368]: KID Deleting chunk 3 
Feb  1 19:22:49 console Bucardo[16368]: KID Deleting chunk 4 
Feb  1 19:22:49 console Bucardo[16368]: KID Deleting chunk 5 
And so repeats the cycle.

On 31/01/2010, at 4:27 PM, Grant Maxwell wrote:

> Hi
> 
> I am doing a master - master - master replication using custom conflict management code. It all works quite well, however
> I get a LOT of conflicts and syncs in some cases have to deal with thousands of records at a time.
> 
> The problem is that each time it deals with a conflict and tells bucardo to "try again" bucardo has to deal with all those thousands of records again. and again.
> 
> 
> I would like to limit how many records bucardo is allowed to replicate in one sync. This would mean we are running more syncs, but more of them would succeed.
> 
> Would another approach be for bucardo to simply mark the conflicts and deal with them at the end so that the bulk of the sync works. It could then feed the conflicts to
> the custom code which would deal with them in a flash.
> 
> Can anyone help with this please ?
> 
> I'm using 4.4.0.
> 
> regards
> Grant
> _______________________________________________
> Bucardo-general mailing list
> Bucardo-general at bucardo.org
> https://mail.endcrypt.com/mailman/listinfo/bucardo-general

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mail.endcrypt.com/pipermail/bucardo-general/attachments/20100201/54a99fd1/attachment.html 