[Bucardo-general] please help with this.

Mon Feb 1 07:54:35 UTC 2010

I haven't looked at the Bucardo code in a while but I think it would
be possible to support this better but would probably require quite a
bit of work.

As I explained in my previous email there is no way to safely
replicate only part of what needs replicating at a time if you want to
guarantee that no one can ever see the database in an inconsistent
state.

Here is my guess as to what features would need to be added to improve
your situation:

1) If a conflict is found in a chunk we need to know which parts of
the chunk were already fine.
2) Then we need to start a list of conflicts that need to be resolved,
or maybe we should call the custom conflict code immediately before we
move on.
3) Know which part of the chunk was NOT tested and try
deleting/updating/copying the rest of the chunk until we run into
another conflict

I'm sure there are a lot of little things required to make this work.

I'd wait on word from Greg whether any of this is feasible.

Adam
Adam Wendt Consulting

On Sun, Jan 31, 2010 at 11:37 PM, Grant Maxwell
<grant.maxwell at maxan.com.au> wrote:
>
> Hi folks
>
> Anyone got more for this for me ? This is crucial not just for me but for
> anyone who has significant numbers of unique constraint violations that need
> to be dealt with. You can see from the log snippet below that we get into a
> loop of deleting 180000 (approx) records (based on 10,000 per chunk),
> copying 170000 + some, discovering a conflict, running the custom code to
> resolve the conflict and then starting the entire set  all over again, only
> to find another conflict and so starting again all over.
>
> Is this a design flaw ? All of the records are being prevented from
> replication by individual events. Surely the majority of the records should
> replicate. This would happen correctly if each chunk was deleted and copied
> as a transaction. some chunks would fail and need correction but it would
> not mean failing every chunk and would not mean copying the entire record
> set again.
>
> please please please look at this.
>
> regards
> Grant
>
> What I am seeing all day in the log is
>
> Feb  1 19:21:21 console Bucardo[16368]: KID Deleting chunk 4
> Feb  1 19:21:21 console Bucardo[16368]: KID Deleting chunk 5
> Feb  1 19:21:22 console Bucardo[16368]: KID Deleting chunk 6
> Feb  1 19:21:22 console Bucardo[16368]: KID Deleting chunk 7
> Feb  1 19:21:22 console Bucardo[16368]: KID Deleting chunk 8
> Feb  1 19:21:22 console Bucardo[16368]: KID Deleting chunk 9
> Feb  1 19:21:22 console Bucardo[16368]: KID Deleting chunk 10
> Feb  1 19:21:22 console Bucardo[16368]: KID Deleting chunk 11
> Feb  1 19:21:22 console Bucardo[16368]: KID Deleting chunk 12
> Feb  1 19:21:22 console Bucardo[16368]: KID Deleting chunk 13
> Feb  1 19:21:23 console Bucardo[16368]: KID Deleting chunk 14
> Feb  1 19:21:23 console Bucardo[16368]: KID Deleting chunk 15
> Feb  1 19:21:23 console Bucardo[16368]: KID Deleting chunk 16
> Feb  1 19:21:23 console Bucardo[16368]: KID Deleting chunk 17
> Feb  1 19:21:23 console Bucardo[16368]: KID Deleting chunk 18
> Feb  1 19:21:23 console Bucardo[16368]: KID Begin COPY to
> public.tblmailaddresslist
> Feb  1 19:21:24 console Bucardo[16368]: KID Copying chunk 1 from t2 to
> messagestore
> Feb  1 19:21:28 console Bucardo[16368]: KID Copying chunk 2 from t2 to
> messagestore
> Feb  1 19:21:32 console Bucardo[16368]: KID Copying chunk 3 from t2 to
> messagestore
> Feb  1 19:21:37 console Bucardo[16368]: KID Copying chunk 4 from t2 to
> messagestore
> Feb  1 19:21:42 console Bucardo[16368]: KID Copying chunk 5 from t2 to
> messagestore
> .......
> Feb  1 19:22:37 console Bucardo[16368]: KID Copying chunk 18 from t2 to
> messagestore
> Feb  1 19:22:38 console Bucardo[16368]: KID Exception caught: DBD::Pg::db
> pg_putcopyend failed: ERROR:  duplicate key value violates unique constraint
> "tblmailaddresslist_emailaddress_key"\nCONTEXT:  COPY tblmailaddresslist,
> line 2: "1281808^Ikellyvenhuizen at hotmail.com^I2009-12-03
> 18:04:35.927361+11^If^I4^I0^I0^I0" at
> /usr/local/share/perl/5.10.0/Bucardo.pm line 4925.
> Feb  1 19:22:38 console Bucardo[16368]: KID Rolling back to target
> savepoint, due to database error: DBD::Pg::db pg_putcopyend failed: ERROR:
>  duplicate key value violates unique constraint
> "tblmailaddresslist_emailaddress_key"\nCONTEXT:  COPY tblmailaddresslist,
> line 2: "1281808^Ikellyvenhuizen at hotmail.com^I2009-12-03
> 18:04:35.927361+11^If^I4^I0^I0^I0" at
> /usr/local/share/perl/5.10.0/Bucardo.pm line 4925.
> Feb  1 19:22:38 console Bucardo[16368]: KID ELS mod set rowinfo pkey to
> 1281808
> Feb  1 19:22:38 console Bucardo[16368]: KID Trying exception code 1:
> mailaddress_in
> Feb  1 19:22:38 console Bucardo[16368]: KID Running exception custom code 1:
> mailaddress_in
> Feb  1 19:22:38 console Bucardo[16368]: KID Finished custom code 1
> Feb  1 19:22:38 console Bucardo[16368]: KID Message from exception code 1:
> ELS updating from db: t2 pkmailid 1281808 to 1533518 for
> addresskellyvenhuizen at hotmail.com
> Feb  1 19:22:38 console Bucardo[16368]: KID Warning! Code exception 1:
> UPDATE affected 1 rows
> Feb  1 19:22:38 console Bucardo[16368]: KID Exception handler thinks we can
> try again
> Feb  1 19:22:47 console Bucardo[16368]: KID Deleting rows from
> public.tblmailaddresslist
> Feb  1 19:22:48 console Bucardo[16368]: KID Deleting chunk 1
> Feb  1 19:22:48 console Bucardo[16368]: KID Deleting chunk 2
> Feb  1 19:22:48 console Bucardo[16368]: KID Deleting chunk 3
> Feb  1 19:22:49 console Bucardo[16368]: KID Deleting chunk 4
> Feb  1 19:22:49 console Bucardo[16368]: KID Deleting chunk 5
> And so repeats the cycle.
>
>
> On 31/01/2010, at 4:27 PM, Grant Maxwell wrote:
>
> Hi
>
> I am doing a master - master - master replication using custom conflict
> management code. It all works quite well, however
>
> I get a LOT of conflicts and syncs in some cases have to deal with thousands
> of records at a time.
>
> The problem is that each time it deals with a conflict and tells bucardo to
> "try again" bucardo has to deal with all those thousands of records again.
> and again.
>
>
> I would like to limit how many records bucardo is allowed to replicate in
> one sync. This would mean we are running more syncs, but more of them would
> succeed.
>
> Would another approach be for bucardo to simply mark the conflicts and deal
> with them at the end so that the bulk of the sync works. It could then feed
> the conflicts to
>
> the custom code which would deal with them in a flash.
>
> Can anyone help with this please ?
>
> I'm using 4.4.0.
>
> regards
>
> Grant
>
> _______________________________________________
>
> Bucardo-general mailing list
>
> Bucardo-general at bucardo.org
>
> https://mail.endcrypt.com/mailman/listinfo/bucardo-general
>
>
> _______________________________________________
> Bucardo-general mailing list
> Bucardo-general at bucardo.org
> https://mail.endcrypt.com/mailman/listinfo/bucardo-general
>
>