[Bucardo-general] indirect bucardo problem - stuck triggers?

Fri Apr 25 20:50:00 UTC 2014

On 4/25/2014 1:10 PM, Greg Sabino Mullane wrote:
> On Fri, Apr 25, 2014 at 12:46:55PM -0700, Paul Theodoropoulos wrote:
>> INFO:  "bucardo_truncate_trigger": found 623880 removable, 181642
>> nonremovable row versions in 1111573 pages
> Interesting. Smells like a bug to me. I'll look into this. So -
> did any of your tables have truncates?

Unfortunately, I've no idea.

Even after making the adjustments I mentioned before, the database 
continued growing - and stopping bucardo and vacuuming brought it back 
down. The errors look just like the ones I was getting last week. On 
exit, i get the familiar 'corrupted double-linked list' backtrace. I'm 
going to upgrade to 4.9.12 and see if it helps.

(30247) [Fri Apr 25 12:17:16 2014] MCP Entering main loop
(30555) [Fri Apr 25 12:17:17 2014] CTL New controller for sync 
"trumgr_group_sync". Relgroup is "trumgr_group_rels", dbs is 
"trumgr_group_dbs". PID=30555
(30558) [Fri Apr 25 12:17:17 2014] CTL New controller for sync 
"trumgr_detailed_tracking_sync". Relgroup is 
"trumgr_detailed_tracking_rels", dbs is "trumgr_detailed_tracking_dbs". 
PID=30558
(30561) [Fri Apr 25 12:17:17 2014] CTL New controller for sync 
"trumgr_main_sync". Relgroup is "trumgr_main_rels", dbs is 
"trumgr_main_dbs". PID=30561
(30565) [Fri Apr 25 12:17:18 2014] KID (trumgr_detailed_tracking_sync) 
New kid, sync "trumgr_detailed_tracking_sync" alive=1 Parent=30558 
PID=30565 kicked=1
(30568) [Fri Apr 25 12:17:19 2014] KID (trumgr_main_sync) New kid, sync 
"trumgr_main_sync" alive=1 Parent=30561 PID=30568 kicked=1
(30571) [Fri Apr 25 12:17:19 2014] KID (trumgr_group_sync) New kid, sync 
"trumgr_group_sync" alive=1 Parent=30555 PID=30571 kicked=1
Exiting eval via redo at /usr/local/share/perl/5.10.1/Bucardo.pm line 4953.
(30571) [Fri Apr 25 13:41:18 2014] KID (trumgr_group_sync) Warning! 
Aborting due to exception for public.unit_history_47:? Error was 
DBD::Pg::db pg_result failed: No asynchronous query is running at 
/usr/local/share/perl/5.10.1/Bucardo.pm line 8720.
DBI::db=HASH(0xb1d5480)->disconnect invalidates 1 active statement 
handle (either destroy statement handles or call finish on them before 
disconnecting) at /usr/local/share/perl/5.10.1/Bucardo.pm line 2291.
DBI::db=HASH(0xb1d8148)->disconnect invalidates 3 active statement 
handles (either destroy statement handles or call finish on them before 
disconnecting) at /usr/local/share/perl/5.10.1/Bucardo.pm line 2291.
(30571) [Fri Apr 25 13:41:18 2014] KID (trumgr_group_sync) DBD::Pg::db 
pg_result failed: No asynchronous query is running at 
/usr/local/share/perl/5.10.1/Bucardo.pm line 8720. Line: 4920 Main DB 
state: ? Error: none DB local_trumgr_group state: ? Error: 7 DB 
remote_trumgr_group state: ? Error: none
(30571) [Fri Apr 25 13:41:18 2014] KID (trumgr_group_sync) Kid 30571 
exiting at cleanup_kid. Sync "trumgr_group_sync" public.unit_history_47 
Reason: DBD::Pg::db pg_result failed: No asynchronous query is running 
at /usr/local/share/perl/5.10.1/Bucardo.pm line 8720. Line: 4920 Main DB 
state: ? Error: none DB local_trumgr_group state: ? Error: 7 DB 
remote_trumgr_group state: ? Error: none
(20964) [Fri Apr 25 13:41:20 2014] KID (trumgr_group_sync) New kid, sync 
"trumgr_group_sync" alive=1 Parent=30555 PID=20964 kicked=1
(30558) [Fri Apr 25 13:44:48 2014] CTL Found stopfile 
"/var/run/bucardo/fullstopbucardo": exiting
(30558) [Fri Apr 25 13:44:48 2014] CTL Warning! Controller for 
"trumgr_detailed_tracking_sync" was killed at line 1762: Found stopfile: 
stop  |
(30555) [Fri Apr 25 13:44:48 2014] CTL Found stopfile 
"/var/run/bucardo/fullstopbucardo": exiting
(30555) [Fri Apr 25 13:44:48 2014] CTL Warning! Controller for 
"trumgr_group_sync" was killed at line 1762: Found stopfile: stop  |
(30565) [Fri Apr 25 13:44:48 2014] KID (trumgr_detailed_tracking_sync) 
Found stopfile "/var/run/bucardo/fullstopbucardo": exiting
(30565) [Fri Apr 25 13:44:48 2014] KID (trumgr_detailed_tracking_sync) 
Kid 30565 exiting at cleanup_kid. Reason: Normal exit
(30561) [Fri Apr 25 13:44:48 2014] CTL Found stopfile 
"/var/run/bucardo/fullstopbucardo": exiting
(30561) [Fri Apr 25 13:44:48 2014] CTL Warning! Controller for 
"trumgr_main_sync" was killed at line 1762: Found stopfile: stop  |
(30247) [Fri Apr 25 13:44:48 2014] MCP Found stopfile 
"/var/run/bucardo/fullstopbucardo": exiting
(30568) [Fri Apr 25 13:44:48 2014] KID (trumgr_main_sync) Found stopfile 
"/var/run/bucardo/fullstopbucardo": exiting
(30568) [Fri Apr 25 13:44:48 2014] KID (trumgr_main_sync) Kid 30568 
exiting at cleanup_kid.  Reason: Normal exit
(20964) [Fri Apr 25 13:44:48 2014] KID (trumgr_group_sync) Found 
stopfile "/var/run/bucardo/fullstopbucardo": exiting
(20964) [Fri Apr 25 13:44:48 2014] KID (trumgr_group_sync) Kid 20964 
exiting at cleanup_kid.  Reason: Normal exit
(30555) [Fri Apr 25 13:44:48 2014] CTL Controller 30555 exiting at 
cleanup_controller. Reason: Found stopfile: stop  |
(30558) [Fri Apr 25 13:44:48 2014] CTL Controller 30558 exiting at 
cleanup_controller. Reason: Found stopfile: stop  |
(30561) [Fri Apr 25 13:44:48 2014] CTL Controller 30561 exiting at 
cleanup_controller. Reason: Found stopfile: stop  |
(30247) [Fri Apr 25 13:44:49 2014] MCP End of cleanup_mcp. Sys time: Fri 
Apr 25 13:44:49 2014. Database time: 2014-04-25 13:44:49.881688-07
(30247) [Fri Apr 25 13:44:49 2014] MCP Exiting
*** glibc detected *** Bucardo VAC.: corrupted double-linked list: 
0x0b3998f0 ***
======= Backtrace: =========
/lib/i686/cmov/libc.so.6(+0x6b381)[0xb76a8381]
/lib/i686/cmov/libc.so.6(+0x6cc31)[0xb76a9c31]
/lib/i686/cmov/libc.so.6(cfree+0x6d)[0xb76accbd]
/usr/lib/libpq.so.5(PQclear+0xf6)[0xb7558756]
/usr/local/lib/perl/5.10.1/auto/DBD/Pg/Pg.so(pg_st_destroy+0x17b)[0xb758b3db]
/usr/local/lib/perl/5.10.1/auto/DBD/Pg/Pg.so(+0xe869)[0xb757f869]
/usr/local/lib/perl/5.10.1/auto/DBI/DBI.so(XS_DBI_dispatch+0x1003)[0xb75ff203]
Bucardo VAC.(Perl_pp_entersub+0x52b)[0x80d5ddb]
Bucardo VAC.(Perl_call_sv+0x5a8)[0x8078c18]
Bucardo VAC.(Perl_sv_clear+0xa0)[0x80e80f0]
Bucardo VAC.(Perl_sv_free2+0x4a)[0x80e883a]
Bucardo VAC.[0x80dd939]
Bucardo VAC.(Perl_sv_clean_objs+0x29)[0x80dd999]
Bucardo VAC.(perl_destruct+0x11bf)[0x807dd0f]
Bucardo VAC.(main+0xdb)[0x80642eb]
/lib/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb7653ca6]
Bucardo VAC.[0x8064171]
======= Memory map: ========
08048000-08176000 r-xp 00000000 08:01 37616 /usr/bin/perl
08176000-08178000 rw-p 0012e000 08:01 37616 /usr/bin/perl

-- 
Paul Theodoropoulos
www.anastrophe.com