[Bucardo-general] New to Bucardo - Issuses with Large Tables

[Wayne Taylor]

Hi Everyone,

I am new to the community and love the tooling and had some questions
regarding some issues we've faced with larger databases.

*As some background:*
We are in the process of migrating from Heroku to AWS Aurora RDS. We
initially tried out DMS but faced issues. We then took the WAL-E approach -
whilst this worked this introduced some unnecessary risks for us such as a
double step migration. Thus intro to Bucardo.

We have successfully migrated several databases - but recently found some
issues with larger databases e.g. we have one database in particular that
has 5 tables consuming 750GB total with the biggest table over 350GB in
size. Whilst the tasks run - we find after 24 hrs (not sure why it takes so
long) the task's timeout and restart again.

We run bucardo on an EC2 instance with Postgres installed (just empty shell)

*Mitigations we are taking right now:*
We have some apps that are 100% completely backed by Kafka and thus we can
take them offline and recover later. For now, a pg_dump and pg_restore with
parallel jobs allow us to achieve a full restore in approx 7 hrs.

*My hopes:*
Determine a way to get Bucardo to run the migration for larger tables
without hitting timeouts - ideally parallelized to speed up time or
settings that prevent the timeout from occurring.

Thank you community and happy to provide more background

Best,
Wayne
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://bucardo.org/pipermail/bucardo-general/attachments/20220121/9846f886/attachment.htm>