[Bucardo-general] Filtering columns while copying

[David Christensen]

> Hi,
> 
> We're looking to use Bucardo to do an upgrade from 9.6 to 11.  The reason we're not using pg_logical or pg_upgrade is thqt the upgrade *also* involves converting from SQL_ASCII to UTF8.  The idea would be to run text columns through Encoding::FixLatin on the fly.  Is this totally implausible, or any other suggestions about this path?

This should work, maybe even without a patched Bucardo to apply a filter to the data as it is copied over.

Probably the tricky part would be applying the filter to just the fields in question, but if you could do something about this at the SQL level instead of Perl level you might be able to make a customcols version of the source SELECT that uses that function for the fields in question.

So say you define a plperl function on your source database “fix_encoding()” that just basically wraps the Encoding::FixLatin routine, then define the customcols for each table in the sync to include instead of “SELECT textfield1, field2 FROM table1”, “SELECT fix_encoding(textfield1), field2 FROM table1”.  It would be easy enough to query the system tables and generate the appropriate customcols definition for each table depending on field type.

Just my initial $0.02.
--
David Christensen
End Point Corporation
david at endpoint.com
785-727-1171



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <https://mail.endcrypt.com/pipermail/bucardo-general/attachments/20190228/218bf0c9/attachment-0001.sig>