PostgreSQL中Slony-I同步复制部署教程

数据库  /  管理员 发布于 5年前   171

前言

本文主要介绍了关于PostgreSQL中Slony-I同步复制部署的相关内容，分享出来供大家参考学习，下面话不多说了，来一起看看详细的介绍吧

本次测试环境

IP    10.189.102.118    10.189.100.195    10.189.100.226    
PGHOME    /usr/local/pgsql    /usr/local/pgsql    /usr/local/pgsql    
Role    Origin,Providers    Subscribers,Providers    Subscribers    
Database    region_il     region_il     region_il     
Table    il_longlive.entity      il_longlive.entity_resultil_longlive.resultil_longlive.result_logbook    il_longlive.entity      il_longlive.entity_resultil_longlive.resultil_longlive.result_logbook    il_longlive.entityil_longlive.entity_resultil_longlive.resultil_longlive.result_logbook    
Sequences    il_longlive.entity_entity_id_seqil_longlive.entity_result_id_seqil_longlive.result_id_seqil_longlive.result_logbook_record_id_seq    il_longlive.entity_entity_id_seqil_longlive.entity_result_id_seqil_longlive.result_id_seqil_longlive.result_logbook_record_id_seq     il_longlive.entity_entity_id_seqil_longlive.entity_result_id_seqil_longlive.result_id_seqil_longlive.result_logbook_record_id_seq    

Slony-I一般要求表有主键,或者唯一键,Slony-I会自动识别主键,如果表没有主键,需要手工指定唯一键,如果没有指定唯一键,则不能复制,下图为Slony-I架构图.

1. 查看主库表信息

region_il=# \d   List of relations Schema |  Name  | Type | Owner -------------+------------------------------+----------+---------- il_longlive | entity   | table | postgres il_longlive | entity_entity_id_seq  | sequence | postgres il_longlive | entity_result  | table | postgres il_longlive | entity_result_id_seq  | sequence | postgres il_longlive | result   | table | postgres il_longlive | result_id_seq  | sequence | postgres il_longlive | result_logbook  | table | postgres il_longlive | result_logbook_record_id_seq | sequence | postgres il_longlive | result_vendor_schema  | table | postgres il_longlive | result_vendor_schema_id_seq | sequence | postgres il_longlive | snapshot   | table | postgres il_longlive | snapshot_id_seq  | sequence | postgres(12 rows)region_il=# \d entity     Table "il_longlive.entity" Column |  Type  |    Modifiers    --------------+------------------------+------------------------------------------------------------ entity_id | integer  | not null default nextval('entity_entity_id_seq'::regclass) country_code | character varying(3) |  state_code | character varying(50) |  name  | character varying(128) |  islive | boolean  | Indexes: "entity_pkey" PRIMARY KEY, btree (entity_id)region_il=# select count(*) from entity; count ---------(1 row)

2. 在所有节点安装Slony-I

$ wget http://www.slony.info/downloads/2.2/source/slony1-2.2.4.tar.bz2$ tar -jxf slony1-2.2.4.tar.bz2$ cd slony1-2.2.4$ ./configure --prefix=/usr/local/pgsql --with-pgconfigdir=/usr/local/pgsql/bin --with-perltools$ make$ make install

3. 在所有节点创建Slony-I使用的专用用户

$ psql -c "CREATE USER slonyadmin WITH SUPERUSER REPLICATION LOGIN PASSWORD '888888';"

4. 在所有节点设置允许Slony-I用户远程登录

$ grep slonyadmin /usr/local/pgsql/data/pg_hba.conf host all  slonyadmin  0.0.0.0/0  md5

5. 在所有节点的对应数据库创建plpgsql语言

$ createlang plpgsql -d region_il 

6. 在主节点创建Slony-I配置文件,并复制到所有其他节点

$ cat /usr/local/pgsql/etc/slon_tools.confif ($ENV{"SLONYNODES"}) { require $ENV{"SLONYNODES"};} else { $CLUSTER_NAME = 'region_il'; $PIDFILE_DIR = '/usr/local/pgsql/data'; $LOGDIR = '/usr/local/pgsql/data/pg_log'; $MASTERNODE = 1; $DEBUGLEVEL = 0;$START_WATCHDOG = 1; $SLEEP_TIME = 60; $WATCHDOG_VERBOSE = 0; add_node(node => 1,  host => '10.189.102.118',  dbname => 'region_il',  port => 5432,  user => 'slonyadmin',  password => '888888',); add_node(node => 2,  parent => 1,  host => '10.189.100.195',  dbname => 'region_il',  port => 5432,  user => 'slonyadmin',  password => '888888',); add_node(node => 3,  parent => 2,  host => '10.189.100.226',  dbname => 'region_il',  port => 5432,  user => 'slonyadmin',  password => '888888',);}$SLONY_SETS = { "il_longlive_schema" => { "set_id" => 1, "table_id" => 101, "sequence_id" => 101, "pkeyedtables" => [    'il_longlive.entity',    'il_longlive.entity_result',    'il_longlive.result',    'il_longlive.result_logbook',    ], "keyedtables" => {}, "sequences" => ['il_longlive.entity_entity_id_seq', 'il_longlive.entity_result_id_seq','il_longlive.result_id_seq','il_longlive.result_logbook_record_id_seq',   ],},};if ($ENV{"SLONYSET"}) { require $ENV{"SLONYSET"};}# Please do not add or change anything below this point. 1;$ scp /usr/local/pgsql/etc/slon_tools.conf [email protected]:/usr/local/pgsql/etc/$ scp /usr/local/pgsql/etc/slon_tools.conf [email protected]:/usr/local/pgsql/etc/

7. 将需要复制的表结构从主节点导出并导入到所有辅节点中

$ pg_dump -U slonyadmin -s -t il_longlive.entity -t il_longlive.entity_result -t il_longlive.result -t il_longlive.result_logbook region_il | psql -h 10.189.100.195 -U slonyadmin region_il$ pg_dump -U slonyadmin -s -t il_longlive.entity -t il_longlive.entity_result -t il_longlive.result -t il_longlive.result_logbook region_il | psql -h 10.189.100.226 -U slonyadmin region_il

 8. 在主节点初始化Slony-I集群

$ /usr/local/pgsql/bin/slonik_init_cluster # INIT CLUSTERcluster name = region_il; node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888'; node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888'; node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888'; init cluster (id = 1, comment = 'Node 1 - [email protected]');# STORE NODE store node (id = 2, event node = 1, comment = 'Node 2 - [email protected]'); store node (id = 3, event node = 1, comment = 'Node 3 - [email protected]'); echo 'Set up replication nodes';# STORE PATH echo 'Next: configure paths for each node/origin'; store path (server = 1, client = 2, conninfo = 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888'); store path (server = 1, client = 3, conninfo = 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888'); store path (server = 2, client = 1, conninfo = 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888'); store path (server = 2, client = 3, conninfo = 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888'); store path (server = 3, client = 1, conninfo = 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888'); store path (server = 3, client = 2, conninfo = 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888'); echo 'Replication nodes prepared'; echo 'Please start a slon replication daemon for each node';$ /usr/local/pgsql/bin/slonik_init_cluster | slonik:7: Possible unsupported PostgreSQL version (90505) 9.5, defaulting to 8.4 support:10: Possible unsupported PostgreSQL version (90505) 9.5, defaulting to 8.4 support:11: Possible unsupported PostgreSQL version (90505) 9.5, defaulting to 8.4 support:12: Set up replication nodes:15: Next: configure paths for each node/origin:22: Replication nodes prepared:23: Please start a slon replication daemon for each node

9. 在所有节点启动slon进程

在主节点启动slon进程

$ slon_start 1Invoke slon for node 1 - /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node1.pid -s 1000 -d0 region_il 'host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888' > /usr/local/pgsql/data/pg_log/node1/region_il-2017-06-14.log 2>&1 &Slon successfully started for cluster region_il, node node1PID [2454]Start the watchdog process as well...$ slon_status 1Slon is running for the 'region_il' cluster on node1.$ ps -ef | grep slonpostgres 2454 1 0 22:54 pts/1 00:00:00 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node1.pid -s 1000 -d0 region_il host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888postgres 2455 2454 0 22:54 pts/1 00:00:00 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node1.pid -s 1000 -d0 region_il host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888postgres 2459 24883 0 22:54 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(24667) idlepostgres 2467 24883 0 22:54 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(24669) idlepostgres 2468 24883 0 22:54 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(24673) idlepostgres 2469 24883 0 22:54 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(24675) idlepostgres 2470 24883 0 22:54 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(24677) idlepostgres 2471 24883 0 22:54 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(24681) idlepostgres 2473 1 0 22:54 pts/1 00:00:00 /usr/bin/perl /usr/local/pgsql/bin/slon_watchdog --config=/usr/local/pgsql/etc/slon_tools.conf node1 60

在辅节点1启动slave进程

$ slon_start 2Invoke slon for node 2 - /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node2.pid -s 1000 -d0 region_il 'host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888' > /usr/local/pgsql/data/pg_log/node2/region_il-2017-06-14.log 2>&1 &Slon successfully started for cluster region_il, node node2PID [15535]Start the watchdog process as well...$ slon_status 2Slon is running for the 'region_il' cluster on node2.$ ps -ef | grep slonpostgres 15515 12843 0 22:54 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(38804) idlepostgres 15535 1 0 22:57 pts/0 00:00:00 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node2.pid -s 1000 -d0 region_il host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888postgres 15536 15535 0 22:57 pts/0 00:00:00 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node2.pid -s 1000 -d0 region_il host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888postgres 15540 12843 0 22:57 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.195(54629) idlepostgres 15546 12843 0 22:57 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.195(54631) idlepostgres 15547 12843 0 22:57 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.195(54633) idlepostgres 15548 12843 0 22:57 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.195(54635) idlepostgres 15549 12843 0 22:57 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.195(54639) idlepostgres 15552 12843 0 22:57 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.195(54643) idlepostgres 15555 1 0 22:57 pts/0 00:00:00 /usr/bin/perl /usr/local/pgsql/bin/slon_watchdog --config=/usr/local/pgsql/etc/slon_tools.conf node2 60postgres 15559 12843 0 22:57 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(38812) idle

在辅节点2启动slave进程

$ slon_start 3Invoke slon for node 3 - /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node3.pid -s 1000 -d0 region_il 'host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888' > /usr/local/pgsql/data/pg_log/node3/region_il-2017-06-14.log 2>&1 &Slon successfully started for cluster region_il, node node3PID [19212]Start the watchdog process as well...$ slon_status 3Slon is running for the 'region_il' cluster on node3.$ ps -ef | grep slonpostgres 6033 1 0 Jun02 ? 00:00:02 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node2.pid -s 1000 -d0 region_il host=ec2t-pgtest-03 dbname=region_il user=slonyadmin port=5432 password=mypna123postgres 6050 1 0 Jun02 ? 00:00:02 /usr/bin/perl /usr/local/pgsql/bin/slon_watchdog --config=/usr/local/pgsql/etc/slon_tools.conf node2 60postgres 19059 15237 0 22:54 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(63615) idlepostgres 19109 15237 0 22:57 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.195(26781) idlepostgres 19212 1 0 23:00 pts/2 00:00:00 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node3.pid -s 1000 -d0 region_il host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888postgres 19213 19212 0 23:00 pts/2 00:00:00 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node3.pid -s 1000 -d0 region_il host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888postgres 19217 15237 0 23:00 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.226(41213) idlepostgres 19225 15237 0 23:00 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.226(41215) idlepostgres 19226 15237 0 23:00 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.226(41219) idlepostgres 19227 15237 0 23:00 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.226(41221) idlepostgres 19228 15237 0 23:00 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.226(41223) idlepostgres 19229 15237 0 23:00 ? 00:00:00 postgres: slonyadmin region_il 10.189.100.226(41227) idlepostgres 19232 1 0 23:00 pts/2 00:00:00 /usr/bin/perl /usr/local/pgsql/bin/slon_watchdog --config=/usr/local/pgsql/etc/slon_tools.conf node3 60postgres 19234 15237 0 23:00 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(63635) idlepostgres 19235 6033 0 23:00 ? 00:00:00 /usr/local/pgsql/bin/slon -p /usr/local/pgsql/data/region_il_node2.pid -s 1000 -d0 region_il host=ec2t-pgtest-03 dbname=region_il user=slonyadmin port=5432 password=mypna123postgres 19239 15237 0 23:00 ? 00:00:00 postgres: slonyadmin region_il 10.189.102.118(63637) idle

10. 在主节点创建复制集

$ /usr/local/pgsql/bin/slonik_create_set 1cluster name = region_il; node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888'; node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888'; node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';# TABLE ADD KEY# CREATE SET create set (id = 1, origin = 1, comment = 'Set 1 (il_longlive_schema) for region_il');# SET ADD TABLE echo 'Subscription set 1 (il_longlive_schema) created'; echo 'Adding tables to the subscription set'; set add table (set id = 1, origin = 1, id = 101,   full qualified name = 'il_longlive.entity',   comment = 'Table il_longlive.entity with primary key'); echo 'Add primary keyed table il_longlive.entity'; set add table (set id = 1, origin = 1, id = 102,   full qualified name = 'il_longlive.entity_result',   comment = 'Table il_longlive.entity_result with primary key'); echo 'Add primary keyed table il_longlive.entity_result'; set add table (set id = 1, origin = 1, id = 103,   full qualified name = 'il_longlive.result',   comment = 'Table il_longlive.result with primary key'); echo 'Add primary keyed table il_longlive.result'; set add table (set id = 1, origin = 1, id = 104,   full qualified name = 'il_longlive.result_logbook',   comment = 'Table il_longlive.result_logbook with primary key'); echo 'Add primary keyed table il_longlive.result_logbook';# SET ADD SEQUENCE echo 'Adding sequences to the subscription set'; set add sequence (set id = 1, origin = 1, id = 101,   full qualified name = 'il_longlive.entity_entity_id_seq',   comment = 'Sequence il_longlive.entity_entity_id_seq'); echo 'Add sequence il_longlive.entity_entity_id_seq'; set add sequence (set id = 1, origin = 1, id = 102,   full qualified name = 'il_longlive.entity_result_id_seq',   comment = 'Sequence il_longlive.entity_result_id_seq'); echo 'Add sequence il_longlive.entity_result_id_seq'; set add sequence (set id = 1, origin = 1, id = 103,   full qualified name = 'il_longlive.result_id_seq',   comment = 'Sequence il_longlive.result_id_seq'); echo 'Add sequence il_longlive.result_id_seq'; set add sequence (set id = 1, origin = 1, id = 104,   full qualified name = 'il_longlive.result_logbook_record_id_seq',   comment = 'Sequence il_longlive.result_logbook_record_id_seq'); echo 'Add sequence il_longlive.result_logbook_record_id_seq'; echo 'All tables added';$ /usr/local/pgsql/bin/slonik_create_set 1 | slonik:12: Subscription set 1 (il_longlive_schema) created:13: Adding tables to the subscription set:17: Add primary keyed table il_longlive.entity:21: Add primary keyed table il_longlive.entity_result:25: Add primary keyed table il_longlive.result:29: Add primary keyed table il_longlive.result_logbook:32: Adding sequences to the subscription set:36: Add sequence il_longlive.entity_entity_id_seq:40: Add sequence il_longlive.entity_result_id_seq:44: Add sequence il_longlive.result_id_seq:48: Add sequence il_longlive.result_logbook_record_id_seq:49: All tables added

11. 在主节点创建复制订阅者

$ slonik_subscribe_set 1 2cluster name = region_il; node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888'; node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888'; node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';  subscribe set (id = 1, provider = 1, receiver = 2, forward = yes); echo 'Subscribed nodes to set 1';$ slonik_subscribe_set 1 2 | slonik:7: Subscribed nodes to set 1$ slonik_subscribe_set 1 3cluster name = region_il; node 1 admin conninfo='host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888'; node 2 admin conninfo='host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888'; node 3 admin conninfo='host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888';  subscribe set (id = 1, provider = 2, receiver = 3, forward = yes); echo 'Subscribed nodes to set 1';$ slonik_subscribe_set 1 3 | slonik:7: Subscribed nodes to set 1

12.查看各节点Slony-I日志信息

主节点查看Slony-I日志信息

$ cat region_il-2017-06-14.log 2017-06-14 22:54:42 PDT CONFIG main: slon version 2.2.4 starting up2017-06-14 22:54:42 PDT INFO slon: watchdog process started2017-06-14 22:54:42 PDT CONFIG slon: watchdog ready - pid = 24542017-06-14 22:54:42 PDT CONFIG slon: worker process created - pid = 24552017-06-14 22:54:42 PDT CONFIG main: Integer option vac_frequency = 32017-06-14 22:54:42 PDT CONFIG main: Integer option log_level = 02017-06-14 22:54:42 PDT CONFIG main: Integer option sync_interval = 10002017-06-14 22:54:42 PDT CONFIG main: Integer option sync_interval_timeout = 100002017-06-14 22:54:42 PDT CONFIG main: Integer option sync_group_maxsize = 202017-06-14 22:54:42 PDT CONFIG main: Integer option syslog = 02017-06-14 22:54:42 PDT CONFIG main: Integer option quit_sync_provider = 02017-06-14 22:54:42 PDT CONFIG main: Integer option remote_listen_timeout = 3002017-06-14 22:54:42 PDT CONFIG main: Integer option monitor_interval = 5002017-06-14 22:54:42 PDT CONFIG main: Integer option explain_interval = 02017-06-14 22:54:42 PDT CONFIG main: Integer option tcp_keepalive_idle = 02017-06-14 22:54:42 PDT CONFIG main: Integer option tcp_keepalive_interval = 02017-06-14 22:54:42 PDT CONFIG main: Integer option tcp_keepalive_count = 02017-06-14 22:54:42 PDT CONFIG main: Integer option apply_cache_size = 1002017-06-14 22:54:42 PDT CONFIG main: Boolean option log_pid = 02017-06-14 22:54:42 PDT CONFIG main: Boolean option log_timestamp = 12017-06-14 22:54:42 PDT CONFIG main: Boolean option tcp_keepalive = 12017-06-14 22:54:42 PDT CONFIG main: Boolean option monitor_threads = 12017-06-14 22:54:42 PDT CONFIG main: Real option real_placeholder = 0.0000002017-06-14 22:54:42 PDT CONFIG main: String option cluster_name = region_il2017-06-14 22:54:42 PDT CONFIG main: String option conn_info = host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=8888882017-06-14 22:54:42 PDT CONFIG main: String option pid_file = /usr/local/pgsql/data/region_il_node1.pid2017-06-14 22:54:42 PDT CONFIG main: String option log_timestamp_format = %Y-%m-%d %H:%M:%S %Z 2017-06-14 22:54:42 PDT CONFIG main: String option archive_dir = [NULL]2017-06-14 22:54:42 PDT CONFIG main: String option sql_on_connection = [NULL]2017-06-14 22:54:42 PDT CONFIG main: String option lag_interval = [NULL]2017-06-14 22:54:42 PDT CONFIG main: String option command_on_logarchive = [NULL]2017-06-14 22:54:42 PDT CONFIG main: String option syslog_facility = LOCAL02017-06-14 22:54:42 PDT CONFIG main: String option syslog_ident = slon2017-06-14 22:54:42 PDT CONFIG main: String option cleanup_interval = 10 minutes2017-06-14 22:54:42 PDT CONFIG main: local node id = 12017-06-14 22:54:42 PDT INFO main: main process started2017-06-14 22:54:42 PDT CONFIG main: launching sched_start_mainloop2017-06-14 22:54:42 PDT CONFIG main: loading current cluster configuration2017-06-14 22:54:42 PDT CONFIG storeNode: no_id=2 no_comment='Node 2 - [email protected]'2017-06-14 22:54:42 PDT CONFIG storeNode: no_id=3 no_comment='Node 3 - [email protected]'2017-06-14 22:54:42 PDT CONFIG storePath: pa_server=2 pa_client=1 pa_conninfo="host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" pa_connretry=102017-06-14 22:54:42 PDT CONFIG storePath: pa_server=3 pa_client=1 pa_conninfo="host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" pa_connretry=102017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=22017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=32017-06-14 22:54:42 PDT CONFIG main: last local event sequence = 50000000072017-06-14 22:54:42 PDT CONFIG main: configuration complete - starting threads2017-06-14 22:54:42 PDT INFO localListenThread: thread starts2017-06-14 22:54:42 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 905052017-06-14 22:54:42 PDT CONFIG enableNode: no_id=22017-06-14 22:54:42 PDT CONFIG enableNode: no_id=32017-06-14 22:54:42 PDT INFO main: running scheduler mainloop2017-06-14 22:54:42 PDT INFO remoteWorkerThread_3: thread starts2017-06-14 22:54:42 PDT INFO remoteListenThread_3: thread starts2017-06-14 22:54:42 PDT CONFIG cleanupThread: thread starts2017-06-14 22:54:42 PDT INFO syncThread: thread starts2017-06-14 22:54:42 PDT INFO monitorThread: thread starts2017-06-14 22:54:42 PDT INFO remoteListenThread_2: thread starts2017-06-14 22:54:42 PDT INFO remoteWorkerThread_2: thread starts2017-06-14 22:54:42 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 905052017-06-14 22:54:42 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 905052017-06-14 22:54:42 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 905052017-06-14 22:54:42 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 905052017-06-14 22:54:42 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 905052017-06-14 22:54:42 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 905052017-06-14 22:54:42 PDT CONFIG cleanupThread: bias = 602017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_2: update provider configuration2017-06-14 22:54:42 PDT CONFIG version for "host=10.189.102.118 dbname=region_il user=slonyadmin port=5432 password=888888" is 905052017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_3: update provider configuration2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=22017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=32017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=22017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_2: update provider configuration2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=22017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=32017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=22017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=32017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_3: update provider configuration2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=22017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=32017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=22017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=32017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_2: update provider configuration2017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=22017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=32017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=22017-06-14 22:54:42 PDT CONFIG storeListen: li_origin=2 li_receiver=1 li_provider=32017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_3: update provider configuration2017-06-14 22:54:42 PDT CONFIG remoteWorkerThread_2: update provider configuration2017-06-14 22:57:50 PDT CONFIG remoteWorkerThread_2: update provider configuration2017-06-14 22:57:50 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 905052017-06-14 22:57:50 PDT INFO remoteWorkerThread_2: SYNC 5000000003 done in 0.014 seconds2017-06-14 22:57:58 PDT INFO remoteWorkerThread_2: SYNC 5000000004 done in 0.003 seconds2017-06-14 22:58:11 PDT INFO remoteWorkerThread_2: SYNC 5000000005 done in 0.002 seconds2017-06-14 22:58:19 PDT INFO remoteWorkerThread_2: SYNC 5000000006 done in 0.002 seconds2017-06-14 22:58:27 PDT INFO remoteWorkerThread_2: SYNC 5000000007 done in 0.003 seconds2017-06-14 22:58:35 PDT INFO remoteWorkerThread_2: SYNC 5000000008 done in 0.002 seconds2017-06-14 22:58:43 PDT INFO remoteWorkerThread_2: SYNC 5000000009 done in 0.002 seconds2017-06-14 22:59:01 PDT INFO remoteWorkerThread_2: SYNC 5000000010 done in 0.003 seconds2017-06-14 22:59:09 PDT INFO remoteWorkerThread_2: SYNC 5000000011 done in 0.002 seconds2017-06-14 22:59:17 PDT INFO remoteWorkerThread_2: SYNC 5000000012 done in 0.002 seconds2017-06-14 22:59:25 PDT INFO remoteWorkerThread_2: SYNC 5000000013 done in 0.002 seconds2017-06-14 22:59:38 PDT INFO remoteWorkerThread_2: SYNC 5000000014 done in 0.003 seconds2017-06-14 22:59:51 PDT INFO remoteWorkerThread_2: SYNC 5000000015 done in 0.002 seconds2017-06-14 22:59:54 PDT INFO remoteWorkerThread_2: SYNC 5000000016 done in 0.002 seconds2017-06-14 23:00:05 PDT INFO remoteWorkerThread_2: SYNC 5000000017 done in 0.002 seconds2017-06-14 23:00:19 PDT INFO remoteWorkerThread_2: SYNC 5000000018 done in 0.003 seconds2017-06-14 23:00:32 PDT INFO remoteWorkerThread_2: SYNC 5000000019 done in 0.003 seconds2017-06-14 23:00:35 PDT INFO remoteWorkerThread_2: SYNC 5000000020 done in 0.003 seconds2017-06-14 23:00:43 PDT CONFIG remoteWorkerThread_3: update provider configuration2017-06-14 23:00:43 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 905052017-06-14 23:00:43 PDT INFO remoteWorkerThread_3: SYNC 5000000003 done in 0.045 seconds2017-06-14 23:00:47 PDT CONFIG remoteWorkerThread_2: update provider configuration2017-06-14 23:00:47 PDT CONFIG version for "host=10.189.100.226 dbname=region_il user=slonyadmin port=5432 password=888888" is 905052017-06-14 23:00:47 PDT INFO remoteWorkerThread_2: SYNC 5000000021 done in 0.014 seconds2017-06-14 23:00:55 PDT INFO remoteWorkerThread_3: SYNC 5000000004 done in 0.002 seconds2017-06-14 23:00:55 PDT INFO remoteWorkerThread_2: SYNC 5000000022 done in 0.005 seconds2017-06-14 23:01:08 PDT INFO remoteWorkerThread_3: SYNC 5000000005 done in 0.002 seconds2017-06-14 23:01:08 PDT INFO remoteWorkerThread_2: SYNC 5000000023 done in 0.005 seconds2017-06-14 23:01:16 PDT INFO remoteWorkerThread_3: SYNC 5000000006 done in 0.002 seconds2017-06-14 23:01:20 PDT INFO remoteWorkerThread_3: SYNC 5000000007 done in 0.002 seconds2017-06-14 23:01:20 PDT INFO remoteWorkerThread_2: SYNC 5000000024 done in 0.005 seconds2017-06-14 23:01:24 PDT INFO remoteWorkerThread_2: SYNC 5000000025 done in 0.005 seconds2017-06-14 23:01:38 PDT INFO remoteWorkerThread_3: SYNC 5000000008 done in 0.003 seconds2017-06-14 23:01:38 PDT INFO remoteWorkerThread_2: SYNC 5000000026 done in 0.005 seconds2017-06-14 23:01:41 PDT INFO remoteWorkerThread_3: SYNC 5000000009 done in 0.002 seconds2017-06-14 23:01:44 PDT INFO remoteWorkerThread_2: SYNC 5000000027 done in 0.004 seconds2017-06-14 23:01:52 PDT INFO remoteWorkerThread_3: SYNC 5000000010 done in 0.003 seconds2017-06-14 23:01:55 PDT INFO remoteWorkerThread_2: SYNC 5000000028 done in 0.005 seconds2017-06-14 23:02:08 PDT CONFIG remoteWorkerThread_3: update provider configuration2017-06-14 23:02:08 PDT CONFIG version for "host=10.189.100.195 dbname=region_il user=slonyadmin port=5432 password=888888" is 905052017-06-14 23:02:08 PDT INFO remoteWorkerThread_2: SYNC 5000000029 done in 0.012 seconds2017-06-14 23:02:08 PDT INFO remoteWorkerThread_3: SYNC 5000000011 done in 0.016 seconds2017-06-14 23:02:11 PDT INFO remoteWorkerThread_3: SYNC 5000000012 done in 0.004 seconds2017-06-14 23:02:19 PDT INFO remoteWorkerThread_2: SYNC 5000000030 done in 0.005 seconds2017-06-14 23:02:22 PDT INFO remoteWorkerThread_3: SYNC 5000000013 done in 0.005 seconds2017-06-14 23:02:25 PDT INFO remoteWorkerThread_2: SYNC 5000000031 done in 0.004 seconds2017-06-14 23:02:38 PDT INFO remoteWorkerThread_2: SYNC 5000000032 done in 0.004 seconds2017-06-14 23:02:38 PDT INFO remoteWorkerThread_3: SYNC 5000000014 done in 0.005 seconds2017-06-14 23:02:46 PDT INFO remoteWorkerThread_3: SYNC 5000000015 done in 0.005 seconds2017-06-14 23:02:50 PDT INFO remoteWorkerThread_3: SYNC 5000000016 done in 0.004 seconds2017-06-14 23:02:50 PDT INFO remoteWorkerThread_2: SYNC 5000000033 done in 0.005 seconds2017-06-14 23:02:54 PDT INFO remoteWorkerThread_2: SYNC 5000000034 done in 0.004 seconds2017-06-14 23:03:08 PDT INFO remoteWorkerThread_3: SYNC 5000000017 done in 0.004 seconds2017-06-14 23:03:08 PDT INFO remoteWorkerThread_2: SYNC 5000000035 done in 0.005 seconds2017-06-14 23:03:11 PDT CONFIG storeSet: set_id=1 set_origin=1 set_comment='Set 1 (il_longlive_schema) for region_il'2017-06-14 23:03:16 PDT INFO remoteWorkerThread_3: SYNC 5000000018 done in 0.005 seconds2017-06-14 23:03:17 PDT INFO remoteWorkerThread_2: SYNC 5000000036 done in 0.005 seconds2017-06-14 23:03:21 PDT INFO remoteWorkerThread_3: SYNC 5000000019 done in 0.004 seconds2017-06-14 23:03:29 PDT INFO remoteWorker