redis数据同步工具redis-shake之两服务器之间数据实时同步案例

Redis  /  管理员 发布于 2年前   2579

redis-shake是阿里开源的用于redis数据同步的工具，我这里主要用于两台redis服务器之间有些key数据要同步一下，所以在测试环境测试一下，写个教程以防下次使用的时候方便

安装redis这点就不用说了

介绍一下我的环境

centos7两台，redis一台4.0，一台5.0

安装redis-shake

下载列表：

https://github.com/alibaba/RedisShake

我这里下载的版本是release-v2.1.1-20210903

https://github.com/alibaba/RedisShake/releases/download/release-v2.1.1-20210903/release-v2.1.1-20210903.tar.gz

下载下来放入被同步的服务器上

我这里是root目录创建redis-shake文件夹 放进这里面，然后解压

[root@hyperf redis-shake]# tar -zxvf release-v2.1.1-20210903.tar.gz 
release-v2.1.1-20210903/
release-v2.1.1-20210903/redis-shake.linux
release-v2.1.1-20210903/redis-shake.conf
release-v2.1.1-20210903/redis-shake.windows
release-v2.1.1-20210903/redis-shake.darwin
[root@hyperf redis-shake]# ll
总用量 58236
-rw-r--r-- 1  502 games    12589 8月  24 2022 redis-shake.conf
-rwxr-xr-x 1  502 games 13176492 9月   3 2021 redis-shake.darwin
-rwxr-xr-x 1  502 games 13235485 9月   3 2021 redis-shake.linux
-rwxr-xr-x 1  502 games 13356032 9月   3 2021 redis-shake.windows
-rw-r--r-- 1 root root  19842840 8月  24 12:49 release-v2.1.1-20210903.tar.gz

配置文件修改，记得把原来的redis-shake.conf备份一下加个.bak后缀

如下修改：

[root@hyperf redis-shake]# cat redis-shake.conf
conf.version = 1
id = redis-shake
log.file = /root/redis-shake/redis-shake.log
log.level = info
pid_path =
system_profile = 9310
http_profile = 9320
parallel = 32
source.type = standalone
source.address = 172.18.1.24:63790
source.password_raw = 123456
source.auth_type = auth
source.tls_enable = false
source.rdb.input =
source.rdb.parallel = 0
source.rdb.special_cloud =
target.type = standalone
target.address = 127.0.0.1:6379
target.password_raw =
target.auth_type = auth
target.db = 0
key_exists = rewrite
filter.db.whitelist = 0
[root@hyperf redis-shake]# 

ps:

我的源redis服务器是172.18.1.24，端口：63790 密码：123456

同步至服务器192.168.1.98 ，端口：6379  密码：无

日志文件记录：/root/redis-shake/redis-shake.log

启动redis-shake服务

[root@hyperf redis-shake]# ./redis-shake.linux -conf=redis-shake.conf -type=sync &
[2] 19824

ps:

这样就启动了，pid：19824，如果修改配置或者要重启服务可以使用kill命令 比如：

kill -9 19824

启动完了，看一下日志文件，我这就截一部分，看看都同步完了

[root@hyperf redis-shake]# ll
总用量 58500
-rw-r--r-- 1  502 games      537 8月  24 13:58 redis-shake.conf
-rw-r--r-- 1 root root     12544 8月  24 13:52 redis-shake.conf.bak
-rwxr-xr-x 1  502 games 13176492 9月   3 2021 redis-shake.darwin
-rwxr-xr-x 1  502 games 13235485 9月   3 2021 redis-shake.linux
-rw-r--r-- 1 root root    136400 8月  24 14:15 redis-shake.log
-rw------- 1 root root         6 8月  24 13:55 redis-shake.pid
-rwxr-xr-x 1  502 games 13356032 9月   3 2021 redis-shake.windows
-rw-r--r-- 1 root root  19842840 8月  24 12:49 release-v2.1.1-20210903.tar.gz
[root@hyperf redis-shake]# cat redis-shake.log
2022/08/24 13:55:40 [WARN] source.auth_type[auth] != auth
2022/08/24 13:55:40 [WARN] target.auth_type[auth] != auth
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] source rdb[172.18.1.24:63790] checksum[yes]
2022/08/24 13:55:40 [WARN] 
______________________________
\                             \           _         ______ |
 \                             \        /   \___-=O'/|O'/__|
  \   RedisShake, here we go !! \_______\          / | /    )
  /                             /        '/-==__ _/__|/__=-|  -GM
 /        Alibaba Cloud        /         *             \ | |
/                             /                        (o)
------------------------------
if you have any problem, please visit https://github.com/alibaba/RedisShake/wiki/FAQ
2022/08/24 13:55:40 [INFO] redis-shake configuration: {"ConfVersion":1,"Id":"redis-shake","LogFile":"/root/redis-shake/redis-shake.log","LogLevel":"info","SystemProfile":9310,"HttpProfile":9320,"Parallel":32,"SourceType":"standalone","SourceAddress":"172.18.1.24:63790","SourcePasswordRaw":"***","SourcePasswordEncoding":"***","SourceAuthType":"auth","SourceTLSEnable":false,"SourceRdbInput":[],"SourceRdbParallel":1,"SourceRdbSpecialCloud":"","TargetAddress":"127.0.0.1:6379","TargetPasswordRaw":"***","TargetPasswordEncoding":"***","TargetDBString":"0","TargetDBMapString":"","TargetAuthType":"auth","TargetType":"standalone","TargetTLSEnable":false,"TargetRdbOutput":"","TargetVersion":"5.0.9","FakeTime":"","KeyExists":"rewrite","FilterDBWhitelist":["0"],"FilterDBBlacklist":null,"FilterKeyWhitelist":null,"FilterKeyBlacklist":null,"FilterSlot":null,"FilterLua":false,"BigKeyThreshold":52428800,"Metric":false,"MetricPrintLog":false,"SenderSize":65535,"SenderCount":1024,"SenderDelayChannelSize":32,"KeepAlive":0,"PidPath":"","ScanKeyNumber":0,"ScanSpecialCloud":"","ScanKeyFile":"","Qps":500000,"ResumeFromBreakPoint":false,"Psync":true,"NCpu":0,"HeartbeatUrl":"","HeartbeatInterval":10,"HeartbeatExternal":"","HeartbeatNetworkInterface":"","ReplaceHashTag":false,"ExtraInfo":false,"SockFileName":"","SockFileSize":0,"FilterKey":null,"FilterDB":"","Rewrite":false,"SourceAddressList":["172.18.1.24:63790"],"TargetAddressList":["127.0.0.1:6379"],"SourceVersion":"4.0.6","HeartbeatIp":"127.0.0.1","ShiftTime":0,"TargetReplace":true,"TargetDB":0,"Version":"develop,cc226f841e2e244c48246ebfcfd5a50396b59710,go1.15.7,2021-09-03_10:06:55","Type":"sync","TargetDBMap":null}
2022/08/24 13:55:40 [INFO] DbSyncer[0] starts syncing data from 172.18.1.24:63790 to [127.0.0.1:6379] with http[9321], enableResumeFromBreakPoint[false], slot boundary[-1, -1]
2022/08/24 13:55:40 [INFO] DbSyncer[0] psync connect '172.18.1.24:63790' with auth type[auth] OK!
2022/08/24 13:55:40 [INFO] DbSyncer[0] psync send listening port[9320] OK!
2022/08/24 13:55:40 [INFO] DbSyncer[0] try to send 'psync' command: run-id[?], offset[-1]
2022/08/24 13:55:40 [INFO] Event:FullSyncStartId:redis-shake
2022/08/24 13:55:40 [INFO] DbSyncer[0] psync runid = a347eaedbfa83634774b5584ed0efe9d152c6d0f, offset = 0, fullsync
2022/08/24 13:55:40 [INFO] DbSyncer[0] rdb file size = 104942
2022/08/24 13:55:40 [INFO] Aux information key:redis-ver value:4.0.6
2022/08/24 13:55:40 [INFO] Aux information key:redis-bits value:64
2022/08/24 13:55:40 [INFO] Aux information key:ctime value:1661320637
2022/08/24 13:55:40 [INFO] Aux information key:used-mem value:2231719
2022/08/24 13:55:40 [INFO] Aux information key:repl-stream-db value:0
2022/08/24 13:55:40 [INFO] Aux information key:repl-id value:a347eaedbfa83634774b5584ed0efe9d152c6d0f
2022/08/24 13:55:40 [INFO] Aux information key:repl-offset value:0
2022/08/24 13:55:40 [INFO] Aux information key:aof-preamble value:0
2022/08/24 13:55:40 [INFO] db_size:20 expire_size:14
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] db_size:2 expire_size:0
2022/08/24 13:55:40 [INFO] Aux information key:lua value:return redis.call('exists',KEYS[1])<1 and redis.call('setex',KEYS[1],ARGV[2],ARGV[1])
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] input password is empty, skip auth address[127.0.0.1:6379] with type[auth].
2022/08/24 13:55:40 [INFO] warning, rewrite key: name
2022/08/24 13:55:40 [INFO] DbSyncer[0] total = 102.48KB -     102.48KB [100%]  entry=20            filter=3           
2022/08/24 13:55:40 [INFO] DbSyncer[0] sync rdb done
...

上面就同步过来了，如果日志没报错的话就ok了；

用redis客户端软件打开检查测试一下

然后在源redis上修改一下数据，在同步redis服务器上看看，同步过来就ok了

测试我就不截图了，比较简单，有兴趣有需要的可以自行测试一下