CCR 同步 Doris 集群数据

说明

  • CCR 功能与使用 MySQL CDC 同步 MySQL 数据类似，可配置同步单库或单表数据
  • 任务执行时会先执行全量备份，全量后监听数据库 binlog，增量同步数据，并自动保证全量增量点位一致
  • 仅同步指定库数据，如果实例内有多个库，需给每个库配置同步任务；目标库用户也需要手动创建

部署

  • 源/目标 Doris 集群 be/fe 节点增加如下配置并重启
1

enable_feature_binlog=true
  • 下载 ccr-syncer
1
2
3

mkdir -p /app/doris/tools/ccr-syncer
cd /app/doris/tools/ccr-syncer
https://apache-doris-releases.oss-accelerate.aliyuncs.com/ccr-release/ccr-syncer-3.0.6-rc05-x64.tar.xz
  • 解压、创建数据存储目录
    • 当前ccr-syncer对高可用无要求，使用sqlite3，如果对高可用有要求，也支持使用 MySQL 作为元数据存储
1
2
3

tar -xvf ccr-syncer-3.0.6-rc05-x64.tar.xz
ln -sfn ccr-syncer-3.0.6-rc05-x64 ccr-syncer
mkdir ccr-data-storage ccr-logs
  • 启动 ccr-syncer
    • –db_type：指定存储元数据的数据库类型
    • –db_dir：指定元数据文件路径
    • –log_dir：指定日志文件路径
    • –daemon：后台运行
1

bin/start_syncer.sh --db_type sqlite3 --db_dir /app/doris/tools/ccr-syncer/ccr-data-storage/ccr.db --log_dir /app/doris/tools/ccr-syncer/ccr-logs/ccr_syncer.log --daemon

使用脚本批量开启源库binlog

  • 要同步的库要分别执行，当前要同步 db1 db2 db3 三个库
1
2
3

./bin/enable_db_binlog.sh -h 192.168.212.11 -p 9030 -u root -P P@55wD111 -d db1
./bin/enable_db_binlog.sh -h 192.168.212.11 -p 9030 -u root -P P@55wD111 -d db2
./bin/enable_db_binlog.sh -h 192.168.212.11 -p 9030 -u root -P P@55wD111 -d db3

提交任务

  • 每个 db 单独提交同步任务
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

$ curl -X POST -H "Content-Type: application/json" -d '{
    "name": "db_db1_sync_job",
    "src": {
    "host": "192.168.212.11",
    "port": "9030",
    "thrift_port": "9020",
    "user": "root",
    "password": "P@55wD111",
    "database": "db1"
    },
    "dest": {
    "host": "192.168.212.12",
    "port": "9030",
    "thrift_port": "9020",
    "user": "root",
    "password": "P@55wD111",
    "database": "db1"
    }' http://127.0.0.1:9190/create_ccr

$ curl -X POST -H "Content-Type: application/json" -d '{
    "name": "db_db2_sync_job",
    "src": {
    "host": "192.168.212.11",
    "port": "9030",
    "thrift_port": "9020",
    "user": "root",
    "password": "P@55wD111",
    "database": "db2"
    },
    "dest": {
    "host": "192.168.212.12",
    "port": "9030",
    "thrift_port": "9020",
    "user": "root",
    "password": "P@55wD111",
    "database": "db2"
    }' http://127.0.0.1:9190/create_ccr

$ curl -X POST -H "Content-Type: application/json" -d '{
    "name": "db_db3_sync_job",
    "src": {
    "host": "192.168.212.11",
    "port": "9030",
    "thrift_port": "9020",
    "user": "root",
    "password": "P@55wD111",
    "database": "db3"
    },
    "dest": {
    "host": "192.168.212.12",
    "port": "9030",
    "thrift_port": "9020",
    "user": "root",
    "password": "P@55wD111",
    "database": "db3"
    }' http://127.0.0.1:9190/create_ccr
  • 查看任务
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# 查看任务
$ curl http://127.0.0.1:9190/list_jobs
# 查看任务状态
$ curl -X POST -H "Content-Type: application/json" -d '{
    "name": "db1"
}' http://127.0.0.1:9190/job_status

$ curl -X POST -H "Content-Type: application/json" -d '{
    "name": "db2"
}' http://127.0.0.1:9190/job_status

$ curl -X POST -H "Content-Type: application/json" -d '{
    "name": "db3"
}' http://127.0.0.1:9190/job_status

CCR 任务管理相关命令

  • 创建任务
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

curl -X POST -H "Content-Type: application/json" -d '{
    "name": "ccr_test",
    "src": {
    "host": "localhost",
    "port": "9030",
    "thrift_port": "9020",
    "user": "root",
    "password": "",
    "database": "demo",
    "table": "example_tbl"
    },
    "dest": {
    "host": "localhost",
    "port": "9030",
    "thrift_port": "9020",
    "user": "root",
    "password": "",
    "database": "ccrt",
    "table": "copy"
    }
}' http://127.0.0.1:9190/create_ccr
  • 查看任务列表
1

curl http://ccr_syncer_host:ccr_syncer_port/list_jobs
  • 查看任务状态
1
2
3

curl -X POST -H "Content-Type: application/json" -d '{
    "name": "job_name"
}' http://ccr_syncer_host:ccr_syncer_port/job_status
  • 暂停任务
1
2
3

curl -X POST -H "Content-Type: application/json" -d '{
    "name": "job_name"
}' http://ccr_syncer_host:ccr_syncer_port/pause
  • 恢复任务
1
2
3

curl -X POST -H "Content-Type: application/json" -d '{
    "name": "job_name"
}' http://ccr_syncer_host:ccr_syncer_port/resume
  • 结束任务
1
2
3

curl -X POST -H "Content-Type: application/json" -d '{
    "name": "job_name"
}' http://ccr_syncer_host:ccr_syncer_port/desync
  • 删除任务
1
2
3

curl -X POST -H "Content-Type: application/json" -d '{
    "name": "job_name"
}' http://ccr_syncer_host:ccr_syncer_port/delete
  • 查看版本
1

curl http://ccr_syncer_host:ccr_syncer_port/version

参考

  • 官网文档：https://doris.apache.org/zh-CN/docs/2.1/admin-manual/data-admin/ccr/overview
    • CCR 同步配置：https://doris.apache.org/zh-CN/docs/2.1/admin-manual/data-admin/ccr/config
    • CCR 对 DDL 支持说明：https://doris.apache.org/zh-CN/docs/2.1/admin-manual/data-admin/ccr/feature
    • CCR 参数说明：https://doris.apache.org/zh-CN/docs/2.1/admin-manual/data-admin/ccr/manual