前提条件

部署Debezium Plugins

创建pvc，用于放connect plugins

1
2
3
4
5
6
7
8
9
10
11
12
13
14
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: kafka-plugins
  namespace: kafka
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  # 使用nfs方便放置
  storageClassName: nfs-storage
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-debezium-connect
  namespace: kafka
  labels:
    app: kafka-debezium-connect
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-debezium-connect
  template:
    metadata:
      labels:
        app: kafka-debezium-connect
    spec:
      containers:
      - name: connect
        image: harbor.basepoint.net/library/connect:2.7
        env:
          - name: GROUP_ID
            value: "1"
          - name: CONFIG_STORAGE_TOPIC
            value: connect-cluster-offsets
          - name: OFFSET_STORAGE_TOPIC
            value: connect-cluster-configs
          - name: STATUS_STORAGE_TOPIC
            value: connect-cluster-status
          - name: KAFKA_CONNECT_PLUGINS_DIR
            value: /kafka/connect/,/usr/local/share/kafka/plugins
          - name: CONNECT_TOPIC_CREATION_ENABLE
            value: "true"
        volumeMounts:
        - name: plugins
          mountPath: /usr/local/share/kafka/plugins
      volumes:
      - name: plugins
        persistentVolumeClaim:
          claimName: kafka-plugins
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-debezium-connect
  namespace: kafka
spec:
  type: NodePort
  selector:
    app: kafka-debezium-connect
  ports:
    - protocol: TCP
      port: 8083
      targetPort: 8083
      nodePort: 31083
      name: restapi

容器已内置了以下插件

1
2
3
4
5
6
7
8
9
10
11
12
13
[kafka@kafka-connect-769747b4bb-vqkld connect]$ ls -l
total 52
drwxr-xr-x 2 kafka kafka 4096 Jul 26 04:26 debezium-connector-db2
drwxr-xr-x 2 kafka kafka 4096 Jul 26 04:26 debezium-connector-ibmi
drwxr-xr-x 2 kafka kafka 4096 Jul 26 04:26 debezium-connector-informix
drwxr-xr-x 2 kafka kafka 4096 Jul 26 04:26 debezium-connector-jdbc
drwxr-xr-x 2 kafka kafka 4096 Jul 26 04:26 debezium-connector-mongodb
drwxr-xr-x 2 kafka kafka 4096 Jul 26 04:26 debezium-connector-mysql
drwxr-xr-x 2 kafka kafka 8192 Jul 26 04:26 debezium-connector-oracle
drwxr-xr-x 2 kafka kafka 4096 Jul 26 04:26 debezium-connector-postgres
drwxr-xr-x 2 kafka kafka 4096 Jul 26 04:26 debezium-connector-spanner
drwxr-xr-x 2 kafka kafka 4096 Jul 26 04:26 debezium-connector-sqlserver
drwxr-xr-x 2 kafka kafka 4096 Jul 26 04:26 debezium-connector-vitess

confluence的jdbc下载地址：JDBC Connector (Source and Sink) | Confluent Hub: Apache Kafka Connectors for Streaming Data

放置pvc目录中，重启connect

已知问题

to many open file

1
2
3
4
5
vi /etc/sysctl.conf

fs.inotify.max_user_instances = 819200
fs.inotify.max_queued_events = 1638400
fs.inotify.max_user_watches = 52428800
1
sysctl -p

部署debezium UI

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: debezium-ui
  namespace: kafka
  labels:
    app: debezium-ui
spec:
  replicas: 1
  selector:
    matchLabels:
      app: debezium-ui
  template:
    metadata:
      labels:
        app: debezium-ui
    spec:
      containers:
      - name: ui
        # 最新源地址：quay.io/debezium/debezium-ui:2.5，版本不兼容
        # image: harbor.basepoint.net/library/debezium-ui:2.5
        # 源地址：debezium/debezium-ui:2.1.2.Final
        image: harbor.basepoint.net/library/debezium-ui:2.1.2.Final
        imagePullPolicy: IfNotPresent
        env:
          - name: KAFKA_CONNECT_URIS
            # connect地址
            value: http://kafka-connect.kafka.svc.cluster.local:8083
          - name: DEPLOYMENT_MODE
            value: default
---
apiVersion: v1
kind: Service
metadata:
  name: debezium-ui
  namespace: kafka
spec:
  type: NodePort
  selector:
    app: debezium-ui
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
      nodePort: 30080
      name: restapi

UI请求有问题请求不到MangoDB.json，先保留后续研究。

结论：debezium-ui镜像版本与debezium-connect镜像版本不兼容，目前ui没有2.7版本，2.5不兼容，最终使用：debezium/debezium-ui:2.1.2.Final镜像版本。

虽然debezium-ui:2.1.2.Final版本还是有报错，但至少能创建Connector了