背景

找一个能查看etcd中存储的解码后的k8s数据的方法或工具。查看开源工具[1]，很久没用维护了，看相关issue，该工具已经加入etcd-io。

编译步骤

根据官方文档[2]操作，下载源码包：

[root@node1]# git clone git@github.com:etcd-io/auger.git
Cloning into 'auger'...
remote: Enumerating objects: 712, done.
remote: Counting objects: 100% (229/229), done.
remote: Compressing objects: 100% (106/106), done.
remote: Total 712 (delta 179), reused 150 (delta 123), pack-reused 483
Receiving objects: 100% (712/712), 247.44 KiB | 186.00 KiB/s, done.
Resolving deltas: 100% (409/409), done.

编译版本：

[root@node1]# cd auger/
[root@node1 auger]# make release
Building release in temp directory /tmp/tmp.VtO7q4KrPY
docker run \
        -v /tmp/tmp.VtO7q4KrPY/auger:/go/src/github.com/etcd-io/auger \
        -w /go/src/github.com/etcd-io/auger \
        golang:1.21.8 \
        /bin/bash -c "make -f /go/src/github.com/etcd-io/auger/Makefile release-docker-build GOARCH=amd64 GOOS=linux"
Unable to find image 'golang:1.21.8' locally
1.21.8: Pulling from library/golang
71215d55680c: Pull complete
3cb8f9c23302: Pull complete
5f899db30843: Pull complete
c29f45468664: Pull complete
6de33e7b6490: Pull complete
6dbaf8e5f127: Pull complete
4f4fb700ef54: Pull complete
Digest: sha256:856073656d1a517517792e6cdd2f7a5ef080d3ca2dff33e518c8412f140fdd2d
Status: Downloaded newer image for golang:1.21.8
export GOPATH=/go
GOOS=linux GOARCH=amd64 GO111MODULE=on go build
go: go.mod requires go >= 1.22.0 (running go 1.21.8; GOTOOLCHAIN=local)
make: *** [/go/src/github.com/etcd-io/auger/Makefile:66: release-docker-build] Error 1
make: *** [release] Error 2

提示go的版本不匹配，更新版本：

[root@node1 auger]# vim Makefile
NAME ?= auger
PKG ?= github.com/etcd-io/$(NAME)
GO_VERSION ?= 1.22.0
...

继续编译：

[root@node1 auger]# make release
Building release in temp directory /tmp/tmp.s0Ue7zvIop
docker run \
        -v /tmp/tmp.s0Ue7zvIop/auger:/go/src/github.com/etcd-io/auger \
        -w /go/src/github.com/etcd-io/auger \
        golang:1.22.0 \
        /bin/bash -c "make -f /go/src/github.com/etcd-io/auger/Makefile release-docker-build GOARCH=amd64 GOOS=linux"
Unable to find image 'golang:1.22.0' locally
1.22.0: Pulling from library/golang
7bb465c29149: Pull complete
...
Digest: sha256:7b297d9abee021bab9046e492506b3c2da8a3722cbf301653186545ecc1e00bb
Status: Downloaded newer image for golang:1.22.0
export GOPATH=/go
GOOS=linux GOARCH=amd64 GO111MODULE=on go build
go: downloading github.com/coreos/etcd v3.1.11+incompatible
go: downloading github.com/google/safetext v0.0.0-20220914124124-e18e3fe012bf
go: downloading github.com/spf13/cobra v1.8.0
go: downloading github.com/coreos/bbolt v1.3.1-coreos.3
go: downloading k8s.io/apimachinery v0.30.0
go: downloading proxy.golang.org/xxx io timeout
...

使用proxy.golang.org代理导致很多依赖包下载失败，修改GOPROXY代理

[root@node1 ~]# docker exec -it 9b41dd00e91a sh
# go env
...
GOPROXY='https://proxy.golang.org,direct'

[root@node1 auger]# vim Makefile
# Build used inside docker by 'release'
release-docker-build:
        export GOPATH=/go
        GOOS=$(GOOS) GOARCH=$(GOARCH) GO111MODULE=on GOPROXY='https://goproxy.cn,direct' go build

继续编译：

[root@node1 auger]# make release
Building release in temp directory /tmp/tmp.34OgmWJGLU
docker run \
        -v /tmp/tmp.34OgmWJGLU/auger:/go/src/github.com/etcd-io/auger \
        -w /go/src/github.com/etcd-io/auger \
        golang:1.22.0 \
        /bin/bash -c "make -f /go/src/github.com/etcd-io/auger/Makefile release-docker-build GOARCH=amd64 GOOS=linux"
export GOPATH=/go
GOOS=linux GOARCH=amd64 GO111MODULE=on GOPROXY='https://goproxy.cn,direct' go build
go: downloading github.com/coreos/etcd v3.1.11+incompatible
...
build/auger built!

编译成功，执行二进制文件测试，提示glibc版本没找到：

[root@node1 auger]# ./build/auger -help
./build/auger: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by ./build/auger)
./build/auger: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by ./build/auger)

查看本地的glibc版本，发现版本不匹配：

[root@node1 l14185]# rpm -qa|grep glibc
glibc-2.17-326.el7_9.x86_64

解决方案有两个：

修改编译使用的镜像，找一个glibc版本跟节点上一致的编译镜像；

直接在节点上编译；

以直接在节点上编译为例，下载指定版本的go安装包，直接执行go build命令：

[root@node1]# GOOS=linux GOARCH=amd64 go build -o build/auger
[root@node1 auger]# ll build/
total 39916
-rwxr-xr-x 1 root root 40871798 May 13 19:12 auger

使用方法

查看帮助信息：

[root@iZbp1esczkzr2k2fughijkZ auger]# ./build/auger
Inspect and analyze kubernetes objects in binary storage
encoding used with etcd 3+ and boltdb.

Usage:
  auger [command]

Available Commands:
  analyze     Analyze kubernetes data from the boltdb '.db' files etcd persists to.
  checksum    Checksum a etcd keyspace.
  completion  Generate the autocompletion script for the specified shell
  decode      Decode objects from the kubernetes binary key-value store encoding.
  encode      Encode objects to the kubernetes binary key-value store encoding.
  extract     Extracts kubernetes data from the boltdb '.db' files etcd persists to.
  help        Help about any command

Flags:
  -h, --help   help for auger

Use "auger [command] --help" for more information about a command.

查看解码后的etcd数据：

[root@node1]# ETCDCTL_API=3 etcdctl get /registry/pods/kube-system/coredns-795cc9c45c-j7nl4 | ./auger decode
apiVersion: v1
kind: Pod
metadata:
  generateName: coredns-795cc9c45c-
  labels:
    k8s-app: kube-dns
    pod-template-hash: 795cc9c45c
  name: coredns-795cc9c45c-j7nl4
  namespace: kube-system
spec:
  containers:
  - args:
    - -conf
    - /etc/coredns/Corefile
    name: coredns
    ports:
  ...
    volumeMounts:
    - mountPath: /etc/coredns
      name: config-volume
      readOnly: true
    - mountPath: /tmp
      name: tmp
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: coredns-token-9dldj
      readOnly: true
  nodeName: node1
  tolerations:
  - key: CriticalAddonsOnly
    operator: Exists
 ...
  volumes:
  - emptyDir: {}
    name: tmp
  ...
status:
  conditions:
  - lastProbeTime: null
    type: Initialized
 ...
  containerStatuses:
  - containerID: docker://f85d0fd1422a3860d574eb88b5dc23c165d5adb3eccb242a1a847bd0cfc98227
  ...
  hostIP: 192.168.10.10
  phase: Running
  podIP: 10.10.166.139
  qosClass: Burstable

注意事项

直接使用auger命令时，需要保证etcd服务未启动，或者把etcd的数据库文件拷贝一份再解析，否则会导致解析卡住。

解析卡住的strace命令现象如下：

[root@node1]# strace ./auger checksum -f /var/lib/etcd/default.etcd/member/snap/db
execve("./auger", ["./auger", "checksum", "-f", "/var/lib/etcd/default.etcd/membe"...], [/* 25 vars */]) = 0
brk(NULL)                               = 0x3e75000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4a242fc000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=37465, ...}) = 0
mmap(NULL, 37465, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f4a242f2000
close(3)                                = 0
...
mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f49da795000
openat(AT_FDCWD, "/var/lib/etcd/default.etcd/member/snap/db", O_RDWR|O_CREAT|O_CLOEXEC, 0400) = 3
fcntl(3, F_GETFL)                       = 0x8002 (flags O_RDWR|O_LARGEFILE)
fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK|O_LARGEFILE) = 0
epoll_ctl(4, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=1581252610, u64=9172183252402700290}}) = -1 EPERM (Operation not permitted)
fcntl(3, F_GETFL)                       = 0x8802 (flags O_RDWR|O_NONBLOCK|O_LARGEFILE)
fcntl(3, F_SETFL, O_RDWR|O_LARGEFILE)   = 0
flock(3, LOCK_EX|LOCK_NB)               = -1 EAGAIN (Resource temporarily unavailable)
futex(0xc000100148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1edd920, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x1edd920, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x1edd920, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x1edd920, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x1edd920, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
...

拷贝一份数据库文件，对比校验结果：

[root@node1]# cp /var/lib/etcd/default.etcd/member/snap/db /root/etcd.db
[root@node1]# ./auger checksum -f /root/etcd.db
checksum: 2125275681
compact-revision: 6609891
revision: 6610932

[root@node2 ~]# cp /var/lib/etcd/default.etcd/member/snap/db /root/etcd.db
[root@node2 ~]# ./auger checksum -f /root/etcd.db -r 6610932
checksum: 2125275681
compact-revision: 6610743
revision: 6610932

参考资料

https://github.com/jpbetz/auger

https://github.com/etcd-io/auger