Containerd启动容器涉及的模块

前面我们梳理了docker、k8s、containerd、runc之间的渊源，Containerd早已脱离Docker成为了一个稳定可靠的容器运行时。

单从containerd的角度来看启动一个容器的过程大致是下图所示的流程:

我们可以使用Containerd的命令行工具`ctr`来启动一个容器，也可以在我们的应用中编写客户端程序为我们的应用加入启动容器的功能。

从containerd作为k8s的容器运行时来看，启动一个容器的过程大致是下图所示流程:

containerd实现了k8s的CRI接口，提供容器运行时核心功能，如镜像管理、容器管理等，当前containerd实现CRI接口的代码位于containerd代码的[pkg/cri]

也就是说containerd同样是一个k8s CRI的实现，可以使用k8s提供的[cri-tools]中的`crictl`命令行工具与containerd的CRI实现交互。

理解了上面这两幅图中各个组件之间的交互关系，接下来就可以部署containerd相关组件。

二进制部署containerd

这里以CentOS 7为例，演示部署containerd的过程，需要禁用selinux。
要部署containerd，需要下载3个包:

wget https://github.com/opencontainers/runc/releases/download/v1.0.0-rc93/runc.amd64
wget https://github.com/containerd/containerd/releases/download/v1.4.4/containerd-1.4.4-linux-amd64.tar.gz
wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.21.0/crictl-v1.21.0-linux-amd64.tar.gz

runc.amd64是runc的二进制文件，containerd-1.4.4-linux-amd64.tar.gz包含containerd和它的containerd-shim以及命令行工具ctr，crictl-v1.21.0-linux-amd64.tar.gz包含CRI的命令行工具crictl。

下一步进行二进制文件的解压和安装:

mkdir /usr/local/containerd
mkdir /etc/containerd

mv runc.amd64 /usr/local/containerd/bin/runc
tar -zxvf containerd-1.4.4-linux-amd64.tar.gz -C /usr/local/containerd
tar -zxvf crictl-v1.21.0-linux-amd64.tar.gz -C /usr/local/containerd/bin

ln -s /usr/local/containerd/bin/containerd /usr/local/bin/containerd
ln -s /usr/local/containerd/bin/ctr /usr/local/bin/ctr
ln -s /usr/local/containerd/bin/runc /usr/local/bin/runc
ln -s /usr/local/containerd/bin/crictl /usr/local/bin/crictl

安装的目录接口如下:

tree /usr/local/containerd/
/usr/local/containerd/
└── bin    
    ├── containerd
    ├── containerd-shim
    ├── containerd-shim-runc-v1    
    ├── containerd-shim-runc-v2
    ├── crictl
    ├── ctr
    └── runc

下一步生成containerd的默认配置文件`/etc/containerd/config.toml`:

containerd config default > /etc/containerd/config.toml

查看一下配置文件(篇幅有限这里没有给出全部内容):

version = 2
root = "/var/lib/containerd"
state = "/run/containerd"
plugin_dir = ""
disabled_plugins = []
required_plugins = []
oom_score = 0

[grpc]
  address = "/run/containerd/containerd.sock"
  tcp_address = ""
  tcp_tls_cert = ""
  tcp_tls_key = ""
  uid = 0
  gid = 0
  ......

创建containerd进程的systemd服务配置containerd.service，containerd源码的根目录中也给出了[containerd.service]这个文件的示例。

cat <<EOF | tee /etc/systemd/system/containerd.service
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd

Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=1048576
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target
EOF

启动containerd:

systemctl start containerd
systemctl status containerd
systemctl enable containerd

ctr命令行工具初始用

打印服务端和客户端版本：

ctr version
Client:
  Version:  v1.4.4
  Revision: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
  Go version: go1.15.8

Server:
  Version:  v1.4.4
  Revision: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
  UUID: dee82270-b4b4-429c-befa-45df1421da7e

pull docker hub中的redis镜像:

# pull镜像
ctr images pull docker.io/library/redis:alpine3.13

# 查看镜像
ctr i ls

pull私有仓库的镜像，需要使用`-u <username>:<password>`给定镜像仓库的用户名和密码:

ctr images pull -u user:password harbor.my.org/library/nginx:1.1

启动这个测试的redis容器:

ctr run -d docker.io/library/redis:alpine3.13 redis

查看容器(container和task):

ctr container ls
CONTAINER    IMAGE                                 RUNTIME
redis        docker.io/library/redis:alpine3.13    io.containerd.runc.v2

ctr task ls
TASK     PID      STATUS
redis    20808    RUNNING

注意: 在containerd中，container和task是分离的，container描述的是容器分配和附加资源的元数据对象，是静态内容，task是任务是系统上一个活动的、正在运行的进程。task应该在每次运行后删除，而container可以被多次使用、更新和查询。这点和docker中container定义是不一样的。

进入到容器中执行redis命令:

ctr task exec -t --exec-id redis-sh redis sh
/data # redis-cli
127.0.0.1:6379> set k1 v1
OK
127.0.0.1:6379> get k1
"v1"

查看一下系统中的进程信息:

ps -ef | grep runc | grep redis
/usr/local/containerd/bin/containerd-shim-runc-v2 -namespace default -id redis -address /run/containerd/containerd.sock

可以看出containerd中是存在namespace概念的，这样可以将不同业务和应用进行隔离，例如k8s使用containerd和直接使用ctr创建的容器可以隔离开。查看一下当前的namespace:

ctr ns ls
NAME    LABELS
default
k8s.io

不同namespace下pull的镜像也是隔离显示的，可以使用`-n`指定具体的namespace：

ctr -n default i ls
ctr -n k8s.io i ls

crictl命令行工具配置和初始用

crictl是k8s cri-tools的一部分，它提供了类似于docker的命令行工具，不需要kubelet就可以通过CRI跟容器运行时通信。crictl是专门为k8s设计的，提供了Pod、容器和镜像等资源的管理命令，可以帮助用户调试容器应用或者排查异常问题。crictl 可以用于所有实现了CRI接口的容器运行时。

尝试使用ctictl命令行工具查看一下镜像，给了一个警告信息：

crictl images
WARN[0000] image connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock]. As the default settings are now deprecated, you should set the endpoint instead.
ERRO[0002] connect endpoint 'unix:///var/run/dockershim.sock', make sure you are running as root and the endpoint has been started: context deadline exceeded
IMAGE               TAG                 IMAGE ID            SIZE

需要我们显示配置默认的endpoints:

crictl config runtime-endpoint unix:///run/containerd/containerd.sock
crictl config image-endpoint unix:///run/containerd/containerd.sock

上面的命令执行完成后将生成配置文件/etc/crictl.yaml:

runtime-endpoint: "unix:///run/containerd/containerd.sock"
image-endpoint: "unix:///run/containerd/containerd.sock"
timeout: 0
debug: false
pull-image-on-create: false
disable-pull-on-run: false

pull镜像:

crictl pull docker.io/library/redis:alpine3.13

crictl images
IMAGE                                       TAG                 IMAGE ID            SIZE
docker.io/library/redis                     alpine3.13          554d20f203657       10.9MB

crictl pull的镜像实际上是在`k8s.io` namespace下，可以使用`ctr -n k8s.io i ls`查看。

crictl不能像ctr那样通过参数给定用户名和密码的方式从开启认证的私有仓库中pull镜像。需要对containerd进行配置。containerd提供的各种功能在其内部都是通过插件实现的，可以使用`ctr plugins ls`查看containerd的插件:

ctr plugins lsTYPE                            ID                       PLATFORMS      STATUS
io.containerd.content.v1        content                  -              ok
io.containerd.snapshotter.v1    aufs                     linux/amd64    error
io.containerd.snapshotter.v1    btrfs                    linux/amd64    error
io.containerd.snapshotter.v1    devmapper                linux/amd64    error
io.containerd.snapshotter.v1    native                   linux/amd64    ok
io.containerd.snapshotter.v1    overlayfs                linux/amd64    ok
io.containerd.snapshotter.v1    zfs                      linux/amd64    error
io.containerd.metadata.v1       bolt                     -              ok
io.containerd.differ.v1         walking                  linux/amd64    ok
io.containerd.gc.v1             scheduler                -              ok
io.containerd.service.v1        introspection-service    -              ok
io.containerd.service.v1        containers-service       -              ok
io.containerd.service.v1        content-service          -              ok
io.containerd.service.v1        diff-service             -              ok
io.containerd.service.v1        images-service           -              ok
io.containerd.service.v1        leases-service           -              ok
io.containerd.service.v1        namespaces-service       -              ok
io.containerd.service.v1        snapshots-service        -              ok
io.containerd.runtime.v1        linux                    linux/amd64    ok
io.containerd.runtime.v2        task                     linux/amd64    ok
io.containerd.monitor.v1        cgroups                  linux/amd64    ok
io.containerd.service.v1        tasks-service            -              ok
io.containerd.internal.v1       restart                  -              ok
io.containerd.grpc.v1           containers               -              ok
io.containerd.grpc.v1           content                  -              ok
io.containerd.grpc.v1           diff                     -              ok
io.containerd.grpc.v1           events                   -              ok
io.containerd.grpc.v1           healthcheck              -              ok
io.containerd.grpc.v1           images                   -              ok
io.containerd.grpc.v1           leases                   -              ok
io.containerd.grpc.v1           namespaces               -              ok
io.containerd.internal.v1       opt                      -              ok
io.containerd.grpc.v1           snapshots                -              ok
io.containerd.grpc.v1           tasks                    -              ok
io.containerd.grpc.v1           version                  -              ok
io.containerd.grpc.v1           cri                      linux/amd64    ok

私有镜像仓库相关的配置在cri插件中，文档[Configure Image Registry](https://github.com/containerd/containerd/blob/master/docs/cri/registry.md)中包含了镜像仓库的配置。

关于私有仓库和认证信息配置示例如下，修改`/etc/containerd/config.toml`：

...
[plugins]
...
  [plugins."io.containerd.grpc.v1.cri"]
  ...
    [plugins."io.containerd.grpc.v1.cri".registry]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
          endpoint = ["https://registry-1.docker.io"]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor.my.org"]
          endpoint = ["https://harbor.my.org"]
      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.my.org".tls]
          insecure_skip_verify = true
        [plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.my.org".auth]
          username = "username"
          password = "passwd"
          # auth = "base64(username:password)"
...

配置完成后重启containerd，就可以使用crictl pull配置的私有仓库的镜像了：

crictl pull harbor.my.org/library/nginx:1.1

总结

我们已经部署了containerd，配置好了ctr和crictl工具，并使用ctr启动了第一个测试容器，但是此时容器还不具备网络能力，只能在容器内部自己玩。下一步，将尝试手工配置containerd和cni及cni插件的集成，为容器加入基本的网络能力。

参考

• 
  [containerd](https://containerd.io/)
• [Container Runtime Interface (CRI) CLI](https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md)
• [containerd Plugins](https://github.com/containerd/containerd/blob/master/docs/PLUGINS.md)
• [Configure Image Registry](https://github.com/containerd/containerd/blob/master/docs/cri/registry.md)