Kubernetes in docker の使い方

Kubernetes in docker (kind) を使えるようになっておこうと思います。今回は DigitalOcean の CentOS 7 で試す。

Docker CE のインストール

Get Docker Engine - Community for CentOS

sudo yum remove docker \
                docker-client \
                docker-client-latest \
                docker-common \
                docker-latest \
                docker-latest-logrotate \
                docker-logrotate \
                docker-engine
sudo yum install -y yum-utils \
  device-mapper-persistent-data \
  lvm2
sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo
sudo yum install docker-ce docker-ce-cli containerd.io
sudo usermod -a -G docker centos
sudo systemctl start docker
sudo systemctl enable docker

kind のインストール

sudo curl -Lo /usr/bin/kind https://github.com/kubernetes-sigs/kind/releases/download/v0.5.1/kind-linux-amd64
sudo chmod +x /usr/bin/kind

kubectl のインストール

sudo curl -Lo /usr/bin/kubectl https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
sudo chmod +x /usr/bin/kubectl

kind の実行

kind create cluster と実行するだけで Kubernetes クラスタが起動する。

$ kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.15.3) 🖼
 ✓ Preparing nodes 📦
 ✓ Creating kubeadm config 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Cluster creation complete. You can now use the cluster with:

export KUBECONFIG="$(kind get kubeconfig-path --name="kind")"
kubectl cluster-info

けど、docker ps では1コンテナ起動してるだけだな

$ docker ps
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS                                  NAMES
260a0db4daf7        kindest/node:v1.15.3   "/usr/local/bin/entr…"   13 minutes ago      Up 13 minutes       37630/tcp, 127.0.0.1:37630->6443/tcp   kind-control-plane

docker exec で ps してみると次のようになっている

# ps auxwwf
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      9013  0.2  0.0   4040  2076 pts/1    Ss   14:25   0:00 bash
root      9122  0.0  0.0   5972  1464 pts/1    R+   14:25   0:00  \_ ps auxwwf
root         1  0.0  0.0  17648  6260 ?        Ss   14:10   0:00 /sbin/init
root        34  0.0  0.0  24640  6648 ?        S<s  14:10   0:00 /lib/systemd/systemd-journald
root        45  2.1  0.6 2360932 48788 ?       Ssl  14:10   0:19 /usr/bin/containerd
root       283  0.0  0.0  10732  3768 ?        Sl   14:10   0:00  \_ containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/03e619f69de43ddc43b1641fb24ac9b0b0c362aa6018999b81b6e894995b72bb -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd
root       349  0.0  0.0   1012     4 ?        Ss   14:10   0:00  |   \_ /pause
root       294  0.0  0.0   9324  3308 ?        Sl   14:10   0:00  \_ containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/6fd68362e810b18a2356142f473499626842e55fd1e38dc60dd602c2b4f918c6 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd
root       329  0.0  0.0   1012     4 ?        Ss   14:10   0:00  |   \_ /pause
root       316  0.0  0.0  10732  3796 ?        Sl   14:10   0:00  \_ containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/9f055513abd1d8723515ad210dfcd3b31ce8d262d3aacbe48bbe575c4e46ac31 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd
root       358  0.0  0.0   1012     4 ?        Ss   14:10   0:00  |   \_ /pause
root       317  0.0  0.0  10732  3564 ?        Sl   14:10   0:00  \_ containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/14df61c598ebf1f6fed8bebc9c4f80b1f11a37a16c29f175d66136525c9e6b60 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd
root       363  0.0  0.0   1012     4 ?        Ss   14:10   0:00  |   \_ /pause
root       491  0.0  0.0  10732  3540 ?        Sl   14:11   0:00  \_ containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/d71810b32bd0ae25b4406449803f7ef496f5ca3cbd6f784557538b654871b709 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd
root       508  0.3  0.3 141480 29196 ?        Ssl  14:11   0:02  |   \_ kube-scheduler --bind-address=127.0.0.1 --kubeconfig=/etc/kubernetes/scheduler.conf --leader-elect=true
root       542  0.0  0.0  10732  3716 ?        Sl   14:11   0:00  \_ containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/b9dcfe2ef38c93bbc0cb9dbde378b4a5824ccc208c86d39629d668dc4cd58489 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd
root       574  1.8  0.9 217568 72104 ?        Ssl  14:11   0:16  |   \_ kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/etc/kubernetes/pki/ca.crt --cluster-cidr=10.244.0.0/16 --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt --cluster-signing-key-file=/etc/kubernetes/pki/ca.key --controllers=*,bootstrapsigner,tokencleaner --enable-hostpath-provisioner=true --kubeconfig=/etc/kubernetes/controller-manager.conf --leader-elect=true --node-cidr-mask-size=24 --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --root-ca-file=/etc/kubernetes/pki/ca.crt --service-account-private-key-file=/etc/kubernetes/pki/sa.key --use-service-account-credentials=true
root       551  0.2  0.0  11788  4804 ?        Sl   14:11   0:02  \_ containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/973aec49f3de28367798e25733cd4863397cccc9aab775d9dbcbe145381f787c -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd
root       580  1.9  0.4 10537600 34244 ?      Ssl  14:11   0:17  |   \_ etcd --advertise-client-urls=https://172.17.0.2:2379 --cert-file=/etc/kubernetes/pki/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/etcd --initial-advertise-peer-urls=https://172.17.0.2:2380 --initial-cluster=kind-control-plane=https://172.17.0.2:2380 --key-file=/etc/kubernetes/pki/etcd/server.key --listen-client-urls=https://127.0.0.1:2379,https://172.17.0.2:2379 --listen-peer-urls=https://172.17.0.2:2380 --name=kind-control-plane --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt --peer-client-cert-auth=true --peer-key-file=/etc/kubernetes/pki/etcd/peer.key --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt --snapshot-count=10000 --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
root       639  0.0  0.0  10732  3628 ?        Sl   14:11   0:00  \_ containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/af9f6dfbc2bb2a8b807e0028de9020c77dea646339b835cc90be74884e6264bd -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd
root       661  3.8  2.8 402920 224844 ?       Ssl  14:11   0:34  |   \_ kube-apiserver --advertise-address=172.17.0.2 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
root       862  0.0  0.0   9324  3536 ?        Sl   14:11   0:00  \_ containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/4879d4a5e3d75990f15f95fb7e54c1afb80396fe59fcfcab4f43efc47e9103ff -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd
root       879  0.0  0.0   1012     4 ?        Ss   14:11   0:00  |   \_ /pause
root       884  0.0  0.0   9324  3400 ?        Sl   14:11   0:00  \_ containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/e9f36ddb48b6571ba9998ac5a5625b92c2b9908b9b7070d8566af02860295d74 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd
root       907  0.0  0.0   1012     4 ?        Ss   14:11   0:00  |   \_ /pause
root       963  0.0  0.0   9324  3240 ?        Sl   14:11   0:00  \_ containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/f6fd15e55e522bc4b48ba3b1672963e2ad652f3d04c69a60fae9bfa0a026924a -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd
root      1002  0.1  0.3 139724 24172 ?        Ssl  14:11   0:01  |   \_ /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=kind-control-plane
root       967  0.0  0.0  10732  5264 ?        Sl   14:11   0:00  \_ containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/bf7aea0752d89b06394670a2ce49da52096604beff7cfeddaf057eb83d6c030d -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd
root       995  0.0  0.1 130224 15516 ?        Ssl  14:11   0:00  |   \_ /bin/kindnetd
root      1246  0.0  0.0  10796  3500 ?        Sl   14:11   0:00  \_ containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/828fb3f0c098eb67209152d7697c468bfe9fa508a440f3bb9b56120ae6336f3e -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd
root      1263  0.0  0.0   1012     4 ?        Ss   14:11   0:00  |   \_ /pause
root      1297  0.0  0.0   9324  3544 ?        Sl   14:11   0:00  \_ containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/3c1bbc921022e0151d4d32ca680462ea14130fd5e823cdfbf8317c275f90b4d3 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd
root      1314  0.3  0.3 142788 24596 ?        Ssl  14:11   0:02  |   \_ /coredns -conf /etc/coredns/Corefile
root      1375  0.0  0.0   9324  3628 ?        Sl   14:11   0:00  \_ containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/11963940dfda62ca5eea86ebaeffc1441113b059423b9a1e4515165fe5d58b92 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd
root      1392  0.0  0.0   1012     4 ?        Ss   14:11   0:00  |   \_ /pause
root      1430  0.0  0.0  10732  3612 ?        Sl   14:11   0:00  \_ containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/bd01f1f3ae96c3b6a392c0055ac4a10afdadff7695078c7b34ce8a6de2decca7 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd
root      1447  0.2  0.3 142788 24136 ?        Ssl  14:11   0:02      \_ /coredns -conf /etc/coredns/Corefile
root       239  2.4  0.8 1623832 68848 ?       Ssl  14:10   0:22 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=remote --container-runtime-endpoint=/run/containerd/containerd.sock --fail-swap-on=false --node-ip=172.17.0.2 --fail-swap-on=false

ふむふむ、Docker in Docker ですね。

kubectl で確認

$ export KUBECONFIG="$(kind get kubeconfig-path --name="kind")"
$ kubectl cluster-info
Kubernetes master is running at https://127.0.0.1:37630
KubeDNS is running at https://127.0.0.1:37630/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
$ kubectl get services
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    443/TCP   33m
[centos@kind ~]$ kubectl get services --all-namespaces
NAMESPACE     NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       kubernetes   ClusterIP   10.96.0.1    443/TCP                  33m
kube-system   kube-dns     ClusterIP   10.96.0.10   53/UDP,53/TCP,9153/TCP   33m 

あ、たまたま kubectl と同じバージョンだったけどバージョン合わせるように気を付ける必要があるか

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-20T18:57:36Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

Multi-node cluster

kind create に –config でコンフィグを渡せば node 数を指定することができる。kind-example-config.yaml

curl -LO https://raw.githubusercontent.com/kubernetes-sigs/kind/master/site/content/docs/user/kind-example-config.yaml
$ kind create cluster --config kind-example-config.yaml
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.15.3) 🖼
 ✓ Preparing nodes 📦📦📦📦
 ✓ Creating kubeadm config 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
 ✓ Joining worker nodes 🚜
Cluster creation complete. You can now use the cluster with:

export KUBECONFIG="$(kind get kubeconfig-path --name="kind")"
kubectl cluster-info
$ kubectl get nodes
NAME                 STATUS   ROLES    AGE     VERSION
kind-control-plane   Ready    master   2m40s   v1.15.3
kind-worker          Ready    <none>   2m4s    v1.15.3
kind-worker2         Ready    <none>   2m4s    v1.15.3
kind-worker3         Ready    <none>   2m3s    v1.15.3

Control Plane も複数台にするには kind-example-config.yaml の nodes を次のように変更して kind create すれば Control Plane が3台になる

nodes:
- role: control-plane
- role: control-plane
- role: control-plane
- role: worker
- role: worker
- role: worker
$ kubectl get nodes
NAME                  STATUS   ROLES    AGE     VERSION
kind-control-plane    Ready    master   2m36s   v1.15.3
kind-control-plane2   Ready    master   2m2s    v1.15.3
kind-control-plane3   Ready    master   86s     v1.15.3
kind-worker           Ready    60s     v1.15.3
kind-worker2          Ready    60s     v1.15.3
kind-worker3          Ready    60s     v1.15.3 

便利ツールだ。kind のドキュメントは https://kind.sigs.k8s.io/ にある。

Built with Hugo
テーマ StackJimmy によって設計されています。