Cilium ってなんだ?
目次
Cilium 1.0: Bringing the BPF Revolution to Kubernetes Networking and Security という記事を見かけた Cilium っていうのが Version 1.0 になったそうだ。はて?これは何をもったらすものだろう? コンテナ化、マイクロサービス化が進みさまざまなサービス同士が通信するようになりますが、守るべき API があるし、どことどこの通信を許可するかというのを iptables による L3, L4 での制御ではスケールしない、ルールが万を超えたり、非常に頻繁に更新する必要もあるため BPF というものでもっとシンプルにしようというもののようです Introduction to Cilium では機能として次のようなものが挙げられています
- Protect and secure APIs transparently
- Secure service to service communication based on identities
- Secure access to and from external services
- Simple Networking
- Load balancing
- Monitoring and Troubleshooting
github.com/cilium/cilium
HTTP, gRPC, and Kafka Aware Security and Networking for Containers with BPF and XDP Cilium Getting Started Guides ここでは Getting Started Using Minikube をなぞってみます
Linux (ubuntu) #
今回は Windows で試すんだけどメモ
kubectl のインストール #
apt-get update && apt-get install -y apt-transport-https
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubectl
minikube のインストール #
Releases · kubernetes/minikube からバイナリを取ってくるだけ
curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.26.1/minikube-linux-amd64 \
&& chmod +x minikube && sudo mv minikube /usr/local/bin/
Windows #
kubectl のインストール #
curl -Lo ~/bin/kubectl.exe \
https://storage.googleapis.com/kubernetes-release/release/v1.10.0/bin/windows/amd64/kubectl.exe
minikube のインストール #
curl -Lo ~/bin/minikube.exe \
https://github.com/kubernetes/minikube/releases/download/v0.26.1/minikube-windows-amd64
minikube で kubernetes 環境をセットアップ #
メモリを4GBにし RBAC を有効にして構築します
minikube start \
--vm-driver=virtualbox \
--network-plugin=cni \
--bootstrapper=localkube \
--memory=4096 \
--extra-config=apiserver.Authorization.Mode=RBAC
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Downloading Minikube ISO
150.53 MB / 150.53 MB 100.00% 0sss
Getting VM IP address...
WARNING: The localkube bootstrapper is now deprecated and support for it
will be removed in a future release. Please consider switching to the kubeadm bootstrapper, which
is intended to replace the localkube bootstrapper. To disable this message, run
[minikube config set ShowBootstrapperDeprecationNotification false]
Moving files into cluster...
Downloading localkube binary
173.54 MB / 173.54 MB 100.00% 0sss
65 B / 65 B 100.00% 0s
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
Kubectl is now configured to use the cluster.
Loading cached images from config file.
(localkube bootstrapper は deprecated で kubeadm bootstrapper 使おうねって WARNING が出てるな) kubectl get cs
で componentstatuses を確認
$ kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health": "true"}
全部 Healthy なので構築できてるようです RBAC
の有効な環境で kube-dns
サービスを有効にできるよう Kubernetes のシステムアカウントを cluster-admin
ロールに紐づけます
$ kubectl create clusterrolebinding kube-system-default-binding-cluster-admin \
--clusterrole=cluster-admin \
--serviceaccount=kube-system:default
clusterrolebinding.rbac.authorization.k8s.io "kube-system-default-binding-cluster-admin" created
standalone-etcd.yaml で Cilium 用の etcd サービスを作成します
$ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/doc-1.0/examples/kubernetes/addons/etcd/standalone-etcd.yaml
service "etcd-cilium" created
statefulset.apps "etcd-cilium" created
$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default etcd-cilium-0 1/1 Running 0 54s
kube-system kube-addon-manager-minikube 1/1 Running 0 3m
kube-system kube-dns-6dcb57bcc8-g58kh 3/3 Running 0 2m
kube-system kubernetes-dashboard-5498ccf677-9xgdz 1/1 Running 3 2m
kube-system storage-provisioner 1/1 Running 0 2m
cilium.yaml で cilium サービスとそれに必要な ConfigMap や Secret、ServiceAccount、ClusterRole などを作成します
$ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/doc-1.0/examples/kubernetes/1.10/cilium.yaml
configmap "cilium-config" created
secret "cilium-etcd-secrets" created
daemonset.apps "cilium" created
clusterrolebinding.rbac.authorization.k8s.io "cilium" created
clusterrole.rbac.authorization.k8s.io "cilium" created
serviceaccount "cilium" created
ContainerCreating
なので Running
になるまでしばらく待つ
$ kubectl get pods --namespace kube-system
NAME READY STATUS RESTARTS AGE
cilium-rg6dc 0/1 ContainerCreating 0 47s
kube-addon-manager-minikube 1/1 Running 0 6m
kube-dns-6dcb57bcc8-g58kh 3/3 Running 0 5m
kubernetes-dashboard-5498ccf677-9xgdz 1/1 Running 3 5m
storage-provisioner 1/1 Running 0 5m
起動しました
$ kubectl get pods --namespace kube-system
NAME READY STATUS RESTARTS AGE
cilium-rg6dc 0/1 Running 0 1m
kube-addon-manager-minikube 1/1 Running 0 6m
kube-dns-6dcb57bcc8-g58kh 3/3 Running 0 6m
kubernetes-dashboard-5498ccf677-9xgdz 1/1 Running 3 6m
storage-provisioner 1/1 Running 0 6m
次に http-sw-app.yaml を使って cilium/starwars イメージを使った deathstar サービス (コンテナ2つ) と tgraf/netperf イメージを使った tiefighter と xwing というコンテナ (Deployment) を作成します
$ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/doc-1.0/examples/minikube/http-sw-app.yaml
service "deathstar" created
deployment.apps "deathstar" created
deployment.apps "tiefighter" created
deployment.apps "xwing" created
Running
になるまでしばらく待つ
$ kubectl get pods,svc
NAME READY STATUS RESTARTS AGE
deathstar-765fd545f9-grr5g 0/1 ContainerCreating 0 46s
deathstar-765fd545f9-pk9h8 0/1 ContainerCreating 0 46s
etcd-cilium-0 1/1 Running 0 5m
tiefighter-787b4ff698-kpmgl 0/1 ContainerCreating 0 46s
xwing-d56b5c5-sdw7q 0/1 ContainerCreating 0 45s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
deathstar ClusterIP 10.105.66.9 <none> 80/TCP 46s
etcd-cilium NodePort 10.109.105.129 <none> 32379:31079/TCP,32380:31080/TCP 5m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 8m
Running
に変わった
$ kubectl get pods,svc
NAME READY STATUS RESTARTS AGE
deathstar-765fd545f9-grr5g 1/1 Running 0 3m
deathstar-765fd545f9-pk9h8 1/1 Running 0 3m
etcd-cilium-0 1/1 Running 0 8m
tiefighter-787b4ff698-kpmgl 1/1 Running 0 3m
xwing-d56b5c5-sdw7q 1/1 Running 0 3m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
deathstar ClusterIP 10.105.66.9 <none> 80/TCP 3m
etcd-cilium NodePort 10.109.105.129 <none> 32379:31079/TCP,32380:31080/TCP 8m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 11m
Cilium の Pod 名確認
$ kubectl -n kube-system get pods -l k8s-app=cilium
NAME READY STATUS RESTARTS AGE
cilium-rg6dc 1/1 Running 0 6m
kubectl exec
で Cilium Pod 内で cilium endpoint list
コマンドを実行するとエンドポイントの一覧が表示されます。deathstar が2つ、tiefighter、xwing が1つずつ
$ kubectl -n kube-system exec cilium-rg6dc cilium endpoint list
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
173 Disabled Disabled 64509 k8s:class=deathstar f00d::a0f:0:0:ad 10.15.42.252 ready
k8s:io.kubernetes.pod.namespace=default
k8s:org=empire
13949 Disabled Disabled 44856 k8s:class=tiefighter f00d::a0f:0:0:367d 10.15.193.100 ready
k8s:io.kubernetes.pod.namespace=default
k8s:org=empire
29381 Disabled Disabled 61850 k8s:class=xwing f00d::a0f:0:0:72c5 10.15.13.37 ready
k8s:io.kubernetes.pod.namespace=default
k8s:org=alliance
29898 Disabled Disabled 51604 reserved:health f00d::a0f:0:0:74ca 10.15.242.54 ready
48896 Disabled Disabled 64509 k8s:class=deathstar f00d::a0f:0:0:bf00 10.15.167.158 ready
k8s:io.kubernetes.pod.namespace=default
k8s:org=empire
xwing から deathstar に curl でアクセスしてみる
$ kubectl exec xwing-d56b5c5-sdw7q -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Ship landed
tiefighter から deathstar に curl でアクセスしてみる
$ kubectl exec tiefighter-787b4ff698-kpmgl -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Ship landed
どちらもアクセス可能でした sw_l3_l4_policy.yaml を使って Cilium のルールを作成します description: "L3-L4 policy to restrict deathstar access to empire ships only"
にあるように deathstar の ingress に制限をいれます。label で org=empire となっている接続元からのみ port 80 へのアクセスを許可します
$ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/doc-1.0/examples/minikube/sw_l3_l4_policy.yaml
ciliumnetworkpolicy.cilium.io "rule1" created
xwing は org=alliance であるため deathstar にアクセスできなくなりました
$ kubectl exec xwing-d56b5c5-sdw7q -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
つながらない 再度 endpoint list を確認します deathstar の POLICY (ingress) ENFORCEMENT
が Enabled
になってます
$ kubectl -n kube-system exec cilium-rg6dc cilium endpoint list
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
173 Enabled Disabled 64509 k8s:class=deathstar f00d::a0f:0:0:ad 10.15.42.252 ready
k8s:io.kubernetes.pod.namespace=default
k8s:org=empire
13949 Disabled Disabled 44856 k8s:class=tiefighter f00d::a0f:0:0:367d 10.15.193.100 ready
k8s:io.kubernetes.pod.namespace=default
k8s:org=empire
29381 Disabled Disabled 61850 k8s:class=xwing f00d::a0f:0:0:72c5 10.15.13.37 ready
k8s:io.kubernetes.pod.namespace=default
k8s:org=alliance
29898 Disabled Disabled 51604 reserved:health f00d::a0f:0:0:74ca 10.15.242.54 ready
48896 Enabled Disabled 64509 k8s:class=deathstar f00d::a0f:0:0:bf00 10.15.167.158 ready
k8s:io.kubernetes.pod.namespace=default
k8s:org=empire
cnp (CiliumNetworkPolicy) を確認
$ kubectl get cnp
NAME AGE
rule1 4m
Describe で詳細確認
$ kubectl describe cnp rule1
Name: rule1
Namespace: default
Labels: <none>
Annotations: <none>
API Version: cilium.io/v2
Kind: CiliumNetworkPolicy
Metadata:
Cluster Name:
Creation Timestamp: 2018-05-08T16:09:44Z
Generation: 1
Resource Version: 1177
Self Link: /apis/cilium.io/v2/namespaces/default/ciliumnetworkpolicies/rule1
UID: 391b40cd-52da-11e8-8ef9-080027a11e00
Spec:
Endpoint Selector:
Match Labels:
Any : Class: deathstar
Any : Org: empire
Ingress:
From Endpoints:
Match Labels:
Any : Org: empire
To Ports:
Ports:
Port: 80
Protocol: TCP
Status:
Nodes:
Minikube:
Enforcing: true
Last Updated: 2018-05-08T16:09:52.34442593Z
Local Policy Revision: 7
Ok: true
Events: <none>
続いて L7 での制限です tiefighter (org=empire) からでも許可したくないエンドポイントが deathstar にあるかもしれません、たとえば次の /v1/exhaust-port
とか
$ kubectl exec tiefighter-787b4ff698-kpmgl -- curl -s -XPUT deathstar.default.svc.cluster.local/v1/exhaust-port
Panic: deathstar exploded
goroutine 1 [running]:
main.HandleGarbage(0x2080c3f50, 0x2, 0x4, 0x425c0, 0x5, 0xa)
/code/src/github.com/empire/deathstar/
temp/main.go:9 +0x64
main.main()
/code/src/github.com/empire/deathstar/
temp/main.go:5 +0x85
sw_l3_l4_l7_policy.yaml で L7 で制御してみます org=empire でかつ /v1/request-landing
への POST
だけを許可するという Rule です
$ kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/doc-1.0/examples/minikube/sw_l3_l4_l7_policy.yaml
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
ciliumnetworkpolicy.cilium.io "rule1" configured
tiefighter から /v1/request-landing
へはアクセスできました
$ kubectl exec tiefighter-787b4ff698-kpmgl -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Ship landed
が、/v1/exhaust-port
では Access denied となりました
$ kubectl exec tiefighter-787b4ff698-kpmgl -- curl -s -XPUT deathstar.default.svc.cluster.local/v1/exhaust-port
Access denied
また cnp (CiliumNetworkPolicy) を確認しておきます
$ kubectl get ciliumnetworkpolicies
NAME AGE
rule1 9m
$ kubectl describe ciliumnetworkpolicies rule1
Name: rule1
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"cilium.io/v2","description":"L7 policy to restrict access to specific HTTP call","kind":"CiliumNetworkPolicy","metadata":{"annotations":...
API Version: cilium.io/v2
Kind: CiliumNetworkPolicy
Metadata:
Cluster Name:
Creation Timestamp: 2018-05-08T16:09:44Z
Generation: 1
Resource Version: 1709
Self Link: /apis/cilium.io/v2/namespaces/default/ciliumnetworkpolicies/rule1
UID: 391b40cd-52da-11e8-8ef9-080027a11e00
Spec:
Endpoint Selector:
Match Labels:
Any : Class: deathstar
Any : Org: empire
Ingress:
From Endpoints:
Match Labels:
Any : Org: empire
To Ports:
Ports:
Port: 80
Protocol: TCP
Rules:
Http:
Method: POST
Path: /v1/request-landing
Status:
Nodes:
Minikube:
Enforcing: true
Last Updated: 2018-05-08T16:17:44.124075147Z
Local Policy Revision: 11
Ok: true
Events: <none>
Cilium Pod 内で cilium policy get
として確認することもできるようです
$ kubectl -n kube-system exec cilium-rg6dc cilium policy get
[
{
"endpointSelector": {
"matchLabels": {
"any:class": "deathstar",
"any:org": "empire",
"k8s:io.kubernetes.pod.namespace": "default"
}
},
"ingress": [
{
"fromEndpoints": [
{
"matchLabels": {
"any:org": "empire",
"k8s:io.kubernetes.pod.namespace": "default"
}
}
],
"toPorts": [
{
"ports": [
{
"port": "80",
"protocol": "TCP"
}
],
"rules": {
"http": [
{
"path": "/v1/request-landing",
"method": "POST"
}
]
}
}
]
}
],
"labels": [
{
"key": "io.cilium.k8s.policy.name",
"value": "rule1",
"source": "unspec"
},
{
"key": "io.cilium.k8s.policy.namespace",
"value": "default",
"source": "unspec"
}
]
}
]
Revision: 11
Prometheus 連携 #
prometheus.yaml (document と path が変わってた) で Prometheus サービスをセットアップします。kubernetes service discovery で cilium コンテナを動的に見つけて cilium-agent から polling するようです
$ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/doc-1.0/examples/kubernetes/prometheus.yaml
namespace "prometheus" created
service "prometheus" created
deployment.extensions "prometheus-core" created
configmap "prometheus-core" created
clusterrolebinding.rbac.authorization.k8s.io "prometheus" created
clusterrole.rbac.authorization.k8s.io "prometheus" created
serviceaccount "prometheus-k8s" created
configmap "cilium-metrics-config" created
Cilium の再起動
$ kubectl replace --force -f https://raw.githubusercontent.com/cilium/cilium/doc-1.0/examples/kubernetes/1.10/cilium.yaml
configmap "cilium-config" deleted
secret "cilium-etcd-secrets" deleted
daemonset.apps "cilium" deleted
clusterrolebinding.rbac.authorization.k8s.io "cilium" deleted
clusterrole.rbac.authorization.k8s.io "cilium" deleted
serviceaccount "cilium" deleted
configmap "cilium-config" replaced
secret "cilium-etcd-secrets" replaced
daemonset.apps "cilium" replaced
clusterrolebinding.rbac.authorization.k8s.io "cilium" replaced
clusterrole.rbac.authorization.k8s.io "cilium" replaced
serviceaccount "cilium" replaced
後片付け #
$ minikube delete
Deleting local Kubernetes cluster...
Machine deleted.
まとめ #
詳しい仕組みは確認してないけど Label を使って送信元、送信先を指定でき、L4, L7 でアクセス制御できる便利ツールだった この方のブロクを見ると BPF について理解が深まるだろうか LinuxのBPF : (1) パケットフィルタ - 睡分不足 - mm_i - はてなブログ