Cilium 1.0: Bringing the BPF Revolution to Kubernetes Networking and Security という記事を見かけた Cilium っていうのが Version 1.0 になったそうだ。はて？これは何をもったらすものだろう？ コンテナ化、マイクロサービス化が進みさまざまなサービス同士が通信するようになりますが、守るべき API があるし、どことどこの通信を許可するかというのを iptables による L3, L4 での制御ではスケールしない、ルールが万を超えたり、非常に頻繁に更新する必要もあるため BPF というものでもっとシンプルにしようというもののようです Introduction to Cilium では機能として次のようなものが挙げられています
- Protect and secure APIs transparently
- Secure service to service communication based on identities
- Secure access to and from external services
- Simple Networking
- Load balancing
- Monitoring and Troubleshooting
github.com/cilium/cilium
HTTP, gRPC, and Kafka Aware Security and Networking for Containers with BPF and XDP Cilium Getting Started Guides ここでは Getting Started Using Minikube をなぞってみます
Linux (ubuntu)
今回は Windows で試すんだけどメモ
kubectl のインストール
| |
minikube のインストール
Releases · kubernetes/minikube からバイナリを取ってくるだけ
| |
Windows
kubectl のインストール
| |
minikube のインストール
| |
minikube で kubernetes 環境をセットアップ
メモリを4GBにし RBAC を有効にして構築します
minikube start \
--vm-driver=virtualbox \
--network-plugin=cni \
--bootstrapper=localkube \
--memory=4096 \
--extra-config=apiserver.Authorization.Mode=RBACStarting local Kubernetes v1.10.0 cluster...
Starting VM...
Downloading Minikube ISO
150.53 MB / 150.53 MB 100.00% 0sss
Getting VM IP address...
WARNING: The localkube bootstrapper is now deprecated and support for it
will be removed in a future release. Please consider switching to the kubeadm bootstrapper, which
is intended to replace the localkube bootstrapper. To disable this message, run
[minikube config set ShowBootstrapperDeprecationNotification false]
Moving files into cluster...
Downloading localkube binary
173.54 MB / 173.54 MB 100.00% 0sss
65 B / 65 B 100.00% 0s
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
Kubectl is now configured to use the cluster.
Loading cached images from config file.(localkube bootstrapper は deprecated で kubeadm bootstrapper 使おうねって WARNING が出てるな) kubectl get cs で componentstatuses を確認
$ kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health": "true"}全部 Healthy なので構築できてるようです RBAC の有効な環境で kube-dns サービスを有効にできるよう Kubernetes のシステムアカウントを cluster-admin ロールに紐づけます
$ kubectl create clusterrolebinding kube-system-default-binding-cluster-admin \
--clusterrole=cluster-admin \
--serviceaccount=kube-system:default
clusterrolebinding.rbac.authorization.k8s.io "kube-system-default-binding-cluster-admin" createdstandalone-etcd.yaml で Cilium 用の etcd サービスを作成します
$ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/doc-1.0/examples/kubernetes/addons/etcd/standalone-etcd.yaml
service "etcd-cilium" created
statefulset.apps "etcd-cilium" created$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default etcd-cilium-0 1/1 Running 0 54s
kube-system kube-addon-manager-minikube 1/1 Running 0 3m
kube-system kube-dns-6dcb57bcc8-g58kh 3/3 Running 0 2m
kube-system kubernetes-dashboard-5498ccf677-9xgdz 1/1 Running 3 2m
kube-system storage-provisioner 1/1 Running 0 2mcilium.yaml で cilium サービスとそれに必要な ConfigMap や Secret、ServiceAccount、ClusterRole などを作成します
$ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/doc-1.0/examples/kubernetes/1.10/cilium.yaml
configmap "cilium-config" created
secret "cilium-etcd-secrets" created
daemonset.apps "cilium" created
clusterrolebinding.rbac.authorization.k8s.io "cilium" created
clusterrole.rbac.authorization.k8s.io "cilium" created
serviceaccount "cilium" createdContainerCreating なので Running になるまでしばらく待つ
$ kubectl get pods --namespace kube-system
NAME READY STATUS RESTARTS AGE
cilium-rg6dc 0/1 ContainerCreating 0 47s
kube-addon-manager-minikube 1/1 Running 0 6m
kube-dns-6dcb57bcc8-g58kh 3/3 Running 0 5m
kubernetes-dashboard-5498ccf677-9xgdz 1/1 Running 3 5m
storage-provisioner 1/1 Running 0 5m起動しました
$ kubectl get pods --namespace kube-system
NAME READY STATUS RESTARTS AGE
cilium-rg6dc 0/1 Running 0 1m
kube-addon-manager-minikube 1/1 Running 0 6m
kube-dns-6dcb57bcc8-g58kh 3/3 Running 0 6m
kubernetes-dashboard-5498ccf677-9xgdz 1/1 Running 3 6m
storage-provisioner 1/1 Running 0 6m次に http-sw-app.yaml を使って cilium/starwars イメージを使った deathstar サービス (コンテナ2つ) と tgraf/netperf イメージを使った tiefighter と xwing というコンテナ (Deployment) を作成します
$ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/doc-1.0/examples/minikube/http-sw-app.yaml
service "deathstar" created
deployment.apps "deathstar" created
deployment.apps "tiefighter" created
deployment.apps "xwing" createdRunning になるまでしばらく待つ
$ kubectl get pods,svc
NAME READY STATUS RESTARTS AGE
deathstar-765fd545f9-grr5g 0/1 ContainerCreating 0 46s
deathstar-765fd545f9-pk9h8 0/1 ContainerCreating 0 46s
etcd-cilium-0 1/1 Running 0 5m
tiefighter-787b4ff698-kpmgl 0/1 ContainerCreating 0 46s
xwing-d56b5c5-sdw7q 0/1 ContainerCreating 0 45s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
deathstar ClusterIP 10.105.66.9 <none> 80/TCP 46s
etcd-cilium NodePort 10.109.105.129 <none> 32379:31079/TCP,32380:31080/TCP 5m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 8mRunning に変わった
$ kubectl get pods,svc
NAME READY STATUS RESTARTS AGE
deathstar-765fd545f9-grr5g 1/1 Running 0 3m
deathstar-765fd545f9-pk9h8 1/1 Running 0 3m
etcd-cilium-0 1/1 Running 0 8m
tiefighter-787b4ff698-kpmgl 1/1 Running 0 3m
xwing-d56b5c5-sdw7q 1/1 Running 0 3m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
deathstar ClusterIP 10.105.66.9 <none> 80/TCP 3m
etcd-cilium NodePort 10.109.105.129 <none> 32379:31079/TCP,32380:31080/TCP 8m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 11mCilium の Pod 名確認
$ kubectl -n kube-system get pods -l k8s-app=cilium
NAME READY STATUS RESTARTS AGE
cilium-rg6dc 1/1 Running 0 6mkubectl exec で Cilium Pod 内で cilium endpoint list コマンドを実行するとエンドポイントの一覧が表示されます。deathstar が2つ、tiefighter、xwing が1つずつ
$ kubectl -n kube-system exec cilium-rg6dc cilium endpoint list
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
173 Disabled Disabled 64509 k8s:class=deathstar f00d::a0f:0:0:ad 10.15.42.252 ready
k8s:io.kubernetes.pod.namespace=default
k8s:org=empire
13949 Disabled Disabled 44856 k8s:class=tiefighter f00d::a0f:0:0:367d 10.15.193.100 ready
k8s:io.kubernetes.pod.namespace=default
k8s:org=empire
29381 Disabled Disabled 61850 k8s:class=xwing f00d::a0f:0:0:72c5 10.15.13.37 ready
k8s:io.kubernetes.pod.namespace=default
k8s:org=alliance
29898 Disabled Disabled 51604 reserved:health f00d::a0f:0:0:74ca 10.15.242.54 ready
48896 Disabled Disabled 64509 k8s:class=deathstar f00d::a0f:0:0:bf00 10.15.167.158 ready
k8s:io.kubernetes.pod.namespace=default
k8s:org=empirexwing から deathstar に curl でアクセスしてみる
$ kubectl exec xwing-d56b5c5-sdw7q -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Ship landedtiefighter から deathstar に curl でアクセスしてみる
$ kubectl exec tiefighter-787b4ff698-kpmgl -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Ship landedどちらもアクセス可能でした sw_l3_l4_policy.yaml を使って Cilium のルールを作成します description: "L3-L4 policy to restrict deathstar access to empire ships only" にあるように deathstar の ingress に制限をいれます。label で org=empire となっている接続元からのみ port 80 へのアクセスを許可します
$ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/doc-1.0/examples/minikube/sw_l3_l4_policy.yaml
ciliumnetworkpolicy.cilium.io "rule1" createdxwing は org=alliance であるため deathstar にアクセスできなくなりました
$ kubectl exec xwing-d56b5c5-sdw7q -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landingつながらない 再度 endpoint list を確認します deathstar の POLICY (ingress) ENFORCEMENT が Enabled になってます
$ kubectl -n kube-system exec cilium-rg6dc cilium endpoint list
ENDPOINT POLICY (ingress) POLICY (egress) IDENTITY LABELS (source:key[=value]) IPv6 IPv4 STATUS
ENFORCEMENT ENFORCEMENT
173 Enabled Disabled 64509 k8s:class=deathstar f00d::a0f:0:0:ad 10.15.42.252 ready
k8s:io.kubernetes.pod.namespace=default
k8s:org=empire
13949 Disabled Disabled 44856 k8s:class=tiefighter f00d::a0f:0:0:367d 10.15.193.100 ready
k8s:io.kubernetes.pod.namespace=default
k8s:org=empire
29381 Disabled Disabled 61850 k8s:class=xwing f00d::a0f:0:0:72c5 10.15.13.37 ready
k8s:io.kubernetes.pod.namespace=default
k8s:org=alliance
29898 Disabled Disabled 51604 reserved:health f00d::a0f:0:0:74ca 10.15.242.54 ready
48896 Enabled Disabled 64509 k8s:class=deathstar f00d::a0f:0:0:bf00 10.15.167.158 ready
k8s:io.kubernetes.pod.namespace=default
k8s:org=empirecnp (CiliumNetworkPolicy) を確認
$ kubectl get cnp
NAME AGE
rule1 4mDescribe で詳細確認
$ kubectl describe cnp rule1
Name: rule1
Namespace: default
Labels: <none>
Annotations: <none>
API Version: cilium.io/v2
Kind: CiliumNetworkPolicy
Metadata:
Cluster Name:
Creation Timestamp: 2018-05-08T16:09:44Z
Generation: 1
Resource Version: 1177
Self Link: /apis/cilium.io/v2/namespaces/default/ciliumnetworkpolicies/rule1
UID: 391b40cd-52da-11e8-8ef9-080027a11e00
Spec:
Endpoint Selector:
Match Labels:
Any : Class: deathstar
Any : Org: empire
Ingress:
From Endpoints:
Match Labels:
Any : Org: empire
To Ports:
Ports:
Port: 80
Protocol: TCP
Status:
Nodes:
Minikube:
Enforcing: true
Last Updated: 2018-05-08T16:09:52.34442593Z
Local Policy Revision: 7
Ok: true
Events: <none>続いて L7 での制限です tiefighter (org=empire) からでも許可したくないエンドポイントが deathstar にあるかもしれません、たとえば次の /v1/exhaust-port とか
$ kubectl exec tiefighter-787b4ff698-kpmgl -- curl -s -XPUT deathstar.default.svc.cluster.local/v1/exhaust-port
Panic: deathstar exploded
goroutine 1 [running]:
main.HandleGarbage(0x2080c3f50, 0x2, 0x4, 0x425c0, 0x5, 0xa)
/code/src/github.com/empire/deathstar/
temp/main.go:9 +0x64
main.main()
/code/src/github.com/empire/deathstar/
temp/main.go:5 +0x85sw_l3_l4_l7_policy.yaml で L7 で制御してみます org=empire でかつ /v1/request-landing への POST だけを許可するという Rule です
$ kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/doc-1.0/examples/minikube/sw_l3_l4_l7_policy.yaml
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
ciliumnetworkpolicy.cilium.io "rule1" configuredtiefighter から /v1/request-landing へはアクセスできました
$ kubectl exec tiefighter-787b4ff698-kpmgl -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Ship landedが、/v1/exhaust-port では Access denied となりました
$ kubectl exec tiefighter-787b4ff698-kpmgl -- curl -s -XPUT deathstar.default.svc.cluster.local/v1/exhaust-port
Access deniedまた cnp (CiliumNetworkPolicy) を確認しておきます
$ kubectl get ciliumnetworkpolicies
NAME AGE
rule1 9m$ kubectl describe ciliumnetworkpolicies rule1
Name: rule1
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"cilium.io/v2","description":"L7 policy to restrict access to specific HTTP call","kind":"CiliumNetworkPolicy","metadata":{"annotations":...
API Version: cilium.io/v2
Kind: CiliumNetworkPolicy
Metadata:
Cluster Name:
Creation Timestamp: 2018-05-08T16:09:44Z
Generation: 1
Resource Version: 1709
Self Link: /apis/cilium.io/v2/namespaces/default/ciliumnetworkpolicies/rule1
UID: 391b40cd-52da-11e8-8ef9-080027a11e00
Spec:
Endpoint Selector:
Match Labels:
Any : Class: deathstar
Any : Org: empire
Ingress:
From Endpoints:
Match Labels:
Any : Org: empire
To Ports:
Ports:
Port: 80
Protocol: TCP
Rules:
Http:
Method: POST
Path: /v1/request-landing
Status:
Nodes:
Minikube:
Enforcing: true
Last Updated: 2018-05-08T16:17:44.124075147Z
Local Policy Revision: 11
Ok: true
Events: <none>Cilium Pod 内で cilium policy get として確認することもできるようです
$ kubectl -n kube-system exec cilium-rg6dc cilium policy get
[
{
"endpointSelector": {
"matchLabels": {
"any:class": "deathstar",
"any:org": "empire",
"k8s:io.kubernetes.pod.namespace": "default"
}
},
"ingress": [
{
"fromEndpoints": [
{
"matchLabels": {
"any:org": "empire",
"k8s:io.kubernetes.pod.namespace": "default"
}
}
],
"toPorts": [
{
"ports": [
{
"port": "80",
"protocol": "TCP"
}
],
"rules": {
"http": [
{
"path": "/v1/request-landing",
"method": "POST"
}
]
}
}
]
}
],
"labels": [
{
"key": "io.cilium.k8s.policy.name",
"value": "rule1",
"source": "unspec"
},
{
"key": "io.cilium.k8s.policy.namespace",
"value": "default",
"source": "unspec"
}
]
}
]
Revision: 11Prometheus 連携
prometheus.yaml (document と path が変わってた) で Prometheus サービスをセットアップします。kubernetes service discovery で cilium コンテナを動的に見つけて cilium-agent から polling するようです
$ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/doc-1.0/examples/kubernetes/prometheus.yaml
namespace "prometheus" created
service "prometheus" created
deployment.extensions "prometheus-core" created
configmap "prometheus-core" created
clusterrolebinding.rbac.authorization.k8s.io "prometheus" created
clusterrole.rbac.authorization.k8s.io "prometheus" created
serviceaccount "prometheus-k8s" created
configmap "cilium-metrics-config" createdCilium の再起動
$ kubectl replace --force -f https://raw.githubusercontent.com/cilium/cilium/doc-1.0/examples/kubernetes/1.10/cilium.yaml
configmap "cilium-config" deleted
secret "cilium-etcd-secrets" deleted
daemonset.apps "cilium" deleted
clusterrolebinding.rbac.authorization.k8s.io "cilium" deleted
clusterrole.rbac.authorization.k8s.io "cilium" deleted
serviceaccount "cilium" deleted
configmap "cilium-config" replaced
secret "cilium-etcd-secrets" replaced
daemonset.apps "cilium" replaced
clusterrolebinding.rbac.authorization.k8s.io "cilium" replaced
clusterrole.rbac.authorization.k8s.io "cilium" replaced
serviceaccount "cilium" replaced後片付け
$ minikube delete
Deleting local Kubernetes cluster...
Machine deleted.まとめ
詳しい仕組みは確認してないけど Label を使って送信元、送信先を指定でき、L4, L7 でアクセス制御できる便利ツールだった この方のブロクを見ると BPF について理解が深まるだろうか LinuxのBPF : (1) パケットフィルタ - 睡分不足 - mm_i - はてなブログ
amazon.co.jp