EKS with Cilium in chaining mode
Cilium can run in chaining mode which allows it to run alongside the AWS-CNI plugin.
This is based on using a sandbox AWS account
The supporting files can be found on Github
Install some extra tools
1curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz{,.sha256sum}
2sha256sum --check cilium-linux-amd64.tar.gz.sha256sum
3sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin
4rm cilium-linux-amd64.tar.gz{,.sha256sum}
5
6
7export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
8curl -L --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz{,.sha256sum}
9sha256sum --check hubble-linux-amd64.tar.gz.sha256sum
10sudo tar xzvfC hubble-linux-amd64.tar.gz /usr/local/bin
11rm hubble-linux-amd64.tar.gz{,.sha256sum}
Deploy an EKS Cluster
1export AWS_REGION=us-east-1
2export AWS_DEFAULT_REGION=us-east-1
3export CLUSTER_NAME=cilium-cluster2
4export ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text)
5export CILIUM_NAMESPACE=kube-system
6
7eksctl create cluster -f - << EOF
8---
9apiVersion: eksctl.io/v1alpha5
10kind: ClusterConfig
11
12metadata:
13 name: ${CLUSTER_NAME}
14 region: ${AWS_DEFAULT_REGION}
15 version: "1.22"
16
17availabilityZones: ["${AWS_DEFAULT_REGION}a","${AWS_DEFAULT_REGION}b"]
18managedNodeGroups:
19 - instanceType: t3.medium
20 name: ${CLUSTER_NAME}-ng
21 desiredCapacity: 2
22 minSize: 1
23 maxSize: 2
24
25EOF
26
27aws eks update-kubeconfig --name ${CLUSTER_NAME} --region=${AWS_DEFAULT_REGION}
Install Cilium in chaining mode
1
2helm repo add cilium https://helm.cilium.io/
3
4helm install cilium cilium/cilium --version 1.11.3 \
5 --namespace kube-system \
6 --set cni.chainingMode=aws-cni \
7 --set enableIPv4Masquerade=false \
8 --set tunnel=disabled \
9 --set hubble.listenAddress=":4244" \
10 --set hubble.relay.enabled=true \
11 --set hubble.ui.enabled=true
12
13 [cloudshell-user@ip-10-0-84-41 cilium]$ helm install cilium cilium/cilium --version 1.11.3 \
14> --namespace kube-system \
15> --set cni.chainingMode=aws-cni \
16> --set enableIPv4Masquerade=false \
17> --set tunnel=disabled \
18> --set hubble.listenAddress=":4244" \
19> --set hubble.relay.enabled=true \
20> --set hubble.ui.enabled=true
21W0512 14:20:59.634900 8693 warnings.go:70] spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[1].matchExpressions[0].key: beta.kubernetes.io/os is deprecated since v1.14; use "kubernetes.io/os" instead
22W0512 14:20:59.634940 8693 warnings.go:70] spec.template.metadata.annotations[scheduler.alpha.kubernetes.io/critical-pod]: non-functional in v1.16+; use the "priorityClassName" field instead
23NAME: cilium
24LAST DEPLOYED: Thu May 12 14:20:58 2022
25NAMESPACE: kube-system
26STATUS: deployed
27REVISION: 1
28TEST SUITE: None
29NOTES:
30You have successfully installed Cilium with Hubble Relay and Hubble UI.
31
32Your release version is 1.11.3.
33
34For any further help, visit https://docs.cilium.io/en/v1.11/gettinghelp
35[cloudshell-user@ip-10-0-84-41 cilium]$ kubectl get pod -A
36NAMESPACE NAME READY STATUS RESTARTS AGE
37kube-system aws-node-5lwk4 1/1 Running 0 9m43s
38kube-system aws-node-8b8zs 1/1 Running 0 9m36s
39kube-system cilium-hm8mf 0/1 Init:0/2 0 5s
40kube-system cilium-operator-66bff48676-6kzpb 0/1 ContainerCreating 0 5s
41kube-system cilium-operator-66bff48676-xdmlx 0/1 ContainerCreating 0 5s
42kube-system cilium-pcjmw 0/1 Init:0/2 0 4s
43kube-system coredns-7f5998f4c-crxct 1/1 Running 0 19m
44kube-system coredns-7f5998f4c-zf6df 1/1 Running 0 19m
45kube-system hubble-relay-8548d8946c-x9jpj 0/1 ContainerCreating 0 5s
46kube-system hubble-ui-5f7cdc86c7-nfbkm 0/3 ContainerCreating 0 5s
47kube-system kube-proxy-89sjz 1/1 Running 0 9m43s
48kube-system kube-proxy-smp5s 1/1 Running 0 9m36s
49[cloudshell-user@ip-10-0-84-41 cilium]$ kubectl get pod -A
50NAMESPACE NAME READY STATUS RESTARTS AGE
51kube-system aws-node-5lwk4 1/1 Running 0 9m54s
52kube-system aws-node-8b8zs 1/1 Running 0 9m47s
53kube-system cilium-hm8mf 1/1 Running 0 16s
54kube-system cilium-operator-66bff48676-6kzpb 1/1 Running 0 16s
55kube-system cilium-operator-66bff48676-xdmlx 1/1 Running 0 16s
56kube-system cilium-pcjmw 0/1 Init:1/2 0 15s
57kube-system coredns-7f5998f4c-jg5zx 1/1 Running 0 7s
58kube-system coredns-7f5998f4c-zf6df 1/1 Running 0 19m
59kube-system hubble-relay-8548d8946c-x9jpj 1/1 Running 0 16s
60kube-system hubble-ui-5f7cdc86c7-nfbkm 3/3 Running 0 16s
61kube-system kube-proxy-89sjz 1/1 Running 0 9m54s
62kube-system kube-proxy-smp5s 1/1 Running 0 9m47s
63[cloudshell-user@ip-10-0-84-41 cilium]$ kubectl get pod -A
64NAMESPACE NAME READY STATUS RESTARTS AGE
65kube-system aws-node-5lwk4 1/1 Running 0 9m58s
66kube-system aws-node-8b8zs 1/1 Running 0 9m51s
67kube-system cilium-hm8mf 1/1 Running 0 20s
68kube-system cilium-operator-66bff48676-6kzpb 1/1 Running 0 20s
69kube-system cilium-operator-66bff48676-xdmlx 1/1 Running 0 20s
70kube-system cilium-pcjmw 0/1 Running 0 19s
71kube-system coredns-7f5998f4c-jg5zx 1/1 Running 0 11s
72kube-system coredns-7f5998f4c-zf6df 1/1 Running 0 19m
73kube-system hubble-relay-8548d8946c-x9jpj 1/1 Running 0 20s
74kube-system hubble-ui-5f7cdc86c7-nfbkm 3/3 Running 0 20s
75kube-system kube-proxy-89sjz 1/1 Running 0 9m58s
76kube-system kube-proxy-smp5s 1/1 Running 0 9m51s
Install the cluster autoscaler
1
2 eksctl utils associate-iam-oidc-provider \
3 --cluster $CLUSTER_NAME \
4 --approve
5
6aws iam create-policy \
7 --policy-name ${CLUSTER_NAME}-k8s-asg-policy \
8 --policy-document file://k8s-asg-policy.json
9
10eksctl create iamserviceaccount \
11 --name cluster-autoscaler \
12 --namespace kube-system \
13 --cluster $CLUSTER_NAME \
14 --attach-policy-arn "arn:aws:iam::${ACCOUNT_ID}:policy/${CLUSTER_NAME}-k8s-asg-policy" \
15 --approve \
16 --override-existing-serviceaccounts
17
18envsubst < cluster-autoscaler-autodiscover.yaml | kubectl apply -f -
19
20kubectl -n kube-system annotate deployment.apps/cluster-autoscaler cluster-autoscaler.kubernetes.io/safe-to-evict="false"
21
22export AUTOSCALER_VERSION=1.22.2
23kubectl -n kube-system \
24 set image deployment.apps/cluster-autoscaler \
25 cluster-autoscaler=us.gcr.io/k8s-artifacts-prod/autoscaling/cluster-autoscaler:v${AUTOSCALER_VERSION}
Check everything is up and running
1kubectl port-forward -n $CILIUM_NAMESPACE svc/hubble-relay --address 0.0.0.0 --address :: 4245:80 &
2
3hubble --server localhost:4245 status
4hubble --server localhost:4245 observe
5
6
7[cloudshell-user@ip-10-0-84-41 cilium]$ hubble --server localhost:4245 status
8Handling connection for 4245
9Healthcheck (via localhost:4245): Ok
10Current/Max Flows: 1,336/8,190 (16.31%)
11Flows/s: 4.46
12Connected Nodes: 2/2
13[cloudshell-user@ip-10-0-84-41 cilium]$ hubble --server localhost:4245 observe
14Handling connection for 4245
15May 12 14:25:43.500: 192.168.37.166:39544 <- kube-system/coredns-7f5998f4c-6tln5:8080 to-stack FORWARDED (TCP Flags: ACK, FIN)
16May 12 14:25:43.500: 192.168.37.166:39542 -> kube-system/coredns-7f5998f4c-6tln5:8080 to-endpoint FORWARDED (TCP Flags: ACK, FIN)
17May 12 14:25:43.500: 192.168.37.166:39544 -> kube-system/coredns-7f5998f4c-6tln5:8080 to-endpoint FORWARDED (TCP Flags: ACK, FIN)
18May 12 14:25:44.236: 192.168.37.166:42346 -> ingress-nginx/ingress-nginx-controller-54d8b558d4-gmz56:10254 to-endpoint FORWARDED (TCP Flags: SYN)
19May 12 14:25:44.236: 192.168.37.166:42346 <- ingress-nginx/ingress-nginx-controller-54d8b558d4-gmz56:10254 to-stack FORWARDED (TCP Flags: SYN, ACK)
20May 12 14:25:44.236: 192.168.37.166:42346 -> ingress-nginx/ingress-nginx-controller-54d8b558d4-gmz56:10254 to-endpoint FORWARDED (TCP Flags: ACK)
21May 12 14:25:44.236: 192.168.37.166:42348 -> ingress-nginx/ingress-nginx-controller-54d8b558d4-gmz56:10254 to-endpoint FORWARDED (TCP Flags: SYN)
22May 12 14:25:44.236: 192.168.37.166:42348 <- ingress-nginx/ingress-nginx-controller-54d8b558d4-gmz56:10254 to-stack FORWARDED (TCP Flags: SYN, ACK)
23May 12 14:25:44.236: 192.168.37.166:42348 -> ingress-nginx/ingress-nginx-controller-54d8b558d4-gmz56:10254 to-endpoint FORWARDED (TCP Flags: ACK)
24May 12 14:25:44.237: 192.168.37.166:42348 -> ingress-nginx/ingress-nginx-controller-54d8b558d4-gmz56:10254 to-endpoint FORWARDED (TCP Flags: ACK, PSH)
25May 12 14:25:44.238: 192.168.37.166:42346 -> ingress-nginx/ingress-nginx-controller-54d8b558d4-gmz56:10254 to-endpoint FORWARDED (TCP Flags: ACK, PSH)
26May 12 14:25:44.239: 192.168.37.166:42348 <- ingress-nginx/ingress-nginx-controller-54d8b558d4-gmz56:10254 to-stack FORWARDED (TCP Flags: ACK, PSH)
27May 12 14:25:44.239: 192.168.37.166:42348 <- ingress-nginx/ingress-nginx-controller-54d8b558d4-gmz56:10254 to-stack FORWARDED (TCP Flags: ACK, FIN)
28May 12 14:25:44.239: 192.168.37.166:42346 <- ingress-nginx/ingress-nginx-controller-54d8b558d4-gmz56:10254 to-stack FORWARDED (TCP Flags: ACK, PSH)
29May 12 14:25:44.239: 192.168.37.166:42346 <- ingress-nginx/ingress-nginx-controller-54d8b558d4-gmz56:10254 to-stack FORWARDED (TCP Flags: ACK, FIN)
30May 12 14:25:44.239: 192.168.37.166:42348 -> ingress-nginx/ingress-nginx-controller-54d8b558d4-gmz56:10254 to-endpoint FORWARDED (TCP Flags: ACK, FIN)
31May 12 14:25:44.239: 192.168.37.166:42346 -> ingress-nginx/ingress-nginx-controller-54d8b558d4-gmz56:10254 to-endpoint FORWARDED (TCP Flags: ACK, FIN)
32May 12 14:25:45.375: ingress-nginx/ingress-nginx-controller-54d8b558d4-gmz56:40654 <- 192.168.108.176:443 to-endpoint FORWARDED (TCP Flags: ACK, PSH)
33May 12 14:25:45.376: ingress-nginx/ingress-nginx-controller-54d8b558d4-gmz56:40654 -> 192.168.108.176:443 to-stack FORWARDED (TCP Flags: ACK)
34May 12 14:25:48.079: kube-system/cluster-autoscaler-85d889ffd8-q789m:55780 <- 54.239.31.45:443 to-endpoint FORWARDED (TCP Flags: ACK, PSH)
35May 12 14:25:48.079: kube-system/cluster-autoscaler-85d889ffd8-q789m:55780 -> 54.239.31.45:443 to-stack FORWARDED (TCP Flags: ACK, PSH)
36May 12 14:25:48.079: kube-system/cluster-autoscaler-85d889ffd8-q789m:55780 -> 54.239.31.45:443 to-stack FORWARDED (TCP Flags: ACK, FIN)
37May 12 14:25:48.080: kube-system/cluster-autoscaler-85d889ffd8-q789m:55780 <- 54.239.31.45:443 to-endpoint FORWARDED (TCP Flags: ACK, FIN)
38May 12 14:25:48.080: kube-system/cluster-autoscaler-85d889ffd8-q789m:55780 -> 54.239.31.45:443 to-stack FORWARDED (TCP Flags: ACK)
39May 12 14:25:48.168: kube-system/cluster-autoscaler-85d889ffd8-q789m:52022 -> 192.168.94.37:443 to-stack FORWARDED (TCP Flags: ACK, PSH)
40May 12 14:25:48.179: kube-system/cluster-autoscaler-85d889ffd8-q789m:52022 <- 192.168.94.37:443 to-endpoint FORWARDED (TCP Flags: ACK, PSH)
41May 12 14:25:48.514: 192.168.14.247:56902 -> kube-system/coredns-7f5998f4c-r84mj:8080 to-endpoint FORWARDED (TCP Flags: SYN)
42May 12 14:25:48.514: 192.168.14.247:56902 <- kube-system/coredns-7f5998f4c-r84mj:8080 to-stack FORWARDED (TCP Flags: SYN, ACK)
43May 12 14:25:48.514: 192.168.14.247:56902 -> kube-system/coredns-7f5998f4c-r84mj:8080 to-endpoint FORWARDED (TCP Flags: ACK)
44May 12 14:25:48.514: 192.168.14.247:56902 -> kube-system/coredns-7f5998f4c-r84mj:8080 to-endpoint FORWARDED (TCP Flags: ACK, PSH)
45May 12 14:25:48.514: 192.168.14.247:56904 -> kube-system/coredns-7f5998f4c-r84mj:8080 to-endpoint FORWARDED (TCP Flags: SYN)
46May 12 14:25:48.514: 192.168.14.247:56904 <- kube-system/coredns-7f5998f4c-r84mj:8080 to-stack FORWARDED (TCP Flags: SYN, ACK)
47May 12 14:25:48.514: 192.168.14.247:56904 -> kube-system/coredns-7f5998f4c-r84mj:8080 to-endpoint FORWARDED (TCP Flags: ACK)
48May 12 14:25:48.514: 192.168.14.247:56902 <- kube-system/coredns-7f5998f4c-r84mj:8080 to-stack FORWARDED (TCP Flags: ACK, PSH)
49May 12 14:25:48.514: 192.168.14.247:56902 -> kube-system/coredns-7f5998f4c-r84mj:8080 to-endpoint FORWARDED (TCP Flags: ACK, FIN)
50May 12 14:25:48.514: 192.168.14.247:56902 <- kube-system/coredns-7f5998f4c-r84mj:8080 to-stack FORWARDED (TCP Flags: ACK, FIN)
51May 12 14:25:48.514: 192.168.14.247:56904 -> kube-system/coredns-7f5998f4c-r84mj:8080 to-endpoint FORWARDED (TCP Flags: ACK, PSH)
52May 12 14:25:48.514: 192.168.14.247:56904 <- kube-system/coredns-7f5998f4c-r84mj:8080 to-stack FORWARDED (TCP Flags: ACK, PSH)
53May 12 14:25:48.514: 192.168.14.247:56904 <- kube-system/coredns-7f5998f4c-r84mj:8080 to-stack FORWARDED (TCP Flags: ACK, FIN)
54May 12 14:25:52.119: ingress-nginx/ingress-nginx-controller-54d8b558d4-gmz56:40654 <- 192.168.108.176:443 to-endpoint FORWARDED (TCP Flags: ACK, PSH)
comments powered by Disqus