Basic EKS Cluster with Cluster Autoscaler
EKSCTL can be used to quickly deploy a AWS EKS Cluster.
This is based on using a sandbox AWS account
The supporting files can be found on Github
Create an EKS deployment file, I tend to create individual nodegroups dedicated to a single AZ
1
2[cloudshell-user@ip-10-1-181-252 cluster-autoscaler]$ cat ca-cluster.yaml
3
4---
5apiVersion: eksctl.io/v1alpha5
6kind: ClusterConfig
7
8metadata:
9 name: ca-cluster
10---
11apiVersion: eksctl.io/v1alpha5
12kind: ClusterConfig
13
14metadata:
15 name: ca-cluster
16 region: us-east-1
17 version: "1.22"
18
19availabilityZones: ["us-east-1a", "us-east-1b", "us-east-1c"]
20
21managedNodeGroups:
22- name: nodegroupA
23 desiredCapacity: 1
24 availabilityZones: ["us-east-1a"]
25 instanceType: t3.small
26 spot: true
27- name: nodegroupB
28 desiredCapacity: 1
29 availabilityZones: ["us-east-1b"]
30 instanceType: t3.small
31 spot: true
32- name: nodegroupC
33 desiredCapacity: 1
34 availabilityZones: ["us-east-1c"]
35 instanceType: t3.small
36 spot: true
Create the cluster (will take a long time)
1[cloudshell-user@ip-10-1-181-252 cluster-autoscaler]$ eksctl create cluster -f ca-cluster.yaml
22022-05-12 08:03:06 [ℹ] eksctl version 0.96.0
32022-05-12 08:03:06 [ℹ] using region us-east-1
42022-05-12 08:03:06 [ℹ] subnets for us-east-1a - public:192.168.0.0/19 private:192.168.96.0/19
52022-05-12 08:03:06 [ℹ] subnets for us-east-1b - public:192.168.32.0/19 private:192.168.128.0/19
62022-05-12 08:03:06 [ℹ] subnets for us-east-1c - public:192.168.64.0/19 private:192.168.160.0/19
72022-05-12 08:03:06 [ℹ] nodegroup "nodegroupA" will use "" [AmazonLinux2/1.22]
82022-05-12 08:03:06 [ℹ] nodegroup "nodegroupB" will use "" [AmazonLinux2/1.22]
92022-05-12 08:03:06 [ℹ] nodegroup "nodegroupC" will use "" [AmazonLinux2/1.22]
102022-05-12 08:03:06 [ℹ] using Kubernetes version 1.22
112022-05-12 08:03:06 [ℹ] creating EKS cluster "ca-cluster" in "us-east-1" region with managed nodes
122022-05-12 08:03:06 [ℹ] 3 nodegroups (nodegroupA, nodegroupB, nodegroupC) were included (based on the include/exclude rules)
132022-05-12 08:03:06 [ℹ] will create a CloudFormation stack for cluster itself and 0 nodegroup stack(s)
142022-05-12 08:03:06 [ℹ] will create a CloudFormation stack for cluster itself and 3 managed nodegroup stack(s)
152022-05-12 08:03:06 [ℹ] if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=us-east-1 --cluster=ca-cluster'
162022-05-12 08:03:06 [ℹ] Kubernetes API endpoint access will use default of {publicAccess=true, privateAccess=false} for cluster "ca-cluster" in "us-east-1"
172022-05-12 08:03:06 [ℹ] CloudWatch logging will not be enabled for cluster "ca-cluster" in "us-east-1"
182022-05-12 08:03:06 [ℹ] you can enable it with 'eksctl utils update-cluster-logging --enable-types={SPECIFY-YOUR-LOG-TYPES-HERE (e.g. all)} --region=us-east-1 --cluster=ca-cluster'
192022-05-12 08:03:06 [ℹ]
202 sequential tasks: { create cluster control plane "ca-cluster",
21 2 sequential sub-tasks: {
22 wait for control plane to become ready,
23 3 parallel sub-tasks: {
24 create managed nodegroup "nodegroupA",
25 create managed nodegroup "nodegroupB",
26 create managed nodegroup "nodegroupC",
27 },
28 }
29}
302022-05-12 08:03:06 [ℹ] building cluster stack "eksctl-ca-cluster-cluster"
312022-05-12 08:03:06 [ℹ] deploying stack "eksctl-ca-cluster-cluster"
322022-05-12 08:03:36 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-cluster"
332022-05-12 08:04:06 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-cluster"
342022-05-12 08:05:06 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-cluster"
352022-05-12 08:06:06 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-cluster"
362022-05-12 08:07:06 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-cluster"
372022-05-12 08:08:06 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-cluster"
382022-05-12 08:09:06 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-cluster"
392022-05-12 08:10:06 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-cluster"
402022-05-12 08:11:07 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-cluster"
412022-05-12 08:12:07 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-cluster"
422022-05-12 08:13:07 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-cluster"
432022-05-12 08:14:07 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-cluster"
442022-05-12 08:16:08 [ℹ] building managed nodegroup stack "eksctl-ca-cluster-nodegroup-nodegroupB"
452022-05-12 08:16:08 [ℹ] building managed nodegroup stack "eksctl-ca-cluster-nodegroup-nodegroupC"
462022-05-12 08:16:08 [ℹ] building managed nodegroup stack "eksctl-ca-cluster-nodegroup-nodegroupA"
472022-05-12 08:16:08 [ℹ] deploying stack "eksctl-ca-cluster-nodegroup-nodegroupA"
482022-05-12 08:16:08 [ℹ] deploying stack "eksctl-ca-cluster-nodegroup-nodegroupB"
492022-05-12 08:16:08 [ℹ] deploying stack "eksctl-ca-cluster-nodegroup-nodegroupC"
502022-05-12 08:16:08 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupB"
512022-05-12 08:16:08 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupC"
522022-05-12 08:16:08 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupA"
532022-05-12 08:16:38 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupB"
542022-05-12 08:16:38 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupA"
552022-05-12 08:16:38 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupC"
562022-05-12 08:17:19 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupB"
572022-05-12 08:17:19 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupC"
582022-05-12 08:17:25 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupA"
592022-05-12 08:18:07 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupC"
602022-05-12 08:18:40 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupC"
612022-05-12 08:18:47 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupB"
622022-05-12 08:19:08 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupA"
632022-05-12 08:20:26 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupC"
642022-05-12 08:20:34 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupB"
652022-05-12 08:20:35 [ℹ] waiting for the control plane availability...
662022-05-12 08:20:35 [✔] saved kubeconfig as "/home/cloudshell-user/.kube/config"
672022-05-12 08:20:35 [ℹ] no tasks
682022-05-12 08:20:35 [✔] all EKS cluster resources for "ca-cluster" have been created
692022-05-12 08:20:35 [ℹ] nodegroup "nodegroupA" has 1 node(s)
702022-05-12 08:20:35 [ℹ] node "ip-192-168-11-66.ec2.internal" is ready
712022-05-12 08:20:35 [ℹ] waiting for at least 1 node(s) to become ready in "nodegroupA"
722022-05-12 08:20:35 [ℹ] nodegroup "nodegroupA" has 1 node(s)
732022-05-12 08:20:35 [ℹ] node "ip-192-168-11-66.ec2.internal" is ready
742022-05-12 08:20:35 [ℹ] nodegroup "nodegroupB" has 1 node(s)
752022-05-12 08:20:35 [ℹ] node "ip-192-168-51-71.ec2.internal" is ready
762022-05-12 08:20:35 [ℹ] waiting for at least 1 node(s) to become ready in "nodegroupB"
772022-05-12 08:20:35 [ℹ] nodegroup "nodegroupB" has 1 node(s)
782022-05-12 08:20:35 [ℹ] node "ip-192-168-51-71.ec2.internal" is ready
792022-05-12 08:20:35 [ℹ] nodegroup "nodegroupC" has 1 node(s)
802022-05-12 08:20:35 [ℹ] node "ip-192-168-73-100.ec2.internal" is ready
812022-05-12 08:20:35 [ℹ] waiting for at least 1 node(s) to become ready in "nodegroupC"
822022-05-12 08:20:35 [ℹ] nodegroup "nodegroupC" has 1 node(s)
832022-05-12 08:20:35 [ℹ] node "ip-192-168-73-100.ec2.internal" is ready
842022-05-12 08:20:35 [✖] parsing kubectl version string (upstream error: error: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1"
85) / "0.0.0": Version string empty
862022-05-12 08:20:35 [ℹ] cluster should be functional despite missing (or misconfigured) client binaries
872022-05-12 08:20:35 [✔] EKS cluster "ca-cluster" in "us-east-1" region is ready
Download the kubeconfig
1[cloudshell-user@ip-10-1-181-252 cluster-autoscaler]$ aws eks update-kubeconfig --name ca-cluster --region=us-east-1
2Added new context arn:aws:eks:us-east-1:566428305604:cluster/ca-cluster to /home/cloudshell-user/.kube/config
Check the cluster is running ok
1[cloudshell-user@ip-10-1-181-252 cluster-autoscaler]$ kubectl get nodes
2NAME STATUS ROLES AGE VERSION
3ip-192-168-11-66.ec2.internal Ready <none> 17m v1.22.6-eks-7d68063
4ip-192-168-51-71.ec2.internal Ready <none> 17m v1.22.6-eks-7d68063
5ip-192-168-73-100.ec2.internal Ready <none> 17m v1.22.6-eks-7d68063
Update the Auto scalling groups to add extra nodes and a limit
1[cloudshell-user@ip-10-1-181-252 cluster-autoscaler]$ aws autoscaling describe-auto-scaling-groups --query "AutoScalingGroups[? Tags[? (Key=='eks:cluster-name') && Value=='$CLUSTER_NAME']].AutoScalingGroupName" --output text > asg.txt
2[cloudshell-user@ip-10-1-181-252 cluster-autoscaler]$ for x in $(cat asg.txt); do aws autoscaling update-auto-scaling-group --auto-scaling-group-name $x --min-size 1 --desired-capacity 1 --max-size 2; done
Create the IAM Policy and ServiceAccount for the cluster autoscaler
1
2[cloudshell-user@ip-10-1-181-252 cluster-autoscaler]$ aws iam create-policy --policy-name ${CLUSTER_NAME}-k8s-asg-policy --policy-document file://k8s-asg-policy.json
3{
4 "Policy": {
5 "PolicyName": "ca-cluster-k8s-asg-policy",
6 "PolicyId": "ANPAYHYOCBDCEETZ2NA33",
7 "Arn": "arn:aws:iam::566428305604:policy/ca-cluster-k8s-asg-policy",
8 "Path": "/",
9 "DefaultVersionId": "v1",
10 "AttachmentCount": 0,
11 "PermissionsBoundaryUsageCount": 0,
12 "IsAttachable": true,
13 "CreateDate": "2022-05-12T08:26:23+00:00",
14 "UpdateDate": "2022-05-12T08:26:23+00:00"
15 }
16}
17
18[cloudshell-user@ip-10-1-181-252 cluster-autoscaler]$ eksctl create iamserviceaccount --name cluster-autoscaler --namespace kube-system --cluster $CLUSTER_NAME --attach-policy-arn "arn:aws:iam::${ACCOUNT_ID}:policy/${CLUSTER_NAME}-k8s-asg-policy" --approve --override-existing-serviceaccounts
Deploy the CA
1
2[cloudshell-user@ip-10-1-181-252 cluster-autoscaler]$ envsubst < cluster-autoscaler-autodiscover.yaml | kubectl apply -f -
3clusterrole.rbac.authorization.k8s.io/cluster-autoscaler created
4role.rbac.authorization.k8s.io/cluster-autoscaler created
5clusterrolebinding.rbac.authorization.k8s.io/cluster-autoscaler created
6rolebinding.rbac.authorization.k8s.io/cluster-autoscaler created
7deployment.apps/cluster-autoscaler created
8
9[cloudshell-user@ip-10-1-181-252 cluster-autoscaler]$ kubectl -n kube-system annotate deployment.apps/cluster-autoscaler cluster-autoscaler.kubernetes.io/safe-to-evict="false"
10deployment.apps/cluster-autoscaler annotated
11
12[cloudshell-user@ip-10-1-181-252 cluster-autoscaler]$ export AUTOSCALER_VERSION=1.22.2
13[cloudshell-user@ip-10-1-181-252 cluster-autoscaler]$ kubectl -n kube-system \
14> set image deployment.apps/cluster-autoscaler \
15> cluster-autoscaler=us.gcr.io/k8s-artifacts-prod/autoscaling/cluster-autoscaler:v${AUTOSCALER_VERSION}
16deployment.apps/cluster-autoscaler image updated
Delete the cluster
1
2[cloudshell-user@ip-10-1-181-252 cluster-autoscaler]$ eksctl delete cluster ca-cluster
32022-05-12 08:36:38 [ℹ] eksctl version 0.96.0
42022-05-12 08:36:38 [ℹ] using region us-east-1
52022-05-12 08:36:38 [ℹ] deleting EKS cluster "ca-cluster"
62022-05-12 08:36:39 [ℹ] will drain 0 unmanaged nodegroup(s) in cluster "ca-cluster"
72022-05-12 08:36:39 [ℹ] deleted 0 Fargate profile(s)
82022-05-12 08:36:39 [✔] kubeconfig has been updated
92022-05-12 08:36:39 [ℹ] cleaning up AWS load balancers created by Kubernetes objects of Kind Service or Ingress
102022-05-12 08:36:48 [ℹ]
113 sequential tasks: {
12 3 parallel sub-tasks: {
13 delete nodegroup "nodegroupC",
14 delete nodegroup "nodegroupA",
15 delete nodegroup "nodegroupB",
16 },
17 2 sequential sub-tasks: {
18 2 sequential sub-tasks: {
19 delete IAM role for serviceaccount "kube-system/cluster-autoscaler",
20 delete serviceaccount "kube-system/cluster-autoscaler",
21 },
22 delete IAM OIDC provider,
23 }, delete cluster control plane "ca-cluster" [async]
24}
252022-05-12 08:36:48 [ℹ] will delete stack "eksctl-ca-cluster-nodegroup-nodegroupB"
262022-05-12 08:36:48 [ℹ] waiting for stack "eksctl-ca-cluster-nodegroup-nodegroupB" to get deleted
272022-05-12 08:36:48 [ℹ] will delete stack "eksctl-ca-cluster-nodegroup-nodegroupC"
282022-05-12 08:36:48 [ℹ] waiting for stack "eksctl-ca-cluster-nodegroup-nodegroupC" to get deleted
292022-05-12 08:36:48 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupB"
302022-05-12 08:36:49 [ℹ] will delete stack "eksctl-ca-cluster-nodegroup-nodegroupA"
312022-05-12 08:36:49 [ℹ] waiting for stack "eksctl-ca-cluster-nodegroup-nodegroupA" to get deleted
322022-05-12 08:36:49 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupC"
332022-05-12 08:36:49 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupA"
342022-05-12 08:37:18 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupB"
352022-05-12 08:37:19 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupC"
362022-05-12 08:37:19 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupA"
372022-05-12 08:37:51 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupC"
382022-05-12 08:37:54 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupA"
392022-05-12 08:38:09 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupB"
402022-05-12 08:38:29 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupA"
412022-05-12 08:39:49 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupC"
422022-05-12 08:39:54 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupB"
432022-05-12 08:40:14 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupA"
442022-05-12 08:40:32 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupC"
452022-05-12 08:40:51 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupA"
462022-05-12 08:41:12 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupB"
472022-05-12 08:41:35 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupC"
482022-05-12 08:42:13 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupB"
492022-05-12 08:42:46 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupA"
502022-05-12 08:42:57 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupC"
512022-05-12 08:43:05 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupB"
522022-05-12 08:43:35 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupA"
532022-05-12 08:44:23 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupA"
542022-05-12 08:44:47 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupC"
552022-05-12 08:45:04 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupB"
562022-05-12 08:46:00 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-nodegroup-nodegroupA"
572022-05-12 08:46:02 [ℹ] will delete stack "eksctl-ca-cluster-addon-iamserviceaccount-kube-system-cluster-autoscaler"
582022-05-12 08:46:02 [ℹ] waiting for stack "eksctl-ca-cluster-addon-iamserviceaccount-kube-system-cluster-autoscaler" to get deleted
592022-05-12 08:46:04 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-addon-iamserviceaccount-kube-system-cluster-autoscaler"
602022-05-12 08:46:34 [ℹ] waiting for CloudFormation stack "eksctl-ca-cluster-addon-iamserviceaccount-kube-system-cluster-autoscaler"
612022-05-12 08:46:34 [ℹ] deleted serviceaccount "kube-system/cluster-autoscaler"
622022-05-12 08:46:34 [ℹ] 1 error(s) occurred while deleting cluster with nodegroup(s)
632022-05-12 08:46:34 [✖] deleting OIDC provider: operation error IAM: DeleteOpenIDConnectProvider, https response error StatusCode: 403, RequestID: 3944137c-c5ff-4263-8156-db113d89d620, api error AccessDenied: User: arn:aws:iam::566428305604:user/cloud_user is not authorized to perform: iam:DeleteOpenIDConnectProvider on resource: arn:aws:iam::566428305604:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/7B0C3F25F3AC9E9E3C2EEDB41AC61FB9 with an explicit deny in a service control policy
64Error: failed to delete cluster with nodegroup(s)
65[cloudshell-user@ip-10-1-181-252 cluster-autoscaler]$
comments powered by Disqus