We have our EKS Cluster and worker nodes already, but we need some Spot Instances configured as workers. We also need a Node Labeling strategy to identify which instances are Spot and which are on-demand so that we can make more intelligent scheduling decisions. We will use AWS CloudFormation to launch new worker nodes that will connect to the EKS cluster.
This template will create a single ASG that leverages the latest feature to mix multiple instance types and purchase as a single K8s nodegroup. Check out this blog: New – EC2 Auto Scaling Groups With Multiple Instance Types & Purchase Options for details.
First, we will need to ensure the ARN Name our workers use is set in our environment:
test -n "$INSTANCE_PROFILE_ARN" && echo INSTANCE_PROFILE_ARN is "$INSTANCE_PROFILE_ARN" || echo INSTANCE_PROFILE_ARN is not set
Copy the Profile ARN for use as a Parameter in the next step. If you receive an error or empty response, expand the steps below to export.
INSTANCE_PROFILE_ARN is not set, please review: /eksctl/test/
# Example Output INSTANCE_PROFILE_ARN is arn:aws:iam::123456789101:instance-profile/eksctl-eksworkshop-eksctl-nodegroup-ng-abcd1234-NodeInstanceProfile-ABCDEF1234
We also need to collect the ID of the security group used with the existing worker nodes.
STACK_NAME=$(aws cloudformation describe-stacks | jq -r '.Stacks.StackName' | grep eksctl-eksworkshop-eksctl-nodegroup) SG_ID=$(aws cloudformation describe-stack-resources --stack-name $STACK_NAME --logical-resource-id SG | jq -r '.StackResources.PhysicalResourceId') echo $SG_ID
# Example Output sg-0d9fb7e709dff5675
We will launch the CloudFormation template as a new set of worker nodes, but it’s also possible to update the nodegroup CloudFormation stack created by the eksctl tool.
Click the Launch button to create the CloudFormation stack in the AWS Management Console.
|EKS Workers - Spot and On Demand||Launch||Download|
Confirm the region is correct based on where you’ve deployed your cluster.
|Cluster Name:||eksworkshop-eksctl (or whatever you named your cluster)|
|ClusterControlPlaneSecurityGroup:||Select from the dropdown. It will contain your cluster name and the words ‘ControlPlaneSecurityGroup’|
|NodeInstanceProfile:||Use the Instance Profile ARN that copied in the step above. (e.g.eks-workshop-nodegroup)|
|UseExistingNodeSecurityGroups:||Leave as ‘Yes’|
|ExistingNodeSecurityGroups:||Use the SG name that copied in the step above. (e.g. sg-0123456789abcdef)|
|NodeImageId:||Visit this link and select the non-GPU image for your region - Check for empty spaces in copy/paste|
|KeyName:||SSH Key Pair created earlier or any valid key will work|
|NodeGroupName:||Leave as spotworkers|
|VpcId:||Select your workshop VPC from the dropdown|
|Subnets:||Select the 3 private subnets for your workshop VPC from the dropdown|
The EKS Bootstrap.sh script is packaged into the EKS Optimized AMI that we are using, and only requires a single input, the EKS Cluster name. The bootstrap script supports setting any
kubelet-extra-args at runtime. We have configured node-labels so that kubernetes knows what type of nodes we have provisioned. We set the lifecycle for the nodes as OnDemand or Ec2Spot. We are also tainting with PreferNoSchedule to prefer pods not be scheduled on Spot Instances. This is a “preference” or “soft” version of NoSchedule – the system will try to avoid placing a pod that does not tolerate the taint on the node, but it is not required.
You can leave the rest of the default parameters as is and continue through the remaining CloudFormation screens. Check the box next to I acknowledge that AWS CloudFormation might create IAM resources and click Create
The creation of the workers will take about 3 minutes.
Confirm that the new nodes joined the cluster correctly. You should see 2-3 more nodes added to the cluster.
kubectl get nodes
You can use the node-labels to identify the lifecycle of the nodes
kubectl get nodes --show-labels --selector=lifecycle=Ec2Spot
The output of this command should return 2 nodes. At the end of the node output, you should see the node label lifecycle=Ec2Spot
Now we will show all nodes with the lifecycle=OnDemand. The output of this command should return 1 node as configured in our CloudFormation template.
kubectl get nodes --show-labels --selector=lifecycle=OnDemand
You can use the
kubectl describe nodes with one of the spot nodes to see the taints applied to the EC2 Spot Instances.