I really like AWS and the way they build their products, I really do. But for some reason, AWS Elastic Kubernetes Service (EKS) provides one of the worst user experiences. There are a couple of huge pain points that I want to cover in this quick blog post.
Cluster deployment
Tooling – AWS CLI
It is just unusual to me how complex the process is with AWS CLI when it comes to EKS. You can’t just run a command and have the cluster up and running; you first need to figure out a lot of details:
1 2 3 4 5 6 7 8 | % aws eks create-cluster --region us-east-2 --name sp-test usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters] To see help text, you can run: aws help aws <command> help aws <command> <subcommand> help aws: error: the following arguments are required: --role-arn, --resources-vpc-config |
What? I just need a k8s cluster!
But even if you figure out these parameters, it will create a Kubernetes control-plane for you but no worker nodes. To create worker nodes, you need another manual!
Werner Vogels talks about creating primitives, not frameworks. But this time, I believe AWS took it too far.
Tooling – eksctl
The community always finds a better way. That is how eksctl appeared. Weaveworks rightfully noticed the issue with EKS provisioning and created a tool to automate all the steps. AWS acknowledged the problem and decided to collaborate with Weaveworks on it. The tool provides a substantially better user experience – single command to provision a simple cluster, no hussle.
But now we have another problem – time to deploy.
Time
eksctl provisions CloudFormation stack to deploy the Kubernetes cluster, the node groups, and everything else required. And it just takes time to provision it all – at least 15 minutes. I deploy and destroy multiple clusters per day; 15 minutes does not sound good for my productivity.
Stateful workloads
To deploy stateful workloads on Kubernetes, you need storage. In the majority of cases, Persistent Volumes are used. They are a standard interface to provision block storage devices and expose those to your applications. Persistent Volumes are created with the help of Container Storage Interface (CSI) drivers. Cloud providers and storage vendors have their own CSIs, and AWS is not an exception here.
But for some reason, AWS decided that it is not something they should install and configure by default. This leads to a variety of issues that we observe in the community.
Persistent Volume Claims are stuck in Pending status
Well, actually, first, you see that your Pod that is using some PVC is stuck in a Pending state with the following error:
1 2 3 4 | % kubectl describe pod my-psmdb-rs0-0 ... Warning FailedScheduling 5m21s (x6 over 55m) default-scheduler running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition |
You start digging and find that PVCs are stuck as well:
1 2 3 4 | % kubectl describe pvc mongod-data-my-psmdb-rs0-0 ... Normal ExternalProvisioning 2m3s (x262 over 67m) persistentvolume-controller waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator |
After some googling you find that you need to install aws-ebs-csi driver.
Install the driver
It is yet another weird user experience. The most common issue is forgetting about permissions or not setting the correct ones. AWS knows about it and rightfully mentions it in their documentation:
An Amazon EBS CSI driver IAM role. If you don’t satisfy this prerequisite, attempting to install the add-on and running kubectl describe pvc will show failed to provision volume with StorageClass along with a could not create volume in EC2: UnauthorizedOperation error. For more information, see Creating the Amazon EBS CSI driver IAM role.
To be fair here, setting it up through eksctl from the start would give you the cluster with the support of EBS, its CSI, and all the roles created. Just specify the following in the YAML manifest that you use with your eksctl:
1 2 3 4 | addons: - name: aws-ebs-csi-driver version: latest resolveConflicts: overwrite |
Unfortunately, the EKS cluster is deployed first, and you get a weird user experience and spend some time googling for a solution. Which might not work from the first try.
Our team opened this GitHub issue to get to the root cause of why aws-ebs-csi-driver is not enabled by default. There might be a good reason for it.
Conclusion
Various surveys (1, 2) indicate that AWS EKS is the leader in cloud-managed Kubernetes services. At the same time, it is surprising how it falls short in delivering a seamless user experience, especially for stateful workloads. The complexities of cluster deployment, the time-consuming nature of provisioning resources, and the additional steps required to manage storage for stateful applications highlight significant areas for improvement.
The Percona Kubernetes Operators lets you easily create and manage highly available, enterprise-ready MySQL, PostgreSQL, and MongoDB clusters on Kubernetes. Experience hassle-free database management and provisioning without the need for manual maintenance or custom in-house scripts.
Learn More About Percona Kubernetes Operators
