There are several tools that allow automatic deployment of Kubernetes clusters in AWS, like kube-aws, kops, kismatic and others. Some of the tools can also provision AWS infrastructure according to your use case.
kube-awsallows you to customise a yaml file and generate a Cloudformation stack that automates the creation of VPCs, subnets, Nat Gateways, security groups, etc. Even if it's easier to deploy everything using kube-aws, I find it safer to manage your network infrastructure separately, especially when using Cloudformation, which sometimes can roll back and mess your entire stack.
The fallowing setup use a base Cloudformation stack to configure Public and Private Subnets, IGW, NatGW, Route Tables and KMS. After the stack is created we'll deploy Kubernetes on top of it using kube-aws.
Features:
- all the nodes are deployed in private subnets
- 3 distinct availability zones
- 3 masters in HA, one per availability zone
- workers configured using node pools, similar to GKE node pools
- HA ETCD with encrypted partitions for data, automatic backups to S3 and automatic recovery from failover
- role based authentication using the RBAC plugin
- users authentication using OpenID Connect Identity (OIDC)
- AWS IAM roles directly assigned to pods using kube2iam
- cluster autoscaling
Click here to enlarge the diagram
Note: all the configurations are available on this github repository
Deploy the Kubernetes cluster
-
Clone my demo repository locally
-
Create the base Cloudformation stack by customising and running
./vpc/deploy.vpc.shscript -
Install kube-aws; in this example I'm using
kube-awsv0.9-7-rc.3 -
Get the output values from the Cloudformation base stack created and update the
./kube-aws/cluster.yamlaws cloudformation describe-stacks --stack-name kube-aws-vpc | jq -r '.Stacks[].Outputs' -
Render the stack and credentials. Go to
./kube-awsdirectory and execute the fallowing command:kube-aws render credentials --generate-ca kube-aws render stack
Please read the kube-aws documentation if you plan to use your own CA.
-
To launch the cluster, first you'll need a S3 bucket. Use an existing bucket or create a new one.
kube-aws up --s3-uri s3://your-bucket-name -
Access your Kubernetes cluster. Since we are creating all the resources in private networks, in order to access it, you'll need a VPN placed in one of the public subnets. I'm using pritunl, which is very easy to configure. In approximately 5 minutes you can get it up and running on a
t2.nano
After the VPN is set, make a quick check to list the nodes and pods in kube-system. You should get something similar to this:
$ kubectl --kubeconfig=./kubeconfig get nodes -o wide
NAME STATUS AGE VERSION EXTERNAL-IP OS-IMAGE KERNEL-VERSION
ip-10-0-1-176.ec2.internal Ready 1h v1.6.4+coreos.0 <none> Container Linux by CoreOS 1353.7.0 (Ladybug) 4.9.24-coreos
ip-10-0-1-219.ec2.internal Ready 1h v1.6.4+coreos.0 <none> Container Linux by CoreOS 1353.7.0 (Ladybug) 4.9.24-coreos
ip-10-0-2-151.ec2.internal Ready 1h v1.6.4+coreos.0 <none> Container Linux by CoreOS 1353.7.0 (Ladybug) 4.9.24-coreos
ip-10-0-2-245.ec2.internal Ready 1h v1.6.4+coreos.0 <none> Container Linux by CoreOS 1353.7.0 (Ladybug) 4.9.24-coreos
ip-10-0-3-124.ec2.internal Ready 1h v1.6.4+coreos.0 <none> Container Linux by CoreOS 1353.7.0 (Ladybug) 4.9.24-coreos
ip-10-0-3-96.ec2.internal Ready 1h v1.6.4+coreos.0 <none> Container Linux by CoreOS 1353.7.0 (Ladybug) 4.9.24-coreos
$ kubectl --kubeconfig=./kubeconfig get pods -o wide -n kube-system
NAME READY STATUS RESTARTS AGE IP NODE
calico-node-0ws56 1/1 Running 0 1h 10.0.2.151 ip-10-0-2-151.ec2.internal
calico-node-12177 1/1 Running 0 1h 10.0.3.124 ip-10-0-3-124.ec2.internal
calico-node-8kxsf 1/1 Running 0 1h 10.0.2.245 ip-10-0-2-245.ec2.internal
calico-node-9sq7p 1/1 Running 0 1h 10.0.1.219 ip-10-0-1-219.ec2.internal
calico-node-hh3l9 1/1 Running 0 1h 10.0.3.96 ip-10-0-3-96.ec2.internal
calico-node-vtzf2 1/1 Running 0 1h 10.0.1.176 ip-10-0-1-176.ec2.internal
calico-policy-controller-2265106533-gkl2q 1/1 Running 0 1h 10.0.3.124 ip-10-0-3-124.ec2.internal
dex-3003102726-2zjp0 1/1 Running 0 1h 10.2.40.5 ip-10-0-3-96.ec2.internal
heapster-v1.3.0-264939892-ggg1c 2/2 Running 0 1h 10.2.43.3 ip-10-0-2-151.ec2.internal
kube-apiserver-ip-10-0-1-219.ec2.internal 1/1 Running 0 1h 10.0.1.219 ip-10-0-1-219.ec2.internal
kube-apiserver-ip-10-0-2-245.ec2.internal 1/1 Running 0 1h 10.0.2.245 ip-10-0-2-245.ec2.internal
kube-apiserver-ip-10-0-3-124.ec2.internal 1/1 Running 0 1h 10.0.3.124 ip-10-0-3-124.ec2.internal
kube-controller-manager-ip-10-0-1-219.ec2.internal 1/1 Running 0 1h 10.0.1.219 ip-10-0-1-219.ec2.internal
kube-controller-manager-ip-10-0-2-245.ec2.internal 1/1 Running 0 1h 10.0.2.245 ip-10-0-2-245.ec2.internal
kube-controller-manager-ip-10-0-3-124.ec2.internal 1/1 Running 0 1h 10.0.3.124 ip-10-0-3-124.ec2.internal
kube-dns-1759312207-2cd6s 3/3 Running 0 1h 10.2.43.2 ip-10-0-2-151.ec2.internal
kube-dns-1759312207-khgqs 3/3 Running 0 1h 10.2.74.2 ip-10-0-1-176.ec2.internal
kube-dns-autoscaler-2188776582-0rzp8 1/1 Running 0 1h 10.2.40.2 ip-10-0-3-96.ec2.internal
kube-proxy-ip-10-0-1-176.ec2.internal 1/1 Running 0 1h 10.0.1.176 ip-10-0-1-176.ec2.internal
kube-proxy-ip-10-0-1-219.ec2.internal 1/1 Running 0 1h 10.0.1.219 ip-10-0-1-219.ec2.internal
kube-proxy-ip-10-0-2-151.ec2.internal 1/1 Running 0 1h 10.0.2.151 ip-10-0-2-151.ec2.internal
kube-proxy-ip-10-0-2-245.ec2.internal 1/1 Running 0 1h 10.0.2.245 ip-10-0-2-245.ec2.internal
kube-proxy-ip-10-0-3-124.ec2.internal 1/1 Running 0 1h 10.0.3.124 ip-10-0-3-124.ec2.internal
kube-proxy-ip-10-0-3-96.ec2.internal 1/1 Running 0 1h 10.0.3.96 ip-10-0-3-96.ec2.internal
kube-rescheduler-3155147949-99k34 1/1 Running 0 1h 10.0.2.151 ip-10-0-2-151.ec2.internal
kube-scheduler-ip-10-0-1-219.ec2.internal 1/1 Running 0 1h 10.0.1.219 ip-10-0-1-219.ec2.internal
kube-scheduler-ip-10-0-2-245.ec2.internal 1/1 Running 0 1h 10.0.2.245 ip-10-0-2-245.ec2.internal
kube-scheduler-ip-10-0-3-124.ec2.internal 1/1 Running 0 1h 10.0.3.124 ip-10-0-3-124.ec2.internal
kubernetes-dashboard-3963616910-fk65g 1/1 Running 0 1h 10.2.40.4 ip-10-0-3-96.ec2.internal
Optionally you can configure your ~/.kube/configaccording to kubeconfig file to avoid passing the --kubeconfig flag on your commands.
Important
In order to expose public services using ELB or Ingress, the public subnets have to be tagged with the cluster name.
Ex. KubeernetesCluser=cluster_name
Add-ons
kube2iam
First we'll configure the kube2iamto allow some of our applications to assume AWS IAM Roles.
When RBAC is enabled kube2iamneeds permissions to list pods and namespaces. We have to grant these permissions.
kubectl create -f ./addons/kube2iam/rbac.yaml
deploy the kube2iam DaemonSet
change the accound ID to yours in ./addons/kube2iam/k2i.ds.yaml, then create the DaemonSet
kubectl create -f ./addons/kube2iam/k2i.ds.yaml
Now your pods can assume all the roles that have a trust relationship configured.
Route53
This add-on is based on ExternalDNS project which allows you to control Route53 DNS records dynamically via Kubernetes resources.
Create a role named k8s-route53using this policy. You also have to establish a trust relationship in order to allow the role to be assumed. An example is provided here.
Now change the values in external-dns.yaml and deploy it.
kubectl create -f ./addons/route53/external-dns.yaml
Nginx Ingress Controller
I'm choosing nginx over traefik because of the Proxy Protocol support. This allows to use the nginx ingress controller in AWS behind an ELB configured with Proxy Protocol. Here are some benefits from it:
- ingress address is associated with your ELB and doesn't change when you replace the workers
- workers can be placed in private networks without public IPs
- better DDOS protection from ELB
- unlimited number of services that can be exposed using only one ELB. It's like a cheap version of AWS ALB
Deploy
kubectl create -f ./addons/ingress/nginx/rbac.yaml
kubectl create -f ./addons/ingress/nginx/nginx.yaml
kube-lego
Kube-Lego automatically requests certificates for Kubernetes Ingress resources from Let's Encrypt.
kubectl create -f ./addons/kube-lego/rbac.yaml
kubectl create -f ./addons/kube-lego/kube-lego.yaml
Dex
Dex runs natively on top of any Kubernetes cluster using Third Party Resources and can drive API server authentication through the OpenID Connect plugin.
By default you have administrator rights using the TLS certificates. If you plan to grant restricted permissions to other users, Dex can facilitate users access using OpenID Connect Tokens.
In this example we use the Github provider to identify the users.
Please configure the./addons/dex/elb/internal-elb.yaml file then expose the service.
kubectl create -f ./addons/dex/elb/internal-elb.yaml
The DNS is configured automatically by ExternalDNS add-on and should be available in approximately 1 minute.
You can now use a client like dex's example-app to obtain a authentication token.
If you prefer, you can use this app as a always running service by configuring and deploying ./addons/kid/kid.yaml
kubectl create secret \
generic kid \
--from-literal=CLIENT_ID=your-client-id \
--from-literal=CLIENT_SECRET=your-client-secret \
-n kube-system
kubectl create -f ./addons/kid/kid.yaml
Please check the dex documentation if you need more informations.
Make a quick test by granting a user permissions to list the pods in kube-system namespace.
kubectl create -f `./examples/rbac/pod-reader.yaml`
kubectl create rolebinding pod-reader --role=pod-reader [email protected] --namespace=kube-system
Example of ~/.kube/config for a user
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: ca.pem_base64_encoded
server: https://kubeapi.example.com
name: your_cluster_name
contexts:
- context:
cluster: your_cluster_name
user: [email protected]
name: your_cluster_name
current-context: your_cluster_name
kind: Config
preferences: {}
users:
- name: [email protected]
user:
auth-provider:
config:
access-token: id_token
client-id: client_id
client-secret: client_secret
extra-scopes: groups
id-token: id_token
idp-issuer-url: https://dex.example.com
refresh-token: refresh_token
name: oidc
If you already have the ~/.kube/config set, you can use this example to configure the user authentication
kubectl config set-credentials [email protected] \
--auth-provider=oidc \
--auth-provider-arg=idp-issuer-url=https://dex.example.com \
--auth-provider-arg=client-id=your_client_id \
--auth-provider-arg=client-secret=your_client_secret \
--auth-provider-arg=refresh-token=your_refresh_token \
--auth-provider-arg=id-token=your_id_token \
--auth-provider-arg=extra-scopes=groups
Once your id_token expires, kubectl will attempt to refresh your id_token using your refresh_token and client_secret storing the new values for the refresh_token and id_token in your ~/.kube/config
At this point you have a pretty secure, highly available, Kubernetes cluster in AWS.
For even better security please also consider using the Pod Security Policy, Calico Network Policy and Istio
Monitoring
A easy to setup, in-cluster, monitoring solution using Prometheus is available here