Camil Blanaru
Camil Blanaru

Systems engineer with a passion for clouds. Based in Barcelona.

Camil Blanaru
Author

Linux Systems Engineer with a passion for clouds. Especially interested in automating the installation and management of enterprise infrastructure.

Share


Tags


avrt
Camil Blanaru

Create a secure Kubernetes HA cluster in AWS using kube-aws

Camil BlanaruCamil Blanaru

There are several tools that allow automatic deployment of Kubernetes clusters in AWS, like kube-aws, kops, kismatic and others. Some of the tools can also provision AWS infrastructure according to your use case.

kube-awsallows you to customise a yaml file and generate a Cloudformation stack that automates the creation of VPCs, subnets, Nat Gateways, security groups, etc. Even if it's easier to deploy everything using kube-aws, I find it safer to manage your network infrastructure separately, especially when using Cloudformation, which sometimes can roll back and mess your entire stack.

The fallowing setup use a base Cloudformation stack to configure Public and Private Subnets, IGW, NatGW, Route Tables and KMS. After the stack is created we'll deploy Kubernetes on top of it using kube-aws.

Features:

alt

Click here to enlarge the diagram

Note: all the configurations are available on this github repository

Deploy the Kubernetes cluster

  1. Clone my demo repository locally

  2. Create the base Cloudformation stack by customising and running ./vpc/deploy.vpc.sh script

  3. Install kube-aws; in this example I'm usingkube-aws v0.9-7-rc.3

  4. Get the output values from the Cloudformation base stack created and update the ./kube-aws/cluster.yaml

    aws cloudformation describe-stacks --stack-name kube-aws-vpc | jq -r '.Stacks[].Outputs'
    
  5. Render the stack and credentials. Go to ./kube-aws directory and execute the fallowing command:

    kube-aws render credentials --generate-ca
    kube-aws render stack
    

    Please read the kube-aws documentation if you plan to use your own CA.

  6. To launch the cluster, first you'll need a S3 bucket. Use an existing bucket or create a new one.

    kube-aws up --s3-uri s3://your-bucket-name 
    
  7. Access your Kubernetes cluster. Since we are creating all the resources in private networks, in order to access it, you'll need a VPN placed in one of the public subnets. I'm using pritunl, which is very easy to configure. In approximately 5 minutes you can get it up and running on a t2.nano

After the VPN is set, make a quick check to list the nodes and pods in kube-system. You should get something similar to this:

    $ kubectl --kubeconfig=./kubeconfig get nodes -o wide
    NAME                         STATUS    AGE       VERSION           EXTERNAL-IP   OS-IMAGE                                       KERNEL-VERSION
    ip-10-0-1-176.ec2.internal   Ready     1h        v1.6.4+coreos.0   <none>        Container Linux by CoreOS 1353.7.0 (Ladybug)   4.9.24-coreos
    ip-10-0-1-219.ec2.internal   Ready     1h        v1.6.4+coreos.0   <none>        Container Linux by CoreOS 1353.7.0 (Ladybug)   4.9.24-coreos
    ip-10-0-2-151.ec2.internal   Ready     1h        v1.6.4+coreos.0   <none>        Container Linux by CoreOS 1353.7.0 (Ladybug)   4.9.24-coreos
    ip-10-0-2-245.ec2.internal   Ready     1h        v1.6.4+coreos.0   <none>        Container Linux by CoreOS 1353.7.0 (Ladybug)   4.9.24-coreos
    ip-10-0-3-124.ec2.internal   Ready     1h        v1.6.4+coreos.0   <none>        Container Linux by CoreOS 1353.7.0 (Ladybug)   4.9.24-coreos
    ip-10-0-3-96.ec2.internal    Ready     1h        v1.6.4+coreos.0   <none>        Container Linux by CoreOS 1353.7.0 (Ladybug)   4.9.24-coreos

    $ kubectl --kubeconfig=./kubeconfig get pods -o wide -n kube-system
    NAME                                                 READY     STATUS    RESTARTS   AGE       IP           NODE
    calico-node-0ws56                                    1/1       Running   0          1h        10.0.2.151   ip-10-0-2-151.ec2.internal
    calico-node-12177                                    1/1       Running   0          1h        10.0.3.124   ip-10-0-3-124.ec2.internal
    calico-node-8kxsf                                    1/1       Running   0          1h        10.0.2.245   ip-10-0-2-245.ec2.internal
    calico-node-9sq7p                                    1/1       Running   0          1h        10.0.1.219   ip-10-0-1-219.ec2.internal
    calico-node-hh3l9                                    1/1       Running   0          1h        10.0.3.96    ip-10-0-3-96.ec2.internal
    calico-node-vtzf2                                    1/1       Running   0          1h        10.0.1.176   ip-10-0-1-176.ec2.internal
    calico-policy-controller-2265106533-gkl2q            1/1       Running   0          1h        10.0.3.124   ip-10-0-3-124.ec2.internal
    dex-3003102726-2zjp0                                 1/1       Running   0          1h        10.2.40.5    ip-10-0-3-96.ec2.internal
    heapster-v1.3.0-264939892-ggg1c                      2/2       Running   0          1h        10.2.43.3    ip-10-0-2-151.ec2.internal
    kube-apiserver-ip-10-0-1-219.ec2.internal            1/1       Running   0          1h        10.0.1.219   ip-10-0-1-219.ec2.internal
    kube-apiserver-ip-10-0-2-245.ec2.internal            1/1       Running   0          1h        10.0.2.245   ip-10-0-2-245.ec2.internal
    kube-apiserver-ip-10-0-3-124.ec2.internal            1/1       Running   0          1h        10.0.3.124   ip-10-0-3-124.ec2.internal
    kube-controller-manager-ip-10-0-1-219.ec2.internal   1/1       Running   0          1h        10.0.1.219   ip-10-0-1-219.ec2.internal
    kube-controller-manager-ip-10-0-2-245.ec2.internal   1/1       Running   0          1h        10.0.2.245   ip-10-0-2-245.ec2.internal
    kube-controller-manager-ip-10-0-3-124.ec2.internal   1/1       Running   0          1h        10.0.3.124   ip-10-0-3-124.ec2.internal
    kube-dns-1759312207-2cd6s                            3/3       Running   0          1h        10.2.43.2    ip-10-0-2-151.ec2.internal
    kube-dns-1759312207-khgqs                            3/3       Running   0          1h        10.2.74.2    ip-10-0-1-176.ec2.internal
    kube-dns-autoscaler-2188776582-0rzp8                 1/1       Running   0          1h        10.2.40.2    ip-10-0-3-96.ec2.internal
    kube-proxy-ip-10-0-1-176.ec2.internal                1/1       Running   0          1h        10.0.1.176   ip-10-0-1-176.ec2.internal
    kube-proxy-ip-10-0-1-219.ec2.internal                1/1       Running   0          1h        10.0.1.219   ip-10-0-1-219.ec2.internal
    kube-proxy-ip-10-0-2-151.ec2.internal                1/1       Running   0          1h        10.0.2.151   ip-10-0-2-151.ec2.internal
    kube-proxy-ip-10-0-2-245.ec2.internal                1/1       Running   0          1h        10.0.2.245   ip-10-0-2-245.ec2.internal
    kube-proxy-ip-10-0-3-124.ec2.internal                1/1       Running   0          1h        10.0.3.124   ip-10-0-3-124.ec2.internal
    kube-proxy-ip-10-0-3-96.ec2.internal                 1/1       Running   0          1h        10.0.3.96    ip-10-0-3-96.ec2.internal
    kube-rescheduler-3155147949-99k34                    1/1       Running   0          1h        10.0.2.151   ip-10-0-2-151.ec2.internal
    kube-scheduler-ip-10-0-1-219.ec2.internal            1/1       Running   0          1h        10.0.1.219   ip-10-0-1-219.ec2.internal
    kube-scheduler-ip-10-0-2-245.ec2.internal            1/1       Running   0          1h        10.0.2.245   ip-10-0-2-245.ec2.internal
    kube-scheduler-ip-10-0-3-124.ec2.internal            1/1       Running   0          1h        10.0.3.124   ip-10-0-3-124.ec2.internal
    kubernetes-dashboard-3963616910-fk65g                1/1       Running   0          1h        10.2.40.4    ip-10-0-3-96.ec2.internal

Optionally you can configure your ~/.kube/configaccording to kubeconfig file to avoid passing the --kubeconfig flag on your commands.

Important

In order to expose public services using ELB or Ingress, the public subnets have to be tagged with the cluster name.

Ex. KubeernetesCluser=cluster_name

Add-ons

kube2iam

First we'll configure the kube2iamto allow some of our applications to assume AWS IAM Roles.

When RBAC is enabled kube2iamneeds permissions to list pods and namespaces. We have to grant these permissions.

kubectl create -f ./addons/kube2iam/rbac.yaml

deploy the kube2iam DaemonSet

change the accound ID to yours in./addons/kube2iam/k2i.ds.yaml, then create the DaemonSet

kubectl create -f ./addons/kube2iam/k2i.ds.yaml

Now your pods can assume all the roles that have a trust relationship configured.

Route53

This add-on is based on ExternalDNS project which allows you to control Route53 DNS records dynamically via Kubernetes resources.

Create a role named k8s-route53using this policy. You also have to establish a trust relationship in order to allow the role to be assumed. An example is provided here.

Now change the values in external-dns.yaml and deploy it.

kubectl create -f ./addons/route53/external-dns.yaml

Nginx Ingress Controller

I'm choosing nginx over traefik because of the Proxy Protocol support. This allows to use the nginx ingress controller in AWS behind an ELB configured with Proxy Protocol. Here are some benefits from it:

Deploy

kubectl create -f ./addons/ingress/nginx/rbac.yaml
kubectl create -f ./addons/ingress/nginx/nginx.yaml

kube-lego

Kube-Lego automatically requests certificates for Kubernetes Ingress resources from Let's Encrypt.

kubectl create -f ./addons/kube-lego/rbac.yaml
kubectl create -f ./addons/kube-lego/kube-lego.yaml

Dex

Dex runs natively on top of any Kubernetes cluster using Third Party Resources and can drive API server authentication through the OpenID Connect plugin.

By default you have administrator rights using the TLS certificates. If you plan to grant restricted permissions to other users, Dex can facilitate users access using OpenID Connect Tokens.

In this example we use the Github provider to identify the users.

Please configure the./addons/dex/elb/internal-elb.yaml file then expose the service.

kubectl create -f ./addons/dex/elb/internal-elb.yaml

The DNS is configured automatically by ExternalDNS add-on and should be available in approximately 1 minute.

You can now use a client like dex's example-app to obtain a authentication token.

If you prefer, you can use this app as a always running service by configuring and deploying ./addons/kid/kid.yaml

kubectl create secret \
generic kid \
--from-literal=CLIENT_ID=your-client-id \
--from-literal=CLIENT_SECRET=your-client-secret \
-n kube-system    

kubectl create -f ./addons/kid/kid.yaml

Please check the dex documentation if you need more informations.

Make a quick test by granting a user permissions to list the pods in kube-system namespace.

kubectl create -f `./examples/rbac/pod-reader.yaml`
kubectl create rolebinding pod-reader --role=pod-reader --user=user@example.com --namespace=kube-system

Example of ~/.kube/config for a user

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ca.pem_base64_encoded
    server: https://kubeapi.example.com
  name: your_cluster_name
contexts:
- context:
    cluster: your_cluster_name
    user: user@example.com
  name: your_cluster_name
current-context: your_cluster_name
kind: Config
preferences: {}
users:
- name: user@example.com
  user:
    auth-provider:
      config:
        access-token: id_token
        client-id: client_id 
        client-secret: client_secret
        extra-scopes: groups
        id-token: id_token
        idp-issuer-url: https://dex.example.com
        refresh-token: refresh_token
      name: oidc

If you already have the ~/.kube/config set, you can use this example to configure the user authentication

kubectl config set-credentials user@example.com \
  --auth-provider=oidc \
  --auth-provider-arg=idp-issuer-url=https://dex.example.com \
  --auth-provider-arg=client-id=your_client_id \
  --auth-provider-arg=client-secret=your_client_secret \
  --auth-provider-arg=refresh-token=your_refresh_token \
  --auth-provider-arg=id-token=your_id_token \
  --auth-provider-arg=extra-scopes=groups

Once your id_token expires, kubectl will attempt to refresh your id_token using your refresh_token and client_secret storing the new values for the refresh_token and id_token in your ~/.kube/config

At this point you have a pretty secure, highly available, Kubernetes cluster in AWS.

For even better security please also consider using the Pod Security Policy, Calico Network Policy and Istio

Monitoring

A easy to setup, in-cluster, monitoring solution using Prometheus is available here

Camil Blanaru

Linux Systems Engineer with a passion for clouds. Especially interested in automating the installation and management of enterprise infrastructure.

Comments