In this post, I'll continue the process of setting up a Kubernetes cluster from my last post by setting up our own load balancer, ingress and certificate manager. This lets us easily deploy web services protected by HTTPS in our Kubernetes cluster.
This article is the second in a series:
- Setting up self-hosted Kubernetes
- Kubernetes HTTPS ingress with ingress-nginx and cert-manager
Installing the load balancer
To let us assign IP addresses to services running in the cluster, we can set up MetalLB and set up
a subnet for the cluster. I've chosen the
172.31.0.0/16 subnet, since it doesn't conflict with
anything else running on my home network, and set up my home network to route all local traffic for
the subnet to the cluster nodes.
The installation is pretty simple, as MetalLB already provides the manifests we need in their installation guide:
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/namespace.yaml kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/metallb.yaml # On first install only kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
In addition, we need to deploy a config map for MetalLB. In ours we will just assign a range of IP addresses that can be assigned to services:
kubectl apply -f - <<EOF apiVersion: v1 kind: ConfigMap metadata: namespace: metallb-system name: config data: config: | address-pools: - name: default protocol: layer2 addresses: - 172.31.0.1-172.31.255.254 EOF
Installing the ingress
The next step is to deploy ingress-nginx. The ingress serves multiple purposes:
- It lets us serve all public HTTP-based services from a single port, routing traffic to the right Kubernetes service based on host and path.
- It terminals SSL traffic, so that we do not need individual services to care about HTTPS. Additionally, when combined with cert-manager later on, it lets us trivially issue TLS certificates for our services.
In this case we also start out by just deploying the manifests for
ingress-nginx found in the
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.44.0/deploy/static/provider/baremetal/deploy.yaml
The default setup will use
NodePort for the service, meaning that HTTP and HTTPS gets assigned to
some arbitrary ports on each node. However, we may want to serve on port 80 and 443 on a static IP
assigned by the load balancer we set up in the last step. To do this, we just need to change the
service type to
LoadBalancer, and optionally specify a static IP from the subnet that was set up
# run this command to edit a deployed service manifest: kubectl edit -n ingress-nginx svc/ingress-nginx-controller
... spec: - type: NodePort + type: LoadBalancer + loadBalancerIP: 172.31.0.1 # static IP for ingress-nginx ...
After this change has been applied, we can test out our ingress by connecting to it. Since we haven't configured any services yet, we'd expect it to just always give a "404 Not Found" error:
# check HTTP: curl http://172.31.0.1 # check HTTPS (-k is necessary since we haven't set up cert-manager yet) curl -k https://172.31.0.1
Deploying a web service using the ingress
Now, let's try to deploy a web service. For this, we'll deploy the
image, which serves a simple "hello world" page.
First, let's make manifests for the deployment and the service:
kubectl apply -f - <<EOF --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-hello labels: app: nginx-hello spec: replicas: 1 selector: matchLabels: app: nginx-hello template: metadata: labels: app: nginx-hello spec: containers: - name: nginx-hello image: nginxdemos/nginx-hello:plain-text ports: - containerPort: 8080 --- apiVersion: v1 kind: Service metadata: name: nginx-hello labels: app: nginx-hello spec: selector: app: nginx-hello ports: - protocol: TCP port: 80 targetPort: 8080 EOF
To use this service with the ingress, we need to create an ingress manifest as well:
kubectl apply -f - <<EOF --- apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: nginx-hello annotations: kubernetes.io/ingress.class: nginx spec: rules: - host: hello.example.com http: paths: - path: / backend: serviceName: nginx-hello servicePort: 80 EOF
In the above manifest, we assume that
hello.example.com is a DNS name that is mapped to the IP
address of the ingress. (In my setup, it's mapped to my home network's public IP, which port
forwards traffic to the MetalLB issued IP for the ingress.)
Once we have deployed the ingress, we can test it with
Server address: 10.42.1.7:80 Server name: nginx-hello-8bb945fb-5m2zs Date: 13/Mar/2021:14:50:46 +0000 URI: / Request ID: 159f091d524ed851ccf85ce13c0ce833
Adding SSL support with cert-manager
Finally, let's deploy cert-manager to let us issue TLS certificates and integrate that with our ingress setup.
As with the other steps, deploying cert-manager is as simple as just applying the manifest file found in the installation guide:
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.2.0/cert-manager.yaml
With cert-manager installed, we can set up a cluster-wide ACME issuer to let us issue certificates for our domains. While you can use HTTP to validate your ownership of a domain name, I prefer to use DNS challenges instead, so that's what I have used in my setup.
I'm using Cloudflare as my DNS provider, so I set up my
ClusterIssuer to automatically set the
TXT records for my domain names as I issue certificates for them.
kubectl -n cert-manager create secret generic cloudflare-api-token --from-literal=api-token=<SOME-API-TOKEN> kubectl -n cert-manager apply -f - <<EOF --- apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: public-ca spec: acme: email: firstname.lastname@example.org server: https://acme-v02.api.letsencrypt.org/directory privateKeySecretRef: name: public-ca-account-key solvers: - dns01: cloudflare: email: email@example.com apiTokenSecretRef: name: cloudflare-api-token key: api-token EOF
Now adding a TLS certificate for our cluster becomes trivial:
kubectl edit ingress/nginx-hello
--- apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: nginx-hello annotations: kubernetes.io/ingress.class: nginx + cert-manager.io/cluster-issuer: public-ca spec: rules: - host: hello.example.com http: paths: - path: / backend: serviceName: nginx-hello servicePort: 80 + tls: + - hosts: + - hello.example.com + secretName: nginx-hello-tls
We can see that a certificate is being requested:
kubectl get certificaterequests
NAME READY AGE nginx-hello-tls-pptqw False 42s
After waiting a while, if all goes well it should become ready, and a certificate should be issued and automatically used by the ingress.
We can try to call our service again, but this time over HTTPS, to check that the certificate is working:
Server address: 10.42.1.9:8080 Server name: nginx-hello-778774fc79-5b6kp Date: 13/Mar/2021:15:32:01 +0000 URI: / Request ID: 221c863e3613b7b431761b3785be5ce3
For a personal Kubrenetes cluster, I've found that this trifecta of services (metallb, ingress-nginx and cert-manager) are all easy to install and configure, and makes it easy to deploy web services in the cluster and have them be protected by HTTPS, without having to do any custom configuration or HTTPS setup for each service: it's all handled automatically when we create the ingresses.