@ -25,30 +25,32 @@ TODO: Twitter, Slack, Issues (tagged with #autocert / special template)...
## Motivation
`Autocert` exists to **make it easy to use mTLS** (mutual TLS) to **improve security** within a cluster and to **secure communication into, out of, and between kubernetes clusters**. The goal is to make right and proper public key infrastructure (PKI) more accessible to people running kubernetes.
`Autocert` exists to **make it easy to use mTLS** ([mutual TLS](#)) to **improve security** within a cluster and to **secure communication into, out of, and between kubernetes clusters**.
TLS (and HTTPS, which is HTTP over TLS) provides _authenticated encryption_: an _identity dialtone_ and _end-to-end encryption_ for your workloads. It's like a secure line with caller ID. This has all sorts of benefits: better security, compliance, and easier auditability for starters. It makes workloads identity-aware, improving observability and enabling granular access control. Perhaps most compelling, [mutual TLS](#) (mTLS) lets you securely communicate with workloads running anywhere, not just inside kubernetes.
TLS (and HTTPS, which is HTTP over TLS) provides _authenticated encryption_: an _identity dialtone_ and _end-to-end encryption_ for your workloads. It's like a secure line with caller ID. This has all sorts of benefits: better security, compliance, and easier auditability for starters. It **makes workloads identity-aware**, improving observability and enabling granular access control. Perhaps most compelling, mTLS lets you securely communicate with workloads running anywhere, not just inside kubernetes.
TODO: Diagram
If you know how to operate and scale DNS and proxy infrastructure then you already know how to scale and operate secure service-to-service communication using mTLS (mutual TLS). There's just one problem: **you need certificates issued by your own certificate authority (CA)**. Building and operating a CA, issuing certificates, and making sure they're renewed before they expire is tricky. `Autocert` does all of this for you.
Unlike VPNs & SDNs, deploying and scaling mTLS is pretty easy. You're (hopefully) already using TLS, and your existing tools and standard libraries will provide most of what you need. If you know how to operate DNS and reverse proxies, you know how to operate mTLS infrastructure.
Because `autocert` is built on [`step certificates`](#) you can easily extend access to developers, endpoints, and workloads running outside your cluster.
There's just one problem: **you need certificates issued by your own certificate authority (CA)**. Building and operating a CA, issuing certificates, and making sure they're renewed before they expire is tricky. `Autocert` does all of this for you.
## Features
First and foremost, `autocert` is easy. You can **get started in minutes**.
`Autocert` uses [`step certificates`](https://github.com/smallstep/certificates) to generate keys and issue certificates. This process is secure and automatic, all you have to do is [install autocert](#install) and [annotate your pods](#annotate-pods).
`Autocert` uses [`step certificates`](https://github.com/smallstep/certificates) to generate keys and issue certificates. This process is secure and automatic, all you have to do is [install autocert](#install) and [annotate your pods](#enable-autocert-per-namespace).
Features include:
* A fully featured private **certificate authority** (CA) for workloads running on kubernetes and elsewhere
* [RFC5280](#) and [CA/Browser Forum](#) compliant certificates that work **for TLS**
* [RFC5280](https://tools.ietf.org/html/rfc5280) and [CA/Browser Forum](https://cabforum.org/baseline-requirements-documents/) compliant certificates that work **for TLS**
* Namespaced installation into the `step` namespace so it's **easy to lock down** your CA
* Short-lived certificates with **fully automated** enrollment and renewal
* Private keys are never transmitted across the network and aren't stored in `etcd`
Because `autocert` is built on [`step certificates`](https://github.com/smallstep/certificates) you can easily [extend access](#connecting-from-outside-the-cluster) to developers, endpoints, and workloads running outside your cluster, too.
To check which namespaces have `autocert` enabled run:
@ -98,9 +105,9 @@ default Active 59m enabled
### Annotate pods to get certificates
To tell `autocert` to issue certificates to a pod's containers you need to [specify a name](RUNBOOK.md#naming-considerations) using the `autocert.step.sm/name` annotation. This name will appear in the issued certificate (as the X.509 common name and SAN).
To get a certificate you need to tell `autocert` your workload's name using the `autocert.step.sm/name` annotation (this name will appear as the X.509 common name and SAN).
To test your installation apply this annotated deployment, which starts a [simple server](examples/hello-mtls/go/server.go) that uses mTLS:
Let's deploy a [simple mTLS server](examples/hello-mtls/go/server.go) named `hello-mtls.default.svc.cluster.local`:
We're done. Our container has a certificate, issued by our CA, and `autocert` will take care of renewal automatically.
We're done. Our container has a certificate, issued by our CA, which `autocert` will automatically renew.
✅ Certificates.
## Hello mTLS
It's easy to deploy certificates automatically with`autocert`, but it's up to you to use them correctly. To get you started, [`hello-mtls`](examples/hello-mtls) demonstrates the right way to use mTLS with various tools and languages (contributions welcome :). If you're a bit fuzzy on how mTLS works, [the `hello-mtls` README](examples/hello-mtls) is a great place to start.
It's easy to deploy certificates using`autocert`, but it's up to you to use them correctly. To get you started, [`hello-mtls`](examples/hello-mtls) demonstrates the right way to use mTLS with various tools and languages (contributions welcome :). If you're a bit fuzzy on how mTLS works, [the `hello-mtls` README](examples/hello-mtls) is a great place to start.
To finish out this tutorial let's keep things simple and try `curl`ing the server we just deployed from inside and outside the cluster.
@ -152,10 +157,10 @@ First, let's expose our workload to the rest of the cluster using a service:
kubectl expose deployment hello-mtls --port 443
```
Now we can `curl` our server from another container using a certificate issued by `autocert`:
Now let's deploy a client, with its own certificate, that [`curl`s our server in a loop](examples/hello-mtls/curl/client.sh):
# AUTOCERT ANNOTATION HERE -^ ######################################
labels: {app: hello-mtls-client}
spec:
containers:
@ -177,7 +182,9 @@ spec:
EOF
```
Once deployed the client logs should indicate that it's successfully connecting to our server using mTLS, which is [echoing the client's name](examples/hello-mtls/go/server.go#L71-L72) in response.
> Note that **the authority portion of the URL** (the `HELLO_MTLS_URL` env var) **matches the name of the server we're connecting to** (both are `hello-mtls.default.svc.cluster.local`). That's required for standard HTTPS and can sometimes require some DNS trickery.
Once deployed we should start seeing the client log responses from the server [saying hello](examples/hello-mtls/go/server.go#L71-L72):
```
$ export HELLO_MTLS_CLIENT=$(kubectl get pods -l app=hello-mtls-client -o jsonpath={$.items[0].metadata.name})
Connecting from outside the cluster is a bit more complicated. We need to handle DNS and obtain a certificate ourselves (tasks which were handled automatically inside the cluster by kubernetes and `autocert`, respectively).
That said, since we're using mTLS our server can be safely exposed directly to the public internet using a [LoadBalancer service type](https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer): **only clients that have a certificate issued by our certificate authority will be allowed to connect**.
Connecting from outside the cluster is a bit more complicated. We need to handle DNS and obtain a certificate ourselves. These tasks were handled automatically inside the cluster by kubernetes and `autocert`, respectively.
That said, because our server uses mTLS **only clients that have a certificate issued by our certificate authority will be allowed to connect**. That means it can be safely and easily exposed directly to the public internet using a [LoadBalancer service type](https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer):
To connect we need a certificate, which we'll need to obtain from the CA. There are a [couple](RUNBOOK.md#federation) [ways](RUNBOOK.md#exposing-the-ca) to do this, but for simplicity we'll just forward a port.
To follow along you'll need to [`install step`](https://github.com/smallstep/cli#installing).
To connect we need a certificate. There are a [couple](RUNBOOK.md#federation) [different](RUNBOOK.md#multiple-intermediates) [ways](RUNBOOK.md#exposing-the-ca) to get one, but for simplicity we'll just forward a port.
In another window we can use `step` to grab the root certificate, generate a key pair, and get a certificate to use with `curl`. You'll need the admin password and CA fingerprint output during installation (see [here](RUNBOOK.md#recovering-fingerprint) and [here](#RUNBOOK.md#recovering-admin-password) if you already lost them :).
In another window we'll use `step` to grab the root certificate, generate a key pair, and get a certificate.
> To follow along you'll need to [`install step`](https://github.com/smallstep/cli#installing) if you haven't already. You'll also need your admin password and CA fingerprint, which were output during installation (see [here](RUNBOOK.md#recover-admin-and-ca-password) and [here](#RUNBOOK.md#recompute-root-certificate-fingerprint) if you already lost them :).
✔ Please enter the password to decrypt the provisioner key: 0QOC9xcq56R1aEyLHPzBqN18Z3WfGZ01
✔ CA: https://127.0.0.1:4443/1.0/sign
✔ Certificate: snarf.crt
✔ Private Key: snarf.key
✔ Certificate: mike.crt
✔ Private Key: mike.key
```
Now we can simply `curl` the service:
> If you're using minikube or docker for mac the load balancer's "IP" might be `localhost`, which won't work. In that case, simply `export HELLO_MTLS_IP=127.0.0.1` and try again.
```
$ export HELLO_MTLS_IP=$(kubectl get svc hello-mtls-lb -ojsonpath={$.status.loadBalancer.ingress[0].ip})
> If you're using minikube or docker for mac the load balancer's "IP" might be `localhost`, which won't work. In that case, simply `export HELLO_MTLS_IP=127.0.0.1` and try again.
> Note that we're using the `--resolve` flag to tell `curl` to resolve the name in our workload's certificate to its public IP address. In a real production infrastructure you could configure DNS manually, or you could propagate DNS to workloads outside kubernetes using something like [ExternalDNS](#).
> Note that we're using `--resolve` to tell `curl` to override DNS and resolve the name in our workload's certificate to its public IP address. In a real production infrastructure you could configure DNS manually, or you could propagate DNS to workloads outside kubernetes using something like [ExternalDNS](https://github.com/kubernetes-incubator/external-dns).
✅ mTLS outside cluster.
<!--- TODO: CTA or Further Reading... Move "How it works" maybe? Or put this below that? --->
### Cleanup & uninstall
To clean up after running through the tutorial remove the `hello-mtls` and `hello-mtls-client` deployments and services:
@ -265,9 +269,7 @@ kubectl delete service hello-mtls
kubectl delete service hello-mtls-lb
```
The runbook contains instructions for [uninstalling `autocert` complete](RUNBOOK.md#uninstalling).
<!--- TODO: CTA or Further Reading... Move "How it works" maybe? Or put this below that? --->
See the runbook for instructions on [uninstalling `autocert`](RUNBOOK.md#uninstalling).
## How it works
@ -279,7 +281,7 @@ The runbook contains instructions for [uninstalling `autocert` complete](RUNBOOK
### Enrollment & renewal
It integrates with [`step certificates`](https://github.com/smallstep/certificates) and uses the [one-time token bootstrap protocol](https://smallstep.com/blog/...) from that project to mutually authenticate a new pod with your certificate authority, and obtain a certificate.
It integrates with [`step certificates`](https://github.com/smallstep/certificates) and uses the [one-time token bootstrap protocol](https://smallstep.com/blog/step-certificates.html#automated-certificate-management) from that project to mutually authenticate a new pod with your certificate authority, and obtain a certificate.
@ -287,30 +289,37 @@ Tokens are [generated by the admission webhook](controller/provisioner.go#L46-L7
## Questions
#### How is this different than [`cert-manager`](https://github.com/jetstack/cert-manager)
#### Wait, so any pod can get a certificate with any identity? How is that secure?
`Cert-manager` is a great project. However, it's designed primarily for managing certificates issued by [Let's Encrypt's](https://letsencrypt.org/) public certificate authority. These certificates are useful for TLS ingress from web browsers. `Autocert` is different. It's purpose-built to manage certificates issued by your own private CA to support the use of mTLS for internal communication (e.g., service-to-service).
1. Don't give people `kubectl` access to your production clusters
2. Use a deploy pipeline based on `git` artifacts
3. Enforce code review on those `git` artifacts
#### Doesn't kubernetes already ship with a certificate authority?
If that doesn't work for you, or if you have a better idea, we'd love to hear! Please [open an issue](https://github.com/smallstep/certificates/issues/new?template=autocert_feature.md)!
#### Why do I have to tell you the name to put in a certificate? Why can't you automatically bind service names?
Mostly because monitoring the API server to figure out which services are associated with which workloads is complicated and somewhat magical. And it might not be what you want.
That said, we're not totally opposed to this idea. If anyone has strong feels and a good design please [open an issue](https://github.com/smallstep/certificates/issues/new?template=autocert_feature.md).
Yes, actually it can have [a bunch of them](https://jvns.ca/blog/2017/08/05/how-kubernetes-certificates-work/) for different sorts of control plane communication.
#### Doesn't kubernetes already ship with a certificate authority?
Wait, no, _actually_
it doesn't _ship_ with _any_ CA. It's complicated. Kubernetes doesn't come with a CA, it has integration points that allow you to use any CA (e.g., [Kubernetes the hard way](https://github.com/kelseyhightower/kubernetes-the-hard-way) [uses CFSSL](https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/2983b28f13b294c6422a5600bb6f14142f5e7a26/docs/02-certificate-authority.md). You could use [`step certificates`](https://github.com/smallstep/certificates), which `autocert` is based on, instead.
Yes, it uses [a bunch of CAs](https://jvns.ca/blog/2017/08/05/how-kubernetes-certificates-work/) for different sorts of control plane communication. Technically, kubernetes doesn't _come with_ a CA. It has integration points that allow you to use any CA (e.g., [Kubernetes the hard way](https://github.com/kelseyhightower/kubernetes-the-hard-way) [uses CFSSL](https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/2983b28f13b294c6422a5600bb6f14142f5e7a26/docs/02-certificate-authority.md). You could use [`step certificates`](https://github.com/smallstep/certificates), which `autocert` is based on, instead.
In any case, none of these CAs are meant for issuing certificates to your workloads for service-to-service communication. Rather, they're meant to secure communication between various control plane components. You could use them for your data plane, but it's probably not a good idea.
In any case, these CAs are meant for control plane communication. You could use them for your service-to-service data plane, but it's probably not a good idea.
#### What permissions does `autocert` require in my cluster and why?
By default we ask for the narrowest permissions we can: the ability to create and delete secrets cluster-wide. You can [check out our RBAC config here](install/03-rbac.yaml).
`Autocert` needs permission to create and delete secrets cluster-wide. You can [check out our RBAC config here](install/03-rbac.yaml). These permissions are needed in order to transmit one-time tokens to workloads using secrets, and to clean up afterwards. We'd love to scope these permissions down further. If anyone has any ideas please [open an issue](https://github.com/smallstep/certificates/issues/new?template=autocert_feature.md).
We need these permissions in order to transmit one-time tokens to workloads using secrets, and to clean up afterwards. We'd love to scope these permissions down further if anyone has any ideas.
#### Why does `autocert` create secrets?
#### Why does the mutating webhook have to create secrets?
The `autocert` admission webhook needs to securely transmit one-time bootstrap tokens to containers. This could be accomplished without using secrets. The webhook returns a [JSONPatch](https://tools.ietf.org/html/rfc6902) response that's applied to the pod spec. This response could patch the literal token value into our init container's environment.
The `autocert` admission webhook needs to securely transmit one-time bootstrap tokens to containers. This could be accomplished without using secrets by simply patching a token directly into the pod's environment via the admission webhook response. Unfortunately, the kubernetes API server does not authenticate itself to admission webhooks by default, and configuring it to do so [requires passing a custom config file](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#authenticate-apiservers) at apiserver startup. This isn't an option for everyone (e.g., on GKE) so we opted not to rely on it.
Unfortunately, the kubernetes API server does not authenticate itself to admission webhooks by default, and configuring it to do so [requires passing a custom config file](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#authenticate-apiservers) at apiserver startup. This isn't an option for everyone (e.g., on GKE) so we opted not to rely on it.
Since our webhook can't tell who is calling it, including bootstrap tokens in patch responses would be dangerous. By using secrets an attacker can still trick us into generating superflous bootstrap tokens, but they'd also need read access to cluster secrets to do anything with them.
Since our webhook can't authenticate callers, including bootstrap tokens in patch responses would be dangerous. By using secrets an attacker can still trick `autocert` into generating superflous bootstrap tokens, but they'd also need read access to cluster secrets to do anything with them.
Hopefully this story will improve with time.
@ -322,7 +331,7 @@ TODO: Link to issue for people who want this.
#### Too. many. containers. Why do you need to install an init container and sidecar?
We don't. Your containers can generate key pairs, exchange them for certificates, and manage renewals themselves. This is pretty easy if you [install `step`](https://github.com/smallstep/cli#installing) in your containers, or integrate with our [golang SDK](https://godoc.org/github.com/smallstep/certificates/ca). To support this we'd need to add the option to inject a bootstrap token without injecting these containers.
We don't. It's just easier for you. Your containers can generate key pairs, exchange them for certificates, and manage renewals themselves. This is pretty easy if you [install `step`](https://github.com/smallstep/cli#installing) in your containers, or integrate with our [golang SDK](https://godoc.org/github.com/smallstep/certificates/ca). To support this we'd need to add the option to inject a bootstrap token without injecting these containers.
TODO: Link to issue for people who want this.
@ -330,9 +339,9 @@ That said, the init container and sidecar are both super lightweight.
#### Why are keys and certificates managed via volume mounts? Why not use a secret or some custom resource?
Because, by default, kubernetes secrets are stored in plaintext in `etcd` and might even be transmitted unencrypted across the network. Even if secrets were properly encrypted, transmitting a private key across the network violates PKI best practices. Key pairs should always be generated where they're used, and private keys should never be shared with anyone but their owners.
Because, by default, kubernetes secrets are stored in plaintext in `etcd` and might even be transmitted unencrypted across the network. Even if secrets were properly encrypted, transmitting a private key across the network violates PKI best practices. Key pairs should always be generated where they're used, and private keys should never be known by anyone but their owners.
That said, there are use cases where a certificate mounted in a secret resource is desirable (e.g., for use with a kubernetes `Ingress`). We may add support for this in the future. However, we think the current method is easier and better.
That said, there are use cases where a certificate mounted in a secret resource is desirable (e.g., for use with a kubernetes `Ingress`). We may add support for this in the future. However, we think the current method is easier and a better default.
TODO: Link to issue for people who want this.
@ -340,11 +349,9 @@ TODO: Link to issue for people who want this.
It's harder and less secure. If any good and simple design exists for securely automating CSR approval using this resource we'd love to see it!
#### Why do I have to tell you the name to put in a certificate? Why can't you automatically bind service names?
Mostly because monitoring the API server to figure out which services are associated with which workloads is complicated and somewhat magical. And it might not be what you want.
#### How is this different than [`cert-manager`](https://github.com/jetstack/cert-manager)
That said, we're not totally opposed to this idea if anyone has strong feels and a good design.
`Cert-manager` is a great project. But it's design is focused on managing Web PKI certificates issued by [Let's Encrypt's](https://letsencrypt.org/) public certificate authority. These certificates are useful for TLS ingress from web browsers. `Autocert` is different. It's purpose-built to manage certificates issued by your own private CA to support the use of mTLS for internal communication (e.g., service-to-service).
#### What sorts of keys are issued and how often are certificates rotated?
#### Recompute your CA's root certificate fingerprint
#### Recompute root certificate fingerprint
```
export CA_POD=$(kubectl -n step get pods -l app=ca -o jsonpath={$.items[0].metadata.name})
@ -67,7 +69,37 @@ for ns in $(kubectl get namespace --selector autocert.step.sm=enabled -o jsonpat
done
```
#### Uninstalling
### TODO:
* Change admin password
* Change autocert password
* Federating with another CA
* DNS tips and tricks
* Multiple SANs
* Getting rid of the sidecar
* Getting logs from the CA (certificates weren't issued)
* Getting logs from the init container / renewer (didn't start properly)
* Adjusting certificate expiration (default 24h)
* Remove label
* Clean up secrets
* Naming considerations (maybe this should be in hello-mtls)
## Federation
TODO: Example of federating a CA running in kubernetes with another CA.
For now, see https://smallstep.com/blog/step-v0.8.3-federation-root-rotation.html
## Multiple intermediates
TODO: Example of creating an additional intermediate signing certificate off of our kubernetes root CA.
For now, see https://smallstep.com/docs/cli/ca/init/ (specifically, the `--root` flag)
## Exposing the CA
Beware that the CA exposes an unauthenticated endpoint that lists your configured provisioners and their encrypted private keys. For this reason, you may not want to expose it directly to the public internet.
## Uninstalling
To uninstall `autocert` completely simply delete the mutating webhook configuration, the `step` namespace and the `autocert` RBAC artifacts:
@ -87,17 +119,4 @@ for ns in $(kubectl get namespace --selector autocert.step.sm=enabled -o jsonpat
done
```
Any remaining sidecar containers will go away once you remove annotations and re-deploy your workloads.
### TODO:
* Change admin password
* Change autocert password
* Federating with another CA
* DNS tips and tricks
* Multiple SANs
* Getting rid of the sidecar
* Getting logs from the CA (certificates weren't issued)
* Getting logs from the init container / renewer (didn't start properly)
* Adjusting certificate expiration (default 24h)
* Remove label
* Clean up secrets
Any remaining sidecar containers will go away once you remove annotations and re-deploy your workloads.
Mike Malone
5 years ago