Delete old autocert

pull/79/head
Mariano Cano 5 years ago
parent 5b84961f42
commit e852b57870

@ -1,180 +0,0 @@
# Installing `autocert`
### Prerequisites
To get started you'll need [`kubectl`](https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl) and a cluster running kubernetes `1.9` or later with [admission webhooks](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#admission-webhooks) enabled:
```bash
$ kubectl version --short
Client Version: v1.13.1
Server Version: v1.10.11
$ kubectl api-versions | grep "admissionregistration.k8s.io/v1beta1"
admissionregistration.k8s.io/v1beta1
```
### Install
The easiest way to install `autocert` is to run:
```bash
kubectl run autocert-init -it --rm --image smallstep/autocert-init --restart Never
```
💥 installation complete.
> You might want to [check out what this command does](init/autocert.sh) before running it.
## Manual install
To install manually you'll need to [install step](https://github.com/smallstep/cli#installing) version `0.8.3` or later.
```
$ step version
Smallstep CLI/0.8.3 (darwin/amd64)
Release Date: 2019-01-16 01:46 UTC
```
### Create a CA
Set your `STEPPATH` to a working directory where we can stage our CA artifacts before we push them to kubernetes. You can delete this directory once installation is complete.
```
$ export STEPPATH=$(mktemp -d /tmp/step.XXX)
$ step path
/tmp/step.0kE
```
Run `step ca init` to generate a root certificate and CA configuration for your cluster. You'll be prompted for a password that will be used to encrypt key material.
```
$ step ca init \
--name Autocert \
--dns "ca.step.svc.cluster.local,127.0.0.1" \
--address ":4443" \
--provisioner admin \
--with-ca-url "ca.step.svc.cluster.local"
```
For older versions of `step` run this command without the flags.
Add provisioning credentials for use by `autocert`. You'll be prompted for a password for `autocert`.
```
$ step ca provisioner add autocert --create
```
For older versions of `step`:
* Run `step ca init` and follow prompts
* Edit `$(step path)/config/ca.json` and change base paths to `/home/step`
* Edit `$(step path)/config/defaults.json` to change base paths to `/home/step` and remove port from CA URL
```
$ sed -i "" "s|$(step path)|/home/step/.step|g" $(step path)/config/ca.json
$ sed -i "" "s|$(step path)|/home/step/.step|g" $(step path)/config/defaults.json
$ sed -i "" "s|ca.step.svc.cluster.local:4443|ca.step.svc.cluster.local|" $(step path)/config/defaults.json
```
### Install the CA in Kubernetes
We'll be creating a new kubernetes namespace and setting up some RBAC rules during installation. You'll need appropriate permissions in your cluster (e.g., you may need to be cluster-admin). GKE, in particular, does not give the cluster owner these rights by default. You can give yourself cluster-admin rights on GKE by running:
```bash
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole cluster-admin \
--user $(gcloud config get-value account)
```
We'll install our CA and the `autocert` controller in the `step` namespace.
```
$ kubectl create namespace step
```
To install the CA we need to configmap the CA certificates, signing keys, and configuration artifacts. Note that key material is encrypted so we don't need to use secrets.
```
$ kubectl -n step create configmap config --from-file $(step path)/config
$ kubectl -n step create configmap certs --from-file $(step path)/certs
$ kubectl -n step create configmap secrets --from-file $(step path)/secrets
```
But we will need to create secrets for the CA and autocert to decrypt their keys:
```
$ kubectl -n step create secret generic ca-password --from-literal password=<ca-password>
$ kubectl -n step create secret generic autocert-password --from-literal password=<autocert-password>
```
Where `<ca-password>` is the password you entered during `step ca init` and `<autocert-password>` is the password you entered during `step ca provisioner add`.
Next, we'll install the CA.
```
$ kubectl apply -f https://raw.githubusercontent.com/smallstep/certificates/master/autocert/install/01-step-ca.yaml
```
Once you've done this you can delete the temporary `$STEPPATH` directory and `unset STEPPATH` (though you may want to retain it as a backup).
### Install `autocert` in Kubernetes
Install the `autocert` controller.
```
$ kubectl apply -f https://raw.githubusercontent.com/smallstep/certificates/master/autocert/install/02-autocert.yaml
```
Autocert creates secrets containing single-use bootstrap tokens for pods to authenticate with the CA and obtain a certificate. The tokens are automatically cleaned up after they expire. To do this, `autocert` needs permission to create and delete secrets in your cluster.
If you have RBAC enabled in your cluster, apply `rbac.yaml` to give `autocert` these permissions.
```
$ kubectl apply -f https://raw.githubusercontent.com/smallstep/certificates/master/autocert/install/03-rbac.yaml
```
Finally, register the `autocert` mutation webhook with kubernetes.
```
$ cat <<EOF | kubectl apply -f -
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
name: autocert-webhook-config
labels: {app: autocert}
webhooks:
- name: autocert.step.sm
clientConfig:
service:
name: autocert
namespace: step
path: "/mutate"
caBundle: $(cat $(step path)/certs/root_ca.crt | base64)
rules:
- operations: ["CREATE"]
apiGroups: [""]
apiVersions: ["v1"]
resources: ["pods"]
failurePolicy: Ignore
namespaceSelector:
matchLabels:
autocert.step.sm: enabled
EOF
```
### Check your work
If everything worked you should have CA and controller pods running in the `step` namespace and your webhook configuration should be installed:
```
$ kubectl -n step get pods
NAME READY STATUS RESTARTS AGE
ca-7577d7d667-vtfq5 1/1 Running 0 1m
controller-86bd99bd96-s9zlc 1/1 Running 0 28s
$ kubectl get mutatingwebhookconfiguration
NAME CREATED AT
autocert-webhook-config 2019-01-17T22:57:57Z
```
### Move on to usage instructions
Make sure to follow the autocert usage steps at https://github.com/smallstep/certificates/tree/master/autocert#usage

@ -1,414 +1,18 @@
![Autocert architecture diagram](https://raw.githubusercontent.com/smallstep/certificates/autocert/autocert/autocert-logo.png)
![Autocert architecture diagram](https://raw.githubusercontent.com/smallstep/autocert/master/autocert-logo.png)
# Autocert
[![GitHub release](https://img.shields.io/github/release/smallstep/certificates.svg)](https://github.com/smallstep/certificates/releases)
[![GitHub release](https://img.shields.io/github/release/smallstep/autocert.svg)](https://github.com/smallstep/autocert/releases)
[![Join the chat at https://gitter.im/smallstep/community](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/smallstep/community)
[![CA Image](https://images.microbadger.com/badges/image/smallstep/step-ca.svg)](https://microbadger.com/images/smallstep/step-ca)
[![Go Report Card](https://goreportcard.com/badge/github.com/smallstep/certificates)](https://goreportcard.com/report/github.com/smallstep/certificates)
[![Go Report Card](https://goreportcard.com/badge/github.com/smallstep/autocert)](https://goreportcard.com/report/github.com/smallstep/autocert)
[![GitHub stars](https://img.shields.io/github/stars/smallstep/certificates.svg?style=social)](https://github.com/smallstep/certificates/stargazers)
[![GitHub stars](https://img.shields.io/github/stars/smallstep/autocert.svg?style=social)](https://github.com/smallstep/autocert/stargazers)
[![Twitter followers](https://img.shields.io/twitter/follow/smallsteplabs.svg?label=Follow&style=social)](https://twitter.com/intent/follow?screen_name=smallsteplabs)
**Autocert** is a kubernetes add-on that automatically injects TLS/HTTPS certificates into your containers.
To get a certificate **simply annotate your pods** with a name. An X.509 (TLS/HTTPS) certificate is automatically created and mounted at `/var/run/autocert.step.sm/` along with a corresponding private key and root certificate (everything you need for [mTLS](#motivation)).
⚠️ Autocert has been promoted to a top level project. Check it out at
https://github.com/smallstep/autocert
We ❤️ feedback. Please [report bugs](https://github.com/smallstep/certificates/issues/new?template=autocert_bug.md) & [suggest enhancements](https://github.com/smallstep/certificates/issues/new?template=autocert_enhancement.md). [Fork](https://github.com/smallstep/certificates/fork) and send a PR. [Give us a ⭐](https://github.com/smallstep/certificates/stargazers) if you like what we're doing.
![Autocert demo gif](https://raw.githubusercontent.com/smallstep/certificates/autocert/autocert/demo.gif)
## Motivation
`Autocert` exists to **make it easy to use mTLS** ([mutual TLS](examples/hello-mtls/README.md#mutual-tls)) to **improve security** within a cluster and to **secure communication into, out of, and between kubernetes clusters**.
TLS (and HTTPS, which is HTTP over TLS) provides _authenticated encryption_: an _identity dialtone_ and _end-to-end encryption_ for your workloads. It's like a secure line with caller ID. This has all sorts of benefits: better security, compliance, and easier auditability for starters. It **makes workloads identity-aware**, improving observability and enabling granular access control. Perhaps most compelling, mTLS lets you securely communicate with workloads running anywhere, not just inside kubernetes.
![Connect with mTLS diagram](https://raw.githubusercontent.com/smallstep/certificates/autocert/autocert/connect-with-mtls.png)
Unlike VPNs & SDNs, deploying and scaling mTLS is pretty easy. You're (hopefully) already using TLS, and your existing tools and standard libraries will provide most of what you need. If you know how to operate DNS and reverse proxies, you know how to operate mTLS infrastructure.
There's just one problem: **you need certificates issued by your own certificate authority (CA)**. Building and operating a CA, issuing certificates, and making sure they're renewed before they expire is tricky. `Autocert` does all of this for you.
## Features
First and foremost, `autocert` is easy. You can **get started in minutes**.
`Autocert` uses [`step certificates`](https://github.com/smallstep/certificates) to generate keys and issue certificates. This process is secure and automatic, all you have to do is [install autocert](#install) and [annotate your pods](#enable-autocert-per-namespace).
Features include:
* A fully featured private **certificate authority** (CA) for workloads running on kubernetes and elsewhere
* [RFC5280](https://tools.ietf.org/html/rfc5280) and [CA/Browser Forum](https://cabforum.org/baseline-requirements-documents/) compliant certificates that work **for TLS**
* Namespaced installation into the `step` namespace so it's **easy to lock down** your CA
* Short-lived certificates with **fully automated** enrollment and renewal
* Private keys are never transmitted across the network and aren't stored in `etcd`
Because `autocert` is built on [`step certificates`](https://github.com/smallstep/certificates) you can easily [extend access](#connecting-from-outside-the-cluster) to developers, endpoints, and workloads running outside your cluster, too.
## Getting Started
> ⚠️ Warning: *this project is in **ALPHA**. DON'T use it for anything mission critical. EXPECT breaking changes in minor revisions with little or no warning. PLEASE [provide feedback](https://github.com/smallstep/certificates/issues/new?template=autocert_enhancement.md).*
### Prerequisites
All you need to get started is [`kubectl`](https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl) and a cluster running kubernetes `1.9` or later with [admission webhooks](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#admission-webhooks) enabled:
```bash
$ kubectl version --short
Client Version: v1.13.1
Server Version: v1.10.11
$ kubectl api-versions | grep "admissionregistration.k8s.io/v1beta1"
admissionregistration.k8s.io/v1beta1
```
### Install
To install `autocert` run:
```bash
kubectl run autocert-init -it --rm --image smallstep/autocert-init --restart Never
```
💥 installation complete.
> You might want to [check out what this command does](init/autocert.sh) before running it. You can also [install `autocert` manually](INSTALL.md#manual-install) if that's your style.
## Usage
Using `autocert` is also easy:
* Enable `autocert` for a namespace by labelling it with `autocert.step.sm=enabled`, then
* Inject certificates into containers by annotating pods with `autocert.step.sm/name: <name>`
### Enable autocert (per namespace)
To enable `autocert` for a namespace it must be labelled `autocert.step.sm=enabled`.
To label the `default` namespace run:
```bash
kubectl label namespace default autocert.step.sm=enabled
```
To check which namespaces have `autocert` enabled run:
```bash
$ kubectl get namespace -L autocert.step.sm
NAME STATUS AGE AUTOCERT.STEP.SM
default Active 59m enabled
...
```
### Annotate pods to get certificates
To get a certificate you need to tell `autocert` your workload's name using the `autocert.step.sm/name` annotation (this name will appear as the X.509 common name and SAN).
Let's deploy a [simple mTLS server](examples/hello-mtls/go/server/server.go) named `hello-mtls.default.svc.cluster.local`:
```yaml
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata: {name: hello-mtls, labels: {app: hello-mtls}}
spec:
replicas: 1
selector: {matchLabels: {app: hello-mtls}}
template:
metadata:
annotations:
# AUTOCERT ANNOTATION HERE -v ###############################
autocert.step.sm/name: hello-mtls.default.svc.cluster.local #
# AUTOCERT ANNOTATION HERE -^ ###############################
labels: {app: hello-mtls}
spec:
containers:
- name: hello-mtls
image: smallstep/hello-mtls-server-go:latest
EOF
```
In our new container we should find a certificate, private key, and root certificate mounted at `/var/run/autocert.step.sm`:
```bash
$ export HELLO_MTLS=$(kubectl get pods -l app=hello-mtls -o jsonpath={$.items[0].metadata.name})
$ kubectl exec -it $HELLO_MTLS -c hello-mtls -- ls /var/run/autocert.step.sm
root.crt site.crt site.key
```
We're done. Our container has a certificate, issued by our CA, which `autocert` will automatically renew.
✅ Certificates.
## Hello mTLS
It's easy to deploy certificates using `autocert`, but it's up to you to use them correctly. To get you started, [`hello-mtls`](examples/hello-mtls) demonstrates the right way to use mTLS with various tools and languages (contributions welcome :). If you're a bit fuzzy on how mTLS works, [the `hello-mtls` README](examples/hello-mtls/README.md) is a great place to start.
To finish out this tutorial let's keep things simple and try `curl`ing the server we just deployed from inside and outside the cluster.
### Connecting from inside the cluster
First, let's expose our workload to the rest of the cluster using a service:
```
kubectl expose deployment hello-mtls --port 443
```
Now let's deploy a client, with its own certificate, that [`curl`s our server in a loop](examples/hello-mtls/curl/client.sh):
```yaml
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata: {name: hello-mtls-client, labels: {app: hello-mtls-client}}
spec:
replicas: 1
selector: {matchLabels: {app: hello-mtls-client}}
template:
metadata:
annotations:
# AUTOCERT ANNOTATION HERE -v ######################################
autocert.step.sm/name: hello-mtls-client.default.pod.cluster.local #
# AUTOCERT ANNOTATION HERE -^ ######################################
labels: {app: hello-mtls-client}
spec:
containers:
- name: hello-mtls-client
image: smallstep/hello-mtls-client-curl:latest
env: [{name: HELLO_MTLS_URL, value: https://hello-mtls.default.svc.cluster.local}]
EOF
```
> Note that **the authority portion of the URL** (the `HELLO_MTLS_URL` env var) **matches the name of the server we're connecting to** (both are `hello-mtls.default.svc.cluster.local`). That's required for standard HTTPS and can sometimes require some DNS trickery.
Once deployed we should start seeing the client log responses from the server [saying hello](examples/hello-mtls/go/server/server.go#L71-L72):
```
$ export HELLO_MTLS_CLIENT=$(kubectl get pods -l app=hello-mtls-client -o jsonpath={$.items[0].metadata.name})
$ kubectl logs $HELLO_MTLS_CLIENT -c hello-mtls-client
Thu Feb 7 23:35:23 UTC 2019: Hello, hello-mtls-client.default.pod.cluster.local!
Thu Feb 7 23:35:28 UTC 2019: Hello, hello-mtls-client.default.pod.cluster.local!
```
For kicks, let's `exec` into this pod and try `curl`ing ourselves:
```
$ kubectl exec $HELLO_MTLS_CLIENT -c hello-mtls-client -- curl -sS \
--cacert /var/run/autocert.step.sm/root.crt \
--cert /var/run/autocert.step.sm/site.crt \
--key /var/run/autocert.step.sm/site.key \
https://hello-mtls.default.svc.cluster.local
Hello, hello-mtls-client.default.pod.cluster.local!
```
✅ mTLS inside cluster.
### Connecting from outside the cluster
Connecting from outside the cluster is a bit more complicated. We need to handle DNS and obtain a certificate ourselves. These tasks were handled automatically inside the cluster by kubernetes and `autocert`, respectively.
That said, because our server uses mTLS **only clients that have a certificate issued by our certificate authority will be allowed to connect**. That means it can be safely and easily exposed directly to the public internet using a [LoadBalancer service type](https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer):
```
kubectl expose deployment hello-mtls --name=hello-mtls-lb --port=443 --type=LoadBalancer
```
To connect we need a certificate. There are a [couple](RUNBOOK.md#federation) [different](RUNBOOK.md#multiple-intermediates) [ways](RUNBOOK.md#exposing-the-ca) to get one, but for simplicity we'll just forward a port.
```
kubectl -n step port-forward $(kubectl -n step get pods -l app=ca -o jsonpath={$.items[0].metadata.name}) 4443:4443
```
In another window we'll use `step` to grab the root certificate, generate a key pair, and get a certificate.
> To follow along you'll need to [`install step`](https://github.com/smallstep/cli#installing) if you haven't already. You'll also need your admin password and CA fingerprint, which were output during installation (see [here](RUNBOOK.md#recover-admin-and-ca-password) and [here](RUNBOOK.md#recompute-root-certificate-fingerprint) if you already lost them :).
```bash
$ export CA_POD=$(kubectl -n step get pods -l app=ca -o jsonpath={$.items[0].metadata.name})
$ step ca root root.crt --ca-url https://127.0.0.1:4443 --fingerprint <fingerprint>
$ step ca certificate mike mike.crt mike.key --ca-url https://127.0.0.1:4443 --root root.crt
✔ Key ID: H4vH5VfvaMro0yrk-UIkkeCoPFqEfjF6vg0GHFdhVyM (admin)
✔ Please enter the password to decrypt the provisioner key: 0QOC9xcq56R1aEyLHPzBqN18Z3WfGZ01
✔ CA: https://127.0.0.1:4443/1.0/sign
✔ Certificate: mike.crt
✔ Private Key: mike.key
```
Now we can simply `curl` the service:
> If you're using minikube or docker for mac the load balancer's "IP" might be `localhost`, which won't work. In that case, simply `export HELLO_MTLS_IP=127.0.0.1` and try again.
```
$ export HELLO_MTLS_IP=$(kubectl get svc hello-mtls-lb -ojsonpath={$.status.loadBalancer.ingress[0].ip})
$ curl --resolve hello-mtls.default.svc.cluster.local:443:$HELLO_MTLS_IP \
--cacert root.crt \
--cert mike.crt \
--key mike.key \
https://hello-mtls.default.svc.cluster.local
Hello, mike!
```
> Note that we're using `--resolve` to tell `curl` to override DNS and resolve the name in our workload's certificate to its public IP address. In a real production infrastructure you could configure DNS manually, or you could propagate DNS to workloads outside kubernetes using something like [ExternalDNS](https://github.com/kubernetes-incubator/external-dns).
✅ mTLS outside cluster.
<!--- TODO: CTA or Further Reading... Move "How it works" maybe? Or put this below that? --->
### Cleanup & uninstall
To clean up after running through the tutorial remove the `hello-mtls` and `hello-mtls-client` deployments and services:
```
kubectl delete deployment hello-mtls
kubectl delete deployment hello-mtls-client
kubectl delete service hello-mtls
kubectl delete service hello-mtls-lb
```
See the runbook for instructions on [uninstalling `autocert`](RUNBOOK.md#uninstalling).
## How it works
### Architecture
`Autocert` is an [admission webhook](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#admission-webhooks) that intercepts and patches pod creation requests with [some YAML](install/02-autocert.yaml#L26-L44) to inject an [init container](bootstrapper/) and [sidecar](renewer/) that handle obtaining and renewing certificates, respectively.
![Autocert architecture diagram](https://raw.githubusercontent.com/smallstep/certificates/autocert/autocert/autocert-arch.png)
### Enrollment & renewal
It integrates with [`step certificates`](https://github.com/smallstep/certificates) and uses the [one-time token bootstrap protocol](https://smallstep.com/blog/step-certificates.html#automated-certificate-management) from that project to mutually authenticate a new pod with your certificate authority, and obtain a certificate.
![Autocert bootstrap protocol diagram](https://raw.githubusercontent.com/smallstep/certificates/autocert/autocert/autocert-bootstrap.png)
Tokens are [generated by the admission webhook](controller/provisioner.go#L46-L72) and [transmitted to the injected init container via a kubernetes secret](controller/main.go#L91-L125). The init container [uses the one-time token](bootstrapper/bootstrapper.sh) to obtain a certificate. A sidecar is also installed to [renew certificates](renewer/Dockerfile#L8) before they expire. Renewal simply uses mTLS with the CA.
## Further Reading
* We tweet [@smallsteplabs](https://twitter.com/smallsteplabs)
* Read [our blog](https://smallstep.com/blog)
* Check out the [runbook](RUNBOOK.md)
* Check out [`step` CLI](https://github.com/smallstep/cli)
## Questions
#### Wait, so any pod can get a certificate with any identity? How is that secure?
1. Don't give people `kubectl` access to your production clusters
2. Use a deploy pipeline based on `git` artifacts
3. Enforce code review on those `git` artifacts
If that doesn't work for you, or if you have a better idea, we'd love to hear! Please [open an issue](https://github.com/smallstep/certificates/issues/new?template=autocert_enhancement.md)!
#### Why do I have to tell you the name to put in a certificate? Why can't you automatically bind service names?
Mostly because monitoring the API server to figure out which services are associated with which workloads is complicated and somewhat magical. And it might not be what you want.
That said, we're not totally opposed to this idea. If anyone has strong feels and a good design please [open an issue](https://github.com/smallstep/certificates/issues/new?template=autocert_enhancement.md).
#### Doesn't kubernetes already ship with a certificate authority?
Yes, it uses [a bunch of CAs](https://jvns.ca/blog/2017/08/05/how-kubernetes-certificates-work/) for different sorts of control plane communication. Technically, kubernetes doesn't _come with_ a CA. It has integration points that allow you to use any CA (e.g., [Kubernetes the hard way](https://github.com/kelseyhightower/kubernetes-the-hard-way) [uses CFSSL](https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/2983b28f13b294c6422a5600bb6f14142f5e7a26/docs/02-certificate-authority.md). You could use [`step certificates`](https://github.com/smallstep/certificates), which `autocert` is based on, instead.
In any case, these CAs are meant for control plane communication. You could use them for your service-to-service data plane, but it's probably not a good idea.
#### What permissions does `autocert` require in my cluster and why?
`Autocert` needs permission to create and delete secrets cluster-wide. You can [check out our RBAC config here](install/03-rbac.yaml). These permissions are needed in order to transmit one-time tokens to workloads using secrets, and to clean up afterwards. We'd love to scope these permissions down further. If anyone has any ideas please [open an issue](https://github.com/smallstep/certificates/issues/new?template=autocert_enhancement.md).
#### Why does `autocert` create secrets?
The `autocert` admission webhook needs to securely transmit one-time bootstrap tokens to containers. This could be accomplished without using secrets. The webhook returns a [JSONPatch](https://tools.ietf.org/html/rfc6902) response that's applied to the pod spec. This response could patch the literal token value into our init container's environment.
Unfortunately, the kubernetes API server does not authenticate itself to admission webhooks by default, and configuring it to do so [requires passing a custom config file](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#authenticate-apiservers) at apiserver startup. This isn't an option for everyone (e.g., on GKE) so we opted not to rely on it.
Since our webhook can't authenticate callers, including bootstrap tokens in patch responses would be dangerous. By using secrets an attacker can still trick `autocert` into generating superflous bootstrap tokens, but they'd also need read access to cluster secrets to do anything with them.
Hopefully this story will improve with time.
#### Why not use kubernetes service accounts instead of bootstrap tokens?
Great idea! This should be pretty easy to add. However, existing service accounts are [somewhat broken](https://github.com/kubernetes/community/pull/1460) for this use case. The upcoming [TokenRequest API](https://github.com/kubernetes/kubernetes/issues/58790) should fix most of these issues.
TODO: Link to issue for people who want this.
#### Too. many. containers. Why do you need to install an init container and sidecar?
We don't. It's just easier for you. Your containers can generate key pairs, exchange them for certificates, and manage renewals themselves. This is pretty easy if you [install `step`](https://github.com/smallstep/cli#installing) in your containers, or integrate with our [golang SDK](https://godoc.org/github.com/smallstep/certificates/ca). To support this we'd need to add the option to inject a bootstrap token without injecting these containers.
TODO: Link to issue for people who want this.
That said, the init container and sidecar are both super lightweight.
#### Why are keys and certificates managed via volume mounts? Why not use a secret or some custom resource?
Because, by default, kubernetes secrets are stored in plaintext in `etcd` and might even be transmitted unencrypted across the network. Even if secrets were properly encrypted, transmitting a private key across the network violates PKI best practices. Key pairs should always be generated where they're used, and private keys should never be known by anyone but their owners.
That said, there are use cases where a certificate mounted in a secret resource is desirable (e.g., for use with a kubernetes `Ingress`). We may add support for this in the future. However, we think the current method is easier and a better default.
TODO: Link to issue for people who want this.
#### Why not use kubernetes CSR resources for this?
It's harder and less secure. If any good and simple design exists for securely automating CSR approval using this resource we'd love to see it!
#### How is this different than [`cert-manager`](https://github.com/jetstack/cert-manager)
`Cert-manager` is a great project. But it's design is focused on managing Web PKI certificates issued by [Let's Encrypt's](https://letsencrypt.org/) public certificate authority. These certificates are useful for TLS ingress from web browsers. `Autocert` is different. It's purpose-built to manage certificates issued by your own private CA to support the use of mTLS for internal communication (e.g., service-to-service).
#### What sorts of keys are issued and how often are certificates rotated?
`Autocert` builds on `step certificates` which issues ECDSA certificates using the P256 curve with ECDSA-SHA256 signatures by default. If this is all Greek to you, rest assured these are safe, sane, and modern defaults that are suitable for the vast majority of environments.
#### What crypto library is under the hood?
https://golang.org/pkg/crypto/
## Building
This project is based on four docker containers. They use [multi-stage builds](https://docs.docker.com/develop/develop-images/multistage-build/) so all you need in order to build them is `docker`.
> Caveat: the `controller` container uses [`dep`](https://github.com/golang/dep) and `dep init` isn't run during the build. You'll need to run `dep init` in the `controller/` subdirectory prior to building, and you'll need to run `dep ensure -update` if you change any dependencies.
Building `autocert-controller` (the admission webhook):
```
cd controller
docker build -t smallstep/autocert-controller:latest .
```
Building `autocert-bootstrapper` (the init container that generates a key pair and exchanges a bootstrap token for a certificate):
```
cd bootstrapper
docker build -t smallstep/autocert-bootstrapper:latest .
```
Building `autocert-renewer` (the sidecar that renews certificates):
```
cd renewer
docker build -t smallstep/autocert-renewer:latest .
```
Building `autocert-init` (the install script):
```
cd init
docker build -t smallstep/autocert-init:latest .
```
If you build your own containers you'll probably need to [install manually](INSTALL.md). You'll also need to adjust which images are deployed in the [deployment yaml](install/02-autocert.yaml).
## Contributing
If you have improvements to `autocert`, send us your pull requests! For those just getting started, GitHub has a [howto](https://help.github.com/articles/about-pull-requests/). A team member will review your pull requests, provide feedback, and merge your changes. In order to accept contributions we do need you to [sign our contributor license agreement](https://cla-assistant.io/smallstep/certificates).
If you want to contribute but you're not sure where to start, take a look at the [issues with the "good first issue" label](https://github.com/smallstep/certificates/issues?q=is%3Aopen+label%3A%22good+first+issue%22+label%3Aarea%2Fautocert). These are issues that we believe are particularly well suited for outside contributions, often because we probably won't get to them right now. If you decide to start on an issue, leave a comment so that other people know that you're working on it. If you want to help out, but not alone, use the issue comment thread to coordinate.
If you've identified a bug or have ideas for improving `autocert` that you don't have time to implement, we'd love to hear about them. Please open an issue to [report a bug](https://github.com/smallstep/certificates/issues/new?template=autocert_bug.md) or [suggest an enhancement](https://github.com/smallstep/certificates/issues/new?template=autocert_enhancement.md)!
## License
Copyright 2019 Smallstep Labs
Licensed under [the Apache License, Version 2.0](https://github.com/smallstep/certificates/blob/master/LICENSE)
If you are looking for hello-mTLS examples look at
https://github.com/smallstep/autocert/tree/master/examples/hello-mtls

@ -1,122 +0,0 @@
# Runbook
## Common admin tasks
#### Recover `admin` and CA password
```
kubectl -n step get secret ca-password -o jsonpath='{$.data.password}' | base64 -D
```
#### Recover `autocert` password
```
kubectl -n step get secret autocert-password -o jsonpath='{$.data.password}' | base64 -D
```
#### Recompute root certificate fingerprint
```
export CA_POD=$(kubectl -n step get pods -l app=ca -o jsonpath={$.items[0].metadata.name})
kubectl -n step exec -it $CA_POD step certificate fingerprint /home/step/.step/certs/root_ca.crt
```
> Tip: Some slight fanciness is necessary to trim this string if you want to put it into an environment variable:
>
> ```
> export FINGERPRINT="$(kubectl -n step exec -it $CA_POD step certificate fingerprint /home/step/.step/certs/root_ca.crt | tr -d '[:space:]')"
> ```
#### Inspect a certificate
```
kubectl exec -it <pod> -c autocert-renewer -- step certificate inspect /var/run/autocert.step.sm/site.crt
```
#### Labelling a namespace (enabling `autocert` for a namespace)
To enable `autocert` for a namespace it must be labelled. To label an existing namespace run:
```
kubectl label namespace <namespace> autocert.step.sm=enabled
```
#### Checking which namespaces are labelled
```
kubectl get namespace -L autocert.step.sm
```
#### Removing a label from a namespace (disabling `autocert` for a namespace)
```
kubectl label namespace <namespace> autocert.step.sm-
```
#### Naming considerations
Use hostnames. Must be global. Everyone who connects to the service using mTLS must use the same hostname. For internal communication it's easy enough to use the FQDN of a service. For stuff you expose publicly you'll need to manage DNS yourself...
In any case, the critical invariant is: ...
Diagram here?
#### Cleaning up one-time token secrets
```
for ns in $(kubectl get namespace --selector autocert.step.sm=enabled -o jsonpath='{$.items[*].metadata.name}'); do
kubectl -n "$ns" delete secrets --selector="autocert.step.sm/token=true"
done
```
### TODO:
* Change admin password
* Change autocert password
* Federating with another CA
* DNS tips and tricks
* Multiple SANs
* Getting rid of the sidecar
* Getting logs from the CA (certificates weren't issued)
* Getting logs from the init container / renewer (didn't start properly)
* Adjusting certificate expiration (default 24h)
* Remove label
* Clean up secrets
* Naming considerations (maybe this should be in hello-mtls)
## Federation
TODO: Example of federating a CA running in kubernetes with another CA.
For now, see https://smallstep.com/blog/step-v0.8.3-federation-root-rotation.html
## Multiple intermediates
TODO: Example of creating an additional intermediate signing certificate off of our kubernetes root CA.
For now, see https://smallstep.com/docs/cli/ca/init/ (specifically, the `--root` flag)
## Exposing the CA
Beware that the CA exposes an unauthenticated endpoint that lists your configured provisioners and their encrypted private keys. For this reason, you may not want to expose it directly to the public internet.
## Uninstalling
To uninstall `autocert` completely simply delete the mutating webhook configuration, the `step` namespace and the `autocert` RBAC artifacts:
```
kubectl delete mutatingwebhookconfiguration autocert-webhook-config
kubectl delete namespace step
kubectl delete clusterrolebinding autocert-controller
kubectl delete clusterrole autocert-controller
```
Remove any namespace labels and clean up any stray secrets that `autocert` hasn't cleaned up yet:
```
for ns in $(kubectl get namespace --selector autocert.step.sm=enabled -o jsonpath='{$.items[*].metadata.name}'); do
kubectl label namespace "$ns" autocert.step.sm-
kubectl -n "$ns" delete secrets --selector="autocert.step.sm/token=true"
done
```
Any remaining sidecar containers will go away once you remove annotations and re-deploy your workloads.

Binary file not shown.

Before

Width:  |  Height:  |  Size: 566 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 396 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 112 KiB

@ -1,10 +0,0 @@
FROM smallstep/step-cli:0.9.0
USER root
ENV CRT="/var/run/autocert.step.sm/site.crt"
ENV KEY="/var/run/autocert.step.sm/site.key"
ENV STEP_ROOT="/var/run/autocert.step.sm/root.crt"
COPY bootstrapper.sh /home/step/
RUN chmod +x /home/step/bootstrapper.sh
CMD ["/home/step/bootstrapper.sh"]

@ -1,7 +0,0 @@
#!/bin/sh
# Download the root certificate and set permissions
step ca certificate $COMMON_NAME $CRT $KEY
chmod 644 $CRT $KEY
step ca root $STEP_ROOT

Binary file not shown.

Before

Width:  |  Height:  |  Size: 669 KiB

@ -1,19 +0,0 @@
# build stage
FROM golang:alpine AS build-env
RUN apk update && apk upgrade && \
apk add --no-cache git
RUN go get -u github.com/golang/dep/cmd/dep
WORKDIR $GOPATH/src/github.com/step-certificates-k8s/controller
# copy dep files and run dep separately from code for better caching
COPY Gopkg.toml Gopkg.lock ./
RUN dep ensure --vendor-only
COPY . ./
RUN go build -o /server .
# final stage
FROM smallstep/step-cli:0.9.0
ENV STEPPATH="/home/step/.step"
ENV PWDPATH="/home/step/password/password"
ENV CONFIGPATH="/home/step/autocert/config.yaml"
COPY --from=build-env /server .
ENTRYPOINT ./server $CONFIGPATH

@ -1,514 +0,0 @@
# This file is autogenerated, do not edit; changes may be undone by the next 'dep ensure'.
[[projects]]
digest = "1:9afc639ef88d907f2e87ab68cbc63117b88d0d84238fd6b08224515d00a8136a"
name = "github.com/alecthomas/gometalinter"
packages = ["."]
pruneopts = "UT"
revision = "df395bfa67c5d0630d936c0044cf07ff05086655"
version = "v3.0.0"
[[projects]]
branch = "master"
digest = "1:c198fdc381e898e8fb62b8eb62758195091c313ad18e52a3067366e1dda2fb3c"
name = "github.com/alecthomas/units"
packages = ["."]
pruneopts = "UT"
revision = "2efee857e7cfd4f3d0138cc3cbb1b4966962b93a"
[[projects]]
branch = "master"
digest = "1:454adc7f974228ff789428b6dc098638c57a64aa0718f0bd61e53d3cd39d7a75"
name = "github.com/chzyer/readline"
packages = ["."]
pruneopts = "UT"
revision = "2972be24d48e78746da79ba8e24e8b488c9880de"
[[projects]]
digest = "1:848ef40f818e59905140552cc49ff3dc1a15f955e4b56d1c5c2cc4b54dbadf0c"
name = "github.com/client9/misspell"
packages = [
".",
"cmd/misspell",
]
pruneopts = "UT"
revision = "b90dc15cfd220ecf8bbc9043ecb928cef381f011"
version = "v0.3.4"
[[projects]]
digest = "1:2cd7915ab26ede7d95b8749e6b1f933f1c6d5398030684e6505940a10f31cfda"
name = "github.com/ghodss/yaml"
packages = ["."]
pruneopts = "UT"
revision = "0ca9ea5df5451ffdf184b4428c902747c2c11cd7"
version = "v1.0.0"
[[projects]]
branch = "master"
digest = "1:41cf598af650689375f647ed7ed87951e3aedb8d5f40400b6e81a84410650626"
name = "github.com/go-chi/chi"
packages = ["."]
pruneopts = "UT"
revision = "d0891661345200ebf8b816cc7de785e9bd570647"
[[projects]]
digest = "1:b7a8552c62868d867795b63eaf4f45d3e92d36db82b428e680b9c95a8c33e5b1"
name = "github.com/gogo/protobuf"
packages = [
"proto",
"sortkeys",
]
pruneopts = "UT"
revision = "342cbe0a04158f6dcb03ca0079991a51a4248c02"
version = "v0.5"
[[projects]]
branch = "travis-1.9"
digest = "1:e8f5d9c09a7209c740e769713376abda388c41b777ba8e9ed52767e21acf379f"
name = "github.com/golang/lint"
packages = [
".",
"golint",
]
pruneopts = "UT"
revision = "883fe33ffc4344bad1ecd881f61afd5ec5d80e0a"
[[projects]]
branch = "master"
digest = "1:3ee90c0d94da31b442dde97c99635aaafec68d0b8a3c12ee2075c6bdabeec6bb"
name = "github.com/google/gofuzz"
packages = ["."]
pruneopts = "UT"
revision = "24818f796faf91cd76ec7bddd72458fbced7a6c1"
[[projects]]
digest = "1:750e747d0aad97b79f4a4e00034bae415c2ea793fd9e61438d966ee9c79579bf"
name = "github.com/google/shlex"
packages = ["."]
pruneopts = "UT"
revision = "6f45313302b9c56850fc17f99e40caebce98c716"
[[projects]]
branch = "master"
digest = "1:824d147914b40e56e9e1eebd602bc6bb9761989d52fd8e4a498428467980eb17"
name = "github.com/gordonklaus/ineffassign"
packages = ["."]
pruneopts = "UT"
revision = "1003c8bd00dc2869cb5ca5282e6ce33834fed514"
[[projects]]
digest = "1:eaefc85d32c03e5f0c2b88ea2f79fce3d993e2c78316d21319575dd4ea9153ca"
name = "github.com/json-iterator/go"
packages = ["."]
pruneopts = "UT"
revision = "ab8a2e0c74be9d3be70b3184d9acc634935ded82"
version = "1.1.4"
[[projects]]
branch = "master"
digest = "1:e51f40f0c19b39c1825eadd07d5c0a98a2ad5942b166d9fc4f54750ce9a04810"
name = "github.com/juju/ansiterm"
packages = [
".",
"tabwriter",
]
pruneopts = "UT"
revision = "720a0952cc2ac777afc295d9861263e2a4cf96a1"
[[projects]]
digest = "1:31e761d97c76151dde79e9d28964a812c46efc5baee4085b86f68f0c654450de"
name = "github.com/konsorten/go-windows-terminal-sequences"
packages = ["."]
pruneopts = "UT"
revision = "f55edac94c9bbba5d6182a4be46d86a2c9b5b50e"
version = "v1.0.2"
[[projects]]
digest = "1:2dc8db55c5b223e6cd50aa7915e698cfcf56d1ddfa89bbbf65e24729e0a0200a"
name = "github.com/lunixbochs/vtclean"
packages = ["."]
pruneopts = "UT"
revision = "88cfb0c2efe8ed7b0ccf0af83db39359829027bb"
version = "v1.0.0"
[[projects]]
digest = "1:2a2a76072bd413b3484a0b5bb2fbb078b0b7dd8950e9276c900e14dce2354679"
name = "github.com/manifoldco/promptui"
packages = [
".",
"list",
"screenbuf",
]
pruneopts = "UT"
revision = "20f2a94120aa14a334121a6de66616a7fa89a5cd"
version = "v0.3.2"
[[projects]]
digest = "1:2fa7b0155cd54479a755c629de26f888a918e13f8857a2c442205d825368e084"
name = "github.com/mattn/go-colorable"
packages = ["."]
pruneopts = "UT"
revision = "3a70a971f94a22f2fa562ffcc7a0eb45f5daf045"
version = "v0.1.1"
[[projects]]
digest = "1:e150b5fafbd7607e2d638e4e5cf43aa4100124e5593385147b0a74e2733d8b0d"
name = "github.com/mattn/go-isatty"
packages = ["."]
pruneopts = "UT"
revision = "c2a7a6ca930a4cd0bc33a3f298eb71960732a3a7"
version = "v0.0.7"
[[projects]]
digest = "1:33422d238f147d247752996a26574ac48dcf472976eda7f5134015f06bf16563"
name = "github.com/modern-go/concurrent"
packages = ["."]
pruneopts = "UT"
revision = "bacd9c7ef1dd9b15be4a9909b8ac7a4e313eec94"
version = "1.0.3"
[[projects]]
digest = "1:c56ad36f5722eb07926c979d5e80676ee007a9e39e7808577b9d87ec92b00460"
name = "github.com/modern-go/reflect2"
packages = ["."]
pruneopts = "UT"
revision = "94122c33edd36123c84d5368cfb2b69df93a0ec8"
version = "v1.0.1"
[[projects]]
digest = "1:266d082179f3a29a4bdcf1dcc49d4a304f5c7107e65bd22d1fecacf45f1ac348"
name = "github.com/newrelic/go-agent"
packages = [
".",
"internal",
"internal/cat",
"internal/jsonx",
"internal/logger",
"internal/sysinfo",
"internal/utilization",
]
pruneopts = "UT"
revision = "f5bce3387232559bcbe6a5f8227c4bf508dac1ba"
version = "v1.11.0"
[[projects]]
digest = "1:07140002dbf37da92090f731b46fa47be4820b82fe5c14a035203b0e813d0ec2"
name = "github.com/nicksnyder/go-i18n"
packages = [
"i18n",
"i18n/bundle",
"i18n/language",
"i18n/translation",
]
pruneopts = "UT"
revision = "0dc1626d56435e9d605a29875701721c54bc9bbd"
version = "v1.10.0"
[[projects]]
digest = "1:95741de3af260a92cc5c7f3f3061e85273f5a81b5db20d4bd68da74bd521675e"
name = "github.com/pelletier/go-toml"
packages = ["."]
pruneopts = "UT"
revision = "c01d1270ff3e442a8a57cddc1c92dc1138598194"
version = "v1.2.0"
[[projects]]
digest = "1:cf31692c14422fa27c83a05292eb5cbe0fb2775972e8f1f8446a71549bd8980b"
name = "github.com/pkg/errors"
packages = ["."]
pruneopts = "UT"
revision = "ba968bfe8b2f7e042a574c888954fccecfa385b4"
version = "v0.8.1"
[[projects]]
digest = "1:2e76a73cb51f42d63a2a1a85b3dc5731fd4faf6821b434bd0ef2c099186031d6"
name = "github.com/rs/xid"
packages = ["."]
pruneopts = "UT"
revision = "15d26544def341f036c5f8dca987a4cbe575032c"
version = "v1.2.1"
[[projects]]
branch = "master"
digest = "1:8baa3b16f20963c54e296627ea1dabfd79d1b486f81baf8759e99d73bddf2687"
name = "github.com/samfoo/ansi"
packages = ["."]
pruneopts = "UT"
revision = "b6bd2ded7189ce35bc02233b554eb56a5146af73"
[[projects]]
digest = "1:9421f6e9e28ef86933e824b5caff441366f2b69bb281085b9dca40e1f27a1602"
name = "github.com/shurcooL/sanitized_anchor_name"
packages = ["."]
pruneopts = "UT"
revision = "7bfe4c7ecddb3666a94b053b422cdd8f5aaa3615"
version = "v1.0.0"
[[projects]]
digest = "1:e4c72127d910a96daf869a44f3dd563b86dbe6931a172863a0e99c5ff04b59e4"
name = "github.com/sirupsen/logrus"
packages = ["."]
pruneopts = "UT"
revision = "dae0fa8d5b0c810a8ab733fbd5510c7cae84eca4"
version = "v1.4.0"
[[projects]]
digest = "1:b66ff4abd77f39e94020219427c0c62bc1474d41edb2b445d2058adede69475f"
name = "github.com/smallstep/certificates"
packages = [
"api",
"authority",
"authority/provisioner",
"ca",
"logging",
"monitoring",
"server",
]
pruneopts = "UT"
revision = "1bb25b517115cebfb8ce9e5ecb48fc7bbea055ec"
version = "v0.9.0"
[[projects]]
digest = "1:8b36444f30009b5e124a3ac48b353558024a95c3fccdf3e6bb557a091e67342b"
name = "github.com/smallstep/cli"
packages = [
"command",
"config",
"crypto/keys",
"crypto/pemutil",
"crypto/randutil",
"crypto/tlsutil",
"crypto/x509util",
"errs",
"jose",
"pkg/blackfriday",
"pkg/x509",
"token",
"token/provision",
"ui",
"usage",
"utils",
]
pruneopts = "UT"
revision = "b8972dd3035caefb9f493c4a2c664cc6cf557f93"
version = "v0.9.0"
[[projects]]
branch = "master"
digest = "1:ba52e5a5fb800ce55108b7a5f181bb809aab71c16736051312b0aa969f82ad39"
name = "github.com/tsenart/deadcode"
packages = ["."]
pruneopts = "UT"
revision = "210d2dc333e90c7e3eedf4f2242507a8e83ed4ab"
[[projects]]
branch = "master"
digest = "1:8286f653bf8b8fd155a0c9c3b9ee3dbc2a95b1b51f7a1dc11fe2c66018454a0c"
name = "github.com/urfave/cli"
packages = ["."]
pruneopts = "UT"
revision = "693af58b4d51b8fcc7f9d89576da170765980581"
[[projects]]
branch = "master"
digest = "1:8cee4cbf1682070ab96818200f7f43b78850d68ada71698b551cb6c351420016"
name = "golang.org/x/crypto"
packages = [
"cryptobyte",
"cryptobyte/asn1",
"ed25519",
"ed25519/internal/edwards25519",
"pbkdf2",
"ssh/terminal",
]
pruneopts = "UT"
revision = "a5d413f7728c81fb97d96a2b722368945f651e78"
[[projects]]
branch = "master"
digest = "1:4ebfd72ba817efd42cb4be720c20c200d5a2f3694e8a3bd243b95d558fc5f423"
name = "golang.org/x/net"
packages = [
"html",
"html/atom",
"http/httpguts",
"http2",
"http2/hpack",
"idna",
]
pruneopts = "UT"
revision = "63eda1eb0650888965ead1296efd04d0b2b61128"
[[projects]]
branch = "master"
digest = "1:6b3e6ddcebac95be1d690dbd53b5aa2e520715becb7e521bb526ccf3b4c53c15"
name = "golang.org/x/sys"
packages = [
"unix",
"windows",
]
pruneopts = "UT"
revision = "f49334f85ddcf0f08d7fb6dd7363e9e6d6b777eb"
[[projects]]
digest = "1:a2ab62866c75542dd18d2b069fec854577a20211d7c0ea6ae746072a1dccdd18"
name = "golang.org/x/text"
packages = [
"collate",
"collate/build",
"internal/colltab",
"internal/gen",
"internal/tag",
"internal/triegen",
"internal/ucd",
"language",
"secure/bidirule",
"transform",
"unicode/bidi",
"unicode/cldr",
"unicode/norm",
"unicode/rangetable",
]
pruneopts = "UT"
revision = "f21a4dfb5e38f5895301dc265a8def02365cc3d0"
version = "v0.3.0"
[[projects]]
branch = "master"
digest = "1:a45ec3bb7c73e52430410dff3e0a5534ce518f72a8eb4355bc8502c546b91ecc"
name = "golang.org/x/tools"
packages = [
"go/ast/astutil",
"go/gcexportdata",
"go/internal/gcimporter",
"go/types/typeutil",
]
pruneopts = "UT"
revision = "8f05a32dce9ff80c9bf11baeb89d26b410f39281"
[[projects]]
branch = "v3-unstable"
digest = "1:39efb07a0d773dc09785b237ada4e10b5f28646eb6505d97bc18f8d2ff439362"
name = "gopkg.in/alecthomas/kingpin.v3-unstable"
packages = ["."]
pruneopts = "UT"
revision = "63abe20a23e29e80bbef8089bd3dee3ac25e5306"
[[projects]]
digest = "1:ef72505cf098abdd34efeea032103377bec06abb61d8a06f002d5d296a4b1185"
name = "gopkg.in/inf.v0"
packages = ["."]
pruneopts = "UT"
revision = "3887ee99ecf07df5b447e9b00d9c0b2adaa9f3e4"
version = "v0.9.0"
[[projects]]
digest = "1:7fbe10f3790dc4e6296c7c844c5a9b35513e5521c29c47e10ba99cd2956a2719"
name = "gopkg.in/square/go-jose.v2"
packages = [
".",
"cipher",
"json",
"jwt",
]
pruneopts = "UT"
revision = "ef984e69dd356202fd4e4910d4d9c24468bdf0b8"
version = "v2.1.9"
[[projects]]
digest = "1:4d2e5a73dc1500038e504a8d78b986630e3626dc027bc030ba5c75da257cdb96"
name = "gopkg.in/yaml.v2"
packages = ["."]
pruneopts = "UT"
revision = "51d6538a90f86fe93ac480b35f37b2be17fef232"
version = "v2.2.2"
[[projects]]
branch = "master"
digest = "1:e0db54f895460bc12675b12b1c2f8931447051736f370f541214236a84d08023"
name = "k8s.io/api"
packages = [
"admission/v1beta1",
"authentication/v1",
"core/v1",
]
pruneopts = "UT"
revision = "92d2ee7fc726fd16632fbd2dff1466f551f5b8b4"
[[projects]]
branch = "master"
digest = "1:8d51d10f66dbcd39c53f17dc65b9113ca623a72ff99c5e52dfad908b49c07f18"
name = "k8s.io/apimachinery"
packages = [
"pkg/api/resource",
"pkg/apis/meta/v1",
"pkg/apis/meta/v1/unstructured",
"pkg/conversion",
"pkg/conversion/queryparams",
"pkg/fields",
"pkg/labels",
"pkg/runtime",
"pkg/runtime/schema",
"pkg/runtime/serializer",
"pkg/runtime/serializer/json",
"pkg/runtime/serializer/protobuf",
"pkg/runtime/serializer/recognizer",
"pkg/runtime/serializer/versioning",
"pkg/selection",
"pkg/types",
"pkg/util/errors",
"pkg/util/framer",
"pkg/util/intstr",
"pkg/util/json",
"pkg/util/naming",
"pkg/util/net",
"pkg/util/runtime",
"pkg/util/sets",
"pkg/util/validation",
"pkg/util/validation/field",
"pkg/util/yaml",
"pkg/watch",
"third_party/forked/golang/reflect",
]
pruneopts = "UT"
revision = "4ceb6b6c5db56a2f8f454dd837c07160a3d6e131"
[[projects]]
digest = "1:69367163a23cd68971724f36a6759a01d50968e58936808b7eb5e5c186a3a382"
name = "k8s.io/klog"
packages = ["."]
pruneopts = "UT"
revision = "8e90cee79f823779174776412c13478955131846"
[[projects]]
digest = "1:7719608fe0b52a4ece56c2dde37bedd95b938677d1ab0f84b8a7852e4c59f849"
name = "sigs.k8s.io/yaml"
packages = ["."]
pruneopts = "UT"
revision = "fd68e9863619f6ec2fdd8625fe1f02e7c877e480"
version = "v1.1.0"
[solve-meta]
analyzer-name = "dep"
analyzer-version = 1
input-imports = [
"github.com/ghodss/yaml",
"github.com/pkg/errors",
"github.com/sirupsen/logrus",
"github.com/smallstep/certificates/authority/provisioner",
"github.com/smallstep/certificates/ca",
"github.com/smallstep/cli/config",
"github.com/smallstep/cli/crypto/pemutil",
"github.com/smallstep/cli/crypto/randutil",
"github.com/smallstep/cli/jose",
"github.com/smallstep/cli/token",
"github.com/smallstep/cli/token/provision",
"k8s.io/api/admission/v1beta1",
"k8s.io/api/core/v1",
"k8s.io/apimachinery/pkg/apis/meta/v1",
"k8s.io/apimachinery/pkg/runtime",
"k8s.io/apimachinery/pkg/runtime/serializer",
]
solver-name = "gps-cdcl"
solver-version = 1

@ -1,62 +0,0 @@
# Gopkg.toml example
#
# Refer to https://golang.github.io/dep/docs/Gopkg.toml.html
# for detailed Gopkg.toml documentation.
#
# required = ["github.com/user/thing/cmd/thing"]
# ignored = ["github.com/user/project/pkgX", "bitbucket.org/user/project/pkgA/pkgY"]
#
# [[constraint]]
# name = "github.com/user/project"
# version = "1.0.0"
#
# [[constraint]]
# name = "github.com/user/project2"
# branch = "dev"
# source = "github.com/myfork/project2"
#
# [[override]]
# name = "github.com/x/y"
# version = "2.4.0"
#
# [prune]
# non-go = false
# go-tests = true
# unused-packages = true
[[constraint]]
name = "github.com/ghodss/yaml"
version = "1.0.0"
[[constraint]]
name = "github.com/pkg/errors"
version = "0.8.1"
[[constraint]]
name = "github.com/sirupsen/logrus"
version = "1.4.0"
[[constraint]]
name = "github.com/smallstep/certificates"
version = "0.9.0"
[[constraint]]
name = "github.com/smallstep/cli"
version = "0.9.0"
[[constraint]]
branch = "master"
name = "k8s.io/api"
[[constraint]]
branch = "master"
name = "k8s.io/apimachinery"
[[override]]
name = "gopkg.in/square/go-jose.v2"
version = "=2.1.9"
[prune]
go-tests = true
unused-packages = true

@ -1,115 +0,0 @@
package main
import (
"crypto/tls"
"crypto/x509"
"fmt"
"io/ioutil"
"net"
"net/http"
"os"
"strings"
"time"
)
const (
serviceAccountToken = "/var/run/secrets/kubernetes.io/serviceaccount/token"
serviceAccountCACert = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
)
// Client is minimal kubernetes client interface
type Client interface {
Do(req *http.Request) (*http.Response, error)
GetRequest(url string) (*http.Request, error)
PostRequest(url, body, contentType string) (*http.Request, error)
DeleteRequest(url string) (*http.Request, error)
Host() string
}
type k8sClient struct {
host string
token string
httpClient *http.Client
}
func (kc *k8sClient) GetRequest(url string) (*http.Request, error) {
if !strings.HasPrefix(url, kc.host) {
url = fmt.Sprintf("%s/%s", kc.host, url)
}
req, err := http.NewRequest("GET", url, nil)
if err != nil {
return nil, err
}
if len(kc.token) > 0 {
req.Header.Set("Authorization", "Bearer "+kc.token)
}
return req, nil
}
func (kc *k8sClient) PostRequest(url string, body string, contentType string) (*http.Request, error) {
if !strings.HasPrefix(url, kc.host) {
url = fmt.Sprintf("%s/%s", kc.host, url)
}
req, err := http.NewRequest("POST", url, strings.NewReader(body))
if err != nil {
return nil, err
}
if len(kc.token) > 0 {
req.Header.Set("Authorization", "Bearer "+kc.token)
}
if contentType != "" {
req.Header.Set("Content-Type", contentType)
}
return req, nil
}
func (kc *k8sClient) DeleteRequest(url string) (*http.Request, error) {
if !strings.HasPrefix(url, kc.host) {
url = fmt.Sprintf("%s/%s", kc.host, url)
}
req, err := http.NewRequest("DELETE", url, nil)
if err != nil {
return nil, err
}
if len(kc.token) > 0 {
req.Header.Set("Authorization", "Bearer "+kc.token)
}
return req, nil
}
func (kc *k8sClient) Do(req *http.Request) (*http.Response, error) {
return kc.httpClient.Do(req)
}
func (kc *k8sClient) Host() string {
return kc.host
}
// NewInClusterK8sClient creates K8sClient if it is inside Kubernetes
func NewInClusterK8sClient() (Client, error) {
host, port := os.Getenv("KUBERNETES_SERVICE_HOST"), os.Getenv("KUBERNETES_SERVICE_PORT")
if len(host) == 0 || len(port) == 0 {
return nil, fmt.Errorf("unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined")
}
token, err := ioutil.ReadFile(serviceAccountToken)
if err != nil {
return nil, err
}
ca, err := ioutil.ReadFile(serviceAccountCACert)
if err != nil {
return nil, err
}
certPool := x509.NewCertPool()
certPool.AppendCertsFromPEM(ca)
transport := &http.Transport{TLSClientConfig: &tls.Config{
MinVersion: tls.VersionTLS10,
RootCAs: certPool,
}}
httpClient := &http.Client{Transport: transport, Timeout: time.Nanosecond * 0}
return &k8sClient{
host: "https://" + net.JoinHostPort(host, port),
token: string(token),
httpClient: httpClient,
}, nil
}

@ -1,647 +0,0 @@
package main
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"os"
"strings"
"time"
"github.com/ghodss/yaml"
"github.com/pkg/errors"
log "github.com/sirupsen/logrus"
"github.com/smallstep/certificates/ca"
"github.com/smallstep/cli/crypto/pemutil"
"k8s.io/api/admission/v1beta1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/runtime/serializer"
)
var (
runtimeScheme = runtime.NewScheme()
codecs = serializer.NewCodecFactory(runtimeScheme)
deserializer = codecs.UniversalDeserializer()
// GetRootCAPath() is broken; points to secrets not certs. So
// we'll hard code instead for now.
//rootCAPath = pki.GetRootCAPath()
rootCAPath = "/home/step/.step/certs/root_ca.crt"
)
const (
admissionWebhookAnnotationKey = "autocert.step.sm/name"
admissionWebhookStatusKey = "autocert.step.sm/status"
provisionerPasswordFile = "/home/step/password/password"
volumeMountPath = "/var/run/autocert.step.sm"
tokenSecretKey = "token"
tokenSecretLabel = "autocert.step.sm/token"
)
// Config options for the autocert admission controller.
type Config struct {
LogFormat string `yaml:"logFormat"`
CaURL string `yaml:"caUrl"`
CertLifetime string `yaml:"certLifetime"`
Bootstrapper corev1.Container `yaml:"bootstrapper"`
Renewer corev1.Container `yaml:"renewer"`
CertsVolume corev1.Volume `yaml:"certsVolume"`
RestrictCertificatesToNamespace bool `yaml:"restrictCertificatesToNamespace"`
ClusterDomain string `yaml:"clusterDomain"`
}
// GetClusterDomain returns the Kubernetes cluster domain, defaults to
// "cluster.local" if not specified in the configuration.
func (c Config) GetClusterDomain() string {
if c.ClusterDomain != "" {
return c.ClusterDomain
}
return "cluster.local"
}
// PatchOperation represents a RFC6902 JSONPatch Operation
type PatchOperation struct {
Op string `json:"op"`
Path string `json:"path"`
Value interface{} `json:"value,omitempty"`
}
// RFC6901 JSONPath Escaping -- https://tools.ietf.org/html/rfc6901
func escapeJSONPath(path string) string {
// Replace`~` with `~0` then `/` with `~1`. Note that the order
// matters otherwise we'll turn a `/` into a `~/`.
path = strings.Replace(path, "~", "~0", -1)
path = strings.Replace(path, "/", "~1", -1)
return path
}
func loadConfig(file string) (*Config, error) {
data, err := ioutil.ReadFile(file)
if err != nil {
return nil, err
}
var cfg Config
if err := yaml.Unmarshal(data, &cfg); err != nil {
return nil, err
}
return &cfg, nil
}
// createTokenSecret generates a kubernetes Secret object containing a bootstrap token
// in the specified namespce. The secret name is randomly generated with a given prefix.
// A goroutine is scheduled to cleanup the secret after the token expires. The secret
// is also labelled for easy identification and manual cleanup.
func createTokenSecret(prefix, namespace, token string) (string, error) {
secret := corev1.Secret{
TypeMeta: metav1.TypeMeta{
Kind: "Secret",
APIVersion: "v1",
},
ObjectMeta: metav1.ObjectMeta{
GenerateName: prefix,
Namespace: namespace,
Labels: map[string]string{
tokenSecretLabel: "true",
},
},
StringData: map[string]string{
tokenSecretKey: token,
},
Type: corev1.SecretTypeOpaque,
}
client, err := NewInClusterK8sClient()
if err != nil {
return "", err
}
body, err := json.Marshal(secret)
if err != nil {
return "", err
}
log.WithField("secret", string(body)).Debug("Creating secret")
req, err := client.PostRequest(fmt.Sprintf("api/v1/namespaces/%s/secrets", namespace), string(body), "application/json")
if err != nil {
return "", err
}
resp, err := client.Do(req)
if err != nil {
log.Errorf("Secret creation error. Response: %v", resp)
return "", errors.Wrap(err, "secret creation")
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
log.Errorf("Secret creation error (!2XX). Response: %v", resp)
var rbody []byte
if resp.Body != nil {
if data, err := ioutil.ReadAll(resp.Body); err == nil {
rbody = data
}
}
log.Error("Error body: ", string(rbody))
return "", errors.New("Not 200")
}
var rbody []byte
if resp.Body != nil {
if data, err := ioutil.ReadAll(resp.Body); err == nil {
rbody = data
}
}
if len(rbody) == 0 {
return "", errors.New("Empty response body")
}
var created *corev1.Secret
if err := json.Unmarshal(rbody, &created); err != nil {
return "", errors.Wrap(err, "Error unmarshalling secret response")
}
// Clean up after ourselves by deleting the Secret after the bootstrap
// token expires. This is best effort -- obviously we'll miss some stuff
// if this process goes away -- but the secrets are also labelled so
// it's also easy to clean them up in bulk using kubectl if we miss any.
go func() {
time.Sleep(tokenLifetime)
req, err := client.DeleteRequest(fmt.Sprintf("api/v1/namespaces/%s/secrets/%s", namespace, created.Name))
ctxLog := log.WithFields(log.Fields{
"name": created.Name,
"namespace": namespace,
})
if err != nil {
ctxLog.WithField("error", err).Error("Error deleting expired boostrap token secret")
return
}
resp, err := client.Do(req)
if err != nil {
ctxLog.WithField("error", err).Error("Error deleting expired boostrap token secret")
return
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
ctxLog.WithFields(log.Fields{
"status": resp.Status,
"statusCode": resp.StatusCode,
}).Error("Error deleting expired boostrap token secret")
return
}
ctxLog.Info("Deleted expired bootstrap token secret")
}()
return created.Name, err
}
// mkBootstrapper generates a bootstrap container based on the template defined in Config. It
// generates a new bootstrap token and mounts it, along with other required coniguration, as
// environment variables in the returned bootstrap container.
func mkBootstrapper(config *Config, commonName string, namespace string, provisioner Provisioner) (corev1.Container, error) {
b := config.Bootstrapper
token, err := provisioner.Token(commonName)
if err != nil {
return b, errors.Wrap(err, "token generation")
}
// Generate CA fingerprint
crt, err := pemutil.ReadCertificate(rootCAPath)
if err != nil {
return b, errors.Wrap(err, "CA fingerprint")
}
sum := sha256.Sum256(crt.Raw)
fingerprint := strings.ToLower(hex.EncodeToString(sum[:]))
secretName, err := createTokenSecret(commonName+"-", namespace, token)
if err != nil {
return b, errors.Wrap(err, "create token secret")
}
log.Infof("Secret name is: %s", secretName)
b.Env = append(b.Env, corev1.EnvVar{
Name: "COMMON_NAME",
Value: commonName,
})
b.Env = append(b.Env, corev1.EnvVar{
Name: "STEP_TOKEN",
ValueFrom: &corev1.EnvVarSource{
SecretKeyRef: &corev1.SecretKeySelector{
LocalObjectReference: corev1.LocalObjectReference{
Name: secretName,
},
Key: tokenSecretKey,
},
},
})
b.Env = append(b.Env, corev1.EnvVar{
Name: "STEP_CA_URL",
Value: config.CaURL,
})
b.Env = append(b.Env, corev1.EnvVar{
Name: "STEP_FINGERPRINT",
Value: fingerprint,
})
b.Env = append(b.Env, corev1.EnvVar{
Name: "STEP_NOT_AFTER",
Value: config.CertLifetime,
})
return b, nil
}
// mkRenewer generates a new renewer based on the template provided in Config.
func mkRenewer(config *Config) corev1.Container {
r := config.Renewer
r.Env = append(r.Env, corev1.EnvVar{
Name: "STEP_CA_URL",
Value: config.CaURL,
})
return r
}
func addContainers(existing, new []corev1.Container, path string) (ops []PatchOperation) {
if len(existing) == 0 {
return []PatchOperation{
{
Op: "add",
Path: path,
Value: new,
},
}
}
for _, add := range new {
ops = append(ops, PatchOperation{
Op: "add",
Path: path + "/-",
Value: add,
})
}
return ops
}
func addVolumes(existing, new []corev1.Volume, path string) (ops []PatchOperation) {
if len(existing) == 0 {
return []PatchOperation{
{
Op: "add",
Path: path,
Value: new,
},
}
}
for _, add := range new {
ops = append(ops, PatchOperation{
Op: "add",
Path: path + "/-",
Value: add,
})
}
return ops
}
func addCertsVolumeMount(volumeName string, containers []corev1.Container) (ops []PatchOperation) {
volumeMount := corev1.VolumeMount{
Name: volumeName,
MountPath: volumeMountPath,
ReadOnly: true,
}
for i, container := range containers {
if len(container.VolumeMounts) == 0 {
ops = append(ops, PatchOperation{
Op: "add",
Path: fmt.Sprintf("/spec/containers/%v/volumeMounts", i),
Value: []corev1.VolumeMount{volumeMount},
})
} else {
ops = append(ops, PatchOperation{
Op: "add",
Path: fmt.Sprintf("/spec/containers/%v/volumeMounts/-", i),
Value: volumeMount,
})
}
}
return ops
}
func addAnnotations(existing, new map[string]string) (ops []PatchOperation) {
if len(existing) == 0 {
return []PatchOperation{
{
Op: "add",
Path: "/metadata/annotations",
Value: new,
},
}
}
for k, v := range new {
if existing[k] == "" {
ops = append(ops, PatchOperation{
Op: "add",
Path: "/metadata/annotations/" + escapeJSONPath(k),
Value: v,
})
} else {
ops = append(ops, PatchOperation{
Op: "replace",
Path: "/metadata/annotations/" + escapeJSONPath(k),
Value: v,
})
}
}
return ops
}
// patch produces a list of patches to apply to a pod to inject a certificate. In particular,
// we patch the pod in order to:
// - Mount the `certs` volume in existing containers defined in the pod
// - Add the autocert-renewer as a container (a sidecar)
// - Add the autocert-bootstrapper as an initContainer
// - Add the `certs` volume definition
// - Annotate the pod to indicate that it's been processed by this controller
// The result is a list of serialized JSONPatch objects (or an error).
func patch(pod *corev1.Pod, namespace string, config *Config, provisioner Provisioner) ([]byte, error) {
var ops []PatchOperation
annotations := pod.ObjectMeta.GetAnnotations()
commonName := annotations[admissionWebhookAnnotationKey]
renewer := mkRenewer(config)
bootstrapper, err := mkBootstrapper(config, commonName, namespace, provisioner)
if err != nil {
return nil, err
}
ops = append(ops, addCertsVolumeMount(config.CertsVolume.Name, pod.Spec.Containers)...)
ops = append(ops, addContainers(pod.Spec.Containers, []corev1.Container{renewer}, "/spec/containers")...)
ops = append(ops, addContainers(pod.Spec.InitContainers, []corev1.Container{bootstrapper}, "/spec/initContainers")...)
ops = append(ops, addVolumes(pod.Spec.Volumes, []corev1.Volume{config.CertsVolume}, "/spec/volumes")...)
ops = append(ops, addAnnotations(pod.Annotations, map[string]string{admissionWebhookStatusKey: "injected"})...)
return json.Marshal(ops)
}
// shouldMutate checks whether a pod is subject to mutation by this admission controller. A pod
// is subject to mutation if it's annotated with the `admissionWebhookAnnotationKey` and if it
// has not already been processed (indicated by `admissionWebhookStatusKey` set to `injected`).
// If the pod requests a certificate with a subject matching a namespace other than its own
// and restrictToNamespace is true, then shouldMutate will return a validation error
// that should be returned to the client.
func shouldMutate(metadata *metav1.ObjectMeta, namespace string, clusterDomain string, restrictToNamespace bool) (bool, error) {
annotations := metadata.GetAnnotations()
if annotations == nil {
annotations = map[string]string{}
}
// Only mutate if the object is annotated appropriately (annotation key set) and we haven't
// mutated already (status key isn't set).
if annotations[admissionWebhookAnnotationKey] == "" || annotations[admissionWebhookStatusKey] == "injected" {
return false, nil
}
if !restrictToNamespace {
return true, nil
}
subject := strings.Trim(annotations[admissionWebhookAnnotationKey], ".")
err := fmt.Errorf("subject \"%s\" matches a namespace other than \"%s\" and is not permitted. This check can be disabled by setting restrictCertificatesToNamespace to false in the autocert-config ConfigMap", subject, namespace)
if strings.HasSuffix(subject, ".svc") && !strings.HasSuffix(subject, fmt.Sprintf(".%s.svc", namespace)) {
return false, err
}
if strings.HasSuffix(subject, fmt.Sprintf(".svc.%s", clusterDomain)) && !strings.HasSuffix(subject, fmt.Sprintf(".%s.svc.%s", namespace, clusterDomain)) {
return false, err
}
return true, nil
}
// mutate takes an `AdmissionReview`, determines whether it is subject to mutation, and returns
// an appropriate `AdmissionResponse` including patches or any errors that occurred.
func mutate(review *v1beta1.AdmissionReview, config *Config, provisioner Provisioner) *v1beta1.AdmissionResponse {
ctxLog := log.WithField("uid", review.Request.UID)
request := review.Request
var pod corev1.Pod
if err := json.Unmarshal(request.Object.Raw, &pod); err != nil {
ctxLog.WithField("error", err).Error("Error unmarshalling pod")
return &v1beta1.AdmissionResponse{
Allowed: false,
UID: request.UID,
Result: &metav1.Status{
Message: err.Error(),
},
}
}
ctxLog = ctxLog.WithFields(log.Fields{
"kind": request.Kind,
"operation": request.Operation,
"name": pod.Name,
"generateName": pod.GenerateName,
"namespace": request.Namespace,
"user": request.UserInfo,
})
mutationAllowed, validationErr := shouldMutate(&pod.ObjectMeta, request.Namespace, config.GetClusterDomain(), config.RestrictCertificatesToNamespace)
if validationErr != nil {
ctxLog.WithField("error", validationErr).Info("Validation error")
return &v1beta1.AdmissionResponse{
Allowed: false,
UID: request.UID,
Result: &metav1.Status{
Message: validationErr.Error(),
},
}
}
if !mutationAllowed {
ctxLog.WithField("annotations", pod.Annotations).Info("Skipping mutation")
return &v1beta1.AdmissionResponse{
Allowed: true,
UID: request.UID,
}
}
patchBytes, err := patch(&pod, request.Namespace, config, provisioner)
if err != nil {
ctxLog.WithField("error", err).Error("Error generating patch")
return &v1beta1.AdmissionResponse{
Allowed: false,
UID: request.UID,
Result: &metav1.Status{
Message: err.Error(),
},
}
}
ctxLog.WithField("patch", string(patchBytes)).Info("Generated patch")
return &v1beta1.AdmissionResponse{
Allowed: true,
Patch: patchBytes,
UID: request.UID,
PatchType: func() *v1beta1.PatchType {
pt := v1beta1.PatchTypeJSONPatch
return &pt
}(),
}
}
func main() {
if len(os.Args) != 2 {
log.Errorf("Usage: %s <config>\n", os.Args[0])
os.Exit(1)
}
config, err := loadConfig(os.Args[1])
if err != nil {
panic(err)
}
log.SetOutput(os.Stdout)
if config.LogFormat == "json" {
log.SetFormatter(&log.JSONFormatter{})
}
if config.LogFormat == "text" {
log.SetFormatter(&log.TextFormatter{})
}
log.WithFields(log.Fields{
"config": config,
}).Info("Loaded config")
provisionerName := os.Getenv("PROVISIONER_NAME")
provisionerKid := os.Getenv("PROVISIONER_KID")
log.WithFields(log.Fields{
"provisionerName": provisionerName,
"provisionerKid": provisionerKid,
}).Info("Loaded provisioner configuration")
provisioner, err := NewProvisioner(provisionerName, provisionerKid, config.CaURL, rootCAPath, provisionerPasswordFile)
if err != nil {
log.Errorf("Error loading provisioner: %v", err)
os.Exit(1)
}
log.WithFields(log.Fields{
"name": provisioner.Name(),
"kid": provisioner.Kid(),
}).Info("Loaded provisioner")
namespace := os.Getenv("NAMESPACE")
if namespace == "" {
log.Errorf("$NAMESPACE not set")
os.Exit(1)
}
name := fmt.Sprintf("autocert.%s.svc", namespace)
token, err := provisioner.Token(name)
if err != nil {
log.WithField("error", err).Errorf("Error generating bootstrap token during controller startup")
os.Exit(1)
}
log.WithField("name", name).Infof("Generated bootstrap token for controller")
// make sure to cancel the renew goroutine
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
srv, err := ca.BootstrapServer(ctx, token, &http.Server{
Addr: ":4443",
Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/healthz" {
log.Info("/healthz")
fmt.Fprintf(w, "ok")
w.WriteHeader(http.StatusOK)
return
}
/*
var name string
if r.TLS != nil && len(r.TLS.PeerCertificates) > 0 {
name = r.TLS.PeerCertificates[0].Subject.CommonName
}
*/
if r.URL.Path != "/mutate" {
log.WithField("path", r.URL.Path).Error("Bad Request: 404 Not Found")
http.NotFound(w, r)
return
}
var body []byte
if r.Body != nil {
if data, err := ioutil.ReadAll(r.Body); err == nil {
body = data
}
}
if len(body) == 0 {
log.Error("Bad Request: 400 (Empty Body)")
http.Error(w, "Bad Request (Empty Body)", http.StatusBadRequest)
return
}
contentType := r.Header.Get("Content-Type")
if contentType != "application/json" {
log.WithField("Content-Type", contentType).Error("Bad Request: 415 (Unsupported Media Type)")
http.Error(w, fmt.Sprintf("Bad Request: 415 Unsupported Media Type (Expected Content-Type 'application/json' but got '%s')", contentType), http.StatusUnsupportedMediaType)
return
}
var response *v1beta1.AdmissionResponse
review := v1beta1.AdmissionReview{}
if _, _, err := deserializer.Decode(body, nil, &review); err != nil {
log.WithFields(log.Fields{
"body": body,
"error": err,
}).Error("Can't decode body")
response = &v1beta1.AdmissionResponse{
Allowed: false,
Result: &metav1.Status{
Message: err.Error(),
},
}
} else {
response = mutate(&review, config, provisioner)
}
resp, err := json.Marshal(v1beta1.AdmissionReview{
Response: response,
})
if err != nil {
log.WithFields(log.Fields{
"uid": review.Request.UID,
"error": err,
}).Info("Marshal error")
http.Error(w, fmt.Sprintf("Marshal Error: %v", err), http.StatusInternalServerError)
} else {
log.WithFields(log.Fields{
"uid": review.Request.UID,
"response": string(resp),
}).Info("Returning review")
if _, err := w.Write(resp); err != nil {
log.WithFields(log.Fields{
"uid": review.Request.UID,
"error": err,
}).Info("Write error")
}
}
}),
}, ca.VerifyClientCertIfGiven())
if err != nil {
panic(err)
}
log.Info("Listening on :4443 ...")
if err := srv.ListenAndServeTLS("", ""); err != nil {
panic(err)
}
}

@ -1,75 +0,0 @@
package main
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"testing"
)
func TestGetClusterDomain(t *testing.T) {
c := Config{}
if c.GetClusterDomain() != "cluster.local" {
t.Errorf("cluster domain should default to cluster.local, not: %s", c.GetClusterDomain())
}
c.ClusterDomain = "mydomain.com"
if c.GetClusterDomain() != "mydomain.com" {
t.Errorf("cluster domain should default to cluster.local, not: %s", c.GetClusterDomain())
}
}
func TestShouldMutate(t *testing.T) {
testCases := []struct {
description string
subject string
namespace string
expected bool
}{
{"full cluster domain", "test.default.svc.cluster.local", "default", true},
{"full cluster domain wrong ns", "test.default.svc.cluster.local", "kube-system", false},
{"left dots get stripped", ".test.default.svc.cluster.local", "default", true},
{"left dots get stripped wrong ns", ".test.default.svc.cluster.local", "kube-system", false},
{"right dots get stripped", "test.default.svc.cluster.local.", "default", true},
{"right dots get stripped wrong ns", "test.default.svc.cluster.local.", "kube-system", false},
{"dots get stripped", ".test.default.svc.cluster.local.", "default", true},
{"dots get stripped wrong ns", ".test.default.svc.cluster.local.", "kube-system", false},
{"partial cluster domain", "test.default.svc.cluster", "default", true},
{"partial cluster domain wrong ns is still allowed because not valid hostname", "test.default.svc.cluster", "kube-system", true},
{"service domain", "test.default.svc", "default", true},
{"service domain wrong ns", "test.default.svc", "kube-system", false},
{"two part domain", "test.default", "default", true},
{"two part domain different ns", "test.default", "kube-system", true},
{"one hostname", "test", "default", true},
{"no subject specified", "", "default", false},
{"three part not cluster", "test.default.com", "kube-system", true},
{"four part not cluster", "test.default.svc.com", "kube-system", true},
{"five part not cluster", "test.default.svc.cluster.com", "kube-system", true},
{"six part not cluster", "test.default.svc.cluster.local.com", "kube-system", true},
}
for _, testCase := range testCases {
t.Run(testCase.description, func(t *testing.T) {
mutationAllowed, validationErr := shouldMutate(&metav1.ObjectMeta{
Annotations: map[string]string{
admissionWebhookAnnotationKey: testCase.subject,
},
}, testCase.namespace, "cluster.local", true)
if mutationAllowed != testCase.expected {
t.Errorf("shouldMutate did not return %t for %s", testCase.expected, testCase.description)
}
if testCase.subject != "" && mutationAllowed == false && validationErr == nil {
t.Errorf("shouldMutate should return validation error for invalid hostname")
}
})
}
}
func TestShouldMutateNotRestrictToNamespace(t *testing.T) {
mutationAllowed, _ := shouldMutate(&metav1.ObjectMeta{
Annotations: map[string]string{
admissionWebhookAnnotationKey: "test.default.svc.cluster.local",
},
}, "kube-system", "cluster.local", false)
if mutationAllowed == false {
t.Errorf("shouldMutate should return true even with a wrong namespace if restrictToNamespace is false.")
}
}

@ -1,197 +0,0 @@
package main
import (
"encoding/json"
"fmt"
"path/filepath"
"time"
"github.com/pkg/errors"
provisioners "github.com/smallstep/certificates/authority/provisioner"
"github.com/smallstep/certificates/ca"
"github.com/smallstep/cli/config"
"github.com/smallstep/cli/crypto/randutil"
"github.com/smallstep/cli/jose"
"github.com/smallstep/cli/token"
"github.com/smallstep/cli/token/provision"
)
const (
tokenLifetime = 5 * time.Minute
)
// Provisioner is an authorized entity that can sign tokens necessary for
// signature requests.
type Provisioner interface {
Name() string
Kid() string
Token(subject string) (string, error)
}
type provisioner struct {
name string
kid string
caURL string
caRoot string
jwk *jose.JSONWebKey
tokenLifetime time.Duration
}
// Name returns the provisioner's name.
func (p *provisioner) Name() string {
return p.name
}
// Kid returns the provisioners key ID.
func (p *provisioner) Kid() string {
return p.kid
}
// Token generates a bootstrap token for a subject.
func (p *provisioner) Token(subject string) (string, error) {
// A random jwt id will be used to identify duplicated tokens
jwtID, err := randutil.Hex(64) // 256 bits
if err != nil {
return "", err
}
notBefore := time.Now()
notAfter := notBefore.Add(tokenLifetime)
signURL := fmt.Sprintf("%v/1.0/sign", p.caURL)
tokOptions := []token.Options{
token.WithJWTID(jwtID),
token.WithKid(p.kid),
token.WithIssuer(p.name),
token.WithAudience(signURL),
token.WithValidity(notBefore, notAfter),
token.WithRootCA(p.caRoot),
}
tok, err := provision.New(subject, tokOptions...)
if err != nil {
return "", err
}
return tok.SignedString(p.jwk.Algorithm, p.jwk.Key)
}
func decryptProvisionerJWK(encryptedKey, passFile string) (*jose.JSONWebKey, error) {
decrypted, err := jose.Decrypt("", []byte(encryptedKey), jose.WithPasswordFile(passFile))
if err != nil {
return nil, err
}
jwk := new(jose.JSONWebKey)
if err := json.Unmarshal(decrypted, jwk); err != nil {
return nil, errors.Wrap(err, "error unmarshalling provisioning key")
}
return jwk, nil
}
// loadProvisionerJWKByKid retrieves a provisioner key from the CA by key ID and
// decrypts it using the specified password file.
func loadProvisionerJWKByKid(kid, caURL, caRoot, passFile string) (*jose.JSONWebKey, error) {
encrypted, err := getProvisionerKey(caURL, caRoot, kid)
if err != nil {
return nil, err
}
return decryptProvisionerJWK(encrypted, passFile)
}
// loadProvisionerJWKByName retrieves the list of provisioners and encrypted key then
// returns the key of the first provisioner with a matching name that can be successfully
// decrypted with the specified password file.
func loadProvisionerJWKByName(name, caURL, caRoot, passFile string) (key *jose.JSONWebKey, err error) {
provisioners, err := getProvisioners(caURL, caRoot)
if err != nil {
err = errors.Wrap(err, "error getting the provisioners")
return
}
for _, provisioner := range provisioners {
if provisioner.GetName() == name {
if _, encryptedKey, ok := provisioner.GetEncryptedKey(); ok {
key, err = decryptProvisionerJWK(encryptedKey, passFile)
if err == nil {
return
}
}
}
}
return nil, errors.Errorf("provisioner '%s' not found (or your password is wrong)", name)
}
// NewProvisioner loads and decrypts key material from the CA for the named
// provisioner. The key identified by `kid` will be used if specified. If `kid`
// is the empty string we'll use the first key for the named provisioner that
// decrypts using `passFile`.
func NewProvisioner(name, kid, caURL, caRoot, passFile string) (Provisioner, error) {
var jwk *jose.JSONWebKey
var err error
if kid != "" {
jwk, err = loadProvisionerJWKByKid(kid, caURL, caRoot, passFile)
} else {
jwk, err = loadProvisionerJWKByName(name, caURL, caRoot, passFile)
}
if err != nil {
return nil, err
}
return &provisioner{
name: name,
kid: jwk.KeyID,
caURL: caURL,
caRoot: caRoot,
jwk: jwk,
tokenLifetime: tokenLifetime,
}, nil
}
// getRootCAPath returns the path where the root CA is stored based on the
// STEPPATH environment variable.
func getRootCAPath() string {
return filepath.Join(config.StepPath(), "certs", "root_ca.crt")
}
// getProvisioners returns the map of provisioners on the given CA.
func getProvisioners(caURL, rootFile string) (provisioners.List, error) {
if len(rootFile) == 0 {
rootFile = getRootCAPath()
}
client, err := ca.NewClient(caURL, ca.WithRootFile(rootFile))
if err != nil {
return nil, err
}
cursor := ""
var provisioners provisioners.List
for {
resp, err := client.Provisioners(ca.WithProvisionerCursor(cursor), ca.WithProvisionerLimit(100))
if err != nil {
return nil, err
}
provisioners = append(provisioners, resp.Provisioners...)
if resp.NextCursor == "" {
return provisioners, nil
}
cursor = resp.NextCursor
}
}
// getProvisionerKey returns the encrypted provisioner key with the for the
// given kid.
func getProvisionerKey(caURL, rootFile, kid string) (string, error) {
if len(rootFile) == 0 {
rootFile = getRootCAPath()
}
client, err := ca.NewClient(caURL, ca.WithRootFile(rootFile))
if err != nil {
return "", err
}
resp, err := client.ProvisionerKey(kid)
if err != nil {
return "", err
}
return resp.Key, nil
}

Binary file not shown.

Before

Width:  |  Height:  |  Size: 10 MiB

@ -1,131 +0,0 @@
# hello-mtls
This repository contains examples of dockerized [m]TLS clients and servers in
various languages. There's a lot of confusion and misinformation regarding how
to do mTLS properly with an internal public key infrastructure. The goal of
this repository is to demonstrate best practices like:
* Properly configuring TLS to use your internal CA's root certificate
* mTLS (client certificates / client authentication)
* Short-lived certificate support (clients and servers automatically load
renewed certificates)
Examples use multi-stage docker builds and can be built via without any
required local dependencies (except `docker`):
```
docker build -f Dockerfile.server -t hello-mtls-server-<lang> .
docker build -f Dockerfile.client -t hello-mtls-client-<lang> .
```
Once built, you should be able to deploy via:
```
kubectl apply -f hello-mtls.server.yaml
kubectl apply -f hello-mtls.client.yaml
```
## Mutual TLS
Unlike the _server auth TLS_ that's typical with web browsers, where the browser authenticates the server but not vice versa, _mutual TLS_ (mTLS) connections have both remote peers (client and server) authenticate to one another by presenting certificates. mTLS is not a different protocol. It's just a variant of TLS that's not usually turned on by default. This repository demonstrates **how to turn on mTLS** with different tools and languages. It also demonstrates other **TLS best practices** like certificate rotation.
mTLS provides _authenticated encryption_: an _identity dialtone_ and _end-to-end encryption_ for your workloads. It's like a secure line with caller ID. This has [all sorts of benefits](https://smallstep.com/blog/use-tls.html): better security, compliance, and easier auditability for starters. It **makes workloads identity-aware**, improving observability and enabling granular access control. Perhaps most compelling, mTLS lets you securely communicate with workloads running anywhere. Code, containers, devices, people, and anything else can connect securely using mTLS as long as they know one anothers' names and can resolve those names to routable IP addresses.
With properly configured mTLS, services can be safely exposed directly to the public internet: **only clients that have a certificate issued by the internal certificate authority will be allowed to connect**.
Here's a rough approximation of how an mTLS handshake works:
![mTLS handshake diagram](https://raw.githubusercontent.com/smallstep/certificates/autocert/autocert/mtls-handshake.png)
A few things to note:
* It's the signing of random numbers that proves we're talking to the right remote. It's the digital equivalent of asking someone to send you a photo of them with today's newspaper.
* The client and server need to have prior knowledge of the root certificate(s) used for signing other certificates.
* The client and server need to be configured to use the correct certificate and private key (the certificate must have been issued by a CA with a trusted root certificate)
* Private keys are never shared. This is the magic of public key cryptography: unlike passwords or access tokens, certificates let you prove who you are without giving anyone the ability to impersonate you.
## Feature matrix
This matrix shows the set of features we'd like to demonstrate in each language
and where each language is. Bug fixes, improvements, and examples in new
languages are appreciated!
[curl/](curl/)
- [X] Client
- [X] mTLS (send client certificate if server asks for it)
- [X] Automatic certificate rotation
- [ ] Restrict to safe ciphersuites and TLS versions
- [ ] TLS stack configuration loaded from `step-ca`
- [ ] Root certificate rotation
[nginx/](nginx/)
- [X] Server
- [X] mTLS (client authentication using internal root certificate)
- [X] Automatic certificate renewal
- [X] Restrict to safe ciphersuites and TLS versions
- [ ] TLS stack configuration loaded from `step-ca`
- [ ] Root certificate rotation
[envoy/](envoy/)
- [X] Server
- [X] mTLS (client authentication using internal root certificate)
- [X] Automatic certificate renewal
- [X] Restrict to safe ciphersuites and TLS versions
- [ ] TLS stack configuration loaded from `step-ca`
- [ ] Root certificate rotation
[go/](go/)
- [X] Server using autocert certificate & key
- [X] mTLS (client authentication using internal root certificate)
- [X] Automatic certificate renewal
- [X] Restrict to safe ciphersuites and TLS versions
- [ ] TLS stack configuration loaded from `step-ca`
- [ ] Root certificate rotation
- [X] Client using autocert root certificate
- [X] mTLS (send client certificate if server asks for it)
- [X] Automatic certificate rotation
- [X] Restrict to safe ciphersuites and TLS versions
- [ ] TLS stack configuration loaded from `step-ca`
- [ ] Root certificate rotation
[go-grpc/](go-grpc/)
- [X] Server using autocert certificate & key
- [X] mTLS (client authentication using internal root certificate)
- [X] Automatic certificate renewal
- [X] Restrict to safe ciphersuites and TLS versions
- [ ] TLS stack configuration loaded from `step-ca`
- [ ] Root certificate rotation
- [X] Client using autocert root certificate
- [X] mTLS (send client certificate if server asks for it)
- [X] Automatic certificate rotation
- [X] Restrict to safe ciphersuites and TLS versions
- [ ] TLS stack configuration loaded from `step-ca`
- [ ] Root certificate rotation
[node/](node/)
- [X] Server
- [X] mTLS (client authentication using internal root certificate)
- [X] Automatic certificate renewal
- [X] Restrict to safe ciphersuites and TLS versions
- [ ] TLS stack configuration loaded from `step-ca`
- [ ] Root certificate rotation
- [X] Client using autocert root certificate
- [X] mTLS (send client certificate if server asks for it)
- [X] Automatic certificate rotation
- [X] Restrict to safe ciphersuites and TLS versions
- [ ] TLS stack configuration loaded from `step-ca`
- [ ] Root certificate rotation
[py-gunicorn/](py-gunicorn/)
- [X] Server (gunicorn + Flask)
- [X] mTLS (client authentication using internal root certificate)
- [X] Automatic certificate renewal
- [X] Restrict to safe ciphersuites and TLS versions
- [ ] TLS stack configuration loaded from `step-ca`
- [ ] Root certificate rotation
- [X] Client using autocert root certificate (python)
- [X] mTLS (send client certificate if server asks for it)
- [X] Automatic certificate rotation
- [X] Restrict to safe ciphersuites and TLS versions
- [ ] TLS stack configuration loaded from `step-ca`
- [ ] Root certificate rotation

@ -1,5 +0,0 @@
FROM alpine
RUN apk add --no-cache bash curl
COPY client.sh .
RUN chmod +x client.sh
ENTRYPOINT ./client.sh

@ -1,11 +0,0 @@
#!/bin/bash
while :
do
response=$(curl -sS \
--cacert /var/run/autocert.step.sm/root.crt \
--cert /var/run/autocert.step.sm/site.crt \
--key /var/run/autocert.step.sm/site.key \
${HELLO_MTLS_URL})
echo "$(date): ${response}"
sleep 5
done

@ -1,22 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-mtls-client
labels: {app: hello-mtls-client}
spec:
replicas: 1
selector: {matchLabels: {app: hello-mtls-client}}
template:
metadata:
annotations:
autocert.step.sm/name: hello-mtls-client.default.pod.cluster.local
labels: {app: hello-mtls-client}
spec:
containers:
- name: hello-mtls-client
image: hello-mtls-client-curl:latest
imagePullPolicy: Never
resources: {requests: {cpu: 10m, memory: 20Mi}}
env:
- name: HELLO_MTLS_URL
value: https://hello-mtls.default.svc.cluster.local

@ -1,21 +0,0 @@
FROM envoyproxy/envoy-alpine
RUN apk update
RUN apk add python3
RUN apk add inotify-tools
RUN mkdir /src
ADD entrypoint.sh /src
ADD certwatch.sh /src
ADD hot-restarter.py /src
ADD start-envoy.sh /src
ADD server.yaml /src
# Flask app
ADD server.py /src
ADD requirements.txt /src
RUN pip3 install -r /src/requirements.txt
# app, certificate watcher and envoy
ENTRYPOINT ["/src/entrypoint.sh"]
CMD ["python3", "/src/hot-restarter.py", "/src/start-envoy.sh"]

@ -1,6 +0,0 @@
#!/bin/sh
while true; do
inotifywait -e modify /var/run/autocert.step.sm/site.crt
kill -HUP 1
done

@ -1,10 +0,0 @@
#!/bin/sh
# start hello world app
python3 /src/server.py &
# watch for the update of the cert and reload nginx
/src/certwatch.sh &
# Run docker CMD
exec "$@"

@ -1,33 +0,0 @@
apiVersion: v1
kind: Service
metadata:
labels: {app: hello-mtls}
name: hello-mtls
spec:
type: ClusterIP
ports:
- port: 443
targetPort: 443
selector: {app: hello-mtls}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-mtls
labels: {app: hello-mtls}
spec:
replicas: 1
selector: {matchLabels: {app: hello-mtls}}
template:
metadata:
annotations:
autocert.step.sm/name: hello-mtls.default.svc.cluster.local
labels: {app: hello-mtls}
spec:
containers:
- name: hello-mtls
image: hello-mtls-server-envoy:latest
imagePullPolicy: Never
resources: {requests: {cpu: 10m, memory: 20Mi}}

@ -1,209 +0,0 @@
#!/usr/bin/env python
from __future__ import print_function
import os
import signal
import sys
import time
# The number of seconds to wait for children to gracefully exit after
# propagating SIGTERM before force killing children.
# NOTE: If using a shutdown mechanism such as runit's `force-stop` which sends
# a KILL after a specified timeout period, it's important to ensure that this
# constant is smaller than the KILL timeout
TERM_WAIT_SECONDS = 30
restart_epoch = 0
pid_list = []
def term_all_children():
""" Iterate through all known child processes, send a TERM signal to each of
them, and then wait up to TERM_WAIT_SECONDS for them to exit gracefully,
exiting early if all children go away. If one or more children have not
exited after TERM_WAIT_SECONDS, they will be forcibly killed """
# First uninstall the SIGCHLD handler so that we don't get called again.
signal.signal(signal.SIGCHLD, signal.SIG_DFL)
global pid_list
for pid in pid_list:
print("sending TERM to PID={}".format(pid))
try:
os.kill(pid, signal.SIGTERM)
except OSError:
print("error sending TERM to PID={} continuing".format(pid))
all_exited = False
# wait for TERM_WAIT_SECONDS seconds for children to exit cleanly
retries = 0
while not all_exited and retries < TERM_WAIT_SECONDS:
for pid in list(pid_list):
ret_pid, exit_status = os.waitpid(pid, os.WNOHANG)
if ret_pid == 0 and exit_status == 0:
# the child is still running
continue
pid_list.remove(pid)
if len(pid_list) == 0:
all_exited = True
else:
retries += 1
time.sleep(1)
if all_exited:
print("all children exited cleanly")
else:
for pid in pid_list:
print("child PID={} did not exit cleanly, killing".format(pid))
force_kill_all_children()
sys.exit(1) # error status because a child did not exit cleanly
def force_kill_all_children():
""" Iterate through all known child processes and force kill them. Typically
term_all_children() should be attempted first to give child processes an
opportunity to clean up state before exiting """
global pid_list
for pid in pid_list:
print("force killing PID={}".format(pid))
try:
os.kill(pid, signal.SIGKILL)
except OSError:
print("error force killing PID={} continuing".format(pid))
pid_list = []
def shutdown():
""" Attempt to gracefully shutdown all child Envoy processes and then exit.
See term_all_children() for further discussion. """
term_all_children()
sys.exit(0)
def sigterm_handler(signum, frame):
""" Handler for SIGTERM. """
print("got SIGTERM")
shutdown()
def sigint_handler(signum, frame):
""" Handler for SIGINT (ctrl-c). The same as the SIGTERM handler. """
print("got SIGINT")
shutdown()
def sighup_handler(signum, frame):
""" Handler for SIGUP. This signal is used to cause the restarter to fork and exec a new
child. """
print("got SIGHUP")
fork_and_exec()
def sigusr1_handler(signum, frame):
""" Handler for SIGUSR1. Propagate SIGUSR1 to all of the child processes """
global pid_list
for pid in pid_list:
print("sending SIGUSR1 to PID={}".format(pid))
try:
os.kill(pid, signal.SIGUSR1)
except OSError:
print("error in SIGUSR1 to PID={} continuing".format(pid))
def sigchld_handler(signum, frame):
""" Handler for SIGCHLD. Iterates through all of our known child processes and figures out whether
the signal/exit was expected or not. Python doesn't have any of the native signal handlers
ability to get the child process info directly from the signal handler so we need to iterate
through all child processes and see what happened."""
print("got SIGCHLD")
kill_all_and_exit = False
global pid_list
pid_list_copy = list(pid_list)
for pid in pid_list_copy:
ret_pid, exit_status = os.waitpid(pid, os.WNOHANG)
if ret_pid == 0 and exit_status == 0:
# This child is still running.
continue
pid_list.remove(pid)
# Now we see how the child exited.
if os.WIFEXITED(exit_status):
exit_code = os.WEXITSTATUS(exit_status)
print("PID={} exited with code={}".format(ret_pid, exit_code))
if exit_code == 0:
# Normal exit. We assume this was on purpose.
pass
else:
# Something bad happened. We need to tear everything down so that whoever started the
# restarter can know about this situation and restart the whole thing.
kill_all_and_exit = True
elif os.WIFSIGNALED(exit_status):
print("PID={} was killed with signal={}".format(ret_pid, os.WTERMSIG(exit_status)))
kill_all_and_exit = True
else:
kill_all_and_exit = True
if kill_all_and_exit:
print("Due to abnormal exit, force killing all child processes and exiting")
# First uninstall the SIGCHLD handler so that we don't get called again.
signal.signal(signal.SIGCHLD, signal.SIG_DFL)
force_kill_all_children()
# Our last child died, so we have no purpose. Exit.
if not pid_list:
print("exiting due to lack of child processes")
sys.exit(1 if kill_all_and_exit else 0)
def fork_and_exec():
""" This routine forks and execs a new child process and keeps track of its PID. Before we fork,
set the current restart epoch in an env variable that processes can read if they care. """
global restart_epoch
os.environ['RESTART_EPOCH'] = str(restart_epoch)
print("forking and execing new child process at epoch {}".format(restart_epoch))
restart_epoch += 1
child_pid = os.fork()
if child_pid == 0:
# Child process
os.execl(sys.argv[1], sys.argv[1])
else:
# Parent process
print("forked new child process with PID={}".format(child_pid))
pid_list.append(child_pid)
def main():
""" Script main. This script is designed so that a process watcher like runit or monit can watch
this process and take corrective action if it ever goes away. """
print("starting hot-restarter with target: {}".format(sys.argv[1]))
signal.signal(signal.SIGTERM, sigterm_handler)
signal.signal(signal.SIGINT, sigint_handler)
signal.signal(signal.SIGHUP, sighup_handler)
signal.signal(signal.SIGCHLD, sigchld_handler)
signal.signal(signal.SIGUSR1, sigusr1_handler)
# Start the first child process and then go into an endless loop since everything else happens via
# signals.
fork_and_exec()
while True:
time.sleep(60)
if __name__ == '__main__':
main()

@ -1,9 +0,0 @@
from flask import Flask
app = Flask(__name__)
@app.route("/")
def hello():
return "Hello World!\n"
if __name__ == "__main__":
app.run(host='127.0.0.1', port=8080, debug=False)

@ -1,50 +0,0 @@
static_resources:
listeners:
- address:
socket_address:
address: 0.0.0.0
port_value: 443
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
codec_type: auto
stat_prefix: ingress_http
route_config:
name: hello
virtual_hosts:
- name: hello
domains:
- "hello-mtls.default.svc.cluster.local"
routes:
- match:
prefix: "/"
route:
cluster: hello-mTLS
http_filters:
- name: envoy.router
config: {}
tls_context:
common_tls_context:
tls_params:
tls_minimum_protocol_version: TLSv1_2
tls_maximum_protocol_version: TLSv1_3
cipher_suites: "[ECDHE-ECDSA-AES128-GCM-SHA256|ECDHE-ECDSA-CHACHA20-POLY1305]"
tls_certificates:
- certificate_chain:
filename: "/var/run/autocert.step.sm/site.crt"
private_key:
filename: "/var/run/autocert.step.sm/site.key"
validation_context:
trusted_ca:
filename: "/var/run/autocert.step.sm/root.crt"
require_client_certificate: true
clusters:
- name: hello-mTLS
connect_timeout: 0.25s
type: strict_dns
lb_policy: round_robin
hosts:
- socket_address:
address: 127.0.0.1
port_value: 8080

@ -1,4 +0,0 @@
#!/bin/sh
ulimit -n 65536
/usr/local/bin/envoy -c /src/server.yaml --service-cluster hello-mTLS --restart-epoch $RESTART_EPOCH

@ -1,16 +0,0 @@
# build stage
FROM golang:alpine AS build-env
RUN apk update
RUN apk add git
RUN mkdir /src
WORKDIR /go/src/github.com/smallstep/certificates/autocert/examples/hello-mtls/go-grpc
ADD client/client.go .
COPY hello hello
RUN go get -d -v ./...
RUN go build -o client
# final stage
FROM alpine
COPY --from=build-env /go/src/github.com/smallstep/certificates/autocert/examples/hello-mtls/go-grpc/client .
CMD ["./client"]

@ -1,164 +0,0 @@
package main
import (
"context"
"crypto/tls"
"crypto/x509"
"errors"
"fmt"
"io/ioutil"
"log"
"os"
"sync"
"time"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials"
"github.com/smallstep/certificates/autocert/examples/hello-mtls/go-grpc/hello"
)
const (
autocertFile = "/var/run/autocert.step.sm/site.crt"
autocertKey = "/var/run/autocert.step.sm/site.key"
autocertRoot = "/var/run/autocert.step.sm/root.crt"
requestFrequency = 5 * time.Second
tickFrequency = 15 * time.Second
)
// Uses techniques from https://diogomonica.com/2017/01/11/hitless-tls-certificate-rotation-in-go/
// to automatically rotate certificates when they're renewed.
type rotator struct {
sync.RWMutex
certificate *tls.Certificate
}
func (r *rotator) getClientCertificate(cri *tls.CertificateRequestInfo) (*tls.Certificate, error) {
r.RLock()
defer r.RUnlock()
return r.certificate, nil
}
func (r *rotator) loadCertificate(certFile, keyFile string) error {
r.Lock()
defer r.Unlock()
c, err := tls.LoadX509KeyPair(certFile, keyFile)
if err != nil {
return err
}
r.certificate = &c
return nil
}
func loadRootCertPool() (*x509.CertPool, error) {
root, err := ioutil.ReadFile(autocertRoot)
if err != nil {
return nil, err
}
pool := x509.NewCertPool()
if ok := pool.AppendCertsFromPEM(root); !ok {
return nil, errors.New("Missing or invalid root certificate")
}
return pool, nil
}
func sayHello(c hello.GreeterClient) error {
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
r, err := c.SayHello(ctx, &hello.HelloRequest{Name: "world"})
if err != nil {
return err
}
log.Printf("Greeting: %s", r.Message)
return nil
}
func sayHelloAgain(c hello.GreeterClient) error {
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
r, err := c.SayHelloAgain(ctx, &hello.HelloRequest{Name: "world"})
if err != nil {
return err
}
log.Printf("Greeting: %s", r.Message)
return nil
}
func main() {
// Read the root certificate for our CA from disk
roots, err := loadRootCertPool()
if err != nil {
log.Fatal(err)
}
// Load certificate
r := &rotator{}
if err := r.loadCertificate(autocertFile, autocertKey); err != nil {
log.Fatal("error loading certificate and key", err)
}
tlsConfig := &tls.Config{
RootCAs: roots,
MinVersion: tls.VersionTLS12,
CurvePreferences: []tls.CurveID{tls.CurveP521, tls.CurveP384, tls.CurveP256},
CipherSuites: []uint16{
tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,
tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
},
// GetClientCertificate is called when a server requests a
// certificate from a client.
//
// In this example keep alives will cause the certificate to
// only be called once, but if we disable them,
// GetClientCertificate will be called on every request.
GetClientCertificate: r.getClientCertificate,
}
// Schedule periodic re-load of certificate
// A real implementation can use something like
// https://github.com/fsnotify/fsnotify
done := make(chan struct{})
go func() {
ticker := time.NewTicker(tickFrequency)
defer ticker.Stop()
for {
select {
case <-ticker.C:
fmt.Println("Checking for new certificate...")
err := r.loadCertificate(autocertFile, autocertKey)
if err != nil {
log.Println("Error loading certificate and key", err)
}
case <-done:
return
}
}
}()
defer close(done)
// Set up a connection to the server.
address := os.Getenv("HELLO_MTLS_URL")
conn, err := grpc.Dial(address, grpc.WithTransportCredentials(credentials.NewTLS(tlsConfig)))
if err != nil {
log.Fatalf("did not connect: %v", err)
}
defer conn.Close()
client := hello.NewGreeterClient(conn)
for {
if err := sayHello(client); err != nil {
log.Fatalf("could not greet: %v", err)
}
if err := sayHelloAgain(client); err != nil {
log.Fatalf("could not greet: %v", err)
}
time.Sleep(requestFrequency)
}
}

@ -1,22 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-mtls-client
labels: {app: hello-mtls-client}
spec:
replicas: 1
selector: {matchLabels: {app: hello-mtls-client}}
template:
metadata:
annotations:
autocert.step.sm/name: hello-mtls-client.default.pod.cluster.local
labels: {app: hello-mtls-client}
spec:
containers:
- name: hello-mtls-client
image: hello-mtls-client-go-grpc:latest
imagePullPolicy: Never
resources: {requests: {cpu: 10m, memory: 20Mi}}
env:
- name: HELLO_MTLS_URL
value: hello-mtls.default.svc.cluster.local:443

@ -1,231 +0,0 @@
// Code generated by protoc-gen-go. DO NOT EDIT.
// source: hello.proto
package hello
import proto "github.com/golang/protobuf/proto"
import fmt "fmt"
import math "math"
import (
context "golang.org/x/net/context"
grpc "google.golang.org/grpc"
)
// Reference imports to suppress errors if they are not otherwise used.
var _ = proto.Marshal
var _ = fmt.Errorf
var _ = math.Inf
// This is a compile-time assertion to ensure that this generated file
// is compatible with the proto package it is being compiled against.
// A compilation error at this line likely means your copy of the
// proto package needs to be updated.
const _ = proto.ProtoPackageIsVersion2 // please upgrade the proto package
// The request message containing the user's name.
type HelloRequest struct {
Name string `protobuf:"bytes,1,opt,name=name" json:"name,omitempty"`
XXX_NoUnkeyedLiteral struct{} `json:"-"`
XXX_unrecognized []byte `json:"-"`
XXX_sizecache int32 `json:"-"`
}
func (m *HelloRequest) Reset() { *m = HelloRequest{} }
func (m *HelloRequest) String() string { return proto.CompactTextString(m) }
func (*HelloRequest) ProtoMessage() {}
func (*HelloRequest) Descriptor() ([]byte, []int) {
return fileDescriptor_hello_4c93420831fe68fb, []int{0}
}
func (m *HelloRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_HelloRequest.Unmarshal(m, b)
}
func (m *HelloRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
return xxx_messageInfo_HelloRequest.Marshal(b, m, deterministic)
}
func (dst *HelloRequest) XXX_Merge(src proto.Message) {
xxx_messageInfo_HelloRequest.Merge(dst, src)
}
func (m *HelloRequest) XXX_Size() int {
return xxx_messageInfo_HelloRequest.Size(m)
}
func (m *HelloRequest) XXX_DiscardUnknown() {
xxx_messageInfo_HelloRequest.DiscardUnknown(m)
}
var xxx_messageInfo_HelloRequest proto.InternalMessageInfo
func (m *HelloRequest) GetName() string {
if m != nil {
return m.Name
}
return ""
}
// The response message containing the greetings
type HelloReply struct {
Message string `protobuf:"bytes,1,opt,name=message" json:"message,omitempty"`
XXX_NoUnkeyedLiteral struct{} `json:"-"`
XXX_unrecognized []byte `json:"-"`
XXX_sizecache int32 `json:"-"`
}
func (m *HelloReply) Reset() { *m = HelloReply{} }
func (m *HelloReply) String() string { return proto.CompactTextString(m) }
func (*HelloReply) ProtoMessage() {}
func (*HelloReply) Descriptor() ([]byte, []int) {
return fileDescriptor_hello_4c93420831fe68fb, []int{1}
}
func (m *HelloReply) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_HelloReply.Unmarshal(m, b)
}
func (m *HelloReply) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
return xxx_messageInfo_HelloReply.Marshal(b, m, deterministic)
}
func (dst *HelloReply) XXX_Merge(src proto.Message) {
xxx_messageInfo_HelloReply.Merge(dst, src)
}
func (m *HelloReply) XXX_Size() int {
return xxx_messageInfo_HelloReply.Size(m)
}
func (m *HelloReply) XXX_DiscardUnknown() {
xxx_messageInfo_HelloReply.DiscardUnknown(m)
}
var xxx_messageInfo_HelloReply proto.InternalMessageInfo
func (m *HelloReply) GetMessage() string {
if m != nil {
return m.Message
}
return ""
}
func init() {
proto.RegisterType((*HelloRequest)(nil), "HelloRequest")
proto.RegisterType((*HelloReply)(nil), "HelloReply")
}
// Reference imports to suppress errors if they are not otherwise used.
var _ context.Context
var _ grpc.ClientConn
// This is a compile-time assertion to ensure that this generated file
// is compatible with the grpc package it is being compiled against.
const _ = grpc.SupportPackageIsVersion4
// Client API for Greeter service
type GreeterClient interface {
// Sends a greeting
SayHello(ctx context.Context, in *HelloRequest, opts ...grpc.CallOption) (*HelloReply, error)
// Sends another greeting
SayHelloAgain(ctx context.Context, in *HelloRequest, opts ...grpc.CallOption) (*HelloReply, error)
}
type greeterClient struct {
cc *grpc.ClientConn
}
func NewGreeterClient(cc *grpc.ClientConn) GreeterClient {
return &greeterClient{cc}
}
func (c *greeterClient) SayHello(ctx context.Context, in *HelloRequest, opts ...grpc.CallOption) (*HelloReply, error) {
out := new(HelloReply)
err := grpc.Invoke(ctx, "/Greeter/SayHello", in, out, c.cc, opts...)
if err != nil {
return nil, err
}
return out, nil
}
func (c *greeterClient) SayHelloAgain(ctx context.Context, in *HelloRequest, opts ...grpc.CallOption) (*HelloReply, error) {
out := new(HelloReply)
err := grpc.Invoke(ctx, "/Greeter/SayHelloAgain", in, out, c.cc, opts...)
if err != nil {
return nil, err
}
return out, nil
}
// Server API for Greeter service
type GreeterServer interface {
// Sends a greeting
SayHello(context.Context, *HelloRequest) (*HelloReply, error)
// Sends another greeting
SayHelloAgain(context.Context, *HelloRequest) (*HelloReply, error)
}
func RegisterGreeterServer(s *grpc.Server, srv GreeterServer) {
s.RegisterService(&_Greeter_serviceDesc, srv)
}
func _Greeter_SayHello_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
in := new(HelloRequest)
if err := dec(in); err != nil {
return nil, err
}
if interceptor == nil {
return srv.(GreeterServer).SayHello(ctx, in)
}
info := &grpc.UnaryServerInfo{
Server: srv,
FullMethod: "/Greeter/SayHello",
}
handler := func(ctx context.Context, req interface{}) (interface{}, error) {
return srv.(GreeterServer).SayHello(ctx, req.(*HelloRequest))
}
return interceptor(ctx, in, info, handler)
}
func _Greeter_SayHelloAgain_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
in := new(HelloRequest)
if err := dec(in); err != nil {
return nil, err
}
if interceptor == nil {
return srv.(GreeterServer).SayHelloAgain(ctx, in)
}
info := &grpc.UnaryServerInfo{
Server: srv,
FullMethod: "/Greeter/SayHelloAgain",
}
handler := func(ctx context.Context, req interface{}) (interface{}, error) {
return srv.(GreeterServer).SayHelloAgain(ctx, req.(*HelloRequest))
}
return interceptor(ctx, in, info, handler)
}
var _Greeter_serviceDesc = grpc.ServiceDesc{
ServiceName: "Greeter",
HandlerType: (*GreeterServer)(nil),
Methods: []grpc.MethodDesc{
{
MethodName: "SayHello",
Handler: _Greeter_SayHello_Handler,
},
{
MethodName: "SayHelloAgain",
Handler: _Greeter_SayHelloAgain_Handler,
},
},
Streams: []grpc.StreamDesc{},
Metadata: "hello.proto",
}
func init() { proto.RegisterFile("hello.proto", fileDescriptor_hello_4c93420831fe68fb) }
var fileDescriptor_hello_4c93420831fe68fb = []byte{
// 141 bytes of a gzipped FileDescriptorProto
0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xe2, 0xe2, 0xce, 0x48, 0xcd, 0xc9,
0xc9, 0xd7, 0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x57, 0x52, 0xe2, 0xe2, 0xf1, 0x00, 0x71, 0x83, 0x52,
0x0b, 0x4b, 0x53, 0x8b, 0x4b, 0x84, 0x84, 0xb8, 0x58, 0xf2, 0x12, 0x73, 0x53, 0x25, 0x18, 0x15,
0x18, 0x35, 0x38, 0x83, 0xc0, 0x6c, 0x25, 0x35, 0x2e, 0x2e, 0xa8, 0x9a, 0x82, 0x9c, 0x4a, 0x21,
0x09, 0x2e, 0xf6, 0xdc, 0xd4, 0xe2, 0xe2, 0xc4, 0x74, 0x98, 0x22, 0x18, 0xd7, 0x28, 0x89, 0x8b,
0xdd, 0xbd, 0x28, 0x35, 0xb5, 0x24, 0xb5, 0x48, 0x48, 0x83, 0x8b, 0x23, 0x38, 0xb1, 0x12, 0xac,
0x4b, 0x88, 0x57, 0x0f, 0xd9, 0x06, 0x29, 0x6e, 0x3d, 0x84, 0x61, 0x4a, 0x0c, 0x42, 0xba, 0x5c,
0xbc, 0x30, 0x95, 0x8e, 0xe9, 0x89, 0x99, 0x79, 0xf8, 0x95, 0x27, 0xb1, 0x81, 0x9d, 0x6d, 0x0c,
0x08, 0x00, 0x00, 0xff, 0xff, 0xa6, 0x84, 0x2d, 0xb6, 0xc5, 0x00, 0x00, 0x00,
}

@ -1,19 +0,0 @@
syntax = "proto3";
// The greeting service definition.
service Greeter {
// Sends a greeting
rpc SayHello (HelloRequest) returns (HelloReply) {}
// Sends another greeting
rpc SayHelloAgain (HelloRequest) returns (HelloReply) {}
}
// The request message containing the user's name.
message HelloRequest {
string name = 1;
}
// The response message containing the greetings
message HelloReply {
string message = 1;
}

@ -1,15 +0,0 @@
# build stage
FROM golang:alpine AS build-env
RUN apk update
RUN apk add git
WORKDIR /go/src/github.com/smallstep/certificates/autocert/examples/hello-mtls/go-grpc
ADD server/server.go .
COPY hello hello
RUN go get -d -v ./...
RUN go build -o server
# final stage
FROM alpine
COPY --from=build-env /go/src/github.com/smallstep/certificates/autocert/examples/hello-mtls/go-grpc/server .
CMD ["./server"]

@ -1,33 +0,0 @@
apiVersion: v1
kind: Service
metadata:
labels: {app: hello-mtls}
name: hello-mtls
spec:
type: ClusterIP
ports:
- port: 443
targetPort: 443
selector: {app: hello-mtls}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-mtls
labels: {app: hello-mtls}
spec:
replicas: 1
selector: {matchLabels: {app: hello-mtls}}
template:
metadata:
annotations:
autocert.step.sm/name: hello-mtls.default.svc.cluster.local
labels: {app: hello-mtls}
spec:
containers:
- name: hello-mtls
image: hello-mtls-server-go-grpc:latest
imagePullPolicy: Never
resources: {requests: {cpu: 10m, memory: 20Mi}}

@ -1,151 +0,0 @@
package main
import (
"context"
"crypto/tls"
"crypto/x509"
"errors"
"fmt"
"io/ioutil"
"log"
"net"
"sync"
"time"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials"
"google.golang.org/grpc/peer"
"github.com/smallstep/certificates/autocert/examples/hello-mtls/go-grpc/hello"
)
const (
autocertFile = "/var/run/autocert.step.sm/site.crt"
autocertKey = "/var/run/autocert.step.sm/site.key"
autocertRoot = "/var/run/autocert.step.sm/root.crt"
tickFrequency = 15 * time.Second
)
// Uses techniques from https://diogomonica.com/2017/01/11/hitless-tls-certificate-rotation-in-go/
// to automatically rotate certificates when they're renewed.
type rotator struct {
sync.RWMutex
certificate *tls.Certificate
}
func (r *rotator) getCertificate(hello *tls.ClientHelloInfo) (*tls.Certificate, error) {
r.RLock()
defer r.RUnlock()
return r.certificate, nil
}
func (r *rotator) loadCertificate(certFile, keyFile string) error {
r.Lock()
defer r.Unlock()
c, err := tls.LoadX509KeyPair(certFile, keyFile)
if err != nil {
return err
}
r.certificate = &c
return nil
}
func loadRootCertPool() (*x509.CertPool, error) {
root, err := ioutil.ReadFile(autocertRoot)
if err != nil {
return nil, err
}
pool := x509.NewCertPool()
if ok := pool.AppendCertsFromPEM(root); !ok {
return nil, errors.New("Missing or invalid root certificate")
}
return pool, nil
}
// Greeter is a service that sends greetings.
type Greeter struct{}
// SayHello sends a greeting
func (g *Greeter) SayHello(ctx context.Context, in *hello.HelloRequest) (*hello.HelloReply, error) {
return &hello.HelloReply{Message: "Hello " + in.Name + " (" + getServerName(ctx) + ")"}, nil
}
// SayHelloAgain sends another greeting
func (g *Greeter) SayHelloAgain(ctx context.Context, in *hello.HelloRequest) (*hello.HelloReply, error) {
return &hello.HelloReply{Message: "Hello again " + in.Name + " (" + getServerName(ctx) + ")"}, nil
}
func getServerName(ctx context.Context) string {
if p, ok := peer.FromContext(ctx); ok {
if tlsInfo, ok := p.AuthInfo.(credentials.TLSInfo); ok {
return tlsInfo.State.ServerName
}
}
return "unknown"
}
func main() {
roots, err := loadRootCertPool()
if err != nil {
log.Fatal(err)
}
// Load certificate
r := &rotator{}
if err := r.loadCertificate(autocertFile, autocertKey); err != nil {
log.Fatal("error loading certificate and key", err)
}
tlsConfig := &tls.Config{
ClientAuth: tls.RequireAndVerifyClientCert,
ClientCAs: roots,
MinVersion: tls.VersionTLS12,
CurvePreferences: []tls.CurveID{tls.CurveP521, tls.CurveP384, tls.CurveP256},
PreferServerCipherSuites: true,
CipherSuites: []uint16{
tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,
tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
},
GetCertificate: r.getCertificate,
}
// Schedule periodic re-load of certificate
// A real implementation can use something like
// https://github.com/fsnotify/fsnotify
done := make(chan struct{})
go func() {
ticker := time.NewTicker(tickFrequency)
defer ticker.Stop()
for {
select {
case <-ticker.C:
fmt.Println("Checking for new certificate...")
err := r.loadCertificate(autocertFile, autocertKey)
if err != nil {
log.Println("Error loading certificate and key", err)
}
case <-done:
return
}
}
}()
defer close(done)
lis, err := net.Listen("tcp", ":443")
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
srv := grpc.NewServer(grpc.Creds(credentials.NewTLS(tlsConfig)))
hello.RegisterGreeterServer(srv, &Greeter{})
log.Println("Listening on :443")
if err := srv.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
}

@ -1,10 +0,0 @@
# build stage
FROM golang:alpine AS build-env
RUN mkdir /src
ADD client.go /src
RUN cd /src && go build -o client
# final stage
FROM alpine
COPY --from=build-env /src/client .
ENTRYPOINT ./client

@ -1,142 +0,0 @@
package main
import (
"crypto/tls"
"crypto/x509"
"errors"
"fmt"
"io/ioutil"
"log"
"net/http"
"os"
"strings"
"sync"
"time"
)
const (
autocertFile = "/var/run/autocert.step.sm/site.crt"
autocertKey = "/var/run/autocert.step.sm/site.key"
autocertRoot = "/var/run/autocert.step.sm/root.crt"
requestFrequency = 5 * time.Second
tickFrequency = 15 * time.Second
)
type rotator struct {
sync.RWMutex
certificate *tls.Certificate
}
func (r *rotator) getClientCertificate(cri *tls.CertificateRequestInfo) (*tls.Certificate, error) {
r.RLock()
defer r.RUnlock()
return r.certificate, nil
}
func (r *rotator) loadCertificate(certFile, keyFile string) error {
r.Lock()
defer r.Unlock()
c, err := tls.LoadX509KeyPair(certFile, keyFile)
if err != nil {
return err
}
r.certificate = &c
return nil
}
func loadRootCertPool() (*x509.CertPool, error) {
root, err := ioutil.ReadFile(autocertRoot)
if err != nil {
return nil, err
}
pool := x509.NewCertPool()
if ok := pool.AppendCertsFromPEM(root); !ok {
return nil, errors.New("Missing or invalid root certificate")
}
return pool, nil
}
func main() {
url := os.Getenv("HELLO_MTLS_URL")
// Read the root certificate for our CA from disk
roots, err := loadRootCertPool()
if err != nil {
log.Fatal(err)
}
// Load certificate
r := &rotator{}
if err := r.loadCertificate(autocertFile, autocertKey); err != nil {
log.Fatal("error loading certificate and key", err)
}
// Create an HTTPS client using our cert, key & pool
client := &http.Client{
Transport: &http.Transport{
TLSClientConfig: &tls.Config{
RootCAs: roots,
MinVersion: tls.VersionTLS12,
CurvePreferences: []tls.CurveID{tls.CurveP521, tls.CurveP384, tls.CurveP256},
CipherSuites: []uint16{
tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,
tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
},
// GetClientCertificate is called when a server requests a
// certificate from a client.
//
// In this example keep alives will cause the certificate to
// only be called once, but if we disable them,
// GetClientCertificate will be called on every request.
GetClientCertificate: r.getClientCertificate,
},
// Add this line to get the certificate on every request.
// DisableKeepAlives: true,
},
}
// Schedule periodic re-load of certificate
// A real implementation can use something like
// https://github.com/fsnotify/fsnotify
done := make(chan struct{})
go func() {
ticker := time.NewTicker(tickFrequency)
defer ticker.Stop()
for {
select {
case <-ticker.C:
fmt.Println("Checking for new certificate...")
err := r.loadCertificate(autocertFile, autocertKey)
if err != nil {
log.Println("Error loading certificate and key", err)
}
case <-done:
return
}
}
}()
defer close(done)
for {
// Make request
r, err := client.Get(url)
if err != nil {
log.Fatal(err)
}
defer r.Body.Close()
body, err := ioutil.ReadAll(r.Body)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s: %s\n", time.Now().Format(time.RFC3339), strings.Trim(string(body), "\n"))
time.Sleep(requestFrequency)
}
}

@ -1,22 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-mtls-client
labels: {app: hello-mtls-client}
spec:
replicas: 1
selector: {matchLabels: {app: hello-mtls-client}}
template:
metadata:
annotations:
autocert.step.sm/name: hello-mtls-client.default.pod.cluster.local
labels: {app: hello-mtls-client}
spec:
containers:
- name: hello-mtls-client
image: hello-mtls-client-go:latest
imagePullPolicy: Never
resources: {requests: {cpu: 10m, memory: 20Mi}}
env:
- name: HELLO_MTLS_URL
value: https://hello-mtls.default.svc.cluster.local

@ -1,10 +0,0 @@
# build stage
FROM golang:alpine AS build-env
RUN mkdir /src
ADD server.go /src
RUN cd /src && go build -o server
# final stage
FROM alpine
COPY --from=build-env /src/server .
ENTRYPOINT ./server

@ -1,33 +0,0 @@
apiVersion: v1
kind: Service
metadata:
labels: {app: hello-mtls}
name: hello-mtls
spec:
type: ClusterIP
ports:
- port: 443
targetPort: 443
selector: {app: hello-mtls}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-mtls
labels: {app: hello-mtls}
spec:
replicas: 1
selector: {matchLabels: {app: hello-mtls}}
template:
metadata:
annotations:
autocert.step.sm/name: hello-mtls.default.svc.cluster.local
labels: {app: hello-mtls}
spec:
containers:
- name: hello-mtls
image: hello-mtls-server-go:latest
imagePullPolicy: Never
resources: {requests: {cpu: 10m, memory: 20Mi}}

@ -1,137 +0,0 @@
package main
import (
"crypto/tls"
"crypto/x509"
"errors"
"fmt"
"io/ioutil"
"log"
"net/http"
"sync"
"time"
)
const (
autocertFile = "/var/run/autocert.step.sm/site.crt"
autocertKey = "/var/run/autocert.step.sm/site.key"
autocertRoot = "/var/run/autocert.step.sm/root.crt"
tickFrequency = 15 * time.Second
)
// Uses techniques from https://diogomonica.com/2017/01/11/hitless-tls-certificate-rotation-in-go/
// to automatically rotate certificates when they're renewed.
type rotator struct {
sync.RWMutex
certificate *tls.Certificate
}
func (r *rotator) getCertificate(hello *tls.ClientHelloInfo) (*tls.Certificate, error) {
r.RLock()
defer r.RUnlock()
return r.certificate, nil
}
func (r *rotator) loadCertificate(certFile, keyFile string) error {
r.Lock()
defer r.Unlock()
c, err := tls.LoadX509KeyPair(certFile, keyFile)
if err != nil {
return err
}
r.certificate = &c
return nil
}
func loadRootCertPool() (*x509.CertPool, error) {
root, err := ioutil.ReadFile(autocertRoot)
if err != nil {
return nil, err
}
pool := x509.NewCertPool()
if ok := pool.AppendCertsFromPEM(root); !ok {
return nil, errors.New("Missing or invalid root certificate")
}
return pool, nil
}
func main() {
mux := http.NewServeMux()
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
if r.TLS == nil || len(r.TLS.PeerCertificates) == 0 {
fmt.Fprintf(w, "Unauthenticated")
} else {
name := r.TLS.PeerCertificates[0].Subject.CommonName
fmt.Fprintf(w, "Hello, %s!\n", name)
}
})
mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Ok\n")
})
roots, err := loadRootCertPool()
if err != nil {
log.Fatal(err)
}
r := &rotator{}
cfg := &tls.Config{
ClientAuth: tls.RequireAndVerifyClientCert,
ClientCAs: roots,
MinVersion: tls.VersionTLS12,
CurvePreferences: []tls.CurveID{tls.CurveP521, tls.CurveP384, tls.CurveP256},
PreferServerCipherSuites: true,
CipherSuites: []uint16{
tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,
tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
},
GetCertificate: r.getCertificate,
}
srv := &http.Server{
Addr: ":443",
Handler: mux,
TLSConfig: cfg,
}
// Load certificate
err = r.loadCertificate(autocertFile, autocertKey)
if err != nil {
log.Fatal("Error loading certificate and key", err)
}
// Schedule periodic re-load of certificate
// A real implementation can use something like
// https://github.com/fsnotify/fsnotify
done := make(chan struct{})
go func() {
ticker := time.NewTicker(tickFrequency)
defer ticker.Stop()
for {
select {
case <-ticker.C:
fmt.Println("Checking for new certificate...")
err := r.loadCertificate(autocertFile, autocertKey)
if err != nil {
log.Println("Error loading certificate and key", err)
}
case <-done:
return
}
}
}()
defer close(done)
log.Println("Listening no :443")
// Start serving HTTPS
err = srv.ListenAndServeTLS("", "")
if err != nil {
log.Fatal("ListenAndServerTLS: ", err)
}
}

@ -1,11 +0,0 @@
FROM nginx:alpine
RUN apk add inotify-tools
RUN mkdir /src
ADD site.conf /etc/nginx/conf.d
ADD certwatch.sh /src
ADD entrypoint.sh /src
# Certificate watcher and nginx
ENTRYPOINT ["/src/entrypoint.sh"]
CMD ["nginx", "-g", "daemon off;"]

@ -1,6 +0,0 @@
#!/bin/sh
while true; do
inotifywait -e modify /var/run/autocert.step.sm/site.crt
nginx -s reload
done

@ -1,7 +0,0 @@
#!/bin/sh
# watch for the update of the cert and reload nginx
/src/certwatch.sh &
# Run docker CMD
exec "$@"

@ -1,33 +0,0 @@
apiVersion: v1
kind: Service
metadata:
labels: {app: hello-mtls}
name: hello-mtls
spec:
type: ClusterIP
ports:
- port: 443
targetPort: 443
selector: {app: hello-mtls}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-mtls
labels: {app: hello-mtls}
spec:
replicas: 1
selector: {matchLabels: {app: hello-mtls}}
template:
metadata:
annotations:
autocert.step.sm/name: hello-mtls.default.svc.cluster.local
labels: {app: hello-mtls}
spec:
containers:
- name: hello-mtls
image: hello-mtls-server-nginx:latest
imagePullPolicy: Never
resources: {requests: {cpu: 10m, memory: 20Mi}}

@ -1,16 +0,0 @@
server {
listen 443 ssl;
server_name localhost;
ssl_protocols TLSv1.2;
ssl_certificate /var/run/autocert.step.sm/site.crt;
ssl_certificate_key /var/run/autocert.step.sm/site.key;
ssl_client_certificate /var/run/autocert.step.sm/root.crt;
ssl_verify_client on;
ssl_prefer_server_ciphers on;
ssl_ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256;
location / {
root /usr/share/nginx/html;
index index.html index.htm;
}
}

@ -1,6 +0,0 @@
FROM node:lts-alpine
RUN mkdir /src
ADD client.js /src
CMD ["node", "/src/client.js"]

@ -1,6 +0,0 @@
FROM node:lts-alpine
RUN mkdir /src
ADD server.js /src
CMD ["node", "/src/server.js"]

@ -1,44 +0,0 @@
const fs = require('fs');
const https = require('https');
const config = {
ca: '/var/run/autocert.step.sm/root.crt',
key: '/var/run/autocert.step.sm/site.key',
cert: '/var/run/autocert.step.sm/site.crt',
url: process.env.HELLO_MTLS_URL,
requestFrequency: 5000
};
var options = {
ca: fs.readFileSync(config.ca),
key: fs.readFileSync(config.key),
cert: fs.readFileSync(config.cert),
ciphers: 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256',
minVersion: 'TLSv1.2',
maxVersion: 'TLSv1.2',
// Not necessary as it defaults to true
rejectUnauthorized: true
};
fs.watch(config.cert, (event, filename) => {
if (event == 'change') {
options.cert = fs.readFileSync(config.cert);
}
});
function loop() {
var req = https.request(config.url, options, function(res) {
res.on('data', (data) => {
process.stdout.write(options.cert)
process.stdout.write(data)
setTimeout(loop, config.requestFrequency);
});
});
req.on('error', (e) => {
process.stderr.write('error: ' + e.message + '\n');
setTimeout(loop, config.requestFrequency);
})
req.end();
}
loop();

@ -1,22 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-mtls-client
labels: {app: hello-mtls-client}
spec:
replicas: 1
selector: {matchLabels: {app: hello-mtls-client}}
template:
metadata:
annotations:
autocert.step.sm/name: hello-mtls-client.default.pod.cluster.local
labels: {app: hello-mtls-client}
spec:
containers:
- name: hello-mtls-client
image: hello-mtls-client-node:latest
imagePullPolicy: Never
resources: {requests: {cpu: 10m, memory: 20Mi}}
env:
- name: HELLO_MTLS_URL
value: https://hello-mtls.default.svc.cluster.local

@ -1,33 +0,0 @@
apiVersion: v1
kind: Service
metadata:
labels: {app: hello-mtls}
name: hello-mtls
spec:
type: ClusterIP
ports:
- port: 443
targetPort: 443
selector: {app: hello-mtls}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-mtls
labels: {app: hello-mtls}
spec:
replicas: 1
selector: {matchLabels: {app: hello-mtls}}
template:
metadata:
annotations:
autocert.step.sm/name: hello-mtls.default.svc.cluster.local
labels: {app: hello-mtls}
spec:
containers:
- name: hello-mtls
image: hello-mtls-server-node:latest
imagePullPolicy: Never
resources: {requests: {cpu: 10m, memory: 20Mi}}

@ -1,42 +0,0 @@
const https = require('https');
const tls = require('tls');
const fs = require('fs');
var config = {
ca: '/var/run/autocert.step.sm/root.crt',
key: '/var/run/autocert.step.sm/site.key',
cert: '/var/run/autocert.step.sm/site.crt',
ciphers: 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256',
minVersion: 'TLSv1.2',
maxVersion: 'TLSv1.2'
};
function createSecureContext() {
return tls.createSecureContext({
ca: fs.readFileSync(config.ca),
key: fs.readFileSync(config.key),
cert: fs.readFileSync(config.cert),
ciphers: config.ciphers,
});
}
var ctx = createSecureContext()
fs.watch(config.cert, (event, filename) => {
if (event == 'change') {
ctx = createSecureContext();
}
});
https.createServer({
requestCert: true,
rejectUnauthorized: true,
SNICallback: (servername, cb) => {
cb(null, ctx);
}
}, (req, res) => {
res.writeHead(200);
res.end('hello nodejs\n');
}).listen(443);
console.log("Listening on :443 ...");

@ -1,9 +0,0 @@
FROM python:alpine
RUN mkdir /src
ADD client.py /src
ADD client.requirements.txt /src
RUN pip3 install -r /src/client.requirements.txt
CMD ["python", "/src/client.py"]

@ -1,14 +0,0 @@
FROM python:alpine
RUN mkdir /src
# Gunicorn configuration
ADD gunicorn.conf /src
# Flask app
ADD server.py /src
ADD requirements.txt /src
RUN pip3 install -r /src/requirements.txt
# app, certificate watcher and envoy
CMD ["gunicorn", "--config", "/src/gunicorn.conf", "--pythonpath", "/src", "server:app"]

@ -1,79 +0,0 @@
#!/usr/bin/env python
import os
import sys
import ssl
import signal
import time
import logging
import threading
import http.client
from watchdog.events import FileSystemEventHandler
from watchdog.observers import Observer
from urllib.parse import urlparse
ca_certs = '/var/run/autocert.step.sm/root.crt'
cert_file = '/var/run/autocert.step.sm/site.crt'
key_file = '/var/run/autocert.step.sm/site.key'
# RenewHandler is an even file system event handler that reloads the certs in
# the context when a file is modified.
class RenewHandler(FileSystemEventHandler):
def __init__(self, ctx):
self.ctx = ctx
super().__init__()
def on_modified(self, event):
logging.info("reloading certs ...")
ctx.load_cert_chain(cert_file, key_file)
# Monitor is a thread that watches for changes in a path and calls to the
# RenewHandler when a file is modified.
class Monitor(threading.Thread):
def __init__(self, handler, path):
super().__init__()
self.handler = handler
self.path = path
def run(self):
observer = Observer()
observer.schedule(self.handler, self.path)
observer.start()
# Signal handler
def handler(signum, frame):
print("exiting ...")
sys.exit(0)
if __name__ == "__main__":
logging.basicConfig(level=logging.INFO, format='%(asctime)s %(message)s')
# Start signal handler to exit
signal.signal(signal.SIGTERM, handler)
# url from the environment
url = urlparse(os.environ['HELLO_MTLS_URL'])
# ssl context
ctx = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
ctx.set_ciphers('ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256')
ctx.load_verify_locations(ca_certs)
ctx.load_cert_chain(cert_file, key_file)
# initialize the renewer with the ssl context
renewer = RenewHandler(ctx)
# start file monitor
monitor = Monitor(renewer, os.path.dirname(cert_file))
monitor.start()
# Do requests
while True:
try:
conn = http.client.HTTPSConnection(url.netloc, context=ctx)
conn.request("GET", url.path)
r = conn.getresponse()
data = r.read()
logging.info("%d - %s - %s", r.status, r.reason, data)
except Exception as err:
print('Something went wrong:', err)
time.sleep(5)

@ -1,14 +0,0 @@
bind = '0.0.0.0:443'
workers = 2
accesslog = '-'
# mTLS configuration with TLSv1.2 and requiring and validating client
# certificates
ssl_version = 5 # ssl.PROTOCOL_TLSv1_2
cert_reqs = 2 # ssl.CERT_REQUIRED
ciphers = 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256'
ca_certs = '/var/run/autocert.step.sm/root.crt'
certfile = '/var/run/autocert.step.sm/site.crt'
keyfile = '/var/run/autocert.step.sm/site.key'

@ -1,22 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-mtls-client
labels: {app: hello-mtls-client}
spec:
replicas: 1
selector: {matchLabels: {app: hello-mtls-client}}
template:
metadata:
annotations:
autocert.step.sm/name: hello-mtls-client.default.pod.cluster.local
labels: {app: hello-mtls-client}
spec:
containers:
- name: hello-mtls-client
image: hello-mtls-client-py-gunicorn:latest
imagePullPolicy: Never
resources: {requests: {cpu: 10m, memory: 20Mi}}
env:
- name: HELLO_MTLS_URL
value: https://hello-mtls.default.svc.cluster.local

@ -1,33 +0,0 @@
apiVersion: v1
kind: Service
metadata:
labels: {app: hello-mtls}
name: hello-mtls
spec:
type: ClusterIP
ports:
- port: 443
targetPort: 443
selector: {app: hello-mtls}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-mtls
labels: {app: hello-mtls}
spec:
replicas: 1
selector: {matchLabels: {app: hello-mtls}}
template:
metadata:
annotations:
autocert.step.sm/name: hello-mtls.default.svc.cluster.local
labels: {app: hello-mtls}
spec:
containers:
- name: hello-mtls
image: hello-mtls-server-py-gunicorn:latest
imagePullPolicy: Never
resources: {requests: {cpu: 10m, memory: 20Mi}}

@ -1,9 +0,0 @@
from flask import Flask
app = Flask(__name__)
@app.route("/")
def hello():
return "Hello World!\n"
if __name__ == "__main__":
app.run(host='127.0.0.1', port=8080, debug=False)

Binary file not shown.

Before

Width:  |  Height:  |  Size: 27 KiB

@ -1,193 +0,0 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:xl="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg" version="1.1" viewBox="985.6917 4772.835 272.38144 274.09766" width="272.38144" height="274.09766">
<defs/>
<metadata> Produced by OmniGraffle 7.9.4
<dc:date>2019-05-24 18:12:06 +0000</dc:date>
</metadata>
<g id="Canvas_3" stroke="none" stroke-opacity="1" fill-opacity="1" fill="none" stroke-dasharray="none">
<title>Canvas 3</title>
<g id="Canvas_3: Layer 1">
<title>Layer 1</title>
<g id="Graphic_1074">
<path d="M 1251.4366 4909.882 C 1251.4366 4981.429 1193.4327 5039.433 1121.8859 5039.433 C 1050.3351 5039.433 992.3351 4981.429 992.3351 4909.882 C 992.3351 4838.331 1050.3351 4780.331 1121.8859 4780.331 C 1193.4327 4780.331 1251.4366 4838.331 1251.4366 4909.882 L 1251.4366 4909.882" fill="#f8fbff"/>
</g>
<g id="Graphic_1073">
<path d="M 1121.8859 4780.331 C 1113.1398 4780.331 1104.5968 4781.202 1096.3351 4782.8546 C 1155.6476 4794.722 1200.3351 4847.077 1200.3351 4909.882 C 1200.3351 4972.683 1155.6476 5025.042 1096.3351 5036.9054 C 1104.5968 5038.558 1113.1398 5039.433 1121.8859 5039.433 C 1193.4327 5039.433 1251.4366 4981.429 1251.4366 4909.882 C 1251.4366 4838.331 1193.4327 4780.331 1121.8859 4780.331 L 1121.8859 4780.331" fill="#e4eaf7"/>
</g>
<g id="Graphic_1071">
<path d="M 1251.4288 4917.5343 C 1247.3156 4917.0656 1243.5929 4920.007 1243.1124 4924.12 C 1235.9757 4985.585 1183.8624 5031.933 1121.8859 5031.933 C 1059.8741 5031.933 1007.757 4985.554 1000.6515 4924.054 C 1000.1749 4919.9406 996.4562 4916.9796 992.339 4917.464 C 988.2257 4917.9406 985.2726 4921.6593 985.7491 4925.7765 C 993.7296 4994.847 1052.257 5046.933 1121.8859 5046.933 C 1191.4757 5046.933 1249.9991 4994.878 1258.0148 4925.847 C 1258.4913 4921.7335 1255.5421 4918.011 1251.4288 4917.5343 L 1251.4288 4917.5343" fill="black"/>
</g>
<g id="Graphic_1070">
<path d="M 1121.8859 4772.835 C 1052.2335 4772.835 993.7023 4824.9406 985.7413 4894.038 C 985.2687 4898.1515 988.2218 4901.87 992.3351 4902.347 C 992.628 4902.382 992.9171 4902.3976 993.2023 4902.3976 C 996.9601 4902.3976 1000.2023 4899.577 1000.6437 4895.753 C 1007.7335 4834.2296 1059.8546 4787.835 1121.8859 4787.835 C 1183.9015 4787.835 1236.0226 4834.218 1243.1202 4895.7257 C 1243.5968 4899.839 1247.3116 4902.792 1251.4327 4902.3156 C 1255.5499 4901.843 1258.4991 4898.12 1258.0226 4894.007 C 1250.0499 4824.929 1191.5226 4772.835 1121.8859 4772.835 L 1121.8859 4772.835" fill="black"/>
</g>
<g id="Graphic_1068">
<path d="M 1142.6322 4838.79 C 1142.6322 4847.0837 1137.7739 4854.223 1130.7533 4857.5105 C 1128.0587 4858.811 1125.0662 4859.5253 1121.8972 4859.5253 C 1118.6978 4859.5253 1115.6764 4858.8137 1112.9803 4857.5105 C 1105.993 4854.194 1101.1621 4847.055 1101.1621 4838.79 C 1101.1621 4827.3264 1110.4362 4818.055 1121.8972 4818.055 C 1133.3307 4818.055 1142.6322 4827.3264 1142.6322 4838.79 M 1130.7836 4838.79 C 1130.7836 4833.8726 1126.7844 4829.904 1121.8972 4829.904 C 1116.9795 4829.904 1113.0107 4833.8726 1113.0107 4838.79 C 1113.0107 4843.708 1116.9795 4847.6767 1121.8972 4847.6767 C 1126.7844 4847.6767 1130.7836 4843.708 1130.7836 4838.79" fill="white"/>
</g>
<g id="Graphic_1067">
<path d="M 1089.9947 4880.289 C 1091.4136 4881.86 1092.2757 4883.934 1092.2757 4886.2136 C 1092.2757 4888.4974 1091.4165 4890.5397 1090.0237 4892.109 C 1088.424 4893.946 1086.0245 4895.1 1083.3892 4895.1 C 1078.4716 4895.1 1074.5027 4891.131 1074.5027 4886.2136 L 1074.5027 4886.1847 C 1074.5317 4881.267 1078.4716 4877.327 1083.3892 4877.327 C 1086.0288 4877.327 1088.3922 4878.454 1089.9947 4880.289" fill="white"/>
</g>
<g id="Graphic_1066">
<path d="M 1169.2612 4886.1847 L 1169.2612 4886.2136 C 1169.2612 4891.131 1165.2924 4895.1 1160.3748 4895.1 C 1157.7352 4895.1 1155.3718 4893.946 1153.7692 4892.109 L 1153.7403 4892.109 C 1152.3475 4890.5397 1151.4883 4888.4945 1151.4883 4886.2136 C 1151.4883 4883.934 1152.3475 4881.86 1153.7692 4880.289 C 1155.4022 4878.454 1157.7713 4877.327 1160.3748 4877.327 C 1165.2924 4877.327 1169.2323 4881.267 1169.2612 4886.1847" fill="white"/>
</g>
<g id="Graphic_1065">
<path d="M 1112.9803 4892.109 L 1090.0237 4892.109 C 1091.4165 4890.5397 1092.2757 4888.4945 1092.2757 4886.2136 C 1092.2757 4883.934 1091.4165 4881.86 1089.9947 4880.289 L 1153.7692 4880.289 C 1152.3503 4881.86 1151.4883 4883.934 1151.4883 4886.2136 C 1151.4883 4888.4974 1152.3475 4890.5397 1153.7403 4892.109 L 1112.9803 4892.109" fill="white"/>
</g>
<g id="Graphic_1064">
<path d="M 1130.7533 4880.2604 L 1130.7533 4857.5105 C 1128.0587 4858.811 1125.0662 4859.5253 1121.8972 4859.5253 C 1118.6978 4859.5253 1115.6764 4858.8137 1112.9803 4857.5105 L 1112.9803 4880.289 L 1130.7533 4880.289 L 1130.7533 4880.2604" fill="#d7e9ff"/>
</g>
<g id="Graphic_1063">
<path d="M 1184.072 4933.579 L 1178.1477 4935.948 L 1178.1477 4939.5034 C 1178.1477 4957.5135 1163.5741 4972.087 1145.564 4972.087 L 1142.6322 4972.087 C 1138.9295 4972.087 1135.6449 4970.399 1133.4493 4967.7624 C 1131.7874 4965.719 1130.7533 4963.1124 1130.7533 4960.2385 L 1130.7533 4892.109 L 1112.9803 4892.109 L 1112.9803 4960.2095 C 1112.9803 4964.326 1110.9077 4967.9375 1107.7677 4970.072 C 1105.8686 4971.3465 1103.5906 4972.087 1101.1317 4972.087 L 1098.2 4972.087 C 1080.1898 4972.087 1065.6163 4957.5135 1065.6163 4939.5034 L 1065.6163 4935.948 L 1059.692 4933.579 L 1053.7677 4935.948 L 1053.7677 4939.5034 C 1053.7677 4968.9774 1077.6153 4992.822 1107.0864 4992.822 L 1113.0107 4992.822 C 1117.898 4992.822 1121.8972 4996.791 1121.8972 5001.7086 C 1121.8972 4996.791 1125.866 4992.822 1130.7836 4992.822 L 1136.6776 4992.822 C 1166.1487 4992.822 1189.9963 4968.9774 1189.9963 4939.5034 L 1189.9963 4935.948 L 1184.072 4933.579" fill="#d7e9ff"/>
</g>
<g id="Graphic_1062">
<path d="M 1184.072 4912.844 L 1198.8827 4939.5034 L 1189.9963 4935.948 L 1184.072 4933.579 L 1178.1477 4935.948 L 1169.2612 4939.5034 L 1184.072 4912.844" fill="white"/>
</g>
<g id="Graphic_1061">
<path d="M 1059.692 4912.844 L 1074.5027 4939.5034 L 1065.6163 4935.948 L 1059.692 4933.579 L 1053.7677 4935.948 L 1044.8812 4939.5034 L 1059.692 4912.844" fill="white"/>
</g>
<g id="Group_1013">
<g id="Graphic_1060">
<path d="M 1121.8827 5004.671 C 1120.244 5004.671 1118.9206 5003.3473 1118.9206 5001.7086 C 1118.9206 4995.174 1124.2345 4989.86 1130.7692 4989.86 C 1132.4064 4989.86 1133.7313 4991.1834 1133.7313 4992.822 C 1133.7313 4994.461 1132.4064 4995.784 1130.7692 4995.784 C 1127.5018 4995.784 1124.8449 4998.441 1124.8449 5001.7086 C 1124.8449 5003.3473 1123.52 5004.671 1121.8827 5004.671" fill="#116fff"/>
</g>
<g id="Graphic_1059">
<path d="M 1121.8827 5004.671 C 1120.244 5004.671 1118.9206 5003.3473 1118.9206 5001.7086 C 1118.9206 4998.441 1116.2621 4995.784 1112.9963 4995.784 C 1111.3575 4995.784 1110.0341 4994.461 1110.0341 4992.822 C 1110.0341 4991.1834 1111.3575 4989.86 1112.9963 4989.86 C 1119.5295 4989.86 1124.8449 4995.174 1124.8449 5001.7086 C 1124.8449 5003.3473 1123.52 5004.671 1121.8827 5004.671" fill="#116fff"/>
</g>
<g id="Graphic_1058">
<path d="M 1136.6689 4995.784 C 1135.0316 4995.784 1133.7067 4994.461 1133.7067 4992.822 C 1133.7067 4991.1834 1135.0316 4989.86 1136.6689 4989.86 C 1164.4362 4989.86 1187.0255 4967.2707 1187.0255 4939.5034 C 1187.0255 4937.8647 1188.3503 4936.541 1189.9876 4936.541 C 1191.6263 4936.541 1192.9498 4937.8647 1192.9498 4939.5034 C 1192.9498 4970.538 1167.7035 4995.784 1136.6689 4995.784 L 1136.6689 4995.784" fill="#116fff"/>
</g>
<g id="Graphic_1057">
<path d="M 1145.5669 4975.049 C 1143.9296 4975.049 1142.6048 4973.726 1142.6048 4972.087 C 1142.6048 4970.4483 1143.9296 4969.125 1145.5669 4969.125 C 1161.9007 4969.125 1175.1884 4955.837 1175.1884 4939.5034 C 1175.1884 4937.8647 1176.5133 4936.541 1178.1506 4936.541 C 1179.7893 4936.541 1181.1127 4937.8647 1181.1127 4939.5034 C 1181.1127 4959.1045 1165.168 4975.049 1145.5669 4975.049" fill="#116fff"/>
</g>
<g id="Graphic_1056">
<path d="M 1136.6689 4995.784 L 1130.7692 4995.784 C 1129.1304 4995.784 1127.807 4994.461 1127.807 4992.822 C 1127.807 4991.1834 1129.1304 4989.86 1130.7692 4989.86 L 1136.6689 4989.86 C 1138.3076 4989.86 1139.631 4991.1834 1139.631 4992.822 C 1139.631 4994.461 1138.3076 4995.784 1136.6689 4995.784" fill="#116fff"/>
</g>
<g id="Graphic_1055">
<path d="M 1169.267 4942.4655 C 1168.781 4942.4655 1168.2864 4942.347 1167.8337 4942.0924 C 1166.4032 4941.2983 1165.8883 4939.495 1166.6809 4938.067 L 1181.4917 4911.399 C 1182.2857 4909.97 1184.0908 4909.4494 1185.5183 4910.2464 C 1186.9488 4911.0404 1187.4637 4912.844 1186.6697 4914.2716 L 1171.8589 4940.9396 C 1171.318 4941.9145 1170.3098 4942.4655 1169.267 4942.4655" fill="#116fff"/>
</g>
<g id="Graphic_1054">
<path d="M 1198.8943 4942.4655 C 1197.853 4942.4655 1196.8448 4941.9145 1196.3024 4940.9396 L 1181.4917 4914.2716 C 1180.6962 4912.844 1181.2111 4911.0375 1182.6444 4910.2464 C 1184.072 4909.4465 1185.88 4909.967 1186.6697 4911.399 L 1201.4804 4938.067 C 1202.2774 4939.495 1201.7625 4941.301 1200.3291 4942.0924 C 1199.875 4942.347 1199.3803 4942.4655 1198.8943 4942.4655 L 1198.8943 4942.4655" fill="#116fff"/>
</g>
<g id="Graphic_1053">
<path d="M 1198.8914 4942.4655 C 1198.524 4942.4655 1198.1509 4942.4005 1197.7907 4942.256 L 1184.0807 4936.764 L 1170.372 4942.256 C 1168.8534 4942.859 1167.1293 4942.124 1166.5189 4940.6055 C 1165.9114 4939.0854 1166.6491 4937.3613 1168.1692 4936.751 L 1182.98 4930.821 C 1183.6901 4930.5374 1184.4726 4930.5374 1185.1828 4930.821 L 1199.9936 4936.751 C 1201.5137 4937.3613 1202.2513 4939.0854 1201.6439 4940.6055 C 1201.1781 4941.764 1200.0673 4942.4655 1198.8914 4942.4655 L 1198.8914 4942.4655" fill="#116fff"/>
</g>
<g id="Graphic_1052">
<path d="M 1189.9876 4942.4655 C 1188.3503 4942.4655 1187.0255 4941.142 1187.0255 4939.5034 L 1187.0255 4935.9425 C 1187.0255 4934.305 1188.3503 4932.9803 1189.9876 4932.9803 C 1191.6263 4932.9803 1192.9498 4934.305 1192.9498 4935.9425 L 1192.9498 4939.5034 C 1192.9498 4941.142 1191.6263 4942.4655 1189.9876 4942.4655" fill="#116fff"/>
</g>
<g id="Graphic_1051">
<path d="M 1178.1506 4942.4655 C 1176.5133 4942.4655 1175.1884 4941.142 1175.1884 4939.5034 L 1175.1884 4935.9425 C 1175.1884 4934.305 1176.5133 4932.9803 1178.1506 4932.9803 C 1179.7893 4932.9803 1181.1127 4934.305 1181.1127 4935.9425 L 1181.1127 4939.5034 C 1181.1127 4941.142 1179.7893 4942.4655 1178.1506 4942.4655" fill="#116fff"/>
</g>
<g id="Graphic_1050">
<path d="M 1107.0951 4995.7814 C 1076.0605 4995.7814 1050.8142 4970.535 1050.8142 4939.5005 C 1050.8142 4937.862 1052.1376 4936.5384 1053.7764 4936.5384 C 1055.4136 4936.5384 1056.7385 4937.862 1056.7385 4939.5005 C 1056.7385 4967.268 1079.3278 4989.857 1107.0951 4989.857 C 1108.7324 4989.857 1110.0572 4991.1805 1110.0572 4992.819 C 1110.0572 4994.4565 1108.7324 4995.7814 1107.0951 4995.7814" fill="#116fff"/>
</g>
<g id="Graphic_1049">
<path d="M 1098.197 4975.0463 C 1078.596 4975.0463 1062.6512 4959.1016 1062.6512 4939.5005 C 1062.6512 4937.862 1063.9747 4936.5384 1065.6134 4936.5384 C 1067.2507 4936.5384 1068.5755 4937.862 1068.5755 4939.5005 C 1068.5755 4955.834 1081.8633 4969.122 1098.197 4969.122 C 1099.8343 4969.122 1101.1592 4970.4454 1101.1592 4972.084 C 1101.1592 4973.7215 1099.8343 4975.0463 1098.197 4975.0463" fill="#116fff"/>
</g>
<g id="Graphic_1048">
<path d="M 1112.9963 4995.7814 L 1107.0951 4995.7814 C 1105.4564 4995.7814 1104.133 4994.4565 1104.133 4992.819 C 1104.133 4991.1805 1105.4564 4989.857 1107.0951 4989.857 L 1112.9963 4989.857 C 1114.6335 4989.857 1115.9584 4991.1805 1115.9584 4992.819 C 1115.9584 4994.4565 1114.6335 4995.7814 1112.9963 4995.7814" fill="#116fff"/>
</g>
<g id="Graphic_1047">
<path d="M 1074.497 4942.4627 C 1073.4541 4942.4627 1072.446 4941.9116 1071.9051 4940.937 L 1057.0943 4914.2687 C 1056.2974 4912.841 1056.8123 4911.0346 1058.2456 4910.2435 C 1059.6732 4909.441 1061.4811 4909.9643 1062.2723 4911.396 L 1077.083 4938.064 C 1077.8786 4939.492 1077.3636 4941.2983 1075.9303 4942.0895 C 1075.4776 4942.344 1074.983 4942.4627 1074.497 4942.4627" fill="#116fff"/>
</g>
<g id="Graphic_1046">
<path d="M 1044.8697 4942.4627 C 1044.3837 4942.4627 1043.889 4942.344 1043.4349 4942.0895 C 1042.0044 4941.2954 1041.4895 4939.492 1042.2836 4938.064 L 1057.0943 4911.396 C 1057.8884 4909.967 1059.692 4909.4436 1061.1195 4910.2435 C 1062.55 4911.0375 1063.0649 4912.841 1062.2723 4914.2687 L 1047.4615 4940.937 C 1046.9192 4941.9116 1045.911 4942.4627 1044.8697 4942.4627 L 1044.8697 4942.4627" fill="#116fff"/>
</g>
<g id="Graphic_1045">
<path d="M 1074.4941 4942.4627 C 1074.1267 4942.4627 1073.7535 4942.3976 1073.392 4942.253 L 1059.6833 4936.761 L 1045.9732 4942.253 C 1044.4545 4942.859 1042.7305 4942.1213 1042.1201 4940.6026 C 1041.5126 4939.0825 1042.2503 4937.3584 1043.7704 4936.748 L 1058.5812 4930.818 C 1059.2913 4930.5345 1060.0738 4930.5345 1060.784 4930.818 L 1075.5947 4936.748 C 1077.1149 4937.3584 1077.8525 4939.0825 1077.245 4940.6026 C 1076.7779 4941.761 1075.6671 4942.4627 1074.4941 4942.4627" fill="#116fff"/>
</g>
<g id="Graphic_1044">
<path d="M 1053.7764 4942.4627 C 1052.1376 4942.4627 1050.8142 4941.138 1050.8142 4939.5005 L 1050.8142 4935.9396 C 1050.8142 4934.302 1052.1376 4932.9774 1053.7764 4932.9774 C 1055.4136 4932.9774 1056.7385 4934.302 1056.7385 4935.9396 L 1056.7385 4939.5005 C 1056.7385 4941.138 1055.4136 4942.4627 1053.7764 4942.4627" fill="#116fff"/>
</g>
<g id="Graphic_1043">
<path d="M 1065.6134 4942.4627 C 1063.9747 4942.4627 1062.6512 4941.138 1062.6512 4939.5005 L 1062.6512 4935.9396 C 1062.6512 4934.302 1063.9747 4932.9774 1065.6134 4932.9774 C 1067.2507 4932.9774 1068.5755 4934.302 1068.5755 4935.9396 L 1068.5755 4939.5005 C 1068.5755 4941.138 1067.2507 4942.4627 1065.6134 4942.4627" fill="#116fff"/>
</g>
<g id="Graphic_1042">
<path d="M 1101.1288 4975.0463 L 1098.197 4975.0463 C 1096.5583 4975.0463 1095.2349 4973.7215 1095.2349 4972.084 C 1095.2349 4970.4454 1096.5583 4969.122 1098.197 4969.122 L 1101.1288 4969.122 C 1102.7676 4969.122 1104.091 4970.4454 1104.091 4972.084 C 1104.091 4973.7215 1102.7676 4975.0463 1101.1288 4975.0463" fill="#116fff"/>
</g>
<g id="Graphic_1041">
<path d="M 1145.5669 4975.049 L 1142.6178 4975.049 C 1140.979 4975.049 1139.6556 4973.726 1139.6556 4972.087 C 1139.6556 4970.4483 1140.979 4969.125 1142.6178 4969.125 L 1145.5669 4969.125 C 1147.2056 4969.125 1148.529 4970.4483 1148.529 4972.087 C 1148.529 4973.726 1147.2056 4975.049 1145.5669 4975.049" fill="#116fff"/>
</g>
<g id="Graphic_1040">
<path d="M 1112.989 4963.183 C 1111.3517 4963.183 1110.0269 4961.8584 1110.0269 4960.221 L 1110.0269 4892.1205 C 1110.0269 4890.483 1111.3517 4889.1584 1112.989 4889.1584 C 1114.6278 4889.1584 1115.9512 4890.483 1115.9512 4892.1205 L 1115.9512 4960.221 C 1115.9512 4961.8584 1114.6278 4963.183 1112.989 4963.183" fill="#116fff"/>
</g>
<g id="Graphic_1039">
<path d="M 1130.7619 4963.186 C 1129.1246 4963.186 1127.7998 4961.861 1127.7998 4960.224 L 1127.7998 4892.1205 C 1127.7998 4890.483 1129.1246 4889.1584 1130.7619 4889.1584 C 1132.4007 4889.1584 1133.7241 4890.483 1133.7241 4892.1205 L 1133.7241 4960.224 C 1133.7241 4961.861 1132.4007 4963.186 1130.7619 4963.186" fill="#116fff"/>
</g>
<g id="Graphic_1038">
<path d="M 1112.989 4883.234 C 1111.3517 4883.234 1110.0269 4881.9107 1110.0269 4880.272 L 1110.0269 4857.5235 C 1110.0269 4855.885 1111.3517 4854.5614 1112.989 4854.5614 C 1114.6278 4854.5614 1115.9512 4855.885 1115.9512 4857.5235 L 1115.9512 4880.272 C 1115.9512 4881.9107 1114.6278 4883.234 1112.989 4883.234" fill="#116fff"/>
</g>
<g id="Graphic_1037">
<path d="M 1130.7619 4883.234 C 1129.1246 4883.234 1127.7998 4881.9107 1127.7998 4880.272 L 1127.7998 4857.5235 C 1127.7998 4855.885 1129.1246 4854.5614 1130.7619 4854.5614 C 1132.4007 4854.5614 1133.7241 4855.885 1133.7241 4857.5235 L 1133.7241 4880.272 C 1133.7241 4881.9107 1132.4007 4883.234 1130.7619 4883.234" fill="#116fff"/>
</g>
<g id="Graphic_1036">
<path d="M 1101.1288 4975.0463 C 1099.4916 4975.0463 1098.1667 4973.7215 1098.1667 4972.084 C 1098.1667 4970.4454 1099.4916 4969.122 1101.1288 4969.122 C 1106.0349 4969.122 1110.0269 4965.1286 1110.0269 4960.221 C 1110.0269 4958.5824 1111.3517 4957.259 1112.989 4957.259 C 1114.6278 4957.259 1115.9512 4958.5824 1115.9512 4960.221 C 1115.9512 4968.396 1109.3022 4975.0463 1101.1288 4975.0463" fill="#116fff"/>
</g>
<g id="Graphic_1035">
<path d="M 1142.6178 4975.0463 C 1134.4415 4975.0463 1127.7882 4968.396 1127.7882 4960.221 C 1127.7882 4958.5824 1129.1131 4957.259 1130.7504 4957.259 C 1132.3891 4957.259 1133.7125 4958.5824 1133.7125 4960.221 C 1133.7125 4965.1286 1137.7088 4969.122 1142.6178 4969.122 C 1144.255 4969.122 1145.5799 4970.4454 1145.5799 4972.084 C 1145.5799 4973.7215 1144.255 4975.0463 1142.6178 4975.0463" fill="#116fff"/>
</g>
<g id="Graphic_1034">
<path d="M 1153.7606 4895.0827 L 1090.0092 4895.0827 C 1088.3705 4895.0827 1087.047 4893.759 1087.047 4892.1205 C 1087.047 4890.483 1088.3705 4889.1584 1090.0092 4889.1584 L 1153.7577 4889.1584 C 1155.3964 4889.1584 1156.7198 4890.483 1156.7198 4892.1205 C 1156.7198 4893.759 1155.3964 4895.0827 1153.7606 4895.0827" fill="#116fff"/>
</g>
<g id="Graphic_1033">
<path d="M 1153.7606 4883.2485 L 1090.0034 4883.2485 C 1088.3647 4883.2485 1087.0413 4881.925 1087.0413 4880.2864 C 1087.0413 4878.649 1088.3647 4877.324 1090.0034 4877.324 L 1153.7577 4877.324 C 1155.3964 4877.324 1156.7198 4878.649 1156.7198 4880.2864 C 1156.7198 4881.925 1155.3964 4883.2485 1153.7606 4883.2485" fill="#116fff"/>
</g>
<g id="Graphic_1032">
<path d="M 1083.3805 4898.0593 C 1076.8459 4898.0593 1071.5319 4892.7454 1071.5319 4886.2107 C 1071.5319 4879.6775 1076.8459 4874.362 1083.3805 4874.362 C 1089.9137 4874.362 1095.2291 4879.6775 1095.2291 4886.2107 C 1095.2291 4892.7454 1089.9137 4898.0593 1083.3805 4898.0593 M 1083.3805 4880.2864 C 1080.1132 4880.2864 1077.4562 4882.9434 1077.4562 4886.2107 C 1077.4562 4889.478 1080.1132 4892.135 1083.3805 4892.135 C 1086.6479 4892.135 1089.3048 4889.478 1089.3048 4886.2107 C 1089.3048 4882.9434 1086.6435 4880.2864 1083.3805 4880.2864" fill="#116fff"/>
</g>
<g id="Graphic_1031">
<path d="M 1160.3834 4898.0593 C 1153.8502 4898.0593 1148.5348 4892.7454 1148.5348 4886.2107 C 1148.5348 4879.6775 1153.8502 4874.362 1160.3834 4874.362 C 1166.9181 4874.362 1172.232 4879.6775 1172.232 4886.2107 C 1172.232 4892.7454 1166.9181 4898.0593 1160.3834 4898.0593 M 1160.3834 4880.2864 C 1157.1161 4880.2864 1154.4591 4882.9434 1154.4591 4886.2107 C 1154.4591 4889.478 1157.1161 4892.135 1160.3834 4892.135 C 1163.6508 4892.135 1166.3078 4889.478 1166.3078 4886.2107 C 1166.3078 4882.9434 1163.6508 4880.2864 1160.3834 4880.2864" fill="#116fff"/>
</g>
<g id="Graphic_1030">
<path d="M 1121.8827 4862.4875 C 1108.8163 4862.4875 1098.1855 4851.8567 1098.1855 4838.79 C 1098.1855 4825.724 1108.8163 4815.093 1121.8827 4815.093 C 1134.9477 4815.093 1145.5799 4825.724 1145.5799 4838.79 C 1145.5799 4851.8567 1134.9477 4862.4875 1121.8827 4862.4875 M 1121.8827 4821.0173 C 1112.0807 4821.0173 1104.1098 4828.988 1104.1098 4838.79 C 1104.1098 4848.592 1112.0807 4856.563 1121.8827 4856.563 C 1131.6833 4856.563 1139.6556 4848.592 1139.6556 4838.79 C 1139.6556 4828.988 1131.6833 4821.0173 1121.8827 4821.0173" fill="#116fff"/>
</g>
<g id="Graphic_1029">
<path d="M 1121.8827 4850.639 C 1115.348 4850.639 1110.0341 4845.325 1110.0341 4838.79 C 1110.0341 4832.2556 1115.348 4826.9416 1121.8827 4826.9416 C 1128.4159 4826.9416 1133.7313 4832.2556 1133.7313 4838.79 C 1133.7313 4845.325 1128.4159 4850.639 1121.8827 4850.639 M 1121.8827 4832.866 C 1118.6154 4832.866 1115.9584 4835.523 1115.9584 4838.79 C 1115.9584 4842.0576 1118.6154 4844.7145 1121.8827 4844.7145 C 1125.1486 4844.7145 1127.807 4842.0576 1127.807 4838.79 C 1127.807 4835.523 1125.1486 4832.866 1121.8827 4832.866" fill="#116fff"/>
</g>
<g id="Graphic_1028">
<path d="M 1163.334 4841.7524 L 1121.8827 4841.7524 C 1120.244 4841.7524 1118.9206 4840.429 1118.9206 4838.79 C 1118.9206 4837.1515 1120.244 4835.828 1121.8827 4835.828 L 1163.334 4835.828 C 1164.9728 4835.828 1166.2962 4837.1515 1166.2962 4838.79 C 1166.2962 4840.429 1164.9728 4841.7524 1163.334 4841.7524" fill="#116fff"/>
</g>
<g id="Graphic_1027">
<path d="M 1163.334 4865.458 C 1161.6967 4865.458 1160.3719 4864.135 1160.3719 4862.496 C 1160.3719 4860.859 1161.6967 4859.534 1163.334 4859.534 C 1168.2343 4859.534 1172.2205 4855.5435 1172.2205 4850.642 C 1172.2205 4845.743 1168.2343 4841.7524 1163.334 4841.7524 C 1161.6967 4841.7524 1160.3719 4840.429 1160.3719 4838.79 C 1160.3719 4837.1515 1161.6967 4835.828 1163.334 4835.828 C 1171.5016 4835.828 1178.1448 4842.4727 1178.1448 4850.642 C 1178.1448 4858.811 1171.5016 4865.458 1163.334 4865.458" fill="#116fff"/>
</g>
<g id="Graphic_1026">
<path d="M 1163.3398 4865.444 L 1130.7692 4865.444 C 1129.1304 4865.444 1127.807 4864.119 1127.807 4862.4817 C 1127.807 4860.843 1129.1304 4859.5195 1130.7692 4859.5195 L 1163.3398 4859.5195 C 1164.9785 4859.5195 1166.302 4860.843 1166.302 4862.4817 C 1166.302 4864.119 1164.9785 4865.444 1163.3398 4865.444" fill="#116fff"/>
</g>
<g id="Graphic_1025">
<path d="M 1074.4883 4889.15 C 1066.3178 4889.15 1059.6703 4882.505 1059.6703 4874.336 C 1059.6703 4866.167 1066.3178 4859.5195 1074.4883 4859.5195 C 1076.1256 4859.5195 1077.4504 4860.843 1077.4504 4862.4817 C 1077.4504 4864.119 1076.1256 4865.444 1074.4883 4865.444 C 1069.5851 4865.444 1065.5946 4869.4343 1065.5946 4874.336 C 1065.5946 4879.235 1069.5851 4883.2254 1074.4883 4883.2254 C 1076.1256 4883.2254 1077.4504 4884.55 1077.4504 4886.1876 C 1077.4504 4887.826 1076.1256 4889.15 1074.4883 4889.15" fill="#116fff"/>
</g>
<g id="Graphic_1024">
<path d="M 1112.9963 4865.458 L 1074.4883 4865.458 C 1072.8496 4865.458 1071.5261 4864.135 1071.5261 4862.496 C 1071.5261 4860.859 1072.8496 4859.534 1074.4883 4859.534 L 1112.9963 4859.534 C 1114.6335 4859.534 1115.9584 4860.859 1115.9584 4862.496 C 1115.9584 4864.135 1114.6335 4865.458 1112.9963 4865.458" fill="#116fff"/>
</g>
<g id="Graphic_1023">
<path d="M 1098.2086 4936.544 C 1090.0396 4936.544 1083.3921 4929.8995 1083.3921 4921.7305 C 1083.3921 4913.5614 1090.0396 4906.914 1098.2086 4906.914 C 1099.8459 4906.914 1101.1708 4908.2374 1101.1708 4909.876 C 1101.1708 4911.5134 1099.8459 4912.838 1098.2086 4912.838 C 1093.3055 4912.838 1089.3164 4916.829 1089.3164 4921.7305 C 1089.3164 4926.6293 1093.3055 4930.62 1098.2086 4930.62 C 1099.8459 4930.62 1101.1708 4931.9447 1101.1708 4933.582 C 1101.1708 4935.221 1099.8459 4936.544 1098.2086 4936.544" fill="#116fff"/>
</g>
<g id="Graphic_1022">
<path d="M 1112.9963 4912.853 L 1098.2086 4912.853 C 1096.5699 4912.853 1095.2465 4911.529 1095.2465 4909.8906 C 1095.2465 4908.253 1096.5699 4906.9284 1098.2086 4906.9284 L 1112.9963 4906.9284 C 1114.6335 4906.9284 1115.9584 4908.253 1115.9584 4909.8906 C 1115.9584 4911.529 1114.6335 4912.853 1112.9963 4912.853" fill="#116fff"/>
</g>
<g id="Graphic_1021">
<path d="M 1169.2771 4912.847 C 1167.6384 4912.847 1166.315 4911.5235 1166.315 4909.885 C 1166.315 4908.2475 1167.6384 4906.9226 1169.2771 4906.9226 C 1174.176 4906.9226 1178.1636 4902.932 1178.1636 4898.0304 C 1178.1636 4893.1315 1174.176 4889.141 1169.2771 4889.141 C 1167.6384 4889.141 1166.315 4887.816 1166.315 4886.179 C 1166.315 4884.54 1167.6384 4883.217 1169.2771 4883.217 C 1177.4433 4883.217 1184.0879 4889.8613 1184.0879 4898.0304 C 1184.0879 4906.1994 1177.4433 4912.847 1169.2771 4912.847" fill="#116fff"/>
</g>
<g id="Graphic_1020">
<path d="M 1169.2771 4912.8325 L 1130.7692 4912.8325 C 1129.1304 4912.8325 1127.807 4911.5076 1127.807 4909.8703 C 1127.807 4908.2316 1129.1304 4906.908 1130.7692 4906.908 L 1169.2771 4906.908 C 1170.9144 4906.908 1172.2393 4908.2316 1172.2393 4909.8703 C 1172.2393 4911.5076 1170.9144 4912.8325 1169.2771 4912.8325" fill="#116fff"/>
</g>
<g id="Graphic_1019">
<path d="M 1112.9963 4936.544 L 1098.2086 4936.544 C 1096.5699 4936.544 1095.2465 4935.221 1095.2465 4933.582 C 1095.2465 4931.9447 1096.5699 4930.62 1098.2086 4930.62 L 1112.9963 4930.62 C 1114.6335 4930.62 1115.9584 4931.9447 1115.9584 4933.582 C 1115.9584 4935.221 1114.6335 4936.544 1112.9963 4936.544" fill="#116fff"/>
</g>
<g id="Graphic_1018">
<path d="M 1139.6195 4960.253 C 1137.9807 4960.253 1136.6573 4958.9295 1136.6573 4957.291 C 1136.6573 4955.6535 1137.9807 4954.3286 1139.6195 4954.3286 C 1144.5212 4954.3286 1148.5117 4950.3396 1148.5117 4945.4364 C 1148.5117 4940.5375 1144.5212 4936.547 1139.6195 4936.547 C 1137.9807 4936.547 1136.6573 4935.2236 1136.6573 4933.585 C 1136.6573 4931.9476 1137.9807 4930.623 1139.6195 4930.623 C 1147.7885 4930.623 1154.436 4937.2673 1154.436 4945.4364 C 1154.436 4953.607 1147.7885 4960.253 1139.6195 4960.253" fill="#116fff"/>
</g>
<g id="Graphic_1017">
<path d="M 1139.6195 4960.2385 L 1130.7619 4960.2385 C 1129.1246 4960.2385 1127.7998 4958.915 1127.7998 4957.2763 C 1127.7998 4955.6376 1129.1246 4954.314 1130.7619 4954.314 L 1139.6195 4954.314 C 1141.2567 4954.314 1142.5816 4955.6376 1142.5816 4957.2763 C 1142.5816 4958.915 1141.2567 4960.2385 1139.6195 4960.2385" fill="#116fff"/>
</g>
<g id="Graphic_1016">
<path d="M 1139.6195 4936.547 L 1130.7619 4936.547 C 1129.1246 4936.547 1127.7998 4935.2236 1127.7998 4933.585 C 1127.7998 4931.9476 1129.1246 4930.623 1130.7619 4930.623 L 1139.6195 4930.623 C 1141.2567 4930.623 1142.5816 4931.9476 1142.5816 4933.585 C 1142.5816 4935.2236 1141.2567 4936.547 1139.6195 4936.547" fill="#116fff"/>
</g>
<g id="Graphic_1015">
<path d="M 1127.794 4986.886 C 1127.0361 4986.886 1126.2782 4986.5955 1125.6997 4986.0184 L 1113.8641 4974.1785 C 1112.7055 4973.02 1112.7055 4971.1484 1113.8641 4969.99 C 1115.0212 4968.8313 1116.8942 4968.8313 1118.0513 4969.99 L 1129.8883 4981.83 C 1131.0469 4982.9883 1131.0469 4984.86 1129.8883 4986.0184 C 1129.3112 4986.5955 1128.5533 4986.886 1127.794 4986.886" fill="#116fff"/>
</g>
<g id="Graphic_1014">
<path d="M 1115.97 4986.886 C 1115.2106 4986.886 1114.4527 4986.5955 1113.8756 4986.0184 C 1112.7171 4984.86 1112.7171 4982.9883 1113.8756 4981.83 L 1125.7127 4969.993 C 1126.8698 4968.834 1128.7428 4968.834 1129.8999 4969.993 C 1131.0584 4971.151 1131.0584 4973.023 1129.8999 4974.1814 L 1118.0643 4986.0184 C 1117.4858 4986.5955 1116.7279 4986.886 1115.97 4986.886" fill="#116fff"/>
</g>
</g>
</g>
</g>
</svg>

Before

Width:  |  Height:  |  Size: 27 KiB

@ -1,18 +0,0 @@
FROM smallstep/step-cli:0.9.0
ENV CA_NAME="Autocert"
ENV CA_DNS="ca.step.svc.cluster.local,127.0.0.1"
ENV CA_ADDRESS=":4443"
ENV CA_DEFAULT_PROVISIONER="admin"
ENV CA_URL="ca.step.svc.cluster.local"
ENV KUBE_LATEST_VERSION="v1.14.0"
USER root
RUN curl -L https://storage.googleapis.com/kubernetes-release/release/${KUBE_LATEST_VERSION}/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl \
&& chmod +x /usr/local/bin/kubectl
RUN apk --update add expect
COPY autocert.sh /home/step/
RUN chmod +x /home/step/autocert.sh
CMD ["/home/step/autocert.sh"]

@ -1,164 +0,0 @@
#!/bin/bash
#set -x
echo "Welcome to Autocert configuration. Press return to begin."
read ANYKEY
STEPPATH=/home/step/.step
CA_PASSWORD=$(head /dev/urandom | tr -dc A-Za-z0-9 | head -c 32 ; echo '')
AUTOCERT_PASSWORD=$(head /dev/urandom | tr -dc A-Za-z0-9 | head -c 32 ; echo '')
echo -e "\e[1mChecking cluster permissions...\e[0m"
function permission_error {
# TODO: Figure out the actual service account instead of assuming default.
echo
echo -e "\033[0;31mPERMISSION ERROR\033[0m"
echo "Set permissions by running the following command, then try again:"
echo -e "\e[1m"
echo " kubectl create clusterrolebinding autocert-init-binding \\"
echo " --clusterrole cluster-admin \\"
echo " --user \"system:serviceaccount:default:default\""
echo -e "\e[0m"
echo "Once setup is complete you can remove this binding by running:"
echo -e "\e[1m"
echo " kubectl delete clusterrolebinding autocert-init-binding"
echo -e "\e[0m"
exit 1
}
echo -n "Checking for permission to create step namespace: "
kubectl auth can-i create namespaces
if [ $? -ne 0 ]; then
permission_error "create step namespace"
fi
echo -n "Checking for permission to create configmaps in step namespace: "
kubectl auth can-i create configmaps --namespace step
if [ $? -ne 0 ]; then
permission_error "create configmaps"
fi
echo -n "Checking for permission to create secrets in step namespace: "
kubectl auth can-i create secrets --namespace step
if [ $? -ne 0 ]; then
permission_error "create secrets"
fi
echo -n "Checking for permission to create deployments in step namespace: "
kubectl auth can-i create deployments --namespace step
if [ $? -ne 0 ]; then
permission_error "create deployments"
fi
echo -n "Checking for permission to create services in step namespace: "
kubectl auth can-i create services --namespace step
if [ $? -ne 0 ]; then
permission_error "create services"
fi
echo -n "Checking for permission to create cluster role: "
kubectl auth can-i create clusterrole
if [ $? -ne 0 ]; then
permission_error "create cluster roles"
fi
echo -n "Checking for permission to create cluster role binding:"
kubectl auth can-i create clusterrolebinding
if [ $? -ne 0 ]; then
permission_error "create cluster role bindings"
exit 1
fi
# Setting this here on purpose, after the above section which explicitly checks
# for and handles exit errors.
set -e
step ca init \
--name "$CA_NAME" \
--dns "$CA_DNS" \
--address "$CA_ADDRESS" \
--provisioner "$CA_DEFAULT_PROVISIONER" \
--with-ca-url "$CA_URL" \
--password-file <(echo "$CA_PASSWORD")
echo
echo -e "\e[1mCreating autocert provisioner...\e[0m"
expect <<EOD
spawn step ca provisioner add autocert --create
expect "Please enter a password to encrypt the provisioner private key? \\\\\\[leave empty and we'll generate one\\\\\\]: "
send "${AUTOCERT_PASSWORD}\n"
expect eof
EOD
echo
echo -e "\e[1mCreating step namespace and preparing environment...\e[0m"
kubectl create namespace step
kubectl -n step create configmap config --from-file $(step path)/config
kubectl -n step create configmap certs --from-file $(step path)/certs
kubectl -n step create configmap secrets --from-file $(step path)/secrets
kubectl -n step create secret generic ca-password --from-literal "password=${CA_PASSWORD}"
kubectl -n step create secret generic autocert-password --from-literal "password=${AUTOCERT_PASSWORD}"
# Deploy CA and wait for rollout to complete
echo
echo -e "\e[1mDeploying certificate authority...\e[0m"
kubectl apply -f https://raw.githubusercontent.com/smallstep/certificates/autocert/autocert/install/01-step-ca.yaml
kubectl -n step rollout status deployment/ca
# Deploy autocert, setup RBAC, and wait for rollout to complete
echo
echo -e "\e[1mDeploying autocert...\e[0m"
kubectl apply -f https://raw.githubusercontent.com/smallstep/certificates/autocert/autocert/install/02-autocert.yaml
kubectl apply -f https://raw.githubusercontent.com/smallstep/certificates/autocert/autocert/install/03-rbac.yaml
kubectl -n step rollout status deployment/autocert
# Some `base64`s wrap lines... no thanks!
CA_BUNDLE=$(cat $(step path)/certs/root_ca.crt | base64 | tr -d '\n')
cat <<EOF | kubectl apply -f -
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
name: autocert-webhook-config
labels: {app: autocert}
webhooks:
- name: autocert.step.sm
clientConfig:
service:
name: autocert
namespace: step
path: "/mutate"
caBundle: $CA_BUNDLE
rules:
- operations: ["CREATE"]
apiGroups: [""]
apiVersions: ["v1"]
resources: ["pods"]
failurePolicy: Ignore
namespaceSelector:
matchLabels:
autocert.step.sm: enabled
EOF
FINGERPRINT=$(step certificate fingerprint $(step path)/certs/root_ca.crt)
echo
echo -e "\e[1mAutocert installed!\e[0m"
echo
echo "Store this information somewhere safe:"
echo " CA & admin provisioner password: ${CA_PASSWORD}"
echo " Autocert password: ${AUTOCERT_PASSWORD}"
echo " CA Fingerprint: ${FINGERPRINT}"
echo

@ -1,87 +0,0 @@
apiVersion: v1
kind: Service
metadata:
labels:
app: ca
name: ca
namespace: step
spec:
type: ClusterIP
ports:
- port: 443
targetPort: 4443
selector:
app: ca
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ca
namespace: step
labels:
app: ca
spec:
replicas: 1
selector:
matchLabels:
app: ca
template:
metadata:
labels:
app: ca
spec:
containers:
- name: ca
image: smallstep/step-ca:0.8.3
env:
- name: PWDPATH
value: /home/step/password/password
resources:
requests:
cpu: 100m
memory: 20Mi
readinessProbe:
httpGet:
path: /health
port: 4443
scheme: HTTPS
initialDelaySeconds: 3
periodSeconds: 3
livenessProbe:
httpGet:
path: /health
port: 4443
scheme: HTTPS
initialDelaySeconds: 3
periodSeconds: 3
volumeMounts:
- name: config
mountPath: /home/step/.step/config
readOnly: true
- name: certs
mountPath: /home/step/.step/certs
readOnly: true
- name: secrets
mountPath: /home/step/.step/secrets
readOnly: true
- name: ca-password
mountPath: /home/step/password
readOnly: true
securityContext:
runAsUser: 1000
allowPrivilegeEscalation: false
volumes:
- name: certs
configMap:
name: certs
- name: config
configMap:
name: config
- name: secrets
configMap:
name: secrets
- name: ca-password
secret:
secretName: ca-password

@ -1,108 +0,0 @@
apiVersion: v1
kind: Service
metadata:
labels: {app: autocert}
name: autocert
namespace: step
spec:
type: ClusterIP
ports:
- port: 443
targetPort: 4443
selector: {app: autocert}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: autocert-config
namespace: step
data:
config.yaml: |
logFormat: json # or text
restrictCertificatesToNamespace: false
clusterDomain: cluster.local
caUrl: https://ca.step.svc.cluster.local
certLifetime: 24h
renewer:
name: autocert-renewer
image: smallstep/autocert-renewer:0.8.3
resources: {requests: {cpu: 10m, memory: 20Mi}}
imagePullPolicy: IfNotPresent
volumeMounts:
- name: certs
mountPath: /var/run/autocert.step.sm
bootstrapper:
name: autocert-bootstrapper
image: smallstep/autocert-bootstrapper:0.8.3
resources: {requests: {cpu: 10m, memory: 20Mi}}
imagePullPolicy: IfNotPresent
volumeMounts:
- name: certs
mountPath: /var/run/autocert.step.sm
certsVolume:
name: certs
emptyDir: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: autocert
namespace: step
labels: {app: autocert}
spec:
replicas: 1
selector: {matchLabels: {app: autocert}}
template:
metadata: {labels: {app: autocert}}
spec:
containers:
- name: autocert
image: smallstep/autocert-controller:0.8.3
resources: {requests: {cpu: 100m, memory: 20Mi}}
env:
- name: PROVISIONER_NAME
value: autocert
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: config
mountPath: /home/step/.step/config
readOnly: true
- name: certs
mountPath: /home/step/.step/certs
readOnly: true
- name: autocert-password
mountPath: /home/step/password
readOnly: true
- name: autocert-config
mountPath: /home/step/autocert
readOnly: true
securityContext:
runAsUser: 1000
allowPrivilegeEscalation: false
livenessProbe:
httpGet:
path: /healthz
port: 4443
scheme: HTTPS
readinessProbe:
httpGet:
path: /healthz
port: 4443
scheme: HTTPS
volumes:
- name: config
configMap: {name: config}
- name: certs
configMap: {name: certs}
- name: autocert-password
secret: {secretName: autocert-password}
- name: autocert-config
configMap: {name: autocert-config}

@ -1,36 +0,0 @@
# Create a ClusterRole for managing autocert secrets, which should
# only exist in namespaces with autocert enabled and should always
# be labeled `autocert.step.sm/token: true`.
#
# To create this ClusterRole you need cluster-admin privileges. On
# GKE you can give yourself cluster-admin privileges using the
# following command:
#
# kubectl create clusterrolebinding cluster-admin-binding \
# --clusterrole cluster-admin \
# --user $(gcloud config get-value account)
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: autocert-controller
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: autocert-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: autocert-controller
subjects:
- kind: ServiceAccount
name: default
namespace: step

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.3 MiB

@ -1,8 +0,0 @@
FROM smallstep/step-cli:0.9.0
USER root
ENV CRT="/var/run/autocert.step.sm/site.crt"
ENV KEY="/var/run/autocert.step.sm/site.key"
ENV STEP_ROOT="/var/run/autocert.step.sm/root.crt"
ENTRYPOINT ["/bin/bash", "-c", "step ca renew --daemon $CRT $KEY"]
Loading…
Cancel
Save