The Cloud Native Edge Router





  • We Believe in Open Source

  • We Deliver Træfik

  • Commercial Support for Træfik

  • 12 people, 90% technical experts

Containous Logo


  • Cloud Native Edge Router

  • Native Ingress Controller

  • LetsEncrypt Integration

Træfik's Logo

Kubernetes Rundown

Assumed Knowledge:

  • Deployments/Services/Pods

  • Ingress Basics (Traffic Flow, Bypassing the service layer)

  • Kubectl and tooling


This Project

End to End Encryption

Why, Mr Anderson?

Why, Mr Anderson?

Who needs it?

  • PCI-DSS Compliance

  • HIPAA Compliance

  • SSAE Compliance

  • gRPC Functionality


Issues with End to End TLS

  • Certificate Generation

    • Especially in dynamic environments

  • File Storage

    • Configuration files, and mounting certificates

  • DNS records

    • For certificate common names, and backend requests

Demo Overview Diagram

This is what I will be showing with the demo


Goals of Demo

  1. No TLS Security disabled

  2. No TLS Verification disabled

  3. No Configuration files

  4. No Manual Certificate file management (Outside of trusting the root)

Parts of a TLS Connection

  • Connection Request is made

  • Certificate is returned:

    • Hostname Validation

    • Trust Chain Validation

  • Then request is sent

Hostname Validation

  • Verifies the CN of the certificate matches the request

    • Easy to ignore

    • If ignored, any public certificate will work

    • This validation is important to prevent MITM attacks

Problems with Hostname Validation in Kubernetes

  • You don’t know the Host/IP Combo in advance

    • How do you generate certificates?

  • Backend proxied requests don’t have the same host as the main request for validation

    • What CN do you use?

Trust Validation

  • Verifying that the certificate is signed by a trusted authority

    • Easy to Ignore

    • Infamous "Self Signed Certs"

    • If ignored, any certificate matching the request name will work

    • This validation is important to prevent MITM attacks

Problems with Trust Validation in Kubernetes

  • CA Certificates are hard to manage, and shouldn’t be distributed

  • Intermediate certificates need to be rotated

  • Generating/Signing certificates is error prone/finicky

What is the solution?


Handle Everything Programmatically

  • Dynamically generate what you need, when you need it

  • Have a Root CA

    • Generated, and stored for safe keeping

    • Used only for signing intermediates

  • Use intermediates for all day-to-day signing

How are we going to achieve this?

  • Two parts:

    • Front End (Client facing) TLS

      • Boulder

    • Back End (Pod Facing) TLS

      • MiniCA

Demo Script

  • Generates Certificates

    • Root CA

    • Intermediates

    • Browser bundle for importing to client trust

  • Bakes intermediates into images (Root also in case of Boulder)

Why use Boulder?

  • Dynamic certificate generation

  • Runs in Docker

  • Complies with ACME spec

  • Leverage Træfik’s ACME client


Our Demo Boulder Image

  • We remap duplicate services to avoid portmapping issues in kubernetes

    • VA, SA, Publisher, etc

  • We mount our custom root + intermediate certs (Baked in)

  • Now we can issue certificates signed with our trust chain

Why use MiniCA?

  • Dynamic certificate generation

  • Written by a member of the LetsEncrypt Team

  • Can use our intermediate to sign certificates

  • Runs in Docker

  • Can run in our pod after the IP and Hostname have been assigned by the kubernetes API

Our Demo MiniCA Image

  • Runs as an init container, and generates our certificates using the downward API

  • Stores the issued certificates in a volume

  • This volume is then mounted and usable by other containers in our pod

  • The signing keys are not available once the init container is gone

  • IP SANs are generated, so the certs are valid for the pod IP

Træfik Glue

  • Træfik is the glue that binds these components together

  • We add our intermediate to the trust store in the Træfik image

    • We bake it in, so we don’t have to manage files

    • It can also be added as a separate file, and specified via CLI arg

  • Træfik will connect to boulder, and will use generated frontend certs

  • Træfik will be able to use trusted certificates to connect to backend pod IPs


  • No certificate files + keys floating around

  • No configuration files floating around

  • MiniCA init container can be use on any application

  • Unlimited (in theory) scaling


Adapting for Production

  • LetsEncrypt already works for frontend certificate generation in production

  • Using MiniCA for internal generation is production-ready

    • Using a custom trust chain is completely production acceptable, assuming:

      • Roots are protected

      • Roots are stored for safe keeping

      • Intermediates are issued for a year or so

      • Leaf certificates are issued for a smaller (90 day) lifetime

Thank You!