In a cloud-native Kubernetes environment, secrets management is a critical aspect of security. HashiCorp Vault is a popular tool for managing secrets and protecting sensitive data. When combined with the Vault Agent Injector for Kubernetes, you can create a powerful and secure platform for running your workloads with seamless secret injection.
Note: This article builds upon the concepts introduced in our previous post about HashiCorp Nomad and Vault: Dynamic Secrets, where we explored similar patterns using HashiCorp Nomad. Here, we translate those concepts to a Kubernetes environment using the Vault Agent Injector.
HashiCorp Vault provides a dynamic secrets engine that generates secrets on-demand. This feature allows you to create short-lived credentials for databases, cloud providers, and other services. By using dynamic secrets, you can reduce the risk of exposure and limit the lifespan of sensitive data.
In this example, we will deploy a simple web application using Kubernetes. The application requires access to a MySQL database, and we will use HashiCorp Vault with the Agent Injector to generate dynamic credentials for the database, use HashiCorp Vault transit engine to encrypt the database values on the fly, and leverage Kubernetes native service discovery.
vault.example.com:8200Since Vault is running externally, we only need to install the Vault Agent Injector:
1helm repo add hashicorp https://helm.releases.hashicorp.com
2helm repo update
3
4# Install only the Agent Injector (no Vault server)
5helm install vault-injector hashicorp/vault \
6 --set "global.externalVaultAddr=https://vault.example.com:8200" \
7 --set "injector.enabled=true" \
8 --set "server.enabled=false" \
9 --set "csi.enabled=false"
Set up Kubernetes authentication in your external Vault cluster:
1# Set Vault address environment variable
2export VAULT_ADDR="https://vault.example.com:8200"
3
4# Enable Kubernetes auth method
5vault auth enable kubernetes
6
7# Get Kubernetes cluster information
8KUBERNETES_HOST=$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.server}')
9KUBERNETES_CA_CERT=$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 -d)
10
11# Create a service account for Vault authentication
12kubectl create serviceaccount vault-auth
13kubectl apply -f - <<EOF
14apiVersion: rbac.authorization.k8s.io/v1
15kind: ClusterRoleBinding
16metadata:
17 name: role-tokenreview-binding
18roleRef:
19 apiGroup: rbac.authorization.k8s.io
20 kind: ClusterRole
21 name: system:auth-delegator
22subjects:
23- kind: ServiceAccount
24 name: vault-auth
25 namespace: default
26EOF
27
28# Get the JWT token
29TOKEN_REVIEWER_JWT=$(kubectl create token vault-auth)
30
31# Configure Kubernetes auth
32vault write auth/kubernetes/config \
33 token_reviewer_jwt="$TOKEN_REVIEWER_JWT" \
34 kubernetes_host="$KUBERNETES_HOST" \
35 kubernetes_ca_cert="$KUBERNETES_CA_CERT"
The actual deployment of the MySQL database is done using the following Kubernetes manifests. The MySQL database is started as a deployment and service with the root password set as an environment variable.
1# mysql-deployment.yaml
2apiVersion: apps/v1
3kind: Deployment
4metadata:
5 name: mysql-server
6 namespace: demo
7 labels:
8 app: mysql-server
9spec:
10 replicas: 1
11 selector:
12 matchLabels:
13 app: mysql-server
14 template:
15 metadata:
16 labels:
17 app: mysql-server
18 spec:
19 containers:
20 - name: mysql
21 image: mysql:9
22 env:
23 - name: MYSQL_ROOT_PASSWORD
24 value: "super-duper-password"
25 ports:
26 - containerPort: 3306
27 name: mysql
28 resources:
29 requests:
30 cpu: 500m
31 memory: 1Gi
32 limits:
33 cpu: 500m
34 memory: 1Gi
35
36---
37# mysql-service.yaml
38apiVersion: v1
39kind: Service
40metadata:
41 name: mysql-server
42 namespace: demo
43spec:
44 selector:
45 app: mysql-server
46 ports:
47 - port: 3306
48 targetPort: 3306
49 name: mysql
50 type: ClusterIP
As a first step, we will deploy the web application with hardcoded credentials. That might be a workflow you are familiar with - start with less secure settings and improve them step by step.
The web application is just a simple Python Flask application that connects to the MySQL database and displays the database values on a web page, and as an alternative view it can display the plain database values.
1# dynamic-app-hardcoded.yaml
2apiVersion: apps/v1
3kind: Deployment
4metadata:
5 name: dynamic-app-hardcoded
6 namespace: demo
7 labels:
8 app: dynamic-app
9 version: hardcoded
10spec:
11 replicas: 1
12 selector:
13 matchLabels:
14 app: dynamic-app
15 version: hardcoded
16 template:
17 metadata:
18 labels:
19 app: dynamic-app
20 version: hardcoded
21 spec:
22 containers:
23 - name: dynamic-app
24 image: ghcr.io/infralovers/nomad-vault-mysql:1.0.0
25 ports:
26 - containerPort: 8080
27 name: web
28 env:
29 - name: CONFIG_FILE
30 value: "/app/config/config.ini"
31 volumeMounts:
32 - name: config
33 mountPath: /app/config
34 resources:
35 requests:
36 cpu: 256m
37 memory: 256Mi
38 limits:
39 cpu: 256m
40 memory: 256Mi
41 livenessProbe:
42 httpGet:
43 path: /health
44 port: 8080
45 initialDelaySeconds: 30
46 periodSeconds: 10
47 readinessProbe:
48 httpGet:
49 path: /health
50 port: 8080
51 initialDelaySeconds: 5
52 periodSeconds: 5
53 volumes:
54 - name: config
55 configMap:
56 name: dynamic-app-config-hardcoded
57
58---
59# ConfigMap with hardcoded credentials
60apiVersion: v1
61kind: ConfigMap
62metadata:
63 name: dynamic-app-config-hardcoded
64 namespace: demo
65data:
66 config.ini: |
67 [DEFAULT]
68 Port = 8080
69
70 [DATABASE]
71 Address = mysql-server.demo.svc.cluster.local
72 Port = 3306
73 Database = my_app
74 User = root
75 Password = super-duper-password
76
77---
78# Service
79apiVersion: v1
80kind: Service
81metadata:
82 name: dynamic-app-hardcoded
83 namespace: demo
84spec:
85 selector:
86 app: dynamic-app
87 version: hardcoded
88 ports:
89 - port: 80
90 targetPort: 8080
91 name: web
92 type: ClusterIP
The next step to improve our journey to get more secure deployments is to use the key value secret engine of HashiCorp Vault. This engine allows you to store and retrieve arbitrary secrets. In this example, we will store the database credentials in Vault and retrieve them at runtime using the Vault Agent Injector.
First, configure Vault:
1# Enable KV secrets engine
2vault secrets enable -path=dynamic-app/kv kv-v2
3
4# Store database credentials
5vault kv put dynamic-app/kv/database username=root password=super-duper-password
Create a Vault policy that grants read access to the dynamic-app/kv path:
1vault policy write dynamic-app-kv - <<EOF
2path "dynamic-app/kv/data/database" {
3 capabilities = ["read"]
4}
5EOF
6
7# Create a Kubernetes role
8vault write auth/kubernetes/role/dynamic-app-kv \
9 bound_service_account_names=dynamic-app-kv \
10 bound_service_account_namespaces=demo \
11 policies=dynamic-app-kv \
12 ttl=24h
Now deploy the application using Vault Agent Injector:
1# dynamic-app-kv.yaml
2apiVersion: v1
3kind: ServiceAccount
4metadata:
5 name: dynamic-app-kv
6 namespace: demo
7
8---
9apiVersion: apps/v1
10kind: Deployment
11metadata:
12 name: dynamic-app-kv
13 namespace: demo
14 labels:
15 app: dynamic-app
16 version: kv
17spec:
18 replicas: 1
19 selector:
20 matchLabels:
21 app: dynamic-app
22 version: kv
23 template:
24 metadata:
25 labels:
26 app: dynamic-app
27 version: kv
28 annotations:
29 vault.hashicorp.com/agent-inject: "true"
30 vault.hashicorp.com/agent-inject-status: "update"
31 vault.hashicorp.com/agent-inject-vault-addr: "https://vault.example.com:8200"
32 vault.hashicorp.com/role: "dynamic-app-kv"
33 vault.hashicorp.com/agent-inject-secret-config.ini: "dynamic-app/kv/data/database"
34 vault.hashicorp.com/agent-inject-template-config.ini: |
35 [DEFAULT]
36 Port = 8080
37
38 [DATABASE]
39 Address = mysql-server.demo.svc.cluster.local
40 Port = 3306
41 Database = my_app
42 User = {{ .Data.data.username }}
43 Password = {{ .Data.data.password }}
44 spec:
45 serviceAccountName: dynamic-app-kv
46 containers:
47 - name: dynamic-app
48 image: ghcr.io/infralovers/nomad-vault-mysql:1.0.0
49 ports:
50 - containerPort: 8080
51 name: web
52 env:
53 - name: CONFIG_FILE
54 value: "/vault/secrets/config.ini"
55 resources:
56 requests:
57 cpu: 256m
58 memory: 256Mi
59 limits:
60 cpu: 256m
61 memory: 256Mi
62 livenessProbe:
63 httpGet:
64 path: /health
65 port: 8080
66 initialDelaySeconds: 30
67 periodSeconds: 10
68 readinessProbe:
69 httpGet:
70 path: /health
71 port: 8080
72 initialDelaySeconds: 5
73 periodSeconds: 5
74
75---
76# Service
77apiVersion: v1
78kind: Service
79metadata:
80 name: dynamic-app-kv
81 namespace: demo
82spec:
83 selector:
84 app: dynamic-app
85 version: kv
86 ports:
87 - port: 80
88 targetPort: 8080
89 name: web
90 type: ClusterIP
The almost final step in our journey is to use the dynamic secrets engine of HashiCorp Vault. This engine generates short-lived credentials for databases, cloud providers, and other services. In this example, we will use the MySQL database secrets engine to generate dynamic credentials for the database.
Configure the database secrets engine in Vault:
1# Enable database secrets engine
2vault secrets enable -path=dynamic-app/db database
3
4# Configure MySQL connection
5vault write dynamic-app/db/config/mysql \
6 plugin_name=mysql-database-plugin \
7 connection_url="{{username}}:{{password}}@tcp(mysql-server.demo.svc.cluster.local:3306)/" \
8 allowed_roles="*" \
9 username="root" \
10 password="super-duper-password"
11
12# Rotate root credentials
13vault write -force dynamic-app/db/database/rotate-root/mysql
14
15# Create a role for dynamic credentials
16vault write dynamic-app/db/roles/app \
17 db_name=mysql \
18 creation_statements="CREATE USER '{{name}}'@'%' IDENTIFIED BY '{{password}}';GRANT ALL ON my_app.* TO '{{name}}'@'%';" \
19 default_ttl="1h" \
20 max_ttl="24h"
Update the Vault policy to allow access to dynamic secrets:
1vault policy write dynamic-app-db - <<EOF
2path "dynamic-app/db/creds/app" {
3 capabilities = ["read"]
4}
5EOF
6
7# Create a Kubernetes role for dynamic secrets
8vault write auth/kubernetes/role/dynamic-app-db \
9 bound_service_account_names=dynamic-app-db \
10 bound_service_account_namespaces=demo \
11 policies=dynamic-app-db \
12 ttl=24h
Deploy the application with dynamic secrets:
1# dynamic-app-db.yaml
2apiVersion: v1
3kind: ServiceAccount
4metadata:
5 name: dynamic-app-db
6 namespace: demo
7
8---
9apiVersion: apps/v1
10kind: Deployment
11metadata:
12 name: dynamic-app-db
13 namespace: demo
14 labels:
15 app: dynamic-app
16 version: dynamic-db
17spec:
18 replicas: 1
19 selector:
20 matchLabels:
21 app: dynamic-app
22 version: dynamic-db
23 template:
24 metadata:
25 labels:
26 app: dynamic-app
27 version: dynamic-db
28 annotations:
29 vault.hashicorp.com/agent-inject: "true"
30 vault.hashicorp.com/agent-inject-status: "update"
31 vault.hashicorp.com/agent-inject-vault-addr: "https://vault.example.com:8200"
32 vault.hashicorp.com/role: "dynamic-app-db"
33 vault.hashicorp.com/agent-inject-secret-config.ini: "dynamic-app/db/creds/app"
34 vault.hashicorp.com/agent-inject-template-config.ini: |
35 [DEFAULT]
36 Port = 8080
37
38 [DATABASE]
39 Address = mysql-server.demo.svc.cluster.local
40 Port = 3306
41 Database = my_app
42 User = {{ .Data.username }}
43 Password = {{ .Data.password }}
44 spec:
45 serviceAccountName: dynamic-app-db
46 containers:
47 - name: dynamic-app
48 image: ghcr.io/infralovers/nomad-vault-mysql:1.0.0
49 ports:
50 - containerPort: 8080
51 name: web
52 env:
53 - name: CONFIG_FILE
54 value: "/vault/secrets/config.ini"
55 resources:
56 requests:
57 cpu: 256m
58 memory: 256Mi
59 limits:
60 cpu: 256m
61 memory: 256Mi
62 livenessProbe:
63 httpGet:
64 path: /health
65 port: 8080
66 initialDelaySeconds: 30
67 periodSeconds: 10
68 readinessProbe:
69 httpGet:
70 path: /health
71 port: 8080
72 initialDelaySeconds: 5
73 periodSeconds: 5
74
75---
76# Service
77apiVersion: v1
78kind: Service
79metadata:
80 name: dynamic-app-db
81 namespace: demo
82spec:
83 selector:
84 app: dynamic-app
85 version: dynamic-db
86 ports:
87 - port: 80
88 targetPort: 8080
89 name: web
90 type: ClusterIP
The final step in our journey is to use the transit engine of HashiCorp Vault to encrypt the database values on the fly. This engine provides a way to encrypt and decrypt data without storing the encryption keys. In this example, we will use the transit engine to encrypt the database information before storing them in the database.
Configure the transit engine:
1# Enable transit secrets engine
2vault secrets enable -path=dynamic-app/transit transit
3
4# Create an encryption key
5vault write -f dynamic-app/transit/keys/app
Update the Vault policy to include transit engine access:
1vault policy write dynamic-app-full - <<EOF
2path "dynamic-app/db/creds/app" {
3 capabilities = ["read"]
4}
5path "dynamic-app/transit/encrypt/app" {
6 capabilities = ["create", "update"]
7}
8path "dynamic-app/transit/decrypt/app" {
9 capabilities = ["create", "update"]
10}
11EOF
12
13# Create a Kubernetes role for full access
14vault write auth/kubernetes/role/dynamic-app-full \
15 bound_service_account_names=dynamic-app-full \
16 bound_service_account_namespaces=demo \
17 policies=dynamic-app-full \
18 ttl=24h
Deploy the application with transit encryption:
1# dynamic-app-full.yaml
2apiVersion: v1
3kind: ServiceAccount
4metadata:
5 name: dynamic-app-full
6 namespace: demo
7
8---
9apiVersion: apps/v1
10kind: Deployment
11metadata:
12 name: dynamic-app-full
13 namespace: demo
14 labels:
15 app: dynamic-app
16 version: full
17spec:
18 replicas: 1
19 selector:
20 matchLabels:
21 app: dynamic-app
22 version: full
23 template:
24 metadata:
25 labels:
26 app: dynamic-app
27 version: full
28 annotations:
29 vault.hashicorp.com/agent-inject: "true"
30 vault.hashicorp.com/agent-inject-status: "update"
31 vault.hashicorp.com/agent-inject-vault-addr: "https://vault.example.com:8200"
32 vault.hashicorp.com/role: "dynamic-app-full"
33 vault.hashicorp.com/agent-inject-secret-config.ini: "dynamic-app/db/creds/app"
34 vault.hashicorp.com/agent-inject-template-config.ini: |
35 [DEFAULT]
36 Port = 8080
37
38 [DATABASE]
39 Address = mysql-server.demo.svc.cluster.local
40 Port = 3306
41 Database = my_app
42 User = {{ .Data.username }}
43 Password = {{ .Data.password }}
44
45 [VAULT]
46 Enabled = True
47 InjectToken = True
48 Namespace =
49 Address = https://vault.example.com:8200
50 KeyPath = dynamic-app/transit
51 KeyName = app
52 spec:
53 serviceAccountName: dynamic-app-full
54 containers:
55 - name: dynamic-app
56 image: ghcr.io/infralovers/nomad-vault-mysql:1.0.0
57 ports:
58 - containerPort: 8080
59 name: web
60 env:
61 - name: CONFIG_FILE
62 value: "/vault/secrets/config.ini"
63 - name: VAULT_ADDR
64 value: "https://vault.example.com:8200"
65 resources:
66 requests:
67 cpu: 256m
68 memory: 256Mi
69 limits:
70 cpu: 256m
71 memory: 256Mi
72 livenessProbe:
73 httpGet:
74 path: /health
75 port: 8080
76 initialDelaySeconds: 30
77 periodSeconds: 10
78 readinessProbe:
79 httpGet:
80 path: /health
81 port: 8080
82 initialDelaySeconds: 5
83 periodSeconds: 5
84
85---
86# Service
87apiVersion: v1
88kind: Service
89metadata:
90 name: dynamic-app-full
91 namespace: demo
92spec:
93 selector:
94 app: dynamic-app
95 version: full
96 ports:
97 - port: 80
98 targetPort: 8080
99 name: web
100 type: ClusterIP
In this article, we have demonstrated how to use HashiCorp Vault Agent Injector with Kubernetes to deploy a secure web application with dynamic secrets. By using the Vault Agent Injector, you can:
Even if you are starting with hardcoded credentials, the Key Value Secret Engine provides a good foundation to improve your security settings step by step. The next step is to use the dynamic secrets engine to generate short-lived credentials for your services. As demonstrated in the example, it requires only minimal changes in your deployment annotations to use the dynamic secrets engine.
Key benefits of this Kubernetes and Vault Agent Injector approach:
You can find all the code examples in a Kubernetes-compatible format to reproduce this example on your own cluster. As a starting point, you can use existing Kubernetes clusters with an external Vault installation.
If you are interested in learning more about HashiCorp Vault and Kubernetes, check out our Cloud Native Essentials and HashiCorp Vault Enterprise courses. These courses will help you master the tools and techniques needed to build secure and scalable cloud-native applications.
This is an introduction to Kubernetes, an open-source system for automating deployment, scaling, and management of containerized applications.
Introduction to HashiCorp Vault.
In a cloud-native Kubernetes environment, secrets management is a critical aspect of security. HashiCorp Vault is a popular tool for managing secrets and
Remember last July when we dug into Ollama and its promise for local AI? Well, a lot has happened since then. Ollama has rolled out some pretty significant
Infrastructure as Code (IaC) has revolutionized the way organizations manage cloud infrastructure, with Terraform leading as a premier tool. HashiCorp Cloud
Understanding Terraform’s New License: What You Can and Cannot Do Under the Business Source License (BSL) HashiCorp’s Terraform has become the go-to
Introduction Trunk-Based Development (TBD) means everyone integrates to a single branch: usually main or trunk. There are almost no long-lived branches or repo
You are interested in our courses or you simply have a question that needs answering? You can contact us at anytime! We will do our best to answer all your questions.
Contact us