Running AWX on Kubernetes with HashiCorp Vault Secrets Injector: A Seamless Integration for Secure Automation

In the ever-evolving landscape of IT automation, AWX serves as a powerful web-based user interface for Ansible, streamlining complex and repetitive tasks within your Kubernetes environment. To enhance security and streamline the deployment process, combining AWX with HashiCorp Vault's Secrets Injector offers a robust solution for managing sensitive data and secrets.

In this post, we'll explore how to deploy AWX on Kubernetes and securely manage your secrets using HashiCorp Vault. We'll also delve into using Instance Groups in AWX for tailored execution environments, which enhance flexibility and efficiency when running Ansible playbooks in combination with HashiCorp Vault.

Prerequisites

Before we begin, ensure that you have the following:

• A Kubernetes cluster (Minikube or any other distribution).
• kubectl and helm installed on your local machine.
• AWX Operator for managing AWX deployments.
• An instance of HashiCorp Vault configured with the Kubernetes Secrets Engine.
• Administrative access to your Kubernetes cluster.

Deploying AWX on Kubernetes

Install the AWX Operator**

The AWX Operator simplifies deploying and managing AWX on Kubernetes. First, install it using kustomize:

1---
2apiVersion: kustomize.config.k8s.io/v1beta1
3kind: Kustomization
4namespace: awx
5resources:
6  - github.com/ansible/awx-operator/config/default?ref=2.19.1

Deploy AWX

Create a custom resource definition (CRD) to deploy AWX:

 1apiVersion: awx.ansible.com/v1beta1
 2kind: AWX
 3metadata:
 4    name: awx
 5    namespace: awx
 6spec:
 7    service_type: ClusterIP
 8    ingress_type: ingress
 9    ingress_hosts:
10    - hostname: awx.example.com
11    postgres_storage_size: 8Gi  # Adjust storage size as needed
12create_preload_data: false

Apply the configuration:

1kubectl apply -f awx-deployment.yaml

Integrating HashiCorp Vault Secrets Injector

Vault's Secrets Injector is a Kubernetes Mutating Admission Webhook for injecting secrets directly into application pods.

Enable the Secrets Injector

Deploy the Vault Agent Injector using Helm:

1helm repo add hashicorp https://helm.releases.hashicorp.com
2
3helm install vault hashicorp/vault \
4    --set "global.externalVaultAddr=https://vault.example.com:8200" \
5    --set "server.enabled=false" \
6    --set "injector.enabled=true" \
7    --set "injector.authPath=auth/kubernetes"

Configure Vault Policies and Roles

Define a policy in Vault to allow access to secrets:

1path "secret/data/*" {
2    capabilities = ["read"]
3}

1vault policy write awx-access ./awx-access.hcl

Enable the Kubernetes authentication method and bind a role to it:

1vault auth enable kubernetes
2
3vault write auth/kubernetes/role/awx \
4    bound_service_account_names=awx-execution \
5    bound_service_account_namespaces=awx \
6    policies=default,awx-access \
7    ttl=1h

Verify injection of secrets into Pods

1apiVersion: v1
2kind: ServiceAccount
3metadata:
4  name: awx-execution
5  namespace: awx

Annotate your test pods to ensure the Vault Injector injects necessary secrets:

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4    name: awx-pod
 5    namespace: awx
 6    annotations:
 7        vault.hashicorp.com/agent-inject: "true"
 8        vault.hashicorp.com/role: "awx"
 9        vault.hashicorp.com/agent-inject-secret-database: "secret/data/mysql"
10spec:
11    serviceAccountName: awx-execution
12    automountServiceAccountToken: true
13    ...

Customizing AWX Execution with Instance Groups

AWX Instance Groups allow you to segregate environments based on custom criteria. Instance groups in AWX can be defined to route tasks based on specific resource requirements or workloads. We are going to use this feature to enhance our exection pod with some annotations for Vault injections.

 1terraform {
 2  required_providers {
 3    awx = {
 4      source = "ilijamt/awx"
 5    }
 6  }
 7}
 8variable "awx_user" {
 9  type    = string
10  default = "admin"
11}
12variable "awx_password" {
13  type      = string
14  ephemeral = true
15  sensitive = true
16}
17provider "awx" {
18  hostname = "https://awx.examples.com"
19  username = var.awx_user
20  password = var.awx_password
21}
22
23resource "awx_instance_group" "vault_enabled_instances" {
24    name = "vault-enabled"
25    is_container_group = true
26    pod_spec_override = <<EOT
27apiVersion: v1
28kind: Pod
29metadata:
30  namespace: awx
31  annotations:
32    "vault.hashicorp.com/agent-inject": "true"
33    "vault.hashicorp.com/role": "awx"
34    "vault-hashicorp-com-agent-inject-token": "true"
35    "vault.hashicorp.com/agent-inject-secret-token.sh: |
36        {{- with secret "auth/token/lookup-self" -}}
37            export VAULT_TOKEN="{{ .Data.id }}"
38        {{- end }}
39spec:
40  serviceAccountName: awx-operator
41  automountServiceAccountToken: true
42  containers:
43    - image: quay.io/ansible/awx-ee:latest
44      name: worker
45      command: [ "source /vault/secrets/token.sh && /bin/bash -c" ]
46      args:
47        - ansible-runner
48        - worker
49        - '--private-data-dir=/runner'
50      resources:
51        requests:
52          cpu: 250m
53          memory: 100Mi
54EOT
55}

Assign Playbooks to Instance Groups

When configuring a job template in AWX, assign it to the newly created instance group to ensure playbooks run in a tailored environment. All the playbooks will have a valid vault token in place to interact with vault to access data related to the policy we created.

A more flexible way would be to pass a token to the pod container that allows the playbooks to authenticate against approle roles. According to the actual role configuration different permissions can be granted.

Or you create multiple instance groups related to kubernetes roles that define different permissions on the vault resources.

Conclusion

Combining AWX with HashiCorp Vault on Kubernetes enhances your infrastructure's automation capabilities while maintaining high security standards for sensitive data. By leveraging AWX's instance groups, you can create highly customizable and efficient environments tailored to your specific automation needs. This integration empowers your DevOps practices, promoting both agility and security within your organizational workflows.

Start building your robust automation solutions today with AWX and HashiCorp Vault!