HashiCorp Nomad and Vault: Dynamic Secrets
In a cloud-native environment, secrets management is a critical aspect of security. HashiCorp Vault is a popular tool for managing secrets and protecting
Most HashiCorp Vault tutorials - even those from HashiCorp - use the commandline tools or even just curl to configure HashiCorp Vault. But do we want to use curl when we can use terraform Code? Terraform already includes a HashiCorp Vault provider which enables us to do most tasks by using code and apply this to our installation - and maintain also our infrastructure.
We gonna use ansible to rollout the Vault SSH One Time Password authentication to all of our servers within the infrastructure. With this change HashiCorp Vault enables us to
So nobody has to copy&paste ssh keys or even worse mail the password of a machine over the network.
Let's start with HashiCorp Vault configuration by terraform to provide an endpoint which provides one time passwords for the authentication on our infrastructure.
The first code block defines our vault installation and creates a secret backend in vault of type ssh
. This is mounted to the path ssh
. This can be changed, but has to be adapted later on within the ansible code. Within the terraform code the backend path is passed through the items. As a default we set the lease time of a secret for 4 hours and a maximum of 1 week.
provider "vault" {
address = "http://vault.local:8200/"
}
resource "vault_mount" "ssh" {
type = "ssh"
path = "ssh"
default_lease_ttl_seconds = "14400" # 4h
max_lease_ttl_seconds = "604800" # 1w
}
Next we gonna create a role within the ssh backend of type otp = One Time Password. Within this resource we set some parameters to allow only certains usernames on the connection - in this sample these are centos
and ubuntu
. We also allow connecting users to forward a port over this ssh connection and open a interactive shell on the servers. The secret backend role is also defined to be valid for a subnet based on cidr "192.168.0.0/24".
And last but not least we limit the lifetime of the otp to 30 minutes.
variable "ssh_cidr" {
default = "192.168.0.0/24"
}
resource "vault_ssh_secret_backend_role" "otp" {
name = "otp"
backend = vault_mount.ssh.path
key_type = "otp"
allowed_extensions = "permit-pty,permit-port-forwarding"
default_user = "centos"
allowed_users = "centos,ubuntu"
cidr_list = var.ssh_cidr
ttl = "30m"
}
So that our clients can create an otp we gonna create and policy to allow them access to this endpoint:
resource "vault_policy" "ssh_client_access" {
name = "ssh_clients"
policy = <<EOT
path "${vault_mount.ssh.path}/creds/otp" {
capabilities = ["create", "read", "update"]
}
EOT
}
Next we can apply this change to our HashiCorp Vault:
1export VAULT_TOKEN="my-vault-token"
2terraform apply
At this point HashiCorp Vault is able to serve One Time Passwords for SSH Authentication and also allow our infrastructure to validate these passwords against HashiCorp Vault.
Ansible has already some community written HashiCorp vault roles, but none of them was up-to-date at the time of writing this article. So we forked a solution and updated this with some fixxes and changes to provide a solution for us. You can use this role by following ansible-galaxy
command:
1ansible-galaxy install https://github.com/infralovers/ansible-vault-otp
The role uses the HashiCorp vault-ssh-helper in its core to reconfigure the infrastructure authentication mechanism. This helper validates the login prompt by ssh with our vault configuration that the provided password is a valid one time password provided by HashiCorp Vault.
To use this role you create a playbook which includes the role and sets the required variables:
1- hosts: all
2 become: yes
3 vars:
4 - vault_addr: http://vault.local:8200
5 - ssh_mount_point: ssh
6 roles:
7 - role: infralovers.vault_otp
When using an https endpoint you also have to set vault_ca_cert_file
variable with a path to the certificate file.
When consuming the OTP funcitonality we have to use curl to get this done - but we can write a function and add this content to e.g. .bashrc
. The following function requests an one time password from HashiCorp Vault for a certain ip address or hostname. The function lookups the hostname within the ansible inventory.
If this did not work the ssh configuration is used for a lookup on the ip address. And if this should not work either, a dns query is done to get the ip address of the host.
As optional parameter also the username of the connection can be passed to the function - but it defaults to ubuntu.
1function vault_otp() {
2
3 VAULT_ADDR="http://vault.local:8200"
4 SSH_MOUNT="ssh"
5 SSH_ROLE="otp"
6 ANSIBLE_INVENTORY=${ANSIBLE_INVENTORY:-/full/path/to/hosts.ini}
7 REMOTE_USER="ubuntu"
8
9 if [ -z "$VAULT_TOKEN" ]; then
10 echo "missing vault token!"
11 return
12 fi
13 HOSTNAME=$1
14 if [ -n "$2" ]; then
15 REMOTE_USER="$2"
16 fi
17 if [ -z "$VAULT_TOKEN" ]; then
18 echo "missing vault token!"
19 return
20 fi
21
22 IP=$(ansible-inventory -i "$ANSIBLE_INVENTORY" --host "$HOSTNAME" 2>/dev/null | jq -r '.ansible_host')
23
24 if [ -z "$IP" ]; then
25 IP=$(ssh -G "$HOSTNAME" | awk '$1 == "hostname" p{ print $2 }')
26 fi
27 CONFIGURED_USER=$(ssh -G "$HOSTNAME" | awk '$1 == "user" p{ print $2 }')
28 if [ -n "$CONFIGURED_USER" ]; then
29 REMOTE_USER="$CONFIGURED_USER"
30 fi
31 if [ -z "$IP" ]; then
32 IP=$HOSTNAME
33 fi
34
35 VALIDATE=$(dig +short "$IP")
36 if [ -n "$VALIDATE" ]; then
37 IP=$VALIDATE
38 fi
39 OTP=$( curl \
40 --header "X-Vault-Token: $VAULT_TOKEN" \
41 --request POST \
42 --data "{ \"username\": \"$REMOTE_USER\", \"ip\": \"${IP}\" }" \
43 $VAULT_ADDR/v1/$SSH_MOUNT/creds/$SSH_ROLE | jq -r '.data.key')
44
45 echo $OTP
46
47}
Afterwards you can use this one time password to open a connection to this host:
1$ vault_otp my-server
2d7a79b69-b3d6-ccce-9d2d-ce9a177b8e73
3$ ssh ubuntu@my-server
4Password: d7a79b69-b3d6-ccce-9d2d-ce9a177b8e73
5Welcome to Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-1025-raspi aarch64)
6
7 * Documentation: https://help.ubuntu.com
8 * Management: https://landscape.canonical.com
9 * Support: https://ubuntu.com/advantage
10
11 System information as of Wed Mar 3 10:16:00 UTC 2021
12
13 System load: 0.08 Temperature: 42.3 C
14 Usage of /: 10.6% of 29.05GB Processes: 147
15 Memory usage: 45% Users logged in: 0
16 Swap usage: 0% IPv4 address for eth0: 1.2.3.4
17
18 * Introducing self-healing high availability clusters in MicroK8s.
19 Simple, hardened, Kubernetes for production, from RaspberryPi to DC.
20
21 https://microk8s.io/high-availability
22
230 updates can be installed immediately.
240 of these updates are security updates.
25
26
27The list of available updates is more than a week old.
28To check for new updates run: sudo apt update
29
30Last login: Wed Mar 3 00:00:00 2021 from 127.0.0.1
31ubuntu at my-server in ~
You are interested in our courses or you simply have a question that needs answering? You can contact us at anytime! We will do our best to answer all your questions.
Contact us