HashiCorp Nomad and Vault: Dynamic Secrets
In a cloud-native environment, secrets management is a critical aspect of security. HashiCorp Vault is a popular tool for managing secrets and protecting
In the last HashiCorp Vault post we described a way to configure Vault and also our Infrastructure to use the One Time Passwords feature of Vault. This time we gonna review a way to use signed public keys.
This method is more useful for automation tasks, but also giving access to users because you must not maintain valid user keys on the servers - this is done by the configuration of your HashiCorp Vault.
As server administrators this method allows you to configure servers with the ability to validate a certain ssh user key against a trusted ca. If the CA signed their users keys, those keys are valid for login. Also these signed keys can have a time to live ( TTL ) set, so those users can only authenticate within 30 minutes after signimg the key.
With the addition that vault can have multiple ssh public key roles with different address ranges set, you can create different end points for different network zones which allow a certain subset of users to authenticate.
We are starting with the HashiCorp Vault configuration by terraform to configure the authentication backend. The first code block defines our vault installation and creates a secret backend in vault of type ssh. This is also mounted to the path ssh
. You can adapt this to your design as it is also possible to use more than one ssh backend. Also the default lease times are set within this configuration.
provider "vault" {
address = "https://vault.local:8200/"
}
resource "vault_mount" "ssh" {
type = "ssh"
path = "ssh_signed_keys"
default_lease_ttl_seconds = "14400" # 4h
max_lease_ttl_seconds = "604800" # 1 week
}
Next we configure the role which allows the key signing. This role has the parameter key_type
set to ca
. It is also necessary to disallow host certificates because users should not be able to generate certificates for our host infrastructure.
How this can be done will be part of the next article.
Within the ssh role we also define a short lifetime of those keys - we want the behavior that users just create a new signed key for a new connection or a another host.
In this code block a policy is included which must be assigned to users so that they are able to use this role and generate signed keys.
resource "vault_ssh_secret_backend_role" "client_keys" {
name = "client_keys"
backend = vault_mount.ssh.path
key_type = "ca"
allow_host_certificates = false
allow_subdomains = false
allow_user_key_ids = false
allow_user_certificates = true
default_extensions = {
"permit-pty" = ""
}
allowed_extensions = "permit-pty,permit-port-forwarding"
default_user = "martin"
allowed_users = "martin,ubuntu"
max_ttl = "30m"
ttl = "10m"
cidr_list = "172.16.0.0/16"
}
resource "vault_policy" "user_signing" {
name = "user_signing"
policy = <<EOT
path "${vault_mount.ssh.path}/sign/${vault_ssh_secret_backend_role.client_keys.name}" {
capabilities = ["create", "read", "update"]
}
EOT
}
Afterwards you can run terraform with exporting a valid vault token as environment variable to apply these changes.
1export VAULT_TOKEN="my-vault-token"
2terraform apply
Since we are using this code within our automation we do not want to use our own vault token to authenticate and read the ssh backends ca. For this purpose we are adding an approle authentication and also a policy which allows this role to read the required data from vault.
resource "vault_auth_backend" "approle" {
type = "approle"
}
resource "vault_approle_auth_backend_role" "automated_access" {
backend = data.vault_auth_backend.approle.path
role_name = "automated_access"
token_policies = [ vault_policy.ssh_ca_read.name ]
}
resource "vault_approle_auth_backend_role_secret_id" "automated_access" {
backend = vault_auth_backend.approle.path
role_name = vault_approle_auth_backend_role.automated_access.role_name
}
resource "vault_policy" "ssh_ca_read" {
name = "ssh_ca_read"
policy = <<EOT
path "${vault_mount.ssh.path}/config/ca" {
capabilities = [ "read" ]
}
EOT
}
output "approle_id" {
value = vault_approle_auth_backend_role.automated_access.role_id
}
output "secret_id" {
value = vault_approle_auth_backend_role_secret_id.automated_access.secret_id
}
And again apply this change to your current vault configuration. On this part we must use our own token because this is assigned with higher permissions than the approle of course.
1export VAULT_TOKEN="my-vault-token"
2terraform apply
Because ansible galaxy has no implementation available about an ansible role to sign ssh keys we just wrote one. You can use this role by following ansible-galaxy
command:
1ansible-galaxy install https://github.com/infralovers/ansible-vault-ssh-signed-keys
The role interacts with the HTTP API of HashiCorp Vault. So you are not forced to install any other tool for this role.
To use the role you create a playbook which includes the role and set these variables. Additionally we suggest to configure ansible vault to provide vault_host_role_id and vault_host_secret_id which were created by the approle authentication configuration. But you can also use the commandline to provide these to parameters. For user key validation the TrusedUserCAKeys option within ssh daemon gets configured:
- hosts: all
become: yes
vars:
- vault_addr: http://vault.local:8200/
- user_ssh_path: "ssh_signed_keys"
roles:
- role: vault-ssh-signed-keys
When using ansible vault to load the variable use the following commandline
1ansible-playbook vault-ssh.yml
If you add the variable by commandline - which is not the prefered way to go
1ansible-playbook vault-ssh.yml -e vault_host_role_id="<your-approle-id>" -e vault_host_secret_id="<your-secret-id>"
After this step all hosts are configured so that users are able to connect with a signed ssh key from HashiCorp Vault.
So now we also want to use our infrastructure configuration as clients to connect to our infrastructure. This is a 2 step process:
Signing this ssh key can be done by the following function. All parameters of the function can be configured as environment variables. You might also review "directory based profiles" to configure/change these variable based on the current directory.
At this point the scripts/functions are expecting an environment variable VAULT_TOKEN to exist. You might combine these functions with another function where the login into vault is handled to get a valid token.
vault_sign_key () {
VAULT_ADDR="${VAULT_ADDR:-http://vault.local:8200}"
VAULT_MOUNT=${VAULT_MOUNT:-signed_keys}
VAULT_ROLE=${VAULT_ROLE:-client_keys}
VAULT_PUBLIC_SSH_KEY=${VAULT_PUBLIC_SSH_KEY:-"$HOME/.ssh/id_rsa.pub"}
VAULT_SIGNED_KEY=${VAULT_SIGNED_KEY:-"$HOME/.ssh/vault_signed_key.pub"}
SSH_USER=${SSH_USER:-ubuntu}
if [[ ! -n "${VAULT_TOKEN}" ]]; then
echo "[ERR] No vault access token found at ${VAULT_TOKEN}"
return
fi
export TMP_DIR=$(mktemp -d)
cat > "$(echo ${TMP_DIR}/ssh-ca.json)" << EOF
{
"public_key": "$(cat ${VAULT_PUBLIC_SSH_KEY})",
"valid_principals": "${SSH_USER}"
}
EOF
if ! curl -s --fail -H "X-Vault-Token: ${VAULT_TOKEN}" -X POST -d @${TMP_DIR}/ssh-ca.json \
${VAULT_ADDR}/v1/${VAULT_MOUNT}/sign/${VAULT_ROLE} | jq -r .data.signed_key > "${VAULT_SIGNED_KEY}" ; then
echo "[ERR] Failed to sign public key."
fi
chmod 0600 $VAULT_SIGNED_KEY
rm -rf $TMP_DIR
}
The upper function should provide a signed ssh key which can be used now to connect to an infrastructure host. So we can take the next step and write a wrapper function for this ssh connection. Within this function the previous function is called to provide an ssh key for the current connection. This key as used with our own private key to connect to the host.
vault_ssh () {
if [[ -z "${1}" ]]; then
echo "[INFO] Usage: vault_ssh user@host [-p 2222]"
return
fi
if [[ "${1}" =~ ^-+ ]]; then
echo "[ERR] Additional SSH flags must be passed after the hostname. e.g. 'vssh user@host -p 2222'"
return
elif [[ "${1}" =~ ^[a-zA-Z]+@[a-zA-Z]+ ]]; then
SSH_USER=$(echo $1 | cut -d'@' -f1)
SSH_HOST=$(echo $1 | cut -d'@' -f2)
else
SSH_USER=$(whoami)
SSH_HOST=${1}
fi
SSH_CONFIG_USER=$(ssh -G "$SSH_HOST" | awk '$1 == "user" p{ print $2 }')
if [ -n "$SSH_CONFIG_USER" ]; then
SSH_USER=$SSH_CONFIG_USER
fi
VAULT_PRIVATE_SSH_KEY=${VAULT_PRIVATE_SSH_KEY:$HOME/.ssh/id_ed25519_private}
VAULT_SIGNED_KEY=$(echo "$HOME/.ssh/vault_signed_key.pub")
# sign the public key
vault_sign_key
# shift arguments one to the left to remove target address
shift 1
# construct an SSH command with the credentials, and append any extra args
ssh -i ${VAULT_SIGNED_KEY} -i ${VAULT_PRIVATE_SSH_KEY} ${SSH_USER}@${SSH_HOST} $@
}
With these functions available you can open a ssh key connection to a server which was configured to use the vault TrustedUserCAKeys
and your ssh key is valid for this address range. But for administrators life the user key must not be configured on this server - this is done by our vault configuation and the ansible rollout of the user ca certificate.
1$ vault_ssh my-server
2Welcome to Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-1025-raspi aarch64)
3
4 * Documentation: https://help.ubuntu.com
5 * Management: https://landscape.canonical.com
6 * Support: https://ubuntu.com/advantage
7
8 System information as of Wed Mar 3 10:16:00 UTC 2021
9
10 System load: 0.08 Temperature: 42.3 C
11 Usage of /: 10.6% of 29.05GB Processes: 147
12 Memory usage: 45% Users logged in: 0
13 Swap usage: 0% IPv4 address for eth0: 1.2.3.4
14
15 * Introducing self-healing high availability clusters in MicroK8s.
16 Simple, hardened, Kubernetes for production, from RaspberryPi to DC.
17
18 https://microk8s.io/high-availability
19
200 updates can be installed immediately.
210 of these updates are security updates.
22
23Last login: Wed Mar 8 00:00:00 2021 from 127.0.0.1
24ubuntu at my-server in ~
You are interested in our courses or you simply have a question that needs answering? You can contact us at anytime! We will do our best to answer all your questions.
Contact us