Martin Buchleitner, Senior IT-Consultant

About the author

Martin Buchleitner is a Senior IT-Consultant for Infralovers and for Commandemy. Twitter github LinkedIn

See all articles by this author

Most HashiCorp Vault tutorials - even those from HashiCorp - use the commandline tools or even just curl to configure HashiCorp Vault. But do we want to use curl when we can use terraform Code? Terraform already includes a HashiCorp Vault provider which enables us to do most tasks by using code and apply this to our installation - and maintain also our infrastructure.

We gonna use ansible to rollout the Vault SSH One Time Password authentication to all of our servers within the infrastructure. With this change HashiCorp Vault enables us to

  • control who is able to access/authenticate on which machine/subnet
  • how long those one time password are valid

So nobody has to copy&paste ssh keys or even worse mail the password of a machine over the network.

Configure HashiCorp Vault with terraform

Let’s start with HashiCorp Vault configuration by terraform to provide an endpoint which provides one time passwords for the authentication on our infrastructure. The first code block defines our vault installation and creates a secret backend in vault of type ssh. This is mounted to the path ssh. This can be changed, but has to be adapted later on within the ansible code. Within the terraform code the backend path is passed through the items. As a default we set the lease time of a secret for 4 hours and a maximum of 1 week.

provider "vault" {
  address = "http://vault.local:8200/"
}

resource "vault_mount" "ssh" {
  type = "ssh"
  path = "ssh"

  default_lease_ttl_seconds = "14400"  # 4h
  max_lease_ttl_seconds     = "604800" # 1w

}

Next we gonna create a role within the ssh backend of type otp = One Time Password. Within this resource we set some parameters to allow only certains usernames on the connection - in this sample these are centos and ubuntu. We also allow connecting users to forward a port over this ssh connection and open a interactive shell on the servers. The secret backend role is also defined to be valid for a subnet based on cidr “192.168.0.0/24”. And last but not least we limit the lifetime of the otp to 30 minutes.

variable "ssh_cidr" {
    default = "192.168.0.0/24"
}

resource "vault_ssh_secret_backend_role" "otp" {
  name     = "otp"
  backend  = vault_mount.ssh.path
  key_type = "otp"

  allowed_extensions = "permit-pty,permit-port-forwarding"
  default_user       = "centos"
  allowed_users      = "centos,ubuntu"
  cidr_list          = var.ssh_cidr

  ttl = "30m"

}

So that our clients can create an otp we gonna create and policy to allow them access to this endpoint:

resource "vault_policy" "ssh_client_access" {
  name   = "ssh_clients"
  policy = <<EOT
path "${vault_mount.ssh.path}/creds/otp" {
    capabilities = ["create", "read", "update"]
}
EOT
}

Next we can apply this change to our HashiCorp Vault:

export VAULT_TOKEN="my-vault-token"
terraform apply

At this point HashiCorp Vault is able to serve One Time Passwords for SSH Authentication and also allow our infrastructure to validate these passwords against HashiCorp Vault.

Ansible to rollout SSH One Time Password authentication

Ansible role

Ansible has already some community written HashiCorp vault roles, but none of them was up-to-date at the time of writing this article. So we forked a solution and updated this with some fixxes and changes to provide a solution for us. You can use this role by following ansible-galaxy command:

ansible-galaxy install https://github.com/infralovers/ansible-vault-otp

The role uses the HashiCorp vault-ssh-helper in its core to reconfigure the infrastructure authentication mechanism. This helper validates the login prompt by ssh with our vault configuration that the provided password is a valid one time password provided by HashiCorp Vault.

Ansible rollout

To use this role you create a playbook which includes the role and sets the required variables:

- hosts: all
  become: yes
  vars:
  - vault_addr: http://vault.local:8200
  - ssh_mount_point: ssh
  roles:
  - role: infralovers.vault_otp

When using an https endpoint you also have to set vault_ca_cert_file variable with a path to the certificate file.

Using HashiCorp Vault OTP

When consuming the OTP funcitonality we have to use curl to get this done - but we can write a function and add this content to e.g. .bashrc. The following function requests an one time password from HashiCorp Vault for a certain ip address or hostname. The function lookups the hostname within the ansible inventory. If this did not work the ssh configuration is used for a lookup on the ip address. And if this should not work either, a dns query is done to get the ip address of the host. As optional parameter also the username of the connection can be passed to the function - but it defaults to ubuntu.

function vault_otp() {

    VAULT_ADDR="http://vault.local:8200"
    SSH_MOUNT="ssh"
    SSH_ROLE="otp"
    ANSIBLE_INVENTORY=${ANSIBLE_INVENTORY:-/full/path/to/hosts.ini}
    REMOTE_USER="ubuntu"

    if [ -z "$VAULT_TOKEN" ]; then
        echo "missing vault token!"
        return
    fi
    HOSTNAME=$1
    if [ -n "$2" ]; then
        REMOTE_USER="$2"
    fi
    if [ -z "$VAULT_TOKEN" ]; then
        echo "missing vault token!"
        return
    fi

    IP=$(ansible-inventory -i "$ANSIBLE_INVENTORY" --host "$HOSTNAME" 2>/dev/null | jq -r '.ansible_host')

    if [ -z "$IP" ]; then
        IP=$(ssh -G "$HOSTNAME" | awk '$1 == "hostname" p{ print $2 }')
    fi
    CONFIGURED_USER=$(ssh -G "$HOSTNAME" | awk '$1 == "user" p{ print $2 }')
    if [ -n "$CONFIGURED_USER" ]; then
      REMOTE_USER="$CONFIGURED_USER"
    fi
    if [ -z "$IP" ]; then
        IP=$HOSTNAME
    fi

    VALIDATE=$(dig +short "$IP")
    if [ -n "$VALIDATE" ]; then
        IP=$VALIDATE
    fi
    OTP=$( curl \
            --header "X-Vault-Token: $VAULT_TOKEN" \
            --request POST \
            --data "{ \"username\": \"$REMOTE_USER\", \"ip\": \"${IP}\" }" \
            $VAULT_ADDR/v1/$SSH_MOUNT/creds/$SSH_ROLE | jq -r '.data.key')

    echo $OTP

}

Afterwards you can use this one time password to open a connection to this host:

$ vault_otp my-server
d7a79b69-b3d6-ccce-9d2d-ce9a177b8e73
$ ssh ubuntu@my-server
Password: d7a79b69-b3d6-ccce-9d2d-ce9a177b8e73
Welcome to Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-1025-raspi aarch64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Wed Mar  3 10:16:00 UTC 2021

  System load:  0.08               Temperature:           42.3 C
  Usage of /:   10.6% of 29.05GB   Processes:             147
  Memory usage: 45%                Users logged in:       0
  Swap usage:   0%                 IPv4 address for eth0: 1.2.3.4

 * Introducing self-healing high availability clusters in MicroK8s.
   Simple, hardened, Kubernetes for production, from RaspberryPi to DC.

     https://microk8s.io/high-availability

0 updates can be installed immediately.
0 of these updates are security updates.


The list of available updates is more than a week old.
To check for new updates run: sudo apt update

Last login: Wed Mar  3 00:00:00 2021 from 127.0.0.1
ubuntu at my-server in ~

You want to learn more about this topic?

You don't learn pure knowledge from books and it is not available in capsule form. The most effective form of exercise is still with sparring partners and a guide. Therefore, our "Commandemy" brand offers training for the IT experts of tomorrow.

Become the undisputed king of code and take a look at our current courses now!

See current courses