Martin Buchleitner, Senior IT-Consultant

About the author

Martin Buchleitner is a Senior IT-Consultant for Infralovers and for Commandemy. Twitter github LinkedIn

See all articles by this author

Terraform with terratest in Gitlab pipeline

With terraform Cloud and terraform Enterprise you are enabled now to us your custom modules in a way that all in your organization or team can use the same building blocks and must not reinvent all functionality. But that also takes more into account that those modules always do what they are used for - you should have your tests in place to ensure the behaviour of the module over time. At this point, Terratest comes on the stage.

Terratest

Terratest is a Go library that makes it easier to write automated tests for your infrastructure code. Terratest was developed at Gruntwork to help maintain the Infrastructure as Code Library, which code is written in Terraform, Go, Python, and Bash. Terratest is written in Go and also all tests must be written in Go.

Terratest provides a collection of helper functions and patterns for common infrastructure testing tasks, like making HTTP requests and using SSH to access a specific virtual machine. The following list describes some of the major advantages of using Terratest:

  • It provides convenient helpers to check infrastructure. This feature is useful when you wasnt to verify your real infrastructure in the real environment.
  • The folder structure is clearly organized. Your test cases are organized clearly and follow the standard Terraform module folder structure.
  • All test cases are written in Go. Most developers who use Terraform are Go developers. If you’re a Go developer, you don’t have to learn another programming language to use Terratest. Also, the only dependencies that are required for you to run test cases in Terratest are Go and Terraform.
  • The infrastructure is highly extensible. You can extend additional functions on top of Terratest

Terratest is designed for integration tests. For that purpose, Terratest provisions real resources in a real environment. Sometimes, integration test jobs can become exceptionally large, especially when you have a large number of resources to provision.

So You should take into account that all tests should always refer to a test account in your cloud provider(s) so that no production code - or even staging environments are not influenced by your development efforts of a terraform module which is under test with Terratest!

Sample AWS Route53 module

Within this post, we gonna try to test a module which encapsulates the creation of subdomains within AWS Route53. Its basic code looks like this:

data "aws_route53_zone" "target_zone" {
  name = var.domain
}

resource "aws_route53_record" "target_record" {
  depends_on = [null_resource.module_dependency]
  zone_id    = data.aws_route53_zone.target_zone.zone_id
  name       = "${var.subdomain}.${var.domain}"
  type       = var.record_type
  ttl        = var.record_ttl
  records    = [var.record_ip]
}

Unit testing with Terratest

Thanks to the flexibility of Terratest, we can use unit tests. Unit tests are local running test cases (although internet access is required). Unit test cases execute terraform init and terraform plan commands to parse the output of terraform plan and look for the attribute values to compare.

To start with Terratest you must init your terraform module path as a Go module with the current snipped and create a path test for your upcoming tests

go mod init $(basename $PWD)
mkdir -p test

In the following unit test, we want to test if a subdomain gets created and nothing gets destroyed, but the zone entry should only be loaded as a data source and never be a resource in our module. The zone entry must be created and maintained in another module.

package test

import (
  "encoding/json"
  "fmt"
  "path"
  "testing"

  "github.com/gruntwork-io/terratest/modules/terraform"
  tfPlan "github.com/hashicorp/terraform/plans/planfile"
)


const domain = "testing.infralovers.com"

type awsRoute53 struct {
  subdomain     string
  record_type   string
  record_ip     string
}

// Test cases for storage account name conversion logic
var testCases = map[string]awsRoute53{
  "terratest.testing.infralovers.com": awsRoute53{subdomain: "terratest", record_type: "A", record_ip: "127.0.0.1"},
}

func TestUT_AWSRoute53(t *testing.T) {
  t.Parallel()

  for expected, input := range testCases {
    // Specify the test case folder and "-var" options
    tfOptions := &terraform.Options{
      TerraformDir: "../",
      Vars: map[string]interface{}{
        "subdomain":    input.subdomain,
        "domain":       domain,
        "record_type":  input.record_type,
        "retcord_ip":   input.record_ip,
      },
    }

    // Terraform init and plan only
    tfPlanOutput := "terraform.tfplan"
    terraform.Init(t, tfOptions)
    terraform.RunTerraformCommand(t, tfOptions, terraform.FormatArgs(tfOptions, "plan", "-out="+tfPlanOutput)...)
    tfOptions.Vars = nil

      // Read and parse the plan output
    reader, err := tfPlan.Open(path.Join(tfOptions.TerraformDir, tfPlanOutput))
    if err != nil {
      t.Fatal(err)
    }
    defer reader.Close()
    plan, _ := reader.ReadPlan()
    if plan.Changes.Empty() {
      t.Fatal("Empty plan outcome")
      continue
    }
    fmt.Printf("Checking %s...", expected)
    for _, res := range plan.Changes.Resources {
      if res.ChangeSrc.Action.String() != "Create" {
        t.Errorf("Found an action which is not create: %s", res.ChangeSrc.Action.String())
        continue
      }
      if res.Addr.String() == "aws_route53_record.target_record" {
        // do some fancy checks ...
      }
    }
  }
}

Go developers probably will notice that the unit test matches the signature of a classic Go test function by accepting an argument of type *testing.T.

With this code, we can now run the following command to check if all resources are just generated and none are updated or even destroyed. We can do this by the following command

go test ./test/

The test will show no problems, but it does not take into account if the correct dns record is generated. To also verify this we gonna add the following code to our terraform module

output "dns" {
  value = aws_route53_record.target_record.fqdn
}

And we must also modify now our test to verify the output in the plan

package test

import (
  "encoding/json"
  "path"
  "testing"

  "github.com/gruntwork-io/terratest/modules/terraform"
  tfPlan "github.com/hashicorp/terraform/plans/planfile"
)

func getJsonMap(m map[string]interface{}, key string) map[string]interface{} {
  raw := m[key]
  sub, ok := raw.(map[string]interface{})
  if !ok {
    return nil
  }
  return sub
}

func TestUT_AWSRoute53(t *testing.T) {
  t.Parallel()

  for expected, input := range testCases {
    // Specify the test case folder and "-var" options
    tfOptions := &terraform.Options{
      TerraformDir: "../",
      Vars: map[string]interface{}{
        "subdomain":   input.subdomain,
        "domain":      domain,
        "target_type": input.record_type,
        "target_ip":   input.record_ip,
      },
    }

    // init and plan
    tfPlanOutput := "terraform.tfplan"
    terraform.Init(t, tfOptions)
    terraform.RunTerraformCommand(t, tfOptions, terraform.FormatArgs(tfOptions, "plan", "-out="+tfPlanOutput)...)
    tfOptions.Vars = nil

    // read the plan as json
    jsonplan, err := terraform.RunTerraformCommandAndGetStdoutE(t, tfOptions, terraform.FormatArgs(tfOptions, "show", "-json", tfPlanOutput)...)
    jsonMap := make(map[string]interface{})
    err = json.Unmarshal([]byte(jsonplan), &jsonMap)
    if err != nil {
      panic(err)
    }
    planned := getJsonMap(jsonMap, "planned_values")
    outputs := getJsonMap(planned, "outputs")
    dns := getJsonMap(outputs, "dns")
    actual := dns["value"]
    if expected != actual {
      t.Errorf("Planned dns output is not valid: %s, expected: %s", actual, expected)
    }
    // Read and parse the plan output
    reader, err := tfPlan.Open(path.Join(tfOptions.TerraformDir, tfPlanOutput))
    if err != nil {
      t.Fatal(err)
    }
    defer reader.Close()
    plan, _ := reader.ReadPlan()
    if plan.Changes.Empty() {
      t.Fatal("Empty plan outcome")
      continue
    }

    for _, res := range plan.Changes.Resources {
      if res.ChangeSrc.Action.String() != "Create" {
        t.Errorf("Found an action which is not create: %s", res.ChangeSrc.Action.String())
        continue
      }
      if res.Addr.String() == "aws_route53_record.target_record" {
        // do some fancy checks ...
      }
    }
  }
}

In the above code, the plan is read as json file because the internal processing of the terraform plan is made by go-cty which created some weird types which must be converted and can cause a headache.

Integration testing with Terratest

For integration testing, the test will go one step further and really create resources - and also destroy those afterwards. The actual test code is smaller because now we can read the output of the terraform applied code to verify the generated dns record

package test

import (
  "testing"

  "github.com/gruntwork-io/terratest/modules/terraform"
)

// Test the Terraform module in examples/complete using Terratest.
func TestIT_AWSRoute53(t *testing.T) {
  t.Parallel()

  for expected, input := range testCases {
    // Specify the test case folder and "-var" options
    tfOptions := &terraform.Options{
      TerraformDir: "../",
      Vars: map[string]interface{}{
        "subdomain":   input.subdomain,
        "domain":      domain,
        "target_type": input.dnstype,
        "target_ip":   input.target,
      },
    }

    defer terraform.Destroy(t, tfOptions)

    // Terraform init and plan only
    terraform.InitAndApply(t, tfOptions)

    actual := terraform.Output(t, tfOptions, "dns")

    if actual != expected {
      t.Errorf("Expect %v, but found %v", expected, actual)
    }

  }
}

When running this test now, make sure you already defined the aws provider credentials as environment variables.

go test ./test/ -run "TestIT_"

This time will only run integration tests with the command line above and they should create a dns record and also by the defer terraform.Destroy() destroy it after the complete test has run through. At this point our written tests are fine and our code is ok. But you also want to run those tests within a pipeline in Gitlab.

Pitfalls at Integration testing

If we add another test to our input, the test will succeed in the first run but will fail in the upcoming

// Test cases for storage account name conversion logic
var testCases = map[string]awsRoute53{
  "terratest.testing.infralovers.com": awsRoute53{subdomain: "terratest", record_type: "A", record_ip: "127.0.0.1"},
  "cnamtest.testing.infralovers.com": awsRoute53{subdomain: "cnamtest", record_type: "CNAME", record_ip: "terratest.testing.infralovers.com"},
}

At the current test code, we will produce resources, which will not be destroyed because the destroy process just uses the latest created terraform plan! In our example now only the CNAME record will be removed correctly, the A type record will still exist. This is of course not a problem of Terraform or Terratest, it is the simple example code which produces this behaviour!

Keep in mind to verify that your integration tests always clean up correctly. Otherwise, they will fail and eventually produce also costs!

Terratest in a Gitlab Pipeline

terratest:
  stage: test
  image:
    name: "hashicorp/terraform:full"
    entrypoint:
      - "/usr/bin/env"
      - "PATH=/go/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
  script:
    - go test ./test/ -run "TestUT_" # running unit tests first
    - go test ./test/ -run "TestIT_" # and afterwards integration tests

A full gitlab Pipeline for terraform and terratest

In the following code snippet is a gitlab pipeline which also validates and lints the containing code. In this example also shellcheck is running for some scripts in the module and the terraform is linted by tflint. The testing is done within an advanced version using mage by the following magefile.

stages:
  - validate
  - lint
  - test

validate:
  stage: validate
  image:
    name: "hashicorp/terraform:0.12.8"
    entrypoint:
      - "/usr/bin/env"
      - "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
  script:
    - terraform init
    - terraform validate
  artifacts:
    paths:
      - .terraform

scriptlint:
  stage: lint
  image:
    name: "koalaman/shellcheck-alpine"
  script:
    - shellcheck scripts/*

terralint:
  stage: lint
  image:
    name: "wata727/tflint"
    entrypoint:
      - "/usr/bin/env"
      - "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
  script:
    - tflint

terratest:
  stage: test
  image:
    name: "hashicorp/terraform:full"
    entrypoint:
      - "/usr/bin/env"
      - "PATH=/go/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
  script:
    - currdir=$(pwd)
    - apk add --no-cache gcc libc-dev bind-tools
    - go get -u -d github.com/magefile/mage
    - magedir=$(find /go/pkg/mod/ -name magefile -type d | grep -v cache)
    - cd $magedir/$(ls $magedir/)
    - go run bootstrap.go
    - cd $currdir
    - mage full

The testing is done with in a advanced version using mage by the following magefile.

package main

import (
  "fmt"
  "os"
  "path/filepath"

  "github.com/magefile/mage/mg"
  "github.com/magefile/mage/sh"
)

// The default target when the command executes `mage` in Cloud Shell
var Default = Full

// A build step that runs Clean, Format, Unit and Integration in sequence
func Full() {
  mg.Deps(Unit)
  mg.Deps(Integration)
}

// A build step that runs unit tests
func Unit() error {
  mg.Deps(Clean)
  mg.Deps(Format)
  fmt.Println("Running unit tests...")
  return sh.RunV("go", "test", "./test/", "-run", "TestUT_", "-v")
}

// A build step that runs integration tests
func Integration() error {
  mg.Deps(Clean)
  mg.Deps(Format)
  fmt.Println("Running integration tests...")
  return sh.RunV("go", "test", "./test/", "-run", "TestIT_", "-v")
}

// A build step that formats both Terraform code and Go code
func Format() error {
  fmt.Println("Formatting...")
  if err := sh.RunV("terraform", "fmt", "."); err != nil {
    return err
  }
  return sh.RunV("go", "fmt", "./test/")
}

// A build step that removes temporary build and test files
func Clean() error {
  fmt.Println("Cleaning...")
  return filepath.Walk(".", func(path string, info os.FileInfo, err error) error {
    if err != nil {
      return err
    }
    if info.IsDir() && info.Name() == "vendor" {
      return filepath.SkipDir
    }
    if info.IsDir() && info.Name() == ".terraform" {
      os.RemoveAll(path)
      fmt.Printf("Removed \"%v\"\n", path)
      return filepath.SkipDir
    }
    if !info.IsDir() && (info.Name() == "terraform.tfstate" ||
      info.Name() == "terraform.tfplan" ||
      info.Name() == "terraform.tfstate.backup") {
      os.Remove(path)
      fmt.Printf("Removed \"%v\"\n", path)
    }
    return nil
  })
}