Understanding DevOps and Cloud Maturity Models: A Guide to Elevating Your IT Strategy
In today’s fast-paced technological landscape, DevOps and Cloud practices are integral to accelerating software delivery and optimizing cloud resources. But as
With terraform Cloud and terraform Enterprise you are enabled now to us your custom modules in a way that all in your organization or team can use the same building blocks and must not reinvent all functionality. But that also takes more into account that those modules always do what they are used for - you should have your tests in place to ensure the behaviour of the module over time. At this point, Terratest comes on the stage.
Terratest is a Go library that makes it easier to write automated tests for your infrastructure code. Terratest was developed at Gruntwork to help maintain the Infrastructure as Code Library, which code is written in Terraform, Go, Python, and Bash. Terratest is written in Go and also all tests must be written in Go.
Terratest provides a collection of helper functions and patterns for common infrastructure testing tasks, like making HTTP requests and using SSH to access a specific virtual machine. The following list describes some of the major advantages of using Terratest:
Terratest is designed for integration tests. For that purpose, Terratest provisions real resources in a real environment. Sometimes, integration test jobs can become exceptionally large, especially when you have a large number of resources to provision.
So You should take into account that all tests should always refer to a test account in your cloud provider(s) so that no production code - or even staging environments are not influenced by your development efforts of a terraform module which is under test with Terratest!
Within this post, we gonna try to test a module which encapsulates the creation of subdomains within AWS Route53. Its basic code looks like this:
1data "aws_route53_zone" "target_zone" {
2 name = var.domain
3}
4
5resource "aws_route53_record" "target_record" {
6 depends_on = [null_resource.module_dependency]
7 zone_id = data.aws_route53_zone.target_zone.zone_id
8 name = "${var.subdomain}.${var.domain}"
9 type = var.record_type
10 ttl = var.record_ttl
11 records = [var.record_ip]
12}
Thanks to the flexibility of Terratest, we can use unit tests. Unit tests are local running test cases (although internet access is required). Unit test cases execute terraform init and terraform plan commands to parse the output of terraform plan and look for the attribute values to compare.
To start with Terratest you must init your terraform module path as a Go module with the current snipped and create a path test for your upcoming tests
1go mod init $(basename $PWD)
2mkdir -p test
In the following unit test, we want to test if a subdomain gets created and nothing gets destroyed, but the zone entry should only be loaded as a data source and never be a resource in our module. The zone entry must be created and maintained in another module.
1package test
2
3import (
4 "encoding/json"
5 "fmt"
6 "path"
7 "testing"
8
9 "github.com/gruntwork-io/terratest/modules/terraform"
10 tfPlan "github.com/hashicorp/terraform/plans/planfile"
11)
12
13
14const domain = "testing.infralovers.com"
15
16type awsRoute53 struct {
17 subdomain string
18 record_type string
19 record_ip string
20}
21
22// Test cases for storage account name conversion logic
23var testCases = map[string]awsRoute53{
24 "terratest.testing.infralovers.com": awsRoute53{subdomain: "terratest", record_type: "A", record_ip: "127.0.0.1"},
25}
26
27func TestUT_AWSRoute53(t *testing.T) {
28 t.Parallel()
29
30 for expected, input := range testCases {
31 // Specify the test case folder and "-var" options
32 tfOptions := &terraform.Options{
33 TerraformDir: "../",
34 Vars: map[string]interface{}{
35 "subdomain": input.subdomain,
36 "domain": domain,
37 "record_type": input.record_type,
38 "retcord_ip": input.record_ip,
39 },
40 }
41
42 // Terraform init and plan only
43 tfPlanOutput := "terraform.tfplan"
44 terraform.Init(t, tfOptions)
45 terraform.RunTerraformCommand(t, tfOptions, terraform.FormatArgs(tfOptions, "plan", "-out="+tfPlanOutput)...)
46 tfOptions.Vars = nil
47
48 // Read and parse the plan output
49 reader, err := tfPlan.Open(path.Join(tfOptions.TerraformDir, tfPlanOutput))
50 if err != nil {
51 t.Fatal(err)
52 }
53 defer reader.Close()
54 plan, _ := reader.ReadPlan()
55 if plan.Changes.Empty() {
56 t.Fatal("Empty plan outcome")
57 continue
58 }
59 fmt.Printf("Checking %s...", expected)
60 for _, res := range plan.Changes.Resources {
61 if res.ChangeSrc.Action.String() != "Create" {
62 t.Errorf("Found an action which is not create: %s", res.ChangeSrc.Action.String())
63 continue
64 }
65 if res.Addr.String() == "aws_route53_record.target_record" {
66 // do some fancy checks ...
67 }
68 }
69 }
70}
Go developers probably will notice that the unit test matches the signature of a classic Go test function by accepting an argument of type *testing.T
.
With this code, we can now run the following command to check if all resources are just generated and none are updated or even destroyed. We can do this by the following command
1go test ./test/
The test will show no problems, but it does not take into account if the correct dns record is generated. To also verify this we gonna add the following code to our terraform module
1output "dns" {
2 value = aws_route53_record.target_record.fqdn
3}
And we must also modify now our test to verify the output in the plan
1package test
2
3import (
4 "encoding/json"
5 "path"
6 "testing"
7
8 "github.com/gruntwork-io/terratest/modules/terraform"
9 tfPlan "github.com/hashicorp/terraform/plans/planfile"
10)
11
12func getJsonMap(m map[string]interface{}, key string) map[string]interface{} {
13 raw := m[key]
14 sub, ok := raw.(map[string]interface{})
15 if !ok {
16 return nil
17 }
18 return sub
19}
20
21func TestUT_AWSRoute53(t *testing.T) {
22 t.Parallel()
23
24 for expected, input := range testCases {
25 // Specify the test case folder and "-var" options
26 tfOptions := &terraform.Options{
27 TerraformDir: "../",
28 Vars: map[string]interface{}{
29 "subdomain": input.subdomain,
30 "domain": domain,
31 "target_type": input.record_type,
32 "target_ip": input.record_ip,
33 },
34 }
35
36 // init and plan
37 tfPlanOutput := "terraform.tfplan"
38 terraform.Init(t, tfOptions)
39 terraform.RunTerraformCommand(t, tfOptions, terraform.FormatArgs(tfOptions, "plan", "-out="+tfPlanOutput)...)
40 tfOptions.Vars = nil
41
42 // read the plan as json
43 jsonplan, err := terraform.RunTerraformCommandAndGetStdoutE(t, tfOptions, terraform.FormatArgs(tfOptions, "show", "-json", tfPlanOutput)...)
44 jsonMap := make(map[string]interface{})
45 err = json.Unmarshal([]byte(jsonplan), &jsonMap)
46 if err != nil {
47 panic(err)
48 }
49 planned := getJsonMap(jsonMap, "planned_values")
50 outputs := getJsonMap(planned, "outputs")
51 dns := getJsonMap(outputs, "dns")
52 actual := dns["value"]
53 if expected != actual {
54 t.Errorf("Planned dns output is not valid: %s, expected: %s", actual, expected)
55 }
56 // Read and parse the plan output
57 reader, err := tfPlan.Open(path.Join(tfOptions.TerraformDir, tfPlanOutput))
58 if err != nil {
59 t.Fatal(err)
60 }
61 defer reader.Close()
62 plan, _ := reader.ReadPlan()
63 if plan.Changes.Empty() {
64 t.Fatal("Empty plan outcome")
65 continue
66 }
67
68 for _, res := range plan.Changes.Resources {
69 if res.ChangeSrc.Action.String() != "Create" {
70 t.Errorf("Found an action which is not create: %s", res.ChangeSrc.Action.String())
71 continue
72 }
73 if res.Addr.String() == "aws_route53_record.target_record" {
74 // do some fancy checks ...
75 }
76 }
77 }
78}
In the above code, the plan is read as json file because the internal processing of the terraform plan is made by go-cty which created some weird types which must be converted and can cause a headache.
For integration testing, the test will go one step further and really create resources - and also destroy those afterwards. The actual test code is smaller because now we can read the output of the terraform applied code to verify the generated dns record
1package test
2
3import (
4 "testing"
5
6 "github.com/gruntwork-io/terratest/modules/terraform"
7)
8
9// Test the Terraform module in examples/complete using Terratest.
10func TestIT_AWSRoute53(t *testing.T) {
11 t.Parallel()
12
13 for expected, input := range testCases {
14 // Specify the test case folder and "-var" options
15 tfOptions := &terraform.Options{
16 TerraformDir: "../",
17 Vars: map[string]interface{}{
18 "subdomain": input.subdomain,
19 "domain": domain,
20 "target_type": input.dnstype,
21 "target_ip": input.target,
22 },
23 }
24
25 defer terraform.Destroy(t, tfOptions)
26
27 // Terraform init and plan only
28 terraform.InitAndApply(t, tfOptions)
29
30 actual := terraform.Output(t, tfOptions, "dns")
31
32 if actual != expected {
33 t.Errorf("Expect %v, but found %v", expected, actual)
34 }
35
36 }
37}
When running this test now, make sure you already defined the aws provider credentials as environment variables.
1go test ./test/ -run "TestIT_"
This time will only run integration tests with the command line above and they should create a dns record and also by the defer terraform.Destroy()
destroy it after the complete test has run through. At this point our written tests are fine and our code is ok. But you also want to run those tests within a pipeline in Gitlab.
If we add another test to our input, the test will succeed in the first run but will fail in the upcoming
1// Test cases for storage account name conversion logic
2var testCases = map[string]awsRoute53{
3 "terratest.testing.infralovers.com": awsRoute53{subdomain: "terratest", record_type: "A", record_ip: "127.0.0.1"},
4 "cnamtest.testing.infralovers.com": awsRoute53{subdomain: "cnamtest", record_type: "CNAME", record_ip: "terratest.testing.infralovers.com"},
5}
At the current test code, we will produce resources, which will not be destroyed because the destroy process just uses the latest created terraform plan! In our example now only the CNAME record will be removed correctly, the A type record will still exist. This is of course not a problem of Terraform or Terratest, it is the simple example code which produces this behaviour!
Keep in mind to verify that your integration tests always clean up correctly. Otherwise, they will fail and eventually produce also costs!
1terratest:
2 stage: test
3 image:
4 name: "hashicorp/terraform:full"
5 entrypoint:
6 - "/usr/bin/env"
7 - "PATH=/go/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
8 script:
9 - go test ./test/ -run "TestUT_" # running unit tests first
10 - go test ./test/ -run "TestIT_" # and afterwards integration tests
In the following code snippet is a gitlab pipeline which also validates and lints the containing code. In this example also shellcheck is running for some scripts in the module and the terraform is linted by tflint. The testing is done within an advanced version using mage by the following magefile.
1stages:
2 - validate
3 - lint
4 - test
5
6validate:
7 stage: validate
8 image:
9 name: "hashicorp/terraform:0.12.8"
10 entrypoint:
11 - "/usr/bin/env"
12 - "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
13 script:
14 - terraform init
15 - terraform validate
16 artifacts:
17 paths:
18 - .terraform
19
20scriptlint:
21 stage: lint
22 image:
23 name: "koalaman/shellcheck-alpine"
24 script:
25 - shellcheck scripts/*
26
27terralint:
28 stage: lint
29 image:
30 name: "wata727/tflint"
31 entrypoint:
32 - "/usr/bin/env"
33 - "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
34 script:
35 - tflint
36
37terratest:
38 stage: test
39 image:
40 name: "hashicorp/terraform:full"
41 entrypoint:
42 - "/usr/bin/env"
43 - "PATH=/go/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
44 script:
45 - currdir=$(pwd)
46 - apk add --no-cache gcc libc-dev bind-tools
47 - go get -u -d github.com/magefile/mage
48 - magedir=$(find /go/pkg/mod/ -name magefile -type d | grep -v cache)
49 - cd $magedir/$(ls $magedir/)
50 - go run bootstrap.go
51 - cd $currdir
52 - mage full
The testing is done with in a advanced version using mage by the following magefile.
1package main
2
3import (
4 "fmt"
5 "os"
6 "path/filepath"
7
8 "github.com/magefile/mage/mg"
9 "github.com/magefile/mage/sh"
10)
11
12// The default target when the command executes `mage` in Cloud Shell
13var Default = Full
14
15// A build step that runs Clean, Format, Unit and Integration in sequence
16func Full() {
17 mg.Deps(Unit)
18 mg.Deps(Integration)
19}
20
21// A build step that runs unit tests
22func Unit() error {
23 mg.Deps(Clean)
24 mg.Deps(Format)
25 fmt.Println("Running unit tests...")
26 return sh.RunV("go", "test", "./test/", "-run", "TestUT_", "-v")
27}
28
29// A build step that runs integration tests
30func Integration() error {
31 mg.Deps(Clean)
32 mg.Deps(Format)
33 fmt.Println("Running integration tests...")
34 return sh.RunV("go", "test", "./test/", "-run", "TestIT_", "-v")
35}
36
37// A build step that formats both Terraform code and Go code
38func Format() error {
39 fmt.Println("Formatting...")
40 if err := sh.RunV("terraform", "fmt", "."); err != nil {
41 return err
42 }
43 return sh.RunV("go", "fmt", "./test/")
44}
45
46// A build step that removes temporary build and test files
47func Clean() error {
48 fmt.Println("Cleaning...")
49 return filepath.Walk(".", func(path string, info os.FileInfo, err error) error {
50 if err != nil {
51 return err
52 }
53 if info.IsDir() && info.Name() == "vendor" {
54 return filepath.SkipDir
55 }
56 if info.IsDir() && info.Name() == ".terraform" {
57 os.RemoveAll(path)
58 fmt.Printf("Removed \"%v\"\n", path)
59 return filepath.SkipDir
60 }
61 if !info.IsDir() && (info.Name() == "terraform.tfstate" ||
62 info.Name() == "terraform.tfplan" ||
63 info.Name() == "terraform.tfstate.backup") {
64 os.Remove(path)
65 fmt.Printf("Removed \"%v\"\n", path)
66 }
67 return nil
68 })
69}
You are interested in our courses or you simply have a question that needs answering? You can contact us at anytime! We will do our best to answer all your questions.
Contact us