Defining and using terraform modules

What is a terraform module?

A terraform module is like a black box. When called, it performs the logic you have encapsulated in it.

Variables can be used as inputs to parametrized the terraform configuration being executed within the black box.

When to use a terraform module?

A terraform module becomes handy when you have to set up the exact same resources multiple times across multiple configuration files. You can see it as a factorization method that prevents you from writing the exact same configuration block every time. Instead, you have one block that you can refer to, passing in some input variables and that can perform the action you want with the provided parameters.

Writing local files without using a module

Let’s say you have to provision your infrastructure, always having to write local files in it. Writing a local file in terraform is dead simple: it only takes a filename and a content as parameters. Terraform even maintains a local_file resource block in their Terraform Registry.

If you need to write two local files, you could add the following lines to a main.tf file:

resource "local_file" "my_file_1" {
    content = "This is the content of file 1."
    filename = "files/my_file_1.txt"
}

resource "local_file" "my_file_2" {
    content = "This is the content of file 2."
    filename = "files/my_file_2.txt"
}

Then run:

zsh> terraform init
zsh> terraform apply
zsh> tree .
.
├── files
│   ├── my_file_1.txt
│   └── my_file_2.txt
├── main.tf
└── terraform.tfstate

As you can see, the two files have been created. You can even check the content:

zsh> cat files/my_file_1.txt
This is the content of file 1.

However, because of the redundancy, this is not a good pattern. To be better of we will have to make good use of the terraform modularity. Hence, let’s move on into the next section!

Writing local files using a module

To reduce the code redundancy, you could write a terraform module to write your local files. The logic remains the same as before. The only thing to change is that, instead of taking hard-encoded filename and content parameters, our local_file resource block will read those values from input variables.

Following up on our aforementioned example, we decide to encapsulate the logic for the local_file creation under the tf-module/ folder:

.
├── tf-module
│   ├── main.tf
│   └── variables.tf
└──main.tf

tf-module/variables.tf:

variable filename {
    type = string
    nullable = false
}

variable content {
    type = string
    nullable = false
}

tf-module/main.tf:

resource "local_file" "file" {
    content = var.content
    filename = var.filename
}

./main.tf:

locals {
    filenames_postfix = toset(["1", "2", "3", "4"])
}

module "local_files" {
    for_each = local.filenames_postfix
    source = "./tf-module"
    filename = "files/my_file_${each.value}.txt"
    content = "This is the content of file ${each.value}."
}

Note: for_each only works with sets and maps. Thus, we need to convert of list of strings into a set element. An alternative is given in the following snippet.

variable "filenames_postfix" {
    type = set(string)
    default = ["1", "2", "3", "4"]
}

module "local_files" {
    for_each = var.filenames_postfix
    source = "./tf-module"
    filename = "files/my_file_${each.value}.txt"
    content = "This is the content of file ${each.value}."
}

Note: a local variable is only accessible within the local module i.e. the same local namespace. On the other hand, a terraform variable is globally accessible even though defined at a local terraform module level. You can either pick one or the other according to what suits your design pattern contract best.

You can now apply the terraform code:

zsh> terraform init
zsh> terraform apply
zsh> tree .
.
├── tf-module
│   ├── main.tf
│   └── variables.tf
├── files
│   ├── my_file_1.txt
│   ├── my_file_2.txt
│   ├── my_file_3.txt
│   └── my_file_4.txt
├── main.tf
└── terraform.tfstate

Using a remote terraform module

The source attribute of the local_files block module allows you to pinpoint at the module you want to refer to. So far, we have only used a local reference (./tf-module in our case) but it is also possible to finger point at a remote terraform module, for instance stored on a remote Gitlab or Github repository.

In our case, we stored our re-usable terraform modules in a Github repository, publicly accessible at github.com/olivierbenard/terraform-modules. Then, wherever we are, we can refer to it and use those module in our local terraform projects:

module "local_files" {
    source = "git::ssh://git@github.com/olivierbenard/terraform-modules.git//tf-local-file?ref=master"
    for_each = toset(["1", "2", "3"])
    filename = "files/my_file_${each.value}.txt"
    content = "This is the content of file ${each.value}."
}

Real-Case Example: Provisioning Secret Variables on GCP

So far we have played around, creating local files. However, in real-life, we might have few but little use of them. A more realistic use of terraform modules might be for instance for provisioning and storing sensitive data (e.g. password, variables…) on Google Cloud Platform Secret Manager using terraform as Infrastructure as Code (IaC).

Those credentials and variables can then be access by other processes, e.g. Airflow DAGs running on Google Cloud Cloud Composer using airflow.models.Variable.

from airflow.models import Variable
RETRIEVED_VARIABLE = "{{var.value.my_variable}}"

Notes:

In the above snippet we have used a Jinja template to retrieve the variable stored on Google Secret Manager. The value is only gonna get replaced at run time. If you need to access the variable during build time, you need to use Variable.get("my_variable") instead.
Your Airflow DAGs can only access the variables stored on Google Secret Manager if you have configured Cloud Composer to do so. More on the official documentation. An important remarque is that, to be visible by your DAGs, the variables and connections stored on Google Cloud Secret Manager need to match the following template: airflow-variables-<your_variable> and airflow-connections-<your_connection>.

The terraform module wrapping up the logic to provision secrets on Google Cloud Platform (GCP) is available at: github.com/olivierbenard/terraform-modules/tf-airflow-variable.

You can re-use it, using similar blocks of codes:

module "airflow_variable" {
    source = "git::ssh://git@github.com/olivierbenard/terraform-modules.git//tf-airflow-variable?ref=master"
    for_each = fileset("${var.root_directory}/path/to/af/vars/files/${var.env}", "*")
    airflow_variable_name = each.value
    airflow_variable_value = file("${var.root_directory}/path/to/af/vars/files/${var.env}/${each.value}")
    airflow_variable_location = "europe-west3"
}