Managing database schemas efficiently is a cornerstone of scalable infrastructure. This is especially true for Azure Data Explorer (ADX), where dynamic and flexible configurations can make a significant impact on performance and maintainability. In this article, we focus on configuring tables to capture Telegraf metrics using Terraform. Telegraf, part of the TICK stack developed by InfluxData, is an open-source, lightweight agent that collects, processes, and sends metrics and events from various sources to monitoring platforms. The tables required to store these metrics share a consistent schema, making them a perfect candidate for automation using Terraform.

In this third part of the series, I’ll demonstrate how to utilize Terraform’s iteration features to define and provision multiple tables with minimal effort. Specifically, I’ll show you how to convert a list of metric names into a map and use for_each to iterate over the map within the adx_table resource block. This approach simplifies the setup process while maintaining clarity and scalability.

Setting Up Telegraf Metrics

Before provisioning the tables, we need a comprehensive list of Telegraf metrics that correspond to the tables to be created in Azure Data Explorer. The list ensures that each metric’s data is stored in a separate table, while all tables share the same schema. Below is the Terraform code that defines the list of metrics:

locals {

  telegraf_metrics = [
    "cpu_raw",
    "disk_raw",
    "diskio_raw",
    "internal_agent",
    "internal_gather",
    "internal_memstats",
    "internal_parser",
    "internal_process",
    "internal_write",
    "kernel_raw",
    "mem_raw",
    "net_raw",
    "netstat_raw",
    "processes_raw",
    "swap_raw",
    "system_raw"
  ]

}

This list captures a wide array of system and application metrics collected by Telegraf. Each table will correspond to one of these metrics, ensuring that the data is organized and easily queryable in ADX.

Creating a Map for Iteration

While the count iterator works well for lists, Terraform’s for_each iterator offers more flexibility and readability when working with maps. To use for_each, we first convert the telegraf_metrics list into a map where the metric names act as keys. Here’s how we define the map:

locals {
  telegraf_metrics_map = tomap({
    for metric in local.telegraf_metrics : metric => metric
  })
}

This transformation creates a map where each key-value pair corresponds to a metric name. Using a map allows us to use for_each, which provides better traceability and alignment with Terraform’s state structure.

Defining the ADX Tables

Now that we have a map of metrics, we can use the for_each iterator to create the necessary tables. Below is the updated Terraform resource block:

resource "adx_table" "telegraf_metric" {

  count = length(local.telegraf_metrics)

  name          = local.telegraf_metrics[count.index]
  database_name = var.database_name

  column {
    name = "fields"
    type = "dynamic"
  }
  column {
    name = "name"
    type = "string"
  }
  column {
    name = "['tags']"
    type = "dynamic"
  }
  column {
    name = "timestamp"
    type = "datetime"
  }

}

Here’s a breakdown of the configuration:

  1. for_each Iterator: The for_each block iterates over the telegraf_metrics_map, creating one table for each metric. The each.key represents the table name, while each.value holds the corresponding metric value (identical in this case).
  2. Table Name: The name attribute uses each.key to dynamically assign the table name for each metric.
  3. Schema Definition:

Each table includes the same set of columns:

  • fields (dynamic): Stores the data fields.
  • name (string): Represents the metric name.
  • [‘tags’] (dynamic): Contains any associated tags.
  • timestamp (datetime): Tracks when the metric was collected.

This approach eliminates the need to define separate resource blocks for each metric and ensures consistency across all tables.

Why This Approach Works

Telegraf metrics follow a uniform schema, making them well-suited for automation. By leveraging Terraform’s map data structure and for_each iterator, we streamline the process of provisioning multiple tables. The resulting configuration is not only concise but also easy to maintain. If new metrics need to be added, you simply update the telegraf_metrics list, and the changes propagate automatically.

Conclusion

Managing Telegraf metrics in Azure Data Explorer can be very tedious and manual without the right tools. By utilizing Terraform’s iteration capabilities and adopting a map-based approach, we’ve demonstrated how to efficiently provision multiple tables with identical schemas. This method saves time, reduces manual effort, and ensures consistency, all while leveraging the power of infrastructure-as-code.

As your environment grows, this technique can be extended to accommodate additional metrics or stamp out new environments, making it a scalable solution for monitoring complex systems. Terraform’s flexibility and declarative nature continue to shine, enabling developers and operators to automate their infrastructure effectively.