This is Part Two of a four-part series about mashing up the Azure and Grafana Terraform providers together. In Part One, I gave an introduction to the genesis of this topic how I ended up giving a presentation about this at HashiDays 2024 and started with the technical bits about setting up an Azure Grafana Managed instance using the azurerm provider.

Now its time to get the Grafana provider setup. Each provider in Terraform is its own independent piece of software. This allows Terraform to conform to the contours of whatever platform that may be under the hood. In that sense, a Terraform provider is fulfilling the role of an adapter bringing what would otherwise be a completely foreign interface into the fold of a Terraform workspace. It’s pretty damn brilliant actually.

Setting up the Grafana Terraform Provider

As usual, all Terraform providers should be registered in the required_providers block. I’d recommend keeping this sort of boilerplate in a file called versions.tf. This is more of a convention that helps improve productivity — think of all those countless hours searching the source code directory for the required providers block that we can put to bed if we just follow this super easy convention!

terraform {
  required_providers {
    grafana = {
      source  = "grafana/grafana"
      version = "~> 1.36.1"
    }
  }
  backend "azurerm" {
  }
}

Of course, I am also using an azurerm backend. You might not be if you are trying to follow along locally but eventually you probably should be.

Like any good root module, when an Terraform provider requires additional configuration we need to declare a provider block to set it up.

The grafana provider falls into a category of Terraform providers that I call “Hosted Endpoint” providers. That is, these providers talk to a specific hosted endpoint at some network accessible URL. When you think about it, the same thing is happening with the big cloud platforms like AWS, Azure and GCP — it’s just that their “endpoint” is so well known that we don’t even need to specify it. Kind of like when I say “What do you think of Prince Harry?” you immediately think I am talking about Henry Charles Albert David Windsor, a very specific member of a very specific royal family. Or when I say “What’s your favorite thing by Prince?” you also immediately know I am talking about the musician named Prince Rogers Nelson, and how he formerly went by…well, that symbol right down there and the noun phrase “The Artist Formerly Known as Prince”.

IMAGE

Those big public clouds don’t need us to give them their full name. We just know — and so do their respective Terraform providers.

Hosted Endpoint providers are like regular people. Each instance of them is not famous. Their names are not well known. In order to communicate to a specific one you need to know their full name and mailing address. In the case of Grafana, that means the Grafana instance URL.

provider "grafana" {
  url  = var.grafana_endpoint
  auth = var.grafana_auth
}

The url uniquely identifies which specific instance of Grafana we want to talk to. The Grafana provider does not care if the URL that you provide is your own instance of Grafana that you manually setup on a laptop under your desk or an EC2 instance in East US 1. All instances of Grafana will have a set of REST APIs that sit at that URL and the grafana provider is going to attempt to talk to that REST API at the url that you specify. Therefore, where ever you are executing Terraform — be it on your local machine or an Azure DevOps build agent — you better make sure that your machine has network line-of-sight to this URL. Otherwise, the Grafana Terraform provider is not going to be able to talk to your Grafana instance at all. That’s why, in Part One of this series I set public_network_access_enabled to true on the azurerm_dashboard_grafana resource that I provisioned to Azure. I didn’t want to mess with the private networking involved. If you do — and you might — then you’ll need to configure the private networking of the Grafana instance and ensure that where ever you are running Terraform your machine can connect to that Virtual Network.

Now that we know who we are talking to, we need to prove that we have permission to do so. That’s where the auth attribute comes in on the grafana provider.

Authenticating with our Azure Managed Grafana

Well this is where we are in a bit of a catch-22. The Grafana provider itself has resources to configure what we need to configure. However, we can’t configure those things unless we can authenticate with the grafana provider.

In order to do that we need to have a valid authentication token for a Grafana Service Account. Working backwards, first we need a Grafana Service Account, then we need a token for said service account.

Setting up a Grafana Service Account for Terraform

As I mentioned previously, the chicken-or-the-egg issue that we ran into is with the grafana Terraform provider we can create ourselves both a Service Account — using grafana_service_account — and a token — using grafana_service_account_token. However, that doesn’t help us getting our first one that we can use to create more, now, does it?

Unfortunately, that means that we will need to find some other means to provision this initial Service Account that Terraform will use to manage our Grafana instance.

Whenever I automate something, I ask myself a very important question: “How do the ‘normies’ do it?”

IMAGE

In this case they ClickOps it into existence using the Organization Configuration screens in the Grafana portal that was provisioned for their Azure Managed Grafana instance. How do they get access to this portal? By being members of the Azure Role Assignment “Grafana Admin”. Previously we already set ourselves and the Terraform user up in this role.

It’s important to note that this is an Azure Role Assignment. This is not a Grafana permission. It’s something you create in Azure that grant’s Entra ID identities access within Grafana. It seems like it should be possible for Azure to then provision service accounts using the azurerm provider. This would absolutely make life so much easier — to the point where this next section would not even be required.

Alas, it is still required. So here we go.

Well, if the normies are going into the Grafana Dashboard and clicking this service account into existence then they are doing it with their Entra ID credentials. That means we have access to do it we just need to figure out how to do it programmatically.

Enter the Azure CLI.

It turns out there is an extension for Azure Managed Grafana that allws you to do a bunch of stuff inside of Grafana such as create Service Accounts and whatnot. Excited? I am.

IMAGE

First we have to install this extension. It’s called amg . No I’m not galking about the Mercedes in your driveway. That’s literally what its called. So let’s login Azure with the CLI and install the “amg” extension for Azure Managed Grafana.

echo 'Azure CLI authN'
az login --service-principal -u $ARM_CLIENT_ID -p $ARM_CLIENT_SECRET --tenant $ARM_TENANT_ID
az extension add --name amg

Now, we should setup a well known Check if the Service Account for Terraform already exists. We’ll setup an environment variable for the Azure Managed Grafana Instance’s name — obtained from azurerm_dashboard_grafana.main.name — called GRAFANA_NAME. By convention this is passed into the –name option on az grafana commands. We’ll also setup another environment variable for the well known name of the Terraform identity GRAFANA_SERVICE_ACCOUNT_NAME. I suppose our bash script could be used for something other than Terraform, no? But for our purposes when we execute this script the value of this environment variable will likely be simply terraform. I mean how many Terraform users do we need, right? It’s a Service Account for a specific identity that we use to provision Grafana Terraform code to this specific instance of Azure Managed Grafana.

echo 'List grafana service-accounts'
if [ $(az grafana service-account list --name $GRAFANA_NAME | jq 'length == 0') == "true" ]; then
    
    echo 'Create grafana service-accounts'
    az grafana service-account create --name $GRAFANA_NAME --service-account $GRAFANA_SERVICE_ACCOUNT_NAME --role Admin --query @.{name:name} | jq '.name'
else
    echo 'Service account already exists!'
fi

The above code checks whether the Service Account already exists, and if it does not, it creates a new one. Now we should do the same thing before creating a token.

echo 'List grafana service-account tokens'
if [ $(az grafana service-account token list --name $GRAFANA_NAME --service-account $GRAFANA_SERVICE_ACCOUNT_NAME | jq 'length == 0' ) == true ]; then
    echo 'Token does not exist'
else
    echo 'Token already exists!'
    az grafana service-account token delete --name $GRAFANA_NAME --service-account $GRAFANA_SERVICE_ACCOUNT_NAME --token $GRAFANA_SERVICE_ACCOUNT_TOKEN_NAME
fi

The above code basically is going to check for an old token and delete it. Thus allowing us to safely create a new one. The goal being that our Grafana Service Account that Terraform uses only ever has one active Token.

echo 'Create grafana service-account token'
GRAFANA_SERVICE_ACCOUNT_TOKEN=$(az grafana service-account token create --name $GRAFANA_NAME --service-account $GRAFANA_SERVICE_ACCOUNT_NAME --token $GRAFANA_SERVICE_ACCOUNT_TOKEN_NAME --time-to-live 15d --query @.{key:key} | jq -r '.key')

Using the az grafana service-account token create command allows us to obtain the Token for our service account and save it to an environment variable of our choosing. Notice the TTL is fifteen (15) days! That is probably too long. Just adjust the –time-to-live command option and rachet that down to minimize the blast radius. Just-In-Time secrets like this should only live as long as a single Terraform Apply takes — and on Grafana, that will be extremely fast, probably less than a couple minutes.

There we go. Now we just need to execute this before we run Terraform for the Grafana layer in order to ensure that we have a valid Service Account and Token to supply the grafana provider with everything it needs to manage all the things on our Azure Managed Grafana!

In Part 3, we’ll get into actually provisioning things to Grafana using this provider that we have now fully operational!

Until then — Happy Azure Terraforming!