Welcome to this deep dive on creating a Custom Event Grid Topic and Subscription for an Azure Function running on .NET 8.0 in the isolated worker model. In this walkthrough, I chronicle my journey of setting up the infrastructure with Terraform while simultaneously writing the application code for two Azure Functions: an HTTP trigger (InternalPublisher) that publishes CloudEvents to the custom topic, and an Event Grid trigger (InternalSubscriber) that processes those events.

I ran into a variety of obstacles stemming from the somewhat confusing Azure documentation around Event Grid, the difficulty in extracting ARM template schema for Azure Functions, and the differences in how Azure handles resources like Event Grid Subscriptions. Additionally, I discovered some frustrations around using Managed Identities for authentication in this scenario, especially when it comes to user-assigned identities. My hope is that by sharing my experience (complete with several screenshots and code snippets), you will have a clearer path forward for building distributed serverless systems in Azure.

Setting up the Event Grid Topic

First, I set up a custom Event Grid Topic that I want to use for internal communication within the Azure Function App. This enables different Azure Functions to trigger each other. For instance, one HTTP-triggered function can spawn an Event Grid event, which in turn invokes another function.

resource "azurerm_eventgrid_topic" "main" {
  name                = "evgt-${var.name}-${var.location}"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
  input_schema        = "CloudEventSchemaV1_0"
  tags                = var.tags
}

At first, I left off the input_schema attribute. This was a mistake. Both the Topic and the Subscription need to be CloudEventSchemaV1_0:

event_delivery_schema = "CloudEventSchemaV1_0"

When I first started my journey with Event Grid, I used the original Event Grid schema. However, it appears that Azure now considers CloudEvents the best path forward. You can see more details about the differences here:

https://learn.microsoft.com/en-us/azure/event-grid/event-schema

Adding the EventGrid Trigger

In my .NET project file (*.csproj) for the Azure Function, I must add a reference to the EventGrid extension:

<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.EventGrid" Version="3.4.2" />

It seems there is a bug in version 3.4.3 of this package that causes the following error:

Azure Functions — Could not load file or assembly ‘System.Memory.Data, Version=6.0.0.0

I found a GitHub issue with recent chatter that confirmed the same and helped me find the solution. The GitHub user cicorieas had the solution:

You should always be cautious at taking latest, event with .NET nuget packages. My EventGrid Function looks like this:

[Function(nameof(InternalSubscriber))]
public void GetEvents([EventGridTrigger] CloudEvent cloudEvent)
{
    _logger.LogInformation("Event type: {type}, Event subject: {subject}", cloudEvent.Type, cloudEvent.Subject);
    _telemetryClient.TrackEvent("Event.ACK");
}

It’s very similar to what you get out of the box with the Visual Studio template. Adding the subscription

Function System Assigned vs. User Assigned Identity

I discovered that the Azure Function Managed Identity is problematic when you try to configure an Event Grid trigger. Specifically, it fails to pick up user-assigned identities (even if there is only one).

Here is the error message I received:

Azure.Identity: ManagedIdentityCredential authentication failed: Service request failed. Status: 400 (Bad Request) Content: {“statusCode”:400,”message”:”Unable to load the proper Managed Identity.”,”correlationId”:”992c8299–14b7–42f8-b4fa-9b96fbc4db22"} Headers: Date: Mon, 30 Dec 2024 13:56:42 GMT Server: Kestrel Transfer-Encoding: chunked X-CORRELATION-ID: REDACTED Content-Type: application/json; charset=utf-8 See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/managedidentitycredential/troubleshoot. Azure.Identity: Service request failed. Status: 400 (Bad Request) Content: {“statusCode”:400,”message”:”Unable to load the proper Managed Identity.”,”correlationId”:”992c8299–14b7–42f8-b4fa-9b96fbc4db22"} Headers: Date: Mon, 30 Dec 2024 13:56:42 GMT Server: Kestrel Transfer-Encoding: chunked X-CORRELATION-ID: REDACTED Content-Type: application/json; charset=utf-8 .

Then, after adding a system-assigned identity but not granting that identity the Storage role assignment:

Azure.Storage.Blobs: This request is not authorized to perform this operation using this permission. RequestId:b075ccdf-501e-0102–6d09–5b7424000000 Time:2024–12–30T22:26:02.9640368Z Status: 403 (This request is not authorized to perform this operation using this permission.) ErrorCode: AuthorizationPermissionMismatch Content: <?xml version=”1.0" encoding=”utf-8"?><Error><Code>AuthorizationPermissionMismatch</Code><Message>This request is not authorized to perform this operation using this permission. RequestId:b075ccdf-501e-0102–6d09–5b7424000000 Time:2024–12–30T22:26:02.9640368Z</Message></Error> Headers: Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0 x-ms-request-id: b075ccdf-501e-0102–6d09–5b7424000000 x-ms-client-request-id: 996a3c68-eaf6–4512-b5bb-0d0b25fa8537 x-ms-version: 2023–11–03 x-ms-error-code: AuthorizationPermissionMismatch Date: Mon, 30 Dec 2024 22:26:02 GMT Content-Length: 279 Content-Type: application/xml .

It’s a helpful reminder that you must make sure the system-assigned identity has the correct permissions to the storage account when dealing with Azure Functions. Storage Queue Subscription

As a test, I set up a storage queue subscription to confirm that my internal event publisher function actually published events to the topic.

resource "azurerm_storage_queue" "events" {
  name                 = "events"
  storage_account_name = azurerm_storage_account.data.name

  # because this storage account has shared access keys disabled
  depends_on = [azurerm_role_assignment.terraform_data_contributor]
}

Now setup the Event Grid Subscription:

resource "azurerm_eventgrid_event_subscription" "queue" {
  name                  = "evgs-${var.name}-${var.location}-queue"
  scope                 = azurerm_eventgrid_topic.main.id
  event_delivery_schema = "CloudEventSchemaV1_0"

  storage_queue_endpoint {
    storage_account_id = azurerm_storage_account.data.id
    queue_name         = azurerm_storage_queue.events.name
  }
}

Delivery Identity

Initially, I tried to use managed identity for all authentication between the Function, the Event Grid Topic, and the Event Grid Subscription, but it did not end well.

There is a setting on the Subscription resource called delivery_identity that allows a managed identity to be used. However, as I discovered, only certain Event Grid topic types are supported, and queue storage and Azure Functions are not among them.

I went to great lengths by adding a separate user-assigned identity and role assignment:

resource "azurerm_user_assigned_identity" "queue_delivery" {
  resource_group_name = azurerm_resource_group.main.name
  location            = azurerm_resource_group.main.location
  name                = "mi-${var.name}${var.number}-eventgrid-queue"
}
resource "azurerm_role_assignment" "eventgrid_queue_data_contributor" {
  scope                = azurerm_storage_account.data.id
  role_definition_name = "Storage Queue Data Contributor"
  principal_id         = azurerm_user_assigned_identity.queue_delivery.principal_id
}

This still didn’t help because Azure Functions currently are not one of the supported endpoint types for this feature.

The Azure Function Endpoint

I did some GitHub searches for how others used the azurerm_eventgrid_event_subscription resource, especially the azure_function_endpoint block. This sometimes yields better real-world examples than official documentation.

Nonetheless, I kept getting a “400 Bad Request” error (InvalidRequest: Invalid ARM Id) during terraform apply, even though my ARM ID clearly looked valid:

│ Error: creating/updating Scoped Event Subscription (Scope: “/subscriptions/9db8d5ac-a7c8–4882–9720-d0c3424699d7/resourceGroups/rg-qonq-youtube-prod-eastus2/providers/Microsoft.EventGrid/topics/evgt-qonq-youtube-prod-eastus2”
54│ Event Subscription Name: “evgs-qonq-youtube-prod-eastus2”): performing CreateOrUpdate: unexpected status 400 (400 Bad Request) with error: InvalidRequest: Invalid ARM Id.
56│ with module.region_stamp_primary.azurerm_eventgrid_event_subscription.internal,
57│ on modules/regional-stamp/eventgrid.tf line 9, in resource “azurerm_eventgrid_event_subscription” “internal”:
58│ 9: resource “azurerm_eventgrid_event_subscription” “internal” {
61Error: Terraform exited with code 1.
62Error: Process completed with exit code 1.

Nonetheless, I kept getting a “400 Bad Request” error (InvalidRequest: Invalid ARM Id) during terraform apply, even though my ARM ID clearly looked valid: InvalidRequest: Invalid ARM Id.

Which is very odd. This message is implying that the following is not a valid ARM ID:

/subscriptions/9db8d5ac-a7c8–4882–9720-d0c3424699d7/resourceGroups/rg-qonq-youtube-prod-eastus2/providers/Microsoft.EventGrid/topics/evgt-qonq-youtube-prod-eastus2

I don’t know about you but…LGTM. That’s an ARM Resource ID if I’ve ever seen one. So I must be missing something.

Upon further investigation, I noticed that in one of the examples I found, they were adding “/functions/" to the function app’s resource ID in the Terraform configuration. This made sense: a single Function App can host multiple functions, and the subscription must know which specific function to route events to. For example:

  azure_function_endpoint {
    function_id = "${azurerm_function_app.fa_ce.id}/functions/EventGridTrigger1"
  }

Meanwhile, in my code, I was simply passing the Function App’s ID without specifying the function name.

  azure_function_endpoint {
    function_id = var.os_type == "Linux" ? azurerm_linux_function_app.main[0].id : azurerm_windows_function_app.main[0].id
  }

The endpoint in the example I found on GitHub didn’t look familiar to me. I had my doubts. So I thought I would go create a subscription manually and interogate it after I had successfully provisioned it.

Now that my function is loading correctly I can access my functions from the overview tab where they are clickable. Clicking on one of your functions takes you to a page that has all sorts of useful utilities. Code+Test where you can execute you function through the azure portal using the “Test/Run” button. The “Get function URL” button is also useful. When clicking that button my function produces this:

https://func-qonq-youtube-prod-eastus2.azurewebsites.net/runtime/webhooks/EventGrid?functionName=InternalSubscriber&code={code}

This URL looked different then the one on the EventGrid Subscription sample as well adding to my suspicions. I needed a more decisive way to determine what the URL should be. I did search for “runtime/webhooks/EventGrid” on Bing and found some documentation about it. This URL also includes the master key for my Azure Function which I don’t think makes sense having that included in the Event Grid subscription either.

Provisioning it Manually

To confirm the correct ARM ID for my function, I decided to create an Event Grid Subscription manually in the Azure Portal. I navigated to Integration tab, found my EventGrid Trigger visually represented, then clicked “Create Event Grid Subscription.” That flow took me to the standard Event Grid Subscription creation screen, pre-filled with my function information.

Alt

I click on the “Event Grid Trigger” and it shows me some details:

Alt

It also has a button called “Create Event Grid subscription” which seems like what I need. So I click that next. That lands me on a familiar screen: the Event Grid Subscription creation screen in the Azure Portal. This time with my Azure Function filled in.

Alt

So I humor it and create the thing. Now where can I get access to the ARM template that provisioned it? Well, it’s not in my resource group so where else can I look?

An AWS-esque Azure Portal Experience

I head over to the “Event Grid Subscriptions” view in my Azure Portal. Every resource in Azure has an isolated view that shows all the resources of that type. This reminds me a lot of AWS where you have to navigate all your resources of disparate types across different pages within the AWS console. You have to set the context of each page by selecting the AWS region that you want to look at. The EventGrid Subscriptions page in the Azure Portal makes me feel like we hired engineers from AWS and asked them to build it because it has a distinct AWS-esque feel.

Alt

Rolling drop downs to set the context at the top, Topic Type, Azure Subscription, and Azure Region all need to be selected in order to view the Event Grid Subscriptions. This is radically different from almost every Azure Portal experience and I’m not quite sure why.

Unfortunately when you look at an EventGrid Subscription there is no left navigation bar like you do on every other Azure resource. So the usual suspects like Export template are no where to be seen! How an I supposed to reverse engineer Terraform if I can’t get the ARM Template schema for the resource?

Let’s Look in Azure Resource Explorer

Let’s try the Azure Resource Explorer. However, when I navigate to the Resource Group that I provisioned it in I didn’t see any Event Grid Subscription resource types. Alt

Ok, maybe it doesn’t actually get provisioned in a Resource Group.

Alt

Still nothing. Only the topics are visible within the ARM Resource Explorer.

Let’s Look in the Azure CLI

When in doubt, turn to the Azure CLI.

az eventgrid event-subscription list --location eastus2

This produces the details that I want.

{
    "deadLetterDestination": null,
    "deadLetterWithResourceIdentity": null,
    "deliveryWithResourceIdentity": null,
    "destination": {
      "endpointType": "StorageQueue",
      "queueMessageTimeToLiveInSeconds": null,
      "queueName": "events",
      "resourceId": "/subscriptions/9db8d5ac-a7c8-4882-9720-d0c3424699d7/resourceGroups/rg-qonq-youtube-prod-eastus2/providers/Microsoft.Storage/storageAccounts/stdatadn35c86q"
    },
    "eventDeliverySchema": "CloudEventSchemaV1_0",
    "expirationTimeUtc": null,
    "filter": {
      "advancedFilters": null,
      "enableAdvancedFilteringOnArrays": null,
      "includedEventTypes": null,
      "isSubjectCaseSensitive": null,
      "subjectBeginsWith": "",
      "subjectEndsWith": ""
    },
    "id": "/subscriptions/9db8d5ac-a7c8-4882-9720-d0c3424699d7/resourceGroups/rg-qonq-youtube-prod-eastus2/providers/Microsoft.EventGrid/topics/evgt-qonq-youtube-prod-eastus2/providers/Microsoft.EventGrid/eventSubscriptions/evgs-qonq-youtube-prod-eastus2-queue",
    "labels": [],
    "name": "evgs-qonq-youtube-prod-eastus2-queue",
    "provisioningState": "Succeeded",
    "resourceGroup": "rg-qonq-youtube-prod-eastus2",
    "retryPolicy": {
      "eventTimeToLiveInMinutes": 1440,
      "maxDeliveryAttempts": 30
    },
    "systemData": null,
    "topic": "/subscriptions/9db8d5ac-a7c8-4882-9720-d0c3424699d7/resourceGroups/rg-qonq-youtube-prod-eastus2/providers/microsoft.eventgrid/topics/evgt-qonq-youtube-prod-eastus2",
    "type": "Microsoft.EventGrid/eventSubscriptions"
  }

this is the storage queue subscription. now let’s look at the azure function subscription:

{
    "deadLetterDestination": null,
    "deadLetterWithResourceIdentity": null,
    "deliveryWithResourceIdentity": null,
    "destination": {
      "deliveryAttributeMappings": null,
      "endpointType": "AzureFunction",
      "maxEventsPerBatch": 1,
      "preferredBatchSizeInKilobytes": 64,
      "resourceId": "/subscriptions/9db8d5ac-a7c8-4882-9720-d0c3424699d7/resourceGroups/rg-qonq-youtube-prod-eastus2/providers/Microsoft.Web/sites/func-qonq-youtube-prod-eastus2/functions/InternalSubscriber"
    },
    "eventDeliverySchema": "CloudEventSchemaV1_0",
    "expirationTimeUtc": null,
    "filter": {
      "advancedFilters": null,
      "enableAdvancedFilteringOnArrays": true,
      "includedEventTypes": null,
      "isSubjectCaseSensitive": null,
      "subjectBeginsWith": "",
      "subjectEndsWith": ""
    },
    "id": "/subscriptions/9db8d5ac-a7c8-4882-9720-d0c3424699d7/resourceGroups/rg-qonq-youtube-prod-eastus2/providers/Microsoft.EventGrid/Topics/evgt-qonq-youtube-prod-eastus2/providers/Microsoft.EventGrid/eventSubscriptions/evgs-foo",
    "labels": [
      "functions-InternalSubscriber"
    ],
    "name": "evgs-foo",
    "provisioningState": "Succeeded",
    "resourceGroup": "rg-qonq-youtube-prod-eastus2",
    "retryPolicy": {
      "eventTimeToLiveInMinutes": 1440,
      "maxDeliveryAttempts": 30
    },
    "systemData": null,
    "topic": "/subscriptions/9db8d5ac-a7c8-4882-9720-d0c3424699d7/resourceGroups/rg-qonq-youtube-prod-eastus2/providers/Microsoft.EventGrid/Topics/evgt-qonq-youtube-prod-eastus2",
    "type": "Microsoft.EventGrid/eventSubscriptions"
  }

I think I found the Azure Function Resource ID that I need. It looks like this:

/subscriptions/9db8d5ac-a7c8–4882–9720-d0c3424699d7/resourceGroups/rg-qonq-youtube-prod-eastus2/providers/Microsoft.Web/sites/func-qonq-youtube-prod-eastus2/functions/InternalSubscriber

The odd thing is that these resource blocks look like the standard fare as far as Azure Resources go. They do look a little different in terms of schema. For example, look at the structure of the EventGrid Topic:

{
    "properties": {
        "provisioningState": "Succeeded",
        "endpoint": "https://evgt-qonq-youtube-prod-eastus2.eastus2-1.eventgrid.azure.net/api/events",
        "inputSchema": "CloudEventSchemaV1_0",
        "metricResourceId": "a40bd43a-b82a-4f31-a436-ff144199d0d8",
        "publicNetworkAccess": "Enabled"
    },
    "sku": {
        "name": "Basic"
    },
    "kind": "Azure",
    "systemData": null,
    "location": "eastus2",
    "tags": {
        "application_name": "qonq-youtube",
        "environment_name": "prod"
    },
    "id": "/subscriptions/9db8d5ac-a7c8-4882-9720-d0c3424699d7/resourceGroups/rg-qonq-youtube-prod-eastus2/providers/Microsoft.EventGrid/topics/evgt-qonq-youtube-prod-eastus2",
    "name": "evgt-qonq-youtube-prod-eastus2",
    "type": "Microsoft.EventGrid/topics"
}

While similar, they have distinct differences. The Event Grid Topic has the standard schema: name, type, location, kind, tags, sku, properties. The Event Grid Subscription does not. There must be something very different about Event Grid Subscriptions that they do not conform to the standard ARM schema. This might have something to do with why the EventGrid Subscription portal experience is so different.

I ended up breaking up the Azure Function URL for the EventGrid endpoint into a local declared nearby the Event Grid Subscription.

locals {
  eventgrid_internal_subscriber_endpoint = var.os_type == "Linux" ? "${azurerm_linux_function_app.main[0].id}/functions/InternalSubscriber" : "${azurerm_windows_function_app.main[0].id}/functions/InternalSubscriber"
}

This allows me to support either Linux or Windows Function Apps (again — why do I care what the Operating System is?). Finally, I update my Event Grid Subscription to use the new endpoint URL.

resource "azurerm_eventgrid_event_subscription" "internal" {
  name                  = "evgs-${var.name}-${var.location}"
  scope                 = azurerm_eventgrid_topic.main.id
  event_delivery_schema = "CloudEventSchemaV1_0"

  azure_function_endpoint {
    function_id = local.eventgrid_internal_subscriber_endpoint
  }
}

Testing the Event Grid Subscription

Now that I made the update to my Terraform code and re-provisioned when I manually invoke my Internal Event Publisher, I see that my Internal Event Subscriber acknowledge the event in the Application Insights Live Metrics.

Alt

Screenshots of the incoming event logs confirmed everything was working.

Conclusion

By combining manual investigation in the Azure Portal with the Azure CLI, I was finally able to piece together the correct Terraform configuration for an Event Grid Subscription that invokes a specific Azure Function (rather than just a Function App). I hope this walkthrough helps you more rapidly navigate the intricacies of Terraforming Event Grid and Azure Functions.

Although there are still concerns about private networking and limitations with Managed Identities, I have confidence that this end-to-end setup — spanning infrastructure-as-code, application code, and real-time debugging — will eventually get put together. My plan is to wrap this into the Azure Functions GitHub AT-AT extensions so it comes out of the box for everyone. Good luck, and…

Happy Azure Terraforming!