Send custom data from Logic Apps to Microsoft Sentinel
How to send custom data from Azure Logic Apps to Microsoft Sentinel using the Logs Ingestion API.
Background
In Azure Logic Apps we’ve long had a pre-built action for sending data to a custom table in Log Analytics Workspace:
This has been a very useful way to create custom integrations to Microsoft Sentinel. For example, I have built a solution for ingesting Shodan Monitor alerts, which is based on a listening HTTP webhook and the “Send data to Log Analytics” action.
Unfortunately, this action uses the old Data Collection API which is being deprecated.
The new API for custom table integrations is the Logs Ingestion API, but there is no prebuilt Logic App action for this. In this article I will give a quick explanation on how to continue sending custom data from Logic Apps to Sentinel with the new API.
To learn more about how the API works, I can recommend reading the official documentation - and also everything MVP Morten Knudsen has written and coded related to data ingestion with the new API.
Solution description
As a brief architecture overview on how integrating custom data from Azure Logic Apps to the new Logs Ingestion API and Log Analytics and Microsoft Sentinel works, you can refer to the following:
Fundamentally what we are doing compared to the old Logic App method is:
Replacing the existing Send data to Log Analytics action with a custom HTTP action that does a HTTP POST to Azure Data Collection Endpoint.
Authorising the Logic App with Managed Identity and Azure RBAC role to write data to Azure Data Collection Rule, instead of using Workspace ID and a shared secret key.
In the examples described here in this article, we create a simple demo environment consisting of the following resources:
Custom Log Analytics table with a very simple schema.
Data Collection Endpoint based on the official Microsoft template.
Data Collection Rule based on a template that matches our table schema.
Logic App to write example events to the table.
Example templates and scripts for the whole solution can be found here:
https://github.com/mikoiv/Example-LogicApp-MicrosoftSentinel/tree/main
You can deploy the examples if you want, or just follow along with the screenshots for an understanding on how this works.
Step 1: Creating the custom table
We can create the custom table with a simple PowerShell script that is found in the linked repository.
When creating running the script, we need to specify our Entra tenant and Log Analytics workspace information:
The schema for our custom table is defined in the script, in this example it is simple:
Step 2: Creating the Data Collection resources
To ingest data via Logs Ingestion API to the new table, we need two resources in Azure:
Data Collection Endpoint (DCE)
Data Collection Rule (DCR)
Example templates for these are in the repository, with deployment screenshots below.
Deploy DCE from template:
Deploy DCR from template:
For the DCR deployment you need to fetch the Resource ID for the Log Analytics workspace and the Data Collection Endpoint. Do this by navigating to the resource in Azure Portal and “JSON view”.
After deploying the DCR, store the DCR Immutable ID. This is found from the DCR resource by selecting “JSON view”.
Step 3: Creating the Logic App
Example Logic App template is found in the linked repository.
The Logic App flow is pretty simple in this example, as you can see from the screenshot:
This flow does not really make sense in the real world, as it just writes the same data over and over every hour. But it provides you with a starting point to build your own integration, such as using a HTTP Webhook to trigger the Logic App.
The only thing that really matters here is the HTTP action that writes to the DCR.
In the variables you need to set the following information:
DCE Ingestion URL
Find this from the DCE resource Overview
DCR Stream Name
Find this from the DCR, my example is “Custom-LogicAppExample”
DCR ImmutableID
Find this from DCR resource JSON View
After the Logic App is created, you need to assign it the Monitoring Metrics Publisher role to the Data Collection Rule, usually at the Resource Group level:
And after running the Logic App, we have our custom data in Microsoft Sentinel:
There you go! A long winded explanation maybe for a simple task, but as I know many rely on Logic Apps for custom integrations, I thought it would be worthwhile to explain it.