Azure Log Analytics (ALA) is one possible tool to use for logging errors that occur within Boomi. Although, the alerts that come out of ALA are not always the most helpful because of how specific the alert needs to be. This article will cover how to create a shareable link within Boomi that can be used to add to an error notification email. Additionally, it will provide a few additional suggestions on ways to query ALA to create a meaningful notification email.

Create a Shareable Link

Figure 1: Build ALA Link Custom Map Function

The shareable link will be created within a custom map function and will include a few process properties so that they can be extended and set with environment extensions. Below is the list of process properties that will be used and can be seen within the custom map function.

Table 1. Process Properties To Be Used to Build an ALA Shareable Link

The script below takes the four process properties and the execution id to create an shareable link to ALA. The script starts off with a generic ALA Kusto query to search for the execution id. That assumed that the Execution Id is a field within the custom log. This query should be updated to fit the needs of your company’s logging. The query is then converted to bytes, GZip compressed, and finally base64 encoded. The base64 encoded query is then URL encoded. The URL is then build with all of the provide information. The base URL for Azure Public Cloud is https://portal.azure.com/#@/. The URL for Azure Government Cloud is https://portal.azure.us/#@/. Update the script as needed.

When the link is clicked, it will open up the Azure Portal and navigate to the Log Analytics workspace and run the query. Additionally, it assumed that the person clicking the link has access to the Azure Portal and the Log Analytics workspace.

// Groovy 2.4
import java.util.zip.GZIPOutputStream
import java.nio.charset.StandardCharsets

/*
INPUTS:
  tenantId : Azure Tenant Id
  subscription: Azure Subscription Id
  resourceGroup: Azure Resource Group
  workspace: Azure Log Analytics Workspace
  executionId: Boomi Execution Id

OUTPUTS:
  buildUrl: Azure Log Analytics Shareable Link
*/

String query = """BoomiIntegrationLogs_CL
| where ExecutionId_s == "${executionId}"
| where TimeStamp_t > ago(365d)
| order by TimeStamp_t asc;
"""


byte[] encodedQueryBytes = query.getBytes(StandardCharsets.UTF_8)
byte[] bytes = compress(encodedQueryBytes)
byte[] base64EncodedBytes = Base64.getEncoder().encode(bytes)
String base64EncodedQuery = new String(base64EncodedBytes)
String UrlEncodedQuery = URLEncoder.encode(base64EncodedQuery, "UTF-8")


buildUrl = "https://portal.azure.us#@" +
        tenantId +
        "/blade/Microsoft_OperationsManagementSuite_Workspace/Logs.ReactView/resourceId/%2Fsubscriptions%2F" +
        subscription +
        "%2Fresourcegroups%2F" +
        resourceGroup +
        "%2Fproviders%2Fmicrosoft.operationalinsights%2Fworkspaces%2F" +
        workspace +
        "/source/LogsBlade.AnalyticsShareLinkToQuery/q/" +
        UrlEncodedQuery


byte[] compress(byte[] input) {
    byte[] compressedQuery = null
    ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream()
    GZIPOutputStream gzipOutputStream = new GZIPOutputStream(byteArrayOutputStream)
    try {
        gzipOutputStream.write(input)
        gzipOutputStream.close()
        compressedQuery = byteArrayOutputStream.toByteArray()
    } finally {
        if (gzipOutputStream != null) {
            gzipOutputStream.close()
        }
    }
    return compressedQuery
}

Working With ALA within Boomi

The first sections of the article are the basic pieces that are needed to send out a link to view errors within ALA. The rest of the article will go over how to configure an HTTP connection to ALA, how to connect to ALA to perform queries, and example queries on ways to get helpful information. Other than how to set up the connector, the rest is more a suggestion of ideas to consider when working with ALA for your error notifications.

Configure the HTTP Connector to Query ALA

More than likely, you will initially query ALA to see if there are any errors. Then from the log entries you will obtain the execution id. The execution id can then be used to create the shareable links from the above sections.

Azure Document provides all of the steps required to make an Azure App that will connect to Azure Log Analytics through HTTPS OAuth2. Once those steps are complete, you will be able to create an HTTP Connector within Boomi.

Azure Documentation: Azure AD App - How to Register an App for API Access

Azure Documentation: ALA Authentication

Figure 2. ALA Azure App HTTP Connector

Table 2. HTTP Connection Configuration

How to Perform an ALA Query within Boomi

Once an HTTP connection has been set up, HTTP operation and payload can be configured. First, configure the HTTP operation and create SEND operation. Provide the following information in the operation.

Figure 3. HTTP Operation to Query ALA

Table 3. HTTP Operation Configuration

The payload is a single JSON element that contains the ALA Kusto query. Create a message shape that contains your query and any variables, such as last successful run date. This query should look exactly like the query that is ran within the ALA UI. Then use a Set Property shape to set a Dynamic Document Property (DDP_CURRENT_DATA) to the current data. Within a map, map DDP_CURRENT_DATA to the query element. The source profile in the map is set to a Flat File profile that only has a single element. This profile is used so that the data will not be parsed.

Figure 4. Build ALA Query within Boomi

Figure 5. Map ALA Query within Boomi

Below is an example of the JSON payload that is sent to ALA and it can be used to import the ALA Query Request profile.

{
  "query": "<map-current-data-here>"
}

Suggested ALA Kusto Queries

Finally, here are a few suggested queries that can be used to get helpful information from ALA. These queries are just suggestions to provide ideas to consider. The queries should be based on the custom tables and schemas used within your ALA workspace.

You might want to perform a simply query to see if any errors have occurred since the last time the error notification integration has ran. This query will find any LogLevel_s == “ERROR”. The TimeStamp_t in the where clause can be set from the last successful run date.

BoomiIntegrationLogs_CL
| where LogLevel_s == "ERROR"
| where TimeStamp_t >= todatetime(''{1}'')

The following query will aggregate error messages based on the process name and the first 80 characters of the error messages, and return a count. Clustering error messages on the first 80 characters reduce the number of unique error messages if there is the possibility of a datetime or id within the error message. Once the unique error messages have been gathered, you can perform an additional query to get the most recent error message and the aggregated count. Once this information has been gathered, it can be used to a create a meaningful error notification email. These two queries should be ran together.

let AggregatedCount  =
BoomiIntegrationLogs_CL
| where LogLevel_s == "ERROR"
| extend LogMessagePrefix = substring(LogMessage_s, 0, 80)
| summarize Count = count() by ProcessName_s, LogMessagePrefix;

BoomiIntegrationLogs_CL
| where LogLevel_s == "ERROR"
| extend LogMessagePrefix = substring(LogMessage_s, 0, 80)
| join kind=inner (AggregatedCount) on ProcessName_s, LogMessagePrefix
| project ProcessName_s, LogMessagePrefix, TimeStamp_t, Count, LogMessage_s,  Environment_s, AppName_s, ExecutionId_s
| where TimeStamp_t >= todatetime(''{1}'')
| where TimeStamp_t <= todatetime(''{2}'')
| summarize arg_max(TimeStamp_t, *) by ProcessName_s, LogMessagePrefix
| order by ProcessName_s asc, TimeStamp_t desc;

References

• Azure AD App - How to Register an App for API Access
• ALA Authentication
• Tobias Zimmergren - Deep-linking Azure Log Analytics and Application Insights queries

The article was originally posted at Boomi Community.