Microsoft Flow HTTP Trigger <> Request Trigger, and you probably don't want to use it

Microsoft Flow has a fairly good UI update today, and with this a few “hidden” built-in triggers appeared.

The GeoFence Trigger is not available yet. But the HTTP Trigger is, and I wanted to write this blog post to explain how it works, and more importantly, why you probably don’t want to use this trigger.


HTTP trigger

Create this, the trigger calls the football API, it fetches back data…

The HTTP trigger is not a “new” trigger - it is something that’s in LogicApps for sometime. In essence, it is a polling trigger. So is this a run-once? Does it run many times? The answer is in the definition.


The Definition

https://docs.microsoft.com/en-us/azure/logic-apps/logic-apps-workflow-actions-triggers#http-trigger

The definition specifies

"HTTP": {
   "type": "Http",
   "inputs": {
      "method": "<method-type>",
      "uri": "<endpoint-URL>",
      "headers": { "<header-content>" },
      "body": "<body-content>",
      "authentication": { "<authentication-method>" },
      "retryPolicy": { "<retry-behavior>" },
      "queries": "<query-parameters>"
   },
   "recurrence": {
      "frequency": "<time-unit>",
      "interval": <number-of-time-units>
   },
   "runtimeConfiguration": {
      "concurrency": {
         "runs": <max-runs>,
         "maximumWaitingRuns": <max-runs-queue>
      }
   },
   "operationOptions": "<operation-option>"
}

But the designer for HTTP trigger does not let us specify the recurrence pattern for this trigger. So it ends up on the default, which is polling at 1 per minute.

That’s probably a really aggressive way to use up your Flow runs :-)

Summary

Four recommendations / notes:

  • Use a Schedule Recurrence trigger to specify what time and frequency we want our polling to run, then call HTTP as an action

  • Wait for a UI update that let us set up the recurrence trigger

  • Use FlowStudio to patch the JSON definition like a hacker. In a future FlowStudio update we will warn when these type of high-run Flows are created in the tenant unintentionally.

  • The Request Trigger turns a Flow into a web service. A HTTP trigger does not accept any requests, having both triggers now in Flow makes describing the correct trigger in text and blogs slightly tricker.

From Office 365 to Azure Event Grid, the events must Flow

Photo by Archana More on Unsplash

In this blog post, we capture all the events across an Office 365 Tenant from multiple event sources, gather them, and send them through an Azure Event Grid.

We then listen, filter and handle our events in a central, unified way.

The events must Flow.


This is also the full write up of this microblog posted to Twitter #FlowNinja earlier this month.


Plan

  1. What benefit do we get from this?

  2. Listen to every event across an Office 365 Tenant

  3. Construct a uniform event message

  4. Send them into a Serverless Event Solution - Azure Event Grid

  5. Filter and catch our events

We want to build 3 Flows

flow-event-grid-1.jpg

What benefit do we get from this?

First, we see the increasing availability of event hooks - we have subscriptions, delta queries, or webhooks, across various different products in Office 365. Some products like SharePoint is getting a SocketIO webhook. There will be more events, and our event handling design must evolve.

Second, we see the cost-effective solutions to handle these ever increasing flood of events in the form of Serverless compute. This is true with Azure Functions, Microsoft Flow or Azure Logic Apps.

We end up with a lot of individual event sources and a lot of individual event receivers. This is a common event handling problem. As number of events we handle increases, the worse the event management problem becomes.

Consider you have a “handle a document uploaded to a library” event - a very typical SharePoint Workflow. Now consider this library is cloned to a hundred project sites.

If we clone the event handler a hundred times, we have a problem.


If you have already done this with Flow, then try https://FlowStudio.app to help you manage them.
Ooh inline product placement!


If you are a developer, then consider this scenario.
Consider a typical event handling in the browser. A decade ago we used jQuery like this:

// 2008
$('button').click(handler);

// 2018
$(global).on('click', 'button', handler);

And gradually we find that unacceptable, because we have buttons, events everywhere, and managing individual event hooks was tedious and error prone. Eventually, we moved to a global handler model, and we filter just prior to event being raised.

The headache-less way to handle events is to set the hooks all at the global root level, and then filter by the event source and event type.

That is the exact reason we need Azure Event Grid.

  • Centrally manage our events

  • Decouple the event source from event handlers

  • Stay sane, with a hard problem

Listen to every event across an Office 365 Tenant

I had previously wrote about listening to Office 365 Management API via the HTTP action and app-only permissions. What I did not realize was that the Office 365 Management API also have fantastic webhooks.

I read about the webhooks from Kent Weare’s post, where he uses this event to get a trigger when Flows are created in the tenant.

https://flow.microsoft.com/en-us/blog/automate-flow-governance/

These are grouped into several categories: AzureAD, Exchange, SharePoint and General (other).

This is the subscriber. One picture of 4 blocks. I’m subscribing to three webhooks at once.



The Office 365 Management API is a fantastic general event source. The downside is that it’s not instant - event handler is called between 10-20minutes after the actual event. So it is great as an audit webhook, or for scheduling files or search or to signal for a bot to re-scan a document. But it’s not an instant webhook.

Of course, if our goal is to send events into an Event Grid - we can work with multiple event sources at the same time. We can add Microsoft Graph events or subscribe to SharePoint list webhooks directly.

This is the top of the Listener

Construct a uniform event message

The event grid has a event JSON structure, it also supports a CloudEvent structure.

When I built my implementation, the Event Grid connector is still in preview and I had troubles publishing a Cloud Event structure. I assume this wouldn’t be a problem anymore as the connector evolves.

This is the final loop design - all done.

Remember, we are running Serverless so abuse/utilize every opportunity to use as many Azure Servers as you can - if you can fan-out to parallelism you must.

Don’t talk to each individual HTTP action one at a time. Do (up to 50) all at the same time.


Send them into a Serverless Event solution - Azure Event Grid

The Azure Event Grid is a serverless event processing pipeline. It decouples our event source(s) from our event handlers.

Here is our first handler.

This catches every event on the Event Grid - in Event Grid, we see we have our first webhook attached - it appears as a LogicApps webhook.

Here are three examples of what it caught:

  • Flow created Event

  • Site Collection created Event

  • File uploaded Event

flow-event-grid-3.jpg

Filter and catch our events

We see the very specific webhook now registered on the event grid - and the filters are also listed

PPTX filter only runs when the file I’ve uploaded is a PowerPoint file.


Summary

I have been talking a lot about Serverless and how our tools and design must evolve. Having a unique Office 365 to Event Grid solution is something I talked about as far back as 2017. I’m glad a year later I’ve finally got a great prototype going.

  • Office 365 Management API is a great webhook source that catches all sorts of events. The downside is that it is an audit webhook, so the delay may not be acceptable to your needs.

  • Using Azure Event Grid to perform filtering and subscription gives us the unique ability to see EVERYTHING that’s going on in our tenant. That has tremendous value.

  • Because event source and event handling is now decoupled - we can add new event sources to push to the same Azure Event Grid. We can do this from Microsoft Graph, we can do this from SharePoint, or we can do this from a whole myriad of triggers available in Microsoft Flow

  • We can write Azure Functions to trigger off the Event Grid, and it would be visible as well.

  • I was reading and appreciating sending events to the new Azure SignalR service from Azure Functions - that would be pretty amazing to convert an event grid message into a websocket event.
    https://twitter.com/nthonyChu/status/1044427579460145152

The possibilities are endless. Our tools and our design must evolve.

The curious tale of the result() function in Flow and LogicApps

Photo by Andrew Neel on Unsplash

Photo by Andrew Neel on Unsplash

I owe this blog post to several persons, so I will list their names first:


I woke up to Kevin’s reply and something jolted me awake. See, I didn’t know there was a result() function, and it really isn’t documented at all in LogicApps Workflow Expressions. I have most of the functions memorized and I have never seen it.

Before I saw Kevin’s answer, I actually assumed it was an old article and the function doesn’t exist anymore.

I tried searching for it - I still don’t find it.

The RESULT() YOU ARE LOOKING FOR IS NOT here

And it looks darn useful.

How often do you find a completely undocumented function in a language and the product manager doesn’t say it’s an incorrect function?

And that’s how I woke up early on Thursday morning.

Go read the example article and see what you think, before you continue. You only get one chance to ‘feel’ that excitement I had when I read the article, that expression, if you continue reading, I will spoil that for you. Go read it and wonder, may be even stop now, and go explore it yourself and then come back :)

We have work to do. If you like, I want to compare notes, my notes are below.

Plan

  • Experiment

  • Understand

  • Apply

Experiment

So we create a Flow and immediately add two Compose to test the result function.

TALK about a ERROR message wow

This isn’t a completely crazy error - it tells me a few things

  • result function exists

  • it expects “actions of type ‘Enumerable < actions > ’

  • that looks like it expects actions that contains actions

Try the example in the docs, we do get it to work and can see its outputs.

Wrapping our test action inside a Scope, result(‘Scope’) returns actual real data.

It is a treasure trove of runtime information. Nested, grouped, runtime information.

Understand

Scope isn’t the only block in Flow that contains other actions, let’s try a few more of them

The Do Until block returns the final block and the repetitions.

The Apply to each block is extremely interesting. Because it returns outputs from every iteration. It is very very long.


Applications

The first application is to use this as a generic catch-all error.

Here’s a fancy Flow that tries to sometimes run a divide by zero.

This one is similar to the Logic Apps blog post - I use a fancy HTML table to show all the rows at once.

The second application is to use this as a very fast array append

This is a very common scenario where we want to group up individual apply-to-each results into an array for use after the apply-to-each block.

On one side - we have the traditional method of appending a line of result to a shared global array variable.

On the other side - we have apply-to-each and then we just output the result we want in a compose action. We use a Select after the apply-to-each to map the output from the result() function.

Apologies - the left/right of these screenshots toggled when I save and reload the Flow. So embarrassingly I have some of the pictures where result method is on the left, and other picture where the method is on the right.

Please refer to the two branches as "(result method)” and “(append method)”
If confused - please check the name of the apply-to-each block in the picture.

Change the apply-to-each to run in parallel


In parallel, the result() function is so amazing.

The 3rd Application is to run complex Select, using Apply-To-Each

and then use result() to gather the results, all without defining an array variable. This means you can run fairly complex apply-to-each block within a scope.

Bug

Because result() is a function that the Flow editor does not understand, it will save and run correctly, but the editor may mistakenly translate the expression to a string.

The main problem is that you may have a working Flow but when re-opening the Flow the editor turns an expression into a string and breaks the Flow, because it doesn’t understand that result() is a valid function.

I hope this bug is fixed soon.

Summary

result() is a cool function, and has some very important applications.

Microsoft Graph Community Call September 2018 - $Batch and Flow

I wanted to write a quick blog post letting everyone know I've put my hands up to present a 10 minute demo on using a Custom Connection for Microsoft Graph $Batch in Microsoft Flow.

Custom Connection = Swagger = WSDL

In Microsoft Flow, Power Apps or Logic Apps, a Custom Connection is a Swagger/Open API definition that allows these low-code solutions access to a well defined API, so it knows what to call and how to call them.

For old API guys that knew what a SOAP Webservice looked like, this is the WSDL that tells other systems how to use this service.

In general, calling Microsoft Graph with App-Only Permissions requires only a client-id and client-secret pair, and we can do it directly via the HTTP action without needing a custom connection.

When we do need to create a custom connection, that's usually because we need to call something with Delegate permissions

Why Flow - Connectors Framework

One of the hidden supper power of Flow and Logic Apps is the connectors framework, which cache, renew and manages connections across various different APIs and auth methods.  

This means - you actually don't need to write any of this code.  Repeat - as a developer, you don't have to write code to manage auth.  Auth is encapsulated away and auth code is no longer your concern.

Why $Batch

You'll have to see the demo (I'll update this later).  Hey, I can't spill all the beans in one blog post.  This is a tease.

The result

If you want to automate any delegate-permission API calls on Microsoft Graph, then Microsoft Flow is the easiest way to do it - far easier than writing code, and this is considered a low-code solution, it's crazy nuts.

Resource

Link to the recording will be updated here once presentation is done and uploaded.

A Thesis on the Parse JSON action in Microsoft Flow

Parse JSON can be both intelligent and dumb.  It can help you a lot and it can hinder you just as much.

This blog post, we go deep and study the behaviour of the Parse JSON action, it's various problems and solutions.

Here we tackle the question - How do you use Parse JSON, actually?

Plan

  • Getting Started with Parse JSON
  • Problem - way too many properties
  • Problem - the array - item object and auto-wrapping
  • Problem - duplicate properties names?
  • Problem - null value properties
  • Problem - missing properties
  • How to quickly fix the schema
  • Result is Parse JSON will make the Dynamic Content Panel actually intelligent

Getting Started with Parse JSON

In general, the reason we use Parse JSON is because Flow doesn't know the format of the data we are receiving in our actions.  So we always start by running the actual method once - even if the Flow has only been partially completed.  We want to get a sample of the data we'll be seeing.

Look at all these properties in the Dynamic Content Panel.  Looks amazing.  Our problems are about to happen.  Don't worry we'll fix all of them.

 

Problem 1 - way too many properties

The schema Parse JSON generates is very very verbose.  So there are a lot of properties that we probably just don't care.

So copy the generated JSON Schema from the Parse JSON action, and use a text editor to have a look at it.  Here I'm using VS Code - switch the editor mode to JSON.

We see it has generated type information for various _links properties, or in this example, the halfTime results or odds of a football game.  We don't need this, so we can delete it from the schema.

 

Problem 2 - the array - item object and auto apply-each wrapping

Notice under the fixtures (array) we have an items property with each object.  This appears under Dynamic Content Panel as "fixtures - Item"

Becareful when clicking on this property, it will immediately unfold with an For-Each block.

It depends whether this is actually what we want - typically, when we aren't in a For-Each block, this is helpful.  But if we are already inside a For-Each block this may create a second nested for-each block and be quite unnecessary and confusing.

 

Problem 3 - duplicate properties names?

Hey why do I see duplicate names?  How can we tell which one is which?

The simplest way to ease this problem is to delete the properties in the schema that you don't want (same as problem 1).  This will reduce the number of duplicate properties that appears.  But sometimes, we still get some duplicate names - often I see the top level object has a 'name' and the child object has another 'name'

In this case, I check with the hover over to make sure I've selected the right choice.

 

Problem 4 - null value properties

(This problem category also applies to all type mismatch errors)

In general, Parse JSON generates the schema based on the first object it sees from the sample.  This is not always correct - since sometimes the properties may be null in the later objects.  For example in the football games data - earlier records has scores as integers, but later records haven't been played yet so scores are null.

This error usually fails the Flow.  Another mutation of this problem is when the number is a decimal, but the schema thinks it's integer.

The easiest way to fix nullable-values is to delete the type information for this property.

Using no type information is better than changing to something like type: "object" because the Dynamic Content Panel will always show these properties.

If goalsAwayTeam was set to object, then it wouldn't be available here.

 

Problem 5 - missing properties

Sometimes, a later object in the array doesn't have a property at all.

The fix is simple - remove the "y" from "required"

But sometimes this is half the problem - because the expression used after this may expect the property to exist and always available.

 

Result

Parse JSON will make the Dynamic Content Panel actually quite intelligent.  But it requires some clean up maintenance on the JSON schema.

It is up to each scenario to decide if there's value to fix the JSON Schema, or to skip using Parse JSON and just use item()?['property-name'] expressions directly.