Make a wishing wand - why we need a MicrosoftFlow Custom Connector for MSGraph $Batch

/

A question that has been in the back of my mind since my late night demo of building a Microsoft Flow Custom Connector for Microsoft Graph is this.

I focused on the "how" - because I wanted to show that the whole thing can be done in about 10 minutes.  And I didn't want to skip steps like "oh here's how you register an Azure AD app".

 

But (and this is my fault, I did only ask for 10 minutes) I should have spent some time to talk about the "why"

Why do we need this?  What problem does this solve?

Why does John think this is super special?

This is special because, a $Batch connector is a "Send HTTP Request to MS Graph" as me.

It is a wishing wand that grants wishes. Creating MSTeam is a wish. We didn't build a wish. We built a wishing wand. Then we waved it around.  MS Teams API currently is delegate permission only, so we call it through a $batch connector.

Planner API is delegate only.  Any time you see a delegate permission MSGraph you remember you have a wishing wand.

The Ingredients:

Microsoft Flow and Logic Apps has concept of connectors (these are essentially API definitions) and connections (these are instances of connectors bound to a resource as well as cached automatic authentication)

A Flow is a JSON description of how actions and triggers on the connection are chained together to perform automation.

We do have a general purpose HTTP action, and it works very well for app-only permissions that needs client-id/client-secret or client-cert.  But HTTP action is not backed by a connectors framework so it can't remember credentials.

To build a general purpose tool and remember credentials for Microsoft Graph, we have to build a custom connection.  This is basically a swagger API file that tells Flow and LogicApps what APIs exist on the resource.  In the case of MSGraph, that list is enormous.

The Recipe:

The crazy part though, is MSGraph provides a $batch endpoint, so the recipe for building the magic wand is simply this:

  1. Build a custom connection with one method - $batch - this is a POST
  2. Pre-authenticate it with a delegate permission
  3. Wrap requests (1 or many) as a JSON array and call them via the $batch connection.
  4. We have a magic wand

Agenda for the September community call included Microsoft Graph updates, Microsoft Graph and Microsoft Teams, and custom connectors for Microsoft Graph $Batch

Where do we go from here?

I demoed this with a Manual button trigger.  Jeremy mentioned (in the video) that this can easily be a SharePoint trigger.

This can also be a HTTP Request trigger, turning your Teams Provisioning into one simple HTTP Webservice.  You can hook that up to external systems, PowerApp or a bot.

A technical note

Sometimes you discover you need extra permissions on that Azure AD app.  We do this by going back to Azure AD portal and add the extra permissions.  You can "grant permission" through the AAD portal.

If the cached connection in Flow refuses to work, I find that we may need to right-click on the custom connection and re-authenticate.  Sometimes this requires us to switch to a different user then switch back.  Because Azure AD dynamic consent is just so hard to do.

But it will work.  Just need to toggle a few times.  I do this a lot.

Resources

 

Microsoft Graph Community Call September 2018 - $Batch and Flow

/

I wanted to write a quick blog post letting everyone know I've put my hands up to present a 10 minute demo on using a Custom Connection for Microsoft Graph $Batch in Microsoft Flow.

Custom Connection = Swagger = WSDL

In Microsoft Flow, Power Apps or Logic Apps, a Custom Connection is a Swagger/Open API definition that allows these low-code solutions access to a well defined API, so it knows what to call and how to call them.

For old API guys that knew what a SOAP Webservice looked like, this is the WSDL that tells other systems how to use this service.

In general, calling Microsoft Graph with App-Only Permissions requires only a client-id and client-secret pair, and we can do it directly via the HTTP action without needing a custom connection.

When we do need to create a custom connection, that's usually because we need to call something with Delegate permissions

Why Flow - Connectors Framework

One of the hidden supper power of Flow and Logic Apps is the connectors framework, which cache, renew and manages connections across various different APIs and auth methods.  

This means - you actually don't need to write any of this code.  Repeat - as a developer, you don't have to write code to manage auth.  Auth is encapsulated away and auth code is no longer your concern.

Why $Batch

You'll have to see the demo (I'll update this later).  Hey, I can't spill all the beans in one blog post.  This is a tease.

The result

If you want to automate any delegate-permission API calls on Microsoft Graph, then Microsoft Flow is the easiest way to do it - far easier than writing code, and this is considered a low-code solution, it's crazy nuts.

Resource

Link to the recording will be updated here once presentation is done and uploaded.

Do bulk insert/update in Microsoft Flow with two simple elegant filter-arrays

/

This is a blog post of a pattern that I've brooded over for months and finally came to an elegant solution.

Plan

  • Scenario: Bulk Insert / Update
  • Doing filter-arrays elegantly
  • Build it in two parts
  • Run it super quickly
  • Bonus: extensions

The style of this blog post mirrors my previous post on Do Group By in Microsoft Flow with two simple elegant loops.  I presented this during the June 2018 Microsoft Graph Community call.  This blog post is the text, searchable, extended version.

Scenario: Bulk Insert / Update

Bulk insert and update is a very common problem with scheduled Microsoft Flow.  Basically, we have a list of resources - it could be a SharePoint list, an Excel file, a CSV import file, a SQL table, result of a Microsoft Graph call... etc, and we want to either update or insert the entire table into a destination - another SharePoint list etc.

In the example, we will do this with Microsoft Graph's List Groups call.

 

First, call Microsoft Graph groups with a GET request - this gets back a JSON array of Office 365 Groups.

 

We need a SharePoint list to write this to - I made a list with a few columns: groupId is important, the rest is just useful: description, visibility, createdDateTime and renewedDateTime.  I also suggest deletedDateTime, since now groups have a recycle period before true deletion.

 

The plan is this: when this Flow runs, we call Microsoft Graph for list of groups, then perform fast-bulk update/insert into SharePoint list.

 

Doing elegantly with filter-arrays

The first step is to select a list of existing groupIDs from the current SharePoint list.  We do this with two actions.

What is this magic trick?  Select action has the ability to map array of objects into... anything we want.  A bit of code may explain this better:

var results = array.map( (item) => { "id": item.groupId } );
// results = [ {"id":"xxx1"}, {"id":"xxx2"} ]

var results = array.map( (item) => item.groupId );
// results = [ "xxx1", "xxx2" ]

The first UI with key/value lets us create result arrays of objects that has key/value properties.  The second UI form with just "value" lets us create flattened array of string, or number, or really complex nested objects.

Anyway, the result of the Select action is now an string array of groupIds that I've already got in my SharePoint list.

Then we follow with two Filter-Array on the JSON result from the source list: the Microsoft Graph array.

So we have two Filter Array actions.

"New FilterArray" contains rows from MS Graph Groups whose "id" does not exist in the array of "groupIDs" from SharePoint.

"Update FilterArray" contains rows from MS Graph Groups whose "id" already exists in the array of "groupIDs" from SharePoint.

 

Build it in two parts

The hard part is already done, the rest is simply wrapping it up.

Hew done!

The benefits:

  • Compose, Filter, Select are native Flow/LogicApps engine expressions and they run super fast.  Use them to do all the filters and sort and array.map
  • Do not check SharePoint for each row - so the SharePoint connector speed is much faster

For-Each is still slow, but we can do more

Run it super quickly

Now, we have two arrays and two loops through them, one does only create and the other does only updates.  They don't need to wait for each other.  In fact, they can all run in parallel.

This dramatically drops the duration

 

Extensions

We can add a really simple extension

  • When there are "new" records, we want an email about this.
  • Connect Flow Management's "List Flows as Admin" and be notified when new Flows are added to your environment by anyone.
  • Listen to Office 365 Service Admin Center messages and be notified when new events happen.  Push them into Microsoft Teams channel for discussion.

 

Send mail as anyone - #MicrosoftGraph and #MicrosoftFlow (bonus: inline image attachments)

/

Sometimes while browsing MS Graph permissions, you come across something like this:

This is an Application Permission, it says "Send mail as any user"
The correct next step is of course we drop whatever we've been working on and immediately play with this.

Plan

  • Explore what is Send mail as any user
  • Do attachments
  • Do inline attachments

First, while still in the Azure Portal - grant this permission to the current service app.

Next, we head over to Microsoft Graph Explorer and grab a sample of the Exchange SendMail 

sendMail is usually on: /v1.0/me/sendMail for delegate permissions, but since we are testing sending email as another user, I replaced the URL to:

/v1.0/users/{user-id-guid}/sendMail

This is really as simple as it looks.

Gandalf is now asking me to try the new cafeteria.

What can you do with this?  Lots of people ask the Flow team how they can send email on behalf of the current user onto their managers.  This will do it easily, without having to worry about delegate permissions.

Extra notes: the service app does not have permission to read the user's inbox or login as that user.

And if it's so easy to send emails via MS Graph, let's try attachments.

Do Attachment

We add the JSON for attachments.  For this we need the $content-type and $content

flow-sendmail-json-attachment.jpg

Result:

 

Now, do inline attachments

Why do we do inline attachments?  There are few options to embed images as part of your rich HTML email.  You can use background image via CSS - this is mostly ignored by everybody.  You can use inline img with dataUri - this pretty much only works in iOS.

The oldest way and still the most supported way is to do inline attachment.  The trick is that your attachment must have an additional content-id (cid:) header with a unique name.  Then the HTML mail body can refer to that image wtih <img src="cid:xxxyyy" />

I left the original attachment as a comparison.  See the second attachment is marked inline and has a contentId - boromir12345.  The contentId is used in the HTML content as a embed reference: cid:boromir12345

Result:

Summary

MSGraph lets you send email as anyone.  
It also gives you much better controls over attachments and inline images.  

Please watch the recent MS-Flow Webinar I did with MS-Flow team focused on working with Binary Data.  That will explain how I take binary values and split them into $content and $content-type.

 

Setting up MSGraph Webhook with HTTP Action in MicrosoftFlow

/

I've tweeted out several small tidbits of using Microsoft Flow's HTTP action to call the Microsoft Graph.

Hundreds of Graph APIs, dozens of Graph webhooks, one HTTP Action.

This little action continues to amaze me, so I'm putting several examples into this one blog post.

Four Techniques, One Action

  • Connect to Microsoft Graph with ONE HTTP Action 
  • Setting up Microsoft Graph Webhook Subscription
  • Paging
  • Retry Policy

One HTTP Action 

When we set Authentication to "Active Directory OAuth" - we can specify the Client ID / Client Secret in one HTTP action - so we don't need to make two separate calls first to Authenticate and get an access token, then call the flow we want and add the bearer header token.

This one action does it and asks no questions.

  • Tenant, Client ID and Client Secret are 3 strings. 
  • Authority should be set to https://login.microsoftonline.com/
  • Audience (resource) should be set to https://graph.microsoft.com

So yep, it's now easier to call any Microsoft Graph API from Flow than C#, PowerShell or JavaScript.

I'm in love with this, because this is way too amazing.

 

Note - this calls Flow via an App-Only Client ID.  If you are looking for delegate calls, you'll need to set up a Custom Connection swagger file.  Follow @skillriver https://gotoguy.blog/2017/12/17/access-microsoft-graph-api-using-custom-connector-in-powerapps-and-flows/

 

Microsoft Graph Webhook - a dozen new triggers

To set up webhook, we need to set up two Flows.  The First one is the subscriber.

The subscriber should be set up with a recurring 3 day schedule.

The notificationUrl is the HTTP Trigger URL of the second Flow.   

The expiry is 4229 minutes into the future from right now.  The maximum value is 4230 minutes.  If you go over the subscription call will fail.

On success, the subscription is set up and we are now listening to changes in our tenant's groups.

The Second one is the listener.

The listener needs to handle the validationToken that Microsoft Flow will call to test if your webservice follows the specs.

Read the trigger query string to pull out the validationToken.  if the value exists - then this is a set up call.  Respond immediately with text/plain 200 text.

Otherwise, we have a real call.  This is an event where our resource (in my case, I'm listening to Unified Groups being created and modified in my tenant) tells us something is happening. 

I call another Flow to start dealing with the change.  The other Flow does not return a response.  So the HTTP action can happen quickly as a trigger, and I can return 202 accept response to Microsoft Graph quickly.

 

This is the message sent from Microsoft Graph to Flow to tell me I've got a new Group created in my tenant.

Episode III in my blog series on Group Management with Flow will cover the webhook in more detail, combining it with a delta query and figuring out what changed.

 

Pagination

It turns out HTTP Pagination is baked in too.

To set this up - first set a $top in the Microsoft Graph call to artificially limit the number of rows returned.

This will also return a $nextLink

https://developer.microsoft.com/en-us/graph/docs/concepts/paging

Flip to the Settings for the HTTP action by clicking on ...
This lets you turn on Pagination and it will accept up to 5000 items.

The result is that the HTTP action will follow next page links automatically, and return you the entire array of the paged data concatenated together.

Magic.  Still one action.

Retry Policy

When calling HTTP (or other ApiConnections, like SharePoint) the default policy of retry 4 times exponentially means that if your action is going to fail, it will fail four times.

When building a Flow and you want it to fail fast - set the Retry Policy to None.

This is useful for calling AzureFunctions as well.  AzureFunction through host.json lets us control de-queuing speed and concurrency, but that requires you to use a Queue trigger.  When we use HTTP action from Flow - we control the retry / ease off policy through this setting.  In this use case, Flow is the orchestrater.

 

Summary

Calling Microsoft Graph directly with as simple as 1 HTTP request action means that a lot of slightly-more complex task of authenticating, then getting an access token, then calling MSGraph with bearer header becomes a whole lot easier.

And as steps get simpler, we can do a whole lot more.