Office 365 Groups management as a service - Flow, Functions and MSGraph

/

Because Office 365 Groups is a key component of group membership in Office 365, there will always be an evolving story on how to manage it, extend it, report on it and automate it.

Microsoft will continue to add more complex features to Azure AD Premium.  And this will be an enterprise grade solution to many customers that wants these features.  But there is a lot of room for partners to build Groups Management solutions.  

Ultimately, we have realized this: Our client's organization has unique rules, and there's a need to customize, fortunately, we have all the tools at our disposal, and they are not hard to do.

Group Management Life Cycle

This is a post in a series about Office 365 Groups management life cycle.  This is the first post - we will discuss the history Office 365 Groups creation.

Office 365 Groups Creation

There have been many blog posts about how we can automate Groups creation.  So this discussion is about understanding the underlying pieces, and how we must see these pieces as the building blocks to solve all your future problems.

Timeline of availability of O365 Groups creation methods

2015

2016

  • MS Graph API for group creation available
  • PnP Sample Solution - Vesa demo'ed in late 2016 with bots and Azure App Service
  • Azure Function enabled very simple hosting of small code

2017

    Connect-PnPMicrosoftGraph (PR by Mikael Svenson Feb 2017)
    New-PnPUnifiedGroup (available October 2016)

One Flow Connector to rule them all

I only know this: Everything we know got simplified

So here we are.  At the end of the first blog post on Groups Management - we need to understand the trend:

First, we have API
Then we have PowerShell, we have AzureFunctions for code or HTTP Request in Flow
Eventually we have a Flow Connector

We must see this pattern.  Everything we want to build is in one of these stages.

When we understand these pieces - there's nothing we can't build, for practically free.  The question isn't "No, this is too hard" or even "This should be free or out of box".

The question we should all be asking is "Where do I find a piece that fits here, right now" because I have a crazy customization need.

In 2017, everything got abstracted and simplified.  This trend will continue into the future - there will be more Flow Connectors.  Azure Functions will get more upgrades - durable functions are absolutely amazing.  MS Graph will get more API endpoints.  Life will be even more amazing.

Future of this series

I am not alone in what we are building for our customers - many consultants, partners and ISVs have already moved on to more complex issues:

  • Creation - Complex approval chains for Groups creation
  • Creation - Which products to enable on Groups creation
  • Creation - Post-Groups creation SharePoint site templating
  • Maintain - Scheduled Groups compliance audit reporting 
  • Maintain - Owner leaving organization scenario
  • Maintain - Members changed roles scenario
  • Closure - Expiration policies
  • Closure - Group archiving and closure

There are a lot of solutions to solve.  I want to cover more of Office 365 Groups life cycle management, with Flow and Functions, all on top of what MS Graph gives us already.

If you are interested in this topic or have shared some of the ideas you are working on - please share them with me and I would be happy to link to your work.

I'm speaking about Serverless Flow and Azure Functions at Collab365 Free Online Conference

/
collab365-watch-my-session.jpg

Have you heard about the virtual Collab365 Global Conference 2017 that’s streaming online November 1st – 2nd?

Join me and 120 other speakers from around the world who will be bringing you the very latest content around SharePoint, Office 365, Flow, PowerApps, Azure, OneDrive for Business and of course the increasingly popular Microsoft Teams. The event is produced by the Collab365 Community and is entirely free to attend.

Places are limited to 5000 so be quick and register now.

During the conference I'd love you to watch my session which is called : 

'Serverless with Microsoft Flow and Azure Functions'

Level up your mastery of Microsoft Flow. Switch to Azure Functions only when you need to. No doubt there will be many sessions on Microsoft Flow, introducing you to its wonderful merits and rough edges. This session is for the advanced users - we will see what Microsoft Flow really is, and bend it to our will.

If you join me, you will learn:

  • Master JSON in a Flow
  • Combining Azure Functions with Flow
  • Failure Recovery, in a Flow
  • How to handle Binary in a Flow
  • How to write HTML and generate PDF in Flow

Topic(s):

  • Azure Functions
  • Microsoft Flow

Audience :

  • Developer
  • Power User

Time :

  • Thursday, November 2 2017 10:00 AM (UTC)
  • Thursday, November 2 2017 09:00 PM (ADST)

How to attend :

  1. Register here.
  2. At the time listed above go here to watch my session. (you can also add me to your own personal planner from the agenda.
  3. Enjoy the demos and ask me questions, I'll put the templates up for download after the session.

Serverless connect-the-dots: MP3 to WAV via ffmpeg.exe in AzureFunctions, for PowerApps and Flow

/

There's no good title to this post, there's just too many pieces we are connecting.  

So, a problem was in my todo list for a while - I'll try to describe the problem quickly, and get into the solution.

  • PowerApps Microphone control records MP3 files
  • Cognitive Speech to Text wants to turn WAV files into JSON
  • Even with Flow, we can't convert the audio file formats. 
  • We need an Azure Function to gluethis one step
  • After a bit of research, it looks like FFMPEG, a popular free utility can be used to do the conversion

Azure Functions and FFMPEG

So my initial thought is that well, I'll just run this utility exe file through PowerShell.  But then I remembered that PowerShell don't handle binary input and output that well.  A quick search nets me several implementations in C# one of them catches my eye: 

Jordan Knight is one of our Australian DX Microsoftie - so of course I start with his code

It actually was really quick to get going, but because Jordan's code is triggered from blob storage - the Azure Functions binding for blob storage has waiting time that I want to shrink, so I rewrite the input and output bindings to turn the whole conversion function into an input/output HTTP request.

https://github.com/johnnliu/function-ffmpeg-mp3-to-wav/blob/master/run.csx

#r "Microsoft.WindowsAzure.Storage"

using Microsoft.WindowsAzure.Storage.Blob;
using System.Diagnostics;
using System.IO;
using System.Net;
using System.Net.Http.Headers;

public static HttpResponseMessage Run(Stream req, TraceWriter log)
{

    var temp = Path.GetTempFileName() + ".mp3";
    var tempOut = Path.GetTempFileName() + ".wav";
    var tempPath = Path.Combine(Path.GetTempPath(), Guid.NewGuid().ToString());

    Directory.CreateDirectory(tempPath);

    using (var ms = new MemoryStream())
    {
        req.CopyTo(ms);
        File.WriteAllBytes(temp, ms.ToArray());
    }

    var bs = File.ReadAllBytes(temp);
    log.Info($"Renc Length: {bs.Length}");


    var psi = new ProcessStartInfo();
    psi.FileName = @"D:\home\site\wwwroot\mp3-to-wave\ffmpeg.exe";
    psi.Arguments = $"-i \"{temp}\" \"{tempOut}\"";
    psi.RedirectStandardOutput = true;
    psi.RedirectStandardError = true;
    psi.UseShellExecute = false;
    
    log.Info($"Args: {psi.Arguments}");
    var process = Process.Start(psi);
    process.WaitForExit((int)TimeSpan.FromSeconds(60).TotalMilliseconds);


    var bytes = File.ReadAllBytes(tempOut);
    log.Info($"Renc Length: {bytes.Length}");


    var response = new HttpResponseMessage(HttpStatusCode.OK);
    response.Content = new StreamContent(new MemoryStream(bytes));
    response.Content.Headers.ContentType = new MediaTypeHeaderValue("audio/wav");

    File.Delete(tempOut);
    File.Delete(temp);
    Directory.Delete(tempPath, true);    

    return response;
}
Trick: You can upload ffmpeg.exe and run them inside an Azure Function

https://github.com/johnnliu/function-ffmpeg-mp3-to-wav/blob/master/function.json

{
  "bindings": [
    {
      "type": "httpTrigger",
      "name": "req",
      "authLevel": "function",
      "methods": [
        "post"
      ],
      "direction": "in"
    },
    {
      "type": "http",
      "name": "$return",
      "direction": "out"
    }
  ],
  "disabled": false
}

Azure Functions Custom Binding

Ling Toh (of Azure Functions) reached out and tells me I can try the new Azure Functions custom bindings for Cognitive Services directly.  But I wanted to try this with Flow.  I need to come back to custom bindings in the future.

https://twitter.com/ling_toh/status/919891283400724482

Set up Cognitive Services - Speech

In Azure Portal, create Cognitive Services for Speech

Need to copy one of the Keys for later

Flow

Take the binary Multipart Body send to this Flow and send that to the Azure Function

base64ToBinary(triggerMultipartBody(0)?['$content'])

Take the binary returned from the Function and send that to Bing Speech API

Flow returns the result from Speech to text which I force into a JSON

json(body('HTTP_Cognitive_Speech'))

Try it:

Swagger

Need this for PowerApps to call Flow
I despise Swagger so much I don't even want to talk about it (the Swagger file takes 4 hours the most problematic part of the whole exercise)

{
  "swagger": "2.0",
  "info": {
    "description": "speech to text",
    "version": "1.0.0",
    "title": "speech-api"
  },
  "host": "prod-03.australiasoutheast.logic.azure.com",
  "basePath": "/workflows",
  "schemes": [
    "https"
  ],
  "paths": {
    "/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/triggers/manual/paths/invoke": {
      "post": {
        "summary": "speech to text",
        "description": "speech to text",
        "operationId": "Speech-to-text",
        "consumes": [
          "multipart/form-data"
        ],
        "parameters": [
          {
            "name": "api-version",
            "in": "query",
            "default": "2016-06-01",
            "required": true,
            "type": "string",
            "x-ms-visibility": "internal"
          },
          {
            "name": "sp",
            "in": "query",
            "default": "/triggers/manual/run",
            "required": true,
            "type": "string",
            "x-ms-visibility": "internal"
          },
          {
            "name": "sv",
            "in": "query",
            "default": "1.0",
            "required": true,
            "type": "string",
            "x-ms-visibility": "internal"
          },
          {
            "name": "sig",
            "in": "query",
            "default": "4h5rHrIm1VyQhwFYtbTDSM_EtcHLyWC2OMLqPkZ31tc",
            "required": true,
            "type": "string",
            "x-ms-visibility": "internal"
          },
          {
            "name": "file",
            "in": "formData",
            "description": "file to upload",
            "required": true,
            "type": "file"
          }
        ],
        "produces": [
          "application/json; charset=utf-8"
        ],
        "responses": {
          "200": {
            "description": "OK",
            "schema": {
              "description": "",
              "type": "object",
              "properties": {
                "RecognitionStatus": {
                  "type": "string"
                },
                "DisplayText": {
                  "type": "string"
                },
                "Offset": {
                  "type": "number"
                },
                "Duration": {
                  "type": "number"
                }
              },
              "required": [
                "RecognitionStatus",
                "DisplayText",
                "Offset",
                "Duration"
              ]            
            }
          }
        }
      }
    }
  }
}

Power Apps

Result

Summary

I expect a few outcomes from this blog post.

  1. ffmpeg.exe is very powerful and can convert multiple audio and video datatypes.  I'm pretty certain we will be using it a lot more for many purposes.
  2. Cognitive Speech API doesn't have a Flow action yet.  I'm sure we will see it soon.
  3. PowerApps or Flow may need a native way of converting audio file formats.  Until such an action is available, we will need to rely on ffmpeg within an Azure Function
  4. The problem of converting MP3 to WAV was raised by Paul Culmsee - the rest of the blog post is just to connect the dots and make sure it works.  I was also blocked on an error on the output of my original swagger file, which I fixed only after Paul sent me a working Swagger file he used for another service - thank you!

 

 

One Connection to Proxy Them All - Microsoft Flow with Azure Functions Proxies

/

Office 365 licensed PowerApps and Microsoft Flow lets you have 1 custom connection.  I think that's cute.  I also think that limitation is absurd, considering how many times we need to break out of existing capabilities every time we want to do something interesting.  So, custom connections shouldn't be a limit, if the BAP team wants us to innovate the crap out of this platform, this is counter productive.  I hope the team reconsider.

Regardless, it is pointless, because we bypass that right away: Azure Functions has Proxies.

This is a post in a series on Microsoft Flow.

  1. JSON cheatsheet for Microsoft Flow
  2. Nested-Flow / Reusable-Function cheatsheet for Microsoft Flow
  3. Building non-JSON webservices with Flow 
  4. One Connection to Proxy Them All - Microsoft Flow with Azure Functions Proxies
  5. Building Binary output service with Microsoft Flow

In this post, we talk about:

  • Azure Functions proxy for Microsoft Flow connections
  • Azure Functions proxy for itself
  • Azure Functions proxy for bypassing CORS
  • Azure Functions proxy for 3rd party APIs
  • One Simple API for me

So in one Connection to proxy them all, in one Request to Bind them.  And in Serverless something about we pay nothing - hey I'm a blogger not a poet.

Azure Functions Proxy for Microsoft Flow

In an earlier post - I talk about we can turn Microsoft Flow into webservices with the Request trigger.  This is awesome, but it leaves you with an endpoint that looks like sin.

https://prod-18.australiasoutheast.logic.azure.com:443/workflows/76232c77d84d424b8e56ab2f88b672c4
/triggers/manual/paths/invoke?api-version=2016-06-01
&sp=%2Ftriggers%2Fmanual%2Frun
&sv=1.0
&sig=FAKE_KEY_NXpUV_FEyhl1BKgKftQ0-rcOPcE

Lets clean this up with Azure Functions Proxies.  First, enable it.

Create a Proxy: set the name to "uploadBinary" and template to "upload".  Chose your methods (POST only for this one), and paste in the backend URL.

Hit create - our proxy URL is generated.  https://funky-demo.../upload

We had a Swagger OpenAPI file for this binary web request - lets tidy it up 

{
  "swagger": "2.0",
  "info": {
    "description": "",
    "version": "1.0.0",
    "title": "UploadBinary-proxy"
  },
  "host": "funky-demo.azurewebsites.net",
  "basePath": "/",
  "schemes": [
    "https"
  ],
  "paths": {
    "upload": {
      "post": {
        "summary": "uploads an image",
        "description": "",
        "operationId": "uploadFile",
        "consumes": [
          "multipart/form-data"
        ],
        "parameters": [
          {
            "name": "file",
            "in": "formData",
            "description": "file to upload",
            "required": true,
            "type": "file"
          }
        ],
        "responses": {
          "200": {
            "description": "successful operation"
          }
        }
      }
    }
  }
}

We can delete all the additional parameters sp, sv, sig, api-version.  We pretty much say we just care about the formData.

Go back to our PowerApps custom connectors - delete the old one and add this new OpenAPI file.
This is what PowerApps now see: the URL is only /upload
and there are no hidden query parameters - just waiting for the body.

Delete the old connection and recreate a new connection.
The method now on the new connection only takes 1 parameter "file".

The result in PowerApps and SharePoint

This test picture is called "Azure Function is On Fire!"

So in using AzureFunctions Proxy for a Flow WebRequest - we've simplified the query string in the URL and kept the functionality.

Azure Functions proxy for Azure Functions

Some of our connections were to Azure Functions, so let's tidy that up too.

This is a pretty simple example.  /listEmail now is hooked up, and activation code is hidden.

Azure Functions proxy for bypassing CORS

Everything you proxy in Azure Functions can be made available to CORS - just need to turn on Azure Functions CORS with your domain

CORS' design goals are to prevent browser based injection/consumption of a service.  But services are made to be consumed.  So CORS blocks a browser, but never blocks a server-based call.

Azure Function runs server, and proxy works because it doesn't see CORS at all.  Switching on Azure Functions CORS is so that your domain is accepted to call this Azure Function.

Anyway, if you see CORS and got annoyed - just use Azure Functions Proxy to keep going.

Azure Functions proxy for 3rd party APIs

Let's add an external Open Weather API

Azure Functions Proxies has additional means of passing through parameters to the underlying proxied URL.  See Documentation: https://docs.microsoft.com/en-us/azure/azure-functions/functions-proxies

So the call to the SAMPLE open weather API is:

http://samples.openweathermap.org/data/2.5/weather
?lat={request.querystring.lat}
&lon={request.querystring.lon}
&appid=b1b15e88fa797225412429c1c50c122a1

So we are exposing two query string parameters in our end point to be passed down.

 

One simple API Connection for me

{
    "swagger": "2.0",
    "info": {
        "description": "",
        "version": "1.0.0",
        "title": "One-Funky-API"
    },
    "host": "funky-demo.azurewebsites.net",
    "basePath": "/",
    "schemes": [
        "https"
    ],
    "paths": {
        "upload": {
            ...
        },
        "listEmail": {
            ...
        },
        "weather": {
            ...
        }
    }
}

In all the examples, we clean up the API we are calling to hide away apiKeys and such and make our end point really easy to consume from Flow, PowerApps or JavaScript CORS.

And because both Microsoft Flow and Azure Functions are Serverless, we still don't care about servers ;-)

 

Speaking at Digital Workplace Conference Australia 2017

/

I'll be speaking at the Digital Workplace Conference Australia!  23-24 August in Sydney.

 

This is a conference that's near and dear to me - and I've had several opportunities in the past to present at this conference, where I covered Silverlight, JavaScript, TypeScript, Modern Office App-ins and now this year - I plan to present a supercharged talk on running Serverless with Office 365.

Parts of the talk - especially how to get started - may seem familiar to many of you that has started down this journey. 

I wanted to focus a bit less on the technical, and more about how this has changed people. 

Azure Functions democratized 'I need to run a bit of code' to everyone.  Suddenly, the cloud is not this scary place where there are a hundred things we don't know, and don't know where to start.  Suddenly, the toys that seems far out of reach are ours.  Suddenly, a cloud subscription that costs less than a coffee per month is something I don't even think about.

To me, that is the power of AzureFunctions and why Serverless is a game changer. 

Do you know there are now brand new categories of design patterns specifically rewritten for the Serverless world.

I will of course still cover the technical bits - but to see all 20+ demos I have with me, you'll have to come find me in the speaker area for a personal demo :-)

In Digital Workplace Conference 2017, I want to talk about Serverless.

And I want to talk about humans.  Us.

I think the future will be amazing.  I hope to see you at the DWC Australia.  Come and grab me and say hello!