Difference between beta, edu and v1.0 of MSGraph #microblog

I find this interesting - source: digging around MicrosoftTeams powershell and Mikael Svenson's blog post on enabling Teams programmatically. 

What's funny, because what I found strange is when Mikael says this:

The creation of the group itself happens against the /edu/groups which I’ve never seen before, but that’s not interesting.

Because that's the bit I found totally interesting.  Why does the PowerShell need the /edu/ endpoint?
Hop over to Graph Explorer we can play with this:

Comparing different endpoint: v1.0

endpoint: beta

endpoint: edu

It becomes clear why we need the /edu/ endpoint.  It has this bit of information:

"creationOptions": [
    "SkypeSpaces",
    "ExchangeProvisioningFlags:481"
],

The hint that a Team is provisioned seems to be the flag "SkypeSpaces"

Interesting.

Naturally, one would ask.  So if I want to enable Yammer, Planner or PowerBI on an existing Unified Group.  Do I POST an update to creationOptions?

Very.  Interesting.

I'm speaking about Serverless Flow and Azure Functions at Collab365 Free Online Conference

Have you heard about the virtual Collab365 Global Conference 2017 that’s streaming online November 1st – 2nd?

Join me and 120 other speakers from around the world who will be bringing you the very latest content around SharePoint, Office 365, Flow, PowerApps, Azure, OneDrive for Business and of course the increasingly popular Microsoft Teams. The event is produced by the Collab365 Community and is entirely free to attend.

Places are limited to 5000 so be quick and register now.

During the conference I'd love you to watch my session which is called : 

'Serverless with Microsoft Flow and Azure Functions'

Level up your mastery of Microsoft Flow. Switch to Azure Functions only when you need to. No doubt there will be many sessions on Microsoft Flow, introducing you to its wonderful merits and rough edges. This session is for the advanced users - we will see what Microsoft Flow really is, and bend it to our will.

If you join me, you will learn:

• Master JSON in a Flow
• Combining Azure Functions with Flow
• Failure Recovery, in a Flow
• How to handle Binary in a Flow
• How to write HTML and generate PDF in Flow

Topic(s):

• Azure Functions
• Microsoft Flow

Audience :

• Developer
• Power User

Time :

• Thursday, November 2 2017 10:00 AM (UTC)
• Thursday, November 2 2017 09:00 PM (ADST)

How to attend :

1. Register here.
2. At the time listed above go here to watch my session. (you can also add me to your own personal planner from the agenda.
3. Enjoy the demos and ask me questions, I'll put the templates up for download after the session.

From Office 365 to Azure to Minecraft, connected with Flow

"John, what's this headline."
This is just a love serenade to Microsoft's many engineers and teams.  Thank you, for making these products that makes these things look easy.

Thank you for showing the world what One Microsoft might look like when everything works together.

Plan

1. Minecraft Windows 10 edition in the latest 1.2 (better together update) included web sockets previously only in the Minecraft Education version.  Web Sockets lets you connect to a Minecraft game, and remotely execute commands.
2. Minecraft Code Connection is an external application that hosts a friendly REST API and translates JSON to web socket.  Previously this was only for Minecraft Education Edition.  An update in early October allows this to work with Minecraft Windows 10.
   https://makecode.com/blog/minecraft/10-18-2017
3. Microsoft Data Gateway allows Flow, PowerApps, PowerBI to talk to on-premises environments.  It also recently gained the ability to execute custom connections.  It is essentially, an enterprise data gateway / reverse proxy that connects your local environment to an Azure Service Bus.  The ability to call your custom local REST endpoint was released in September.
   https://flow.microsoft.com/en-us/blog/q3-2017-update/
4. Cloud based services like Flow/LogicApps, PowerApps, and PowerBI can talk to this connection online, via the magic of Azure Service Bus.
5. And because I'm a SharePoint MVP - we are triggering this Flow from SharePoint Online.  Because the world needs this.

Setting up Minecraft & Code Connection

https://www.microsoft.com/en-au/store/p/minecraft-for-windows-10/9nblggh2jhxj?rtc=1 
Buy or download Minecraft for Windows 10

https://education.minecraft.net/get-started/download/
Download Minecraft Code Connection

https://education.minecraft.net/support/knowledge-base/code-connection-api-documentation/
The API for the JSON messages to send to Code Connection is here.
 

View fullsize

Code Connection uses Minecraft commands - the option is called "activate cheats" - do this in a creative world.

Code Connection is a separate executable.  Run it outside of Minecraft - it'll ask you to enter the command into Minecraft

Use Postman to test your localhost:8080 and send a REST request.  You'll see your agent bot move.

Congrats - you now have a REST endpoint that can send game commands to your Minecraft game.

Setting up Flow Data Gateway

https://flow.microsoft.com/en-us/documentation/gateway-reference/ 

The data gateway also calls your local REST endpoints via a custom connector.  This was a feature that was silently released in the deluge of news from MSIgnite.  I need to thank @pratapladhani (PowerApps PM) for sending me the link of the announce.  It is REALLY obscure.

https://flow.microsoft.com/en-us/blog/q3-2017-update/

“On-premises connectivity to any HTTP API - Finally, users can now connect their own on-premises APIs with Custom Connectors - leveraging the On-Premises Data Gateway. For example, if you have a service that’s only available on your local network, you can now author a custom connector that can read data from or push data to that local service.”

Congrats - this is your reverse proxy.

Setting up custom connection via this Swagger File I've prepared for you

https://github.com/johnnliu/flow/blob/master/mc-cc.swagger.json 

Grab this swagger.json and add a custom connection to Flow

Note the host is localhost:8080
Tick: Connect via on-premise data gateway

Note the custom connection swagger file points to localhost:8080 - this will work if your data gateway is on the same machine as the Code Connection exe.  Otherwise, you should use the LAN IP Address of the Code Connection server.

Add a  connection, and select On-Premises data gateway

Note the connection type is "On-premises"

Write my Flow

These wonderful methods with dropdowns are defined in Swagger file.  It's hard work.  Contributions welcome!

Trigger it from Office 365 / SharePoint

Hi bot agent

Uploaded by John Liu on 2017-10-25.

Code-Less

I want you to understand, all these are done, without me writing a single line of C# or JavaScript.

Cost?

Office 365 E3+ license allows you to have:

• 2000 Flow runs per user
• 1 Custom Connection
• On-Premises data gateway allowed

https://australia.flow.microsoft.com/en-us/pricing/ 

So there's no extra cost for an Office 365 customer E3+

Summary Key Points

• Minecraft Win10 has websockets
• Code Connection does JSON to websocket
• Data Gateway is amazing, also calls your local REST endpoints
• Microsoft Flow just unlocks it to the rest of the world, if you like, LogicApps will work just as well.
• You can have a PowerApp calling the function via the Custom Connection, without Flow.
• Extremely code-less

Where do we go next?

Well, when I detect an Active Directory User deletion via webhook - I'm going to teleport this Zombie Villager into lava.

Update: I can't see a use case for this

So, some people have commented to me directly - John this is very cute, but there's no business use case for this.

Let me explain the head-fake

Flow can reach into your organization and call any of your REST endpoints with no code

Microsoft Flow blog posted https://flow.microsoft.com/en-us/blog/on-premise-apis/ explaining how you can build your own on-premises API service and call it from Flow via the data gateway.  I was bothering them with explanations for the data gateway and I think they were ready to publish anyway, they told me they'll share it in a few days.

I beat them slightly to publish since I worked out how to do the Minecraft bit.

I also think they need more imagination.  Minecraft is a hundred times better example and I didn't even have to write a REST service ;-)

<3

Flow Admin Center - Do a spring cleaning of your Flows

I spewed coffee all over my screen.

Each user in Office 365 gets 2000 Flow runs pooled.  This is my personal playground tenant, I've got 10 users licenses but it's really just me running around sending Flows to myself, and I.

So, I'm looking around Flow Admin center.  I see a tab Quotas.  I clicked on it and sees oh it's got a bar telling me how many runs left.  That's good.

Then I see the number: 6500+ runs.

I chocked.  What.  I run a number of Flows but really NOT that many.  Is that lifetime total?  It's per month.  OK.  We need to interrupt everything and sort this out.

View fullsize

So hmm, it does look peculiar.  (BTW, I don't know how 8000+4000 = 6500)

But I do want to go into these two examples:

First Bad Thing: Scheduled Recurrence

Sometimes, I use a Recurrence trigger to test something - and it default to 1 per minute.  I had changed this to 5, but this was way too often.  

TIP: Always run test Flows via a HTTP Request Trigger - poke it with Postman instead.

Second Bad Thing: Infinity Loops

View fullsize

This one is can be in different forms.  But the idea is simple - you trigger from a SharePoint list item, and you update that same item.

The more tricker version to spot is if the update was hidden within a Conditional Block, so it only "sometimes" trigger, but not all the time.  The problem is, that sometimes started happening, and now it's looping endlessly without telling you about it.

You get the idea.  This second one was worse than the first problem - this one clocked in 8000 runs on its own.

Only you can save Yourself (and your Free runs)

I'm sure Flow product team will be thrilled if you don't check your Flows.

But you might not be.  So you need to proactively do the next step.

We create a new recurring monthly Flow that creates a TODO task to review the Flow Admin Quotas.

View fullsize

This logic is perfect*
If you see the error, let me know in the comments ;-)

I've been working on an epic Flow story and that's gonna take some time.  But this short episode was too good not to write up.  It is now Friday 7 PM so I'm signing off have a great weekend everyone.

Serverless connect-the-dots: MP3 to WAV via ffmpeg.exe in AzureFunctions, for PowerApps and Flow

There's no good title to this post, there's just too many pieces we are connecting.  

So, a problem was in my todo list for a while - I'll try to describe the problem quickly, and get into the solution.

• PowerApps Microphone control records MP3 files
• Cognitive Speech to Text wants to turn WAV files into JSON
• Even with Flow, we can't convert the audio file formats. 
• We need an Azure Function to gluethis one step
• After a bit of research, it looks like FFMPEG, a popular free utility can be used to do the conversion

PowerApps records MP3. Cognitive Speech2Text wants WAV.
I think there's a path thru this maze. Try run ffmpeg in @AzureFunctions powershell.

— John Liu (@johnnliu) October 16, 2017

Azure Functions and FFMPEG

So my initial thought is that well, I'll just run this utility exe file through PowerShell.  But then I remembered that PowerShell don't handle binary input and output that well.  A quick search nets me several implementations in C# one of them catches my eye: 

So I decided to do mp3 to wav in @AzureFunctions with ffmpeg. And of course @jakkaj had the code I can borrow :-)https://t.co/5M2L2S2SPs

— John Liu (@johnnliu) October 16, 2017

Jordan Knight is one of our Australian DX Microsoftie - so of course I start with his code

It actually was really quick to get going, but because Jordan's code is triggered from blob storage - the Azure Functions binding for blob storage has waiting time that I want to shrink, so I rewrite the input and output bindings to turn the whole conversion function into an input/output HTTP request.

https://github.com/johnnliu/function-ffmpeg-mp3-to-wav/blob/master/run.csx

#r "Microsoft.WindowsAzure.Storage"

using Microsoft.WindowsAzure.Storage.Blob;
using System.Diagnostics;
using System.IO;
using System.Net;
using System.Net.Http.Headers;

public static HttpResponseMessage Run(Stream req, TraceWriter log)
{

    var temp = Path.GetTempFileName() + ".mp3";
    var tempOut = Path.GetTempFileName() + ".wav";
    var tempPath = Path.Combine(Path.GetTempPath(), Guid.NewGuid().ToString());

    Directory.CreateDirectory(tempPath);

    using (var ms = new MemoryStream())
    {
        req.CopyTo(ms);
        File.WriteAllBytes(temp, ms.ToArray());
    }

    var bs = File.ReadAllBytes(temp);
    log.Info($"Renc Length: {bs.Length}");


    var psi = new ProcessStartInfo();
    psi.FileName = @"D:\home\site\wwwroot\mp3-to-wave\ffmpeg.exe";
    psi.Arguments = $"-i \"{temp}\" \"{tempOut}\"";
    psi.RedirectStandardOutput = true;
    psi.RedirectStandardError = true;
    psi.UseShellExecute = false;
    
    log.Info($"Args: {psi.Arguments}");
    var process = Process.Start(psi);
    process.WaitForExit((int)TimeSpan.FromSeconds(60).TotalMilliseconds);


    var bytes = File.ReadAllBytes(tempOut);
    log.Info($"Renc Length: {bytes.Length}");


    var response = new HttpResponseMessage(HttpStatusCode.OK);
    response.Content = new StreamContent(new MemoryStream(bytes));
    response.Content.Headers.ContentType = new MediaTypeHeaderValue("audio/wav");

    File.Delete(tempOut);
    File.Delete(temp);
    Directory.Delete(tempPath, true);    

    return response;
}

Trick: You can upload ffmpeg.exe and run them inside an Azure Function

https://github.com/johnnliu/function-ffmpeg-mp3-to-wav/blob/master/function.json

{
  "bindings": [
    {
      "type": "httpTrigger",
      "name": "req",
      "authLevel": "function",
      "methods": [
        "post"
      ],
      "direction": "in"
    },
    {
      "type": "http",
      "name": "$return",
      "direction": "out"
    }
  ],
  "disabled": false
}

Azure Functions Custom Binding

Ling Toh (of Azure Functions) reached out and tells me I can try the new Azure Functions custom bindings for Cognitive Services directly.  But I wanted to try this with Flow.  I need to come back to custom bindings in the future.

https://twitter.com/ling_toh/status/919891283400724482

Set up Cognitive Services - Speech

In Azure Portal, create Cognitive Services for Speech

View fullsize

Need to copy one of the Keys for later

Flow

View fullsize

Take the binary Multipart Body send to this Flow and send that to the Azure Function

base64ToBinary(triggerMultipartBody(0)?['$content'])

Take the binary returned from the Function and send that to Bing Speech API

View fullsize

https://docs.microsoft.com/en-us/azure/cognitive-services/speech/getstarted/getstartedrest?tabs=Powershell#build-recognition-request-and-send-it-to-the-speech-service

You'll need the key from the Cognitive Azure App earlier for Ocp-Apim-Subscription-Key

View fullsize

Flow returns the result from Speech to text which I force into a JSON

json(body('HTTP_Cognitive_Speech'))

Try it:

View fullsize

View fullsize

Swagger

Need this for PowerApps to call Flow
I despise Swagger so much I don't even want to talk about it (the Swagger file takes 4 hours the most problematic part of the whole exercise)

{
  "swagger": "2.0",
  "info": {
    "description": "speech to text",
    "version": "1.0.0",
    "title": "speech-api"
  },
  "host": "prod-03.australiasoutheast.logic.azure.com",
  "basePath": "/workflows",
  "schemes": [
    "https"
  ],
  "paths": {
    "/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/triggers/manual/paths/invoke": {
      "post": {
        "summary": "speech to text",
        "description": "speech to text",
        "operationId": "Speech-to-text",
        "consumes": [
          "multipart/form-data"
        ],
        "parameters": [
          {
            "name": "api-version",
            "in": "query",
            "default": "2016-06-01",
            "required": true,
            "type": "string",
            "x-ms-visibility": "internal"
          },
          {
            "name": "sp",
            "in": "query",
            "default": "/triggers/manual/run",
            "required": true,
            "type": "string",
            "x-ms-visibility": "internal"
          },
          {
            "name": "sv",
            "in": "query",
            "default": "1.0",
            "required": true,
            "type": "string",
            "x-ms-visibility": "internal"
          },
          {
            "name": "sig",
            "in": "query",
            "default": "4h5rHrIm1VyQhwFYtbTDSM_EtcHLyWC2OMLqPkZ31tc",
            "required": true,
            "type": "string",
            "x-ms-visibility": "internal"
          },
          {
            "name": "file",
            "in": "formData",
            "description": "file to upload",
            "required": true,
            "type": "file"
          }
        ],
        "produces": [
          "application/json; charset=utf-8"
        ],
        "responses": {
          "200": {
            "description": "OK",
            "schema": {
              "description": "",
              "type": "object",
              "properties": {
                "RecognitionStatus": {
                  "type": "string"
                },
                "DisplayText": {
                  "type": "string"
                },
                "Offset": {
                  "type": "number"
                },
                "Duration": {
                  "type": "number"
                }
              },
              "required": [
                "RecognitionStatus",
                "DisplayText",
                "Offset",
                "Duration"
              ]            
            }
          }
        }
      }
    }
  }
}

Power Apps

View fullsize

Result

View fullsize

Summary

I expect a few outcomes from this blog post.

1. ffmpeg.exe is very powerful and can convert multiple audio and video datatypes.  I'm pretty certain we will be using it a lot more for many purposes.
2. Cognitive Speech API doesn't have a Flow action yet.  I'm sure we will see it soon.
3. PowerApps or Flow may need a native way of converting audio file formats.  Until such an action is available, we will need to rely on ffmpeg within an Azure Function
4. The problem of converting MP3 to WAV was raised by Paul Culmsee - the rest of the blog post is just to connect the dots and make sure it works.  I was also blocked on an error on the output of my original swagger file, which I fixed only after Paul sent me a working Swagger file he used for another service - thank you!