One Connection to Proxy Them All - Microsoft Flow with Azure Functions Proxies

Office 365 licensed PowerApps and Microsoft Flow lets you have 1 custom connection.  I think that's cute.  I also think that limitation is absurd, considering how many times we need to break out of existing capabilities every time we want to do something interesting.  So, custom connections shouldn't be a limit, if the BAP team wants us to innovate the crap out of this platform, this is counter productive.  I hope the team reconsider.

Regardless, it is pointless, because we bypass that right away: Azure Functions has Proxies.

This is a post in a series on Microsoft Flow.

  1. JSON cheatsheet for Microsoft Flow
  2. Nested-Flow / Reusable-Function cheatsheet for Microsoft Flow
  3. Building non-JSON webservices with Flow 
  4. One Connection to Proxy Them All - Microsoft Flow with Azure Functions Proxies
  5. Building Binary output service with Microsoft Flow

In this post, we talk about:

  • Azure Functions proxy for Microsoft Flow connections
  • Azure Functions proxy for itself
  • Azure Functions proxy for bypassing CORS
  • Azure Functions proxy for 3rd party APIs
  • One Simple API for me

So in one Connection to proxy them all, in one Request to Bind them.  And in Serverless something about we pay nothing - hey I'm a blogger not a poet.

Azure Functions Proxy for Microsoft Flow

In an earlier post - I talk about we can turn Microsoft Flow into webservices with the Request trigger.  This is awesome, but it leaves you with an endpoint that looks like sin.

https://prod-18.australiasoutheast.logic.azure.com:443/workflows/76232c77d84d424b8e56ab2f88b672c4
/triggers/manual/paths/invoke?api-version=2016-06-01
&sp=%2Ftriggers%2Fmanual%2Frun
&sv=1.0
&sig=FAKE_KEY_NXpUV_FEyhl1BKgKftQ0-rcOPcE

Lets clean this up with Azure Functions Proxies.  First, enable it.

Create a Proxy: set the name to "uploadBinary" and template to "upload".  Chose your methods (POST only for this one), and paste in the backend URL.

Hit create - our proxy URL is generated.  https://funky-demo.../upload

We had a Swagger OpenAPI file for this binary web request - lets tidy it up 

{
  "swagger": "2.0",
  "info": {
    "description": "",
    "version": "1.0.0",
    "title": "UploadBinary-proxy"
  },
  "host": "funky-demo.azurewebsites.net",
  "basePath": "/",
  "schemes": [
    "https"
  ],
  "paths": {
    "upload": {
      "post": {
        "summary": "uploads an image",
        "description": "",
        "operationId": "uploadFile",
        "consumes": [
          "multipart/form-data"
        ],
        "parameters": [
          {
            "name": "file",
            "in": "formData",
            "description": "file to upload",
            "required": true,
            "type": "file"
          }
        ],
        "responses": {
          "200": {
            "description": "successful operation"
          }
        }
      }
    }
  }
}

We can delete all the additional parameters sp, sv, sig, api-version.  We pretty much say we just care about the formData.

Go back to our PowerApps custom connectors - delete the old one and add this new OpenAPI file.
This is what PowerApps now see: the URL is only /upload
and there are no hidden query parameters - just waiting for the body.

Delete the old connection and recreate a new connection.
The method now on the new connection only takes 1 parameter "file".

The result in PowerApps and SharePoint

This test picture is called "Azure Function is On Fire!"

So in using AzureFunctions Proxy for a Flow WebRequest - we've simplified the query string in the URL and kept the functionality.

Azure Functions proxy for Azure Functions

Some of our connections were to Azure Functions, so let's tidy that up too.

This is a pretty simple example.  /listEmail now is hooked up, and activation code is hidden.

Azure Functions proxy for bypassing CORS

Everything you proxy in Azure Functions can be made available to CORS - just need to turn on Azure Functions CORS with your domain

CORS' design goals are to prevent browser based injection/consumption of a service.  But services are made to be consumed.  So CORS blocks a browser, but never blocks a server-based call.

Azure Function runs server, and proxy works because it doesn't see CORS at all.  Switching on Azure Functions CORS is so that your domain is accepted to call this Azure Function.

Anyway, if you see CORS and got annoyed - just use Azure Functions Proxy to keep going.

Azure Functions proxy for 3rd party APIs

Let's add an external Open Weather API

Azure Functions Proxies has additional means of passing through parameters to the underlying proxied URL.  See Documentation: https://docs.microsoft.com/en-us/azure/azure-functions/functions-proxies

So the call to the SAMPLE open weather API is:

http://samples.openweathermap.org/data/2.5/weather
?lat={request.querystring.lat}
&lon={request.querystring.lon}
&appid=b1b15e88fa797225412429c1c50c122a1

So we are exposing two query string parameters in our end point to be passed down.

 

One simple API Connection for me

{
    "swagger": "2.0",
    "info": {
        "description": "",
        "version": "1.0.0",
        "title": "One-Funky-API"
    },
    "host": "funky-demo.azurewebsites.net",
    "basePath": "/",
    "schemes": [
        "https"
    ],
    "paths": {
        "upload": {
            ...
        },
        "listEmail": {
            ...
        },
        "weather": {
            ...
        }
    }
}

In all the examples, we clean up the API we are calling to hide away apiKeys and such and make our end point really easy to consume from Flow, PowerApps or JavaScript CORS.

And because both Microsoft Flow and Azure Functions are Serverless, we still don't care about servers ;-)

 

Speaking at Digital Workplace Conference Australia 2017

I'll be speaking at the Digital Workplace Conference Australia!  23-24 August in Sydney.

 

This is a conference that's near and dear to me - and I've had several opportunities in the past to present at this conference, where I covered Silverlight, JavaScript, TypeScript, Modern Office App-ins and now this year - I plan to present a supercharged talk on running Serverless with Office 365.

Parts of the talk - especially how to get started - may seem familiar to many of you that has started down this journey. 

I wanted to focus a bit less on the technical, and more about how this has changed people. 

Azure Functions democratized 'I need to run a bit of code' to everyone.  Suddenly, the cloud is not this scary place where there are a hundred things we don't know, and don't know where to start.  Suddenly, the toys that seems far out of reach are ours.  Suddenly, a cloud subscription that costs less than a coffee per month is something I don't even think about.

To me, that is the power of AzureFunctions and why Serverless is a game changer. 

Do you know there are now brand new categories of design patterns specifically rewritten for the Serverless world.

I will of course still cover the technical bits - but to see all 20+ demos I have with me, you'll have to come find me in the speaker area for a personal demo :-)

In Digital Workplace Conference 2017, I want to talk about Serverless.

And I want to talk about humans.  Us.

I think the future will be amazing.  I hope to see you at the DWC Australia.  Come and grab me and say hello!

Reusing functions in PowerShell AzureFunctions

This is a pretty simple blog.  Take one of the examples I've been using often: 

https://github.com/johnnliu/azure-functions-o365/blob/master/sharepoint-list-email.ps1

This PowerShell uses PnP-PowerShell to:

  • Connect to SharePoint Online
  • Pulls a list from Document Library
  • Format as HTML Table
  • Send to an email with SPO SendMail utility endpoint

One of the most common, repeated step I do in almost every Function is to get credential and authenticate.  So I decided to put it into a shared function.

Refactor this into a separate function

Use the file navigator on the right hand pane - add a new file, call it "shared.psm1" this is a PowerShell Module file.

Create a function get-cred() and put the 4 lines of getting $username and encode PSCredential into this function, then return $creds

Finally, reference the module via:

Import-Module "D:\home\site\wwwroot\get-list-and-email\shared.psm1"

# call get-cred inline here
Connect-PnPOnline -url $siteUrl -Credentials (get-cred)

The path is the name of the function, that points to the current directory.
From now on, in every other function, you can just import that module to share the function.

If you use a separate shared directory under the wwwroot\shared level - that's a good place to put these shared modules too.  But note that you can't access that area via the Files right-hand pane.  You'll need to go there via the Kudu interface.

 

I consider putting get-cred away as a preparation step for one day in the future where I would replace that function with a call to Azure KeyVault to obtain the PSCredential object. When that refactoring happens, I will only need to update one place.

 

AzureFunctions Work Fan-out with Azure Queue in PowerShell

So I really like using PnP-PowerShell to chain up and perform complex operations in Office 365, and linking them up with AzureFunctions and Flow.

Scenario - scan all tenant's site collections

I bump into another problem today - I needed to scan all my site collections within the tenancy and start a Flow that will notify and apply site closure policy and lock the site.

We can scan all the site collections in a tenant in one request Get-PnPTenantSites - but we want to make sure the job doesn't time out.

So we need to fan-out the workload to a queue, and trigger multiple AzureFunctions to scan each site collection in parallel.

Problem - PoSH Queue binding only 1 output

As soon as I started writing the PoSH - I remembered, with the default PoSH Queue binding - you can only write 1 message to the Queue.

Unlike C# where you could do multiple:

foreach(var message in messages) {
    await outQueue.AddAsync(message);
}

In PoSH - if you are using the default integration tab to set up an Output Binding to AzureQueue.  Then you can only write one message to the Queue.

 

How to write multiple messages to Queue in PoSH?

It turns out I've already solved this once before in April, but I had completely forgotten this, because I DIDN'T BLOG IT.
Let that be a lesson to all developers - Always blog something cool that you did.  Because you will need it in two months when your memory failed you.

# if you have been using the storage in other functions 
# you will already have the connection string in your 
# function's app settings - reuse it

$storeAuthContext = New-AzureStorageContext -ConnectionString $env:azurefunctions3a585851_STORAGE 

$outQueue = Get-AzureStorageQueue –Name 'my-queue-name' -Context $storeAuthContext
if ($outQueue -eq $null) {
    $outQueue = New-AzureStorageQueue –Name 'my-queue-name' -Context $storeAuthContext
}

# this example isn't scanning sites - just going through files in a library
$items | % {
    
    $item = @{
        source = $_.FieldValues.FileRef;
        target = ($destination + "/" + $_.FieldValues.FileLeafRef)
    }

    # Create a new message using a constructor of the CloudQueueMessage class.
    $queueMessage = New-Object `
        -TypeName Microsoft.WindowsAzure.Storage.Queue.CloudQueueMessage `
        -ArgumentList (ConvertTo-Json $item)

    # Add a new message to the queue.
    $outQueue.CloudQueue.AddMessage($queueMessage)
}

 

 

 

Are you Cloud-Curious or Cloud-Serious? Azure Functions in DWCNZ 2017

I had a fantastic time at Digital Workplace Conference in NZ.

Highlight Sessions

There are many other great sessions, I wasn't able to be in multiple places at once!

My Own Session

I presented Azure Functions in Office 365 - Building Serverless Solutions

There were a few things that I didn't managed to get through.  I wanted to list them here, and hope you will accept my apologies.  I've had several conversations with you all over the two days of the conference, many wanted deeper details into certain aspects of using Azure Functions.

 

Demo: Timer Based Alert with Email

https://github.com/johnnliu/azure-functions-o365/blob/master/sharepoint-list-email.ps1

This demo outlines a very simple script that will connect to a SharePoint list (or document library), query and fetch list items, format them into HTML and email to user from the System Account.

Combined with a schedule, this is an extremely common scenario in SharePoint Online: you want to schedule a smart alert email once a week, based on a filter to a list.

 

Using Recurring event in Flow instead of Azure Functions Timer-Trigger

While you can schedule tasks in Azure Function via a Timer Trigger, Microsoft Flow's recurrent trigger has several benefits:

  • You can create a Team Flow - so multiple users can be owners and configure the recurrence trigger.
  • The UI for setting up a time for the trigger is more obvious for power users.
  • You can easily see past runs from within Flow
  • You can easily re-run a Flow

 

The Severless "Specturm"

From my own experiences and from reading and understanding the greater scope of Serverless solutions that are being designed in the world, I wanted to present the spectrum of Serverless solutions.  We start on one side - from the Cloud-Curious, to the experts - the Cloud-Serious.

 

Cloud Curious

The majority of the presentation is pitched for the cloud-curious.  You have heard of Azure Functions and Serverless.  The demos presented how to get going really quickly.

Functions are thus:

  1. Micro (web) services for everyone.  So many people I talked to has given up on programming, thinking writing microservices or complex architecture isn't for them.  It's for the young'in dev teams now.
    AzureFunctions, especially with PowerShell - flipped the whole thing upside down.  Now, many 'ex-developers' suddenly find themselves build amazing service end points, connecting them to webhooks and Azure Blob Queues.  It is an amazing resurgence and move to microservices.  And everyone's having fun playing with really cool new toys.
     
  2. Use your favourite language!
    C#?  JS?  PoSH?  F#?  You can even use TS or VB.NET compiled.  Nobody can tell you what language you can and can't use.
     
  3. Perfect solution for many problems in SharePoint customizations
    Elevate permissions
    Webhook and event receivers
    Timer Jobs
    Extending Flow (as custom workflow action)

    If you are bringing customizations in SharePoint On-Premises to SharePoint Online - Azure Functions is a solution that must be evaluated.  It fits so many scenarios that you need to bring your On-Premises customizations forward, without breaking the bank, or needing complex re-development.

 

Cloud Serious

For the cloud serious - you are already using simple functions.  You want to know what's next.

  1. "Idempotent" - this is the keyword that will define the entire Serverless Framework.  You want to design functions that has no side-effects if you rerun.  A function can fail, it will automatically retry until success.  Your function must be built to be retry-safe.
     
  2. Use message queues and service bus to scale your Function.
    In Serverless, you are bound by duration.  You are not bound by parallel compute.
    To scale your long running process, split into a Queue and spawn infinite parallel compute.
     
  3. In a serial code, we wanted to catch all our exceptions to speed up long running tasks.  When we convert to parallel compute - we no longer really care about exceptions.  If you fail, you want to fail fast.  Throw exceptions freely and as fast as possible.  Terminate the function.  Let Queue retry automatically.
     
  4. With the new Azure Functions Proxies, we can create Serverless Web Applications - which is essentially combining a CDN to host static resources, and Functions to run server side code.

    Future of Serverless web apps is basically: CDN + Functions
    Both scale in parallel infinitely, by default, by design.  But is easy to understand and accept in concept.

    You do no worry about scaling VMs, AppPools, IIS, WebJobs, WebSites... 

Your solutions sits on top of all of those things - but there is no fear.  A fast messaging queue with built-in retries and a thousand atomic hammers will carry your workload from now to infinity.  And it'll cost less than your coffee.

 

Slide downloads

https://github.com/johnnliu/pptx