Speaking and Hackathon at Digital Workplace Conference Australia - Melbourne

/
DWCAU-Landscape-Temp.png

In a little less than a month on August 15-16, I'll be presenting in Melbourne at the Digital Workplace Conference Australia 2018.  

www.dwcau.com.au

(I still fondly remember this conference as the previously annual Australian SharePoint Conference - but all things evolve.  SharePoint to SharePoint Online to Office 365, and the workplace evolved from portals to intranet to social platforms to conversational platforms).

Flow and Functions: level up our Serverless Toolkit in Office 365

I will be presenting an amped up talk covering the implementation and automation with Microsoft Flow and Azure Functions.  This is one of several sessions on Microsoft Flow at this conference so I will be covering implementation and mastery at level 200+  (But because it's Flow, it'll still look deceivingly simple!)

John join forces with Paul Culmsee and Ashlee at #DWCAU pre-conf hackathon

I want to also mention that I'm helping out with Paul Culmsee and Ashlee's PowerApps and Flow Workshop/Hackathon on 14 August 2018

http://www.dwcau.com.au/workshops/powerapps-hackathon/

I personally don't know how much PowerApps and Flow raw potential would be in that room in the hackathon.  I really want to find out and I hope you would too.  Space for this is limited - please consider this one day pre-conference workshop.

"Learn PowerApps and Flow from experts and build an app that you need." says John.

"Two MVPs for the price of one" says Paul.  I'll just leave this here.

Whether at the conference, and/or the hackathon, I hope to see you there in Melbourne.

Run Any PnP-PowerShell via Drag and Drop Zip to AzureFunctions

/
pexels-birthday-box-celebration-45238.jpg

So I previously asked aloud "Can we simplify this further" - It turns out you can.  You can deploy a PnP-PowerShell from scratch in 1 minute.

And since we are targetting completely non-devs.  We do this via dragging and dropping one Zip file.

 

Plan

In my mind - yes.  Yes that installation could get simpler.  App Services (because of the Kudu backend) supports Zip Deploy.

https://docs.microsoft.com/en-us/azure/azure-functions/deployment-zip-push

But a quick read of the article it targets DevOps doing zipdeploy with Azure CLI, REST or PowerShell.  There's nothing for a yet-to-be-developer.  What I wanted to do was drag and drop zip deploy (so I quickly reached out to Ling Toh and turns out she says she does it all the time).

Steps in Pictures

1. Grab this ZIP file 

https://github.com/johnnliu/azure-functions-o365/blob/master/Run-Any-PnP-PowerShell.zip

2. Create a new Functions App

3. Drag and Drop zip file into Kudu

4. Configure App Settings

Back to Functions page under Azure Portal.  Refresh and you'll see the function appear.
Go to app settings to set up the USER/PW variables we need for the test.

5. Test it

Summary

That's it really.  Drag and drop deploy of a function via ZIP file.
Feel free to open up the zip and see how all the pieces are laid out, including the entire PnP-PowerShell module dependency.

Run any PnP-PowerShell in one AzureFunction from Microsoft Flow

/
pexels-photo-tool.jpg

Are you missing a SharePoint connector in Flow?  Do you need that one extra thing that's only in PnP-PowerShell (because there are 300 cmdlets in PnP-PowerShell)?  Frustrated with having to write another AzureFunction just for this one thing?

 

(Trumpet please) Behold!  I present the last AzureFunction (for PnP-PowerShell) you'll ever need to write.

Run-Any-PnP-PowerShell

And then, we'll call it from Flow.  Hmm wait, I think we can do better with the title...
(Trumpet please) Behold!

Run-Any-PowerShell in Flow

Wait, wait, no, why stop there with the title...
(Trumpet please) Behold!

Run Any Compute You Want in Flow

 

Plan

  • Create our Run-Any-PnP-PowerShell script
  • Call it from Flow
  • Call it to do some random compute for us

Create our Run-Any-PnP-PowerShell script

First - create an AzureFunctions app.  Install PnP-PowerShell in it. 

https://docs.microsoft.com/en-us/sharepoint/dev/declarative-customization/site-design-pnp-provisioning#upload-the-pnp-powershell-module-for-your-azure-function

I have an old post in 2016 that has similar steps, but the ones from SharePoint PnP Team is much more current.

If you have one already - great, just add another method. 

Select PowerShell from Experimental.  I call it Run-Any-PnP-PowerShell

Write the method.

# we need to import PnP-PowerShell cmdlets
Import-Module "D:\home\site\wwwroot\modules\SharePointPnPPowerShellOnline.psd1"

# POST method: $req
$script = Get-Content $req -Raw

# prepare credentials from environment USER/PW
$securepw = ConvertTo-SecureString $env:PW -AsPlainText -Force
$credentials = New-Object System.Management.Automation.PSCredential ($env:USER, $securepw)

# run the script from POST body
Invoke-Expression $script -OutVariable out | Tee-Object -Variable out

# catch output and send it back
Out-File -Encoding Ascii -FilePath $res -inputObject $out

6 lines of magic.

Call this from Flow

This works because Azure Functions is pretty clever about guessing the output of the Function, and changes the content type to JSON

Call it to do some random compute

Here's the same Function doing complex maths.  It'll also handle XML, JSON, .NET Framework and run any of the existing cmdlets or any extra ones we decide to import.

I think this ASCII module isn't very good.

 

Security [updated]

The feedback I got immediately was we'll need to secure that.  That's fair.  Here's a bunch of pretty important links to secure such a crazy "weapon of mass scripting".

Guard your AzureFunction activation key

Because you are calling AzureFunction from within a Flow - technically, only people that use the Flow can see the URL.

IP Filtering

Turn on IP Address Filtering for Azure Functions https://docs.microsoft.com/en-us/azure/app-service/app-service-ip-restrictions 
And the list of IP Addresses for your Flow runs from https://docs.microsoft.com/en-us/flow/limits-and-config#ip-address-configuration 

Secure by Azure AD

You can secure the Function with Azure AD https://docs.microsoft.com/en-us/azure/app-service/app-service-mobile-how-to-configure-active-directory-authentication

Your Flow HTTP request will need to authenticate as well before it can call your AzureFunction.  https://twitter.com/johnnliu/status/943761628671090688

Implementing Azure Functions Managed Service Identity

https://docs.microsoft.com/en-us/azure/app-service/app-service-managed-service-identity

This prevents the username/password being stored in the environment which can be read by the script.  With MSI, the credentials are stored in KeyVault.

Switch to Azure Automation

Azure Automation's native integration with KeyVault may offer the extra security you want.  I personally find it slower to start up, and the connector to retrieve the output is one extra step.  But this is definitely possible.

Add Persistent Logging

Because all the scripts has to come through your Flow - add a logging step that ensures scripts that are run are stored away with a log.

 

Finally

I do apologize I published this initially without talking about some security options.  Over the next day it became pretty clear that we need to secure such a crazy technique.

On the other hand, there was some pretty good discussions - so may be leaving out the security section for one day was a good idea after all.

Two complementary MicrosoftFlow podcasts in March, and Two Plateaus of MicrosoftFlow

/
Two complementary MicrosoftFlow podcasts in March, and Two Plateaus of MicrosoftFlow

John, you just recommended people learn Expressions and you cheered when Jon says Expressions is going away?!  Why are you so inconsistent?!

I finished listening to two parallel podcasts and there's some great contrasts between them that I wanted to point out, and write them down.

Read More

Betting on 2018 - level up our Serverless in Azure

/

A recent conversation got me thinking about making some predictions for 2018.  This isn't so much a "ha look I'm right in 2019" post.  This is more about internalizing and verbalize my choices and I think there's value is sharing all this thinking.

So here it is, all of it: notes, wishlist, observations, what other people are doing, what we should be doing.  All in one overview blog post.  Happy 2018.

serverless-levelup.png

 

 

Bet on Serverless

You can't look sideways without seeing "Serverless" it's a silly term, but I need to start with a definition by Serverless experts on "Serverless"

  • Use a compute service to execute code on demand
  • Write single-purpose stateless functions
  • Design push-based event-driven pipelines
  • Create thicker front-ends
  • Embrace third-party services

I started on this path in 2016 and I can't look back.  Being able to run your code, anytime in the cloud is a life changing experience for many of us - it abstracts the operations part of hosting code in the cloud, and lets us get back quickly into code.

Applications for this technique are far and wide.  From simple services to augment the endless front-end applications we were building in 2016, to finally having a great way to handle remote events or permission escalation.  And look beyond to the bot-framework.  A little blog post I wrote in 2016 about Serverless site provisioning is now officially best practice in SharePoint's Site Design - I'm a little glad it was useful :-)  At times it feels like I just hack and cobble things together and behold, wow people do like this.

https://docs.microsoft.com/en-us/sharepoint/dev/declarative-customization/site-design-overview#pnp-provisioning-and-customization-using-microsoft-flow

So, what's next?

 

Serverless Orchestration

Invest into Serverless orchestration.  Azure Functions are not the right place to do our orchestration.  Yes, Durable Functions will help this a lot.  But the product we should be looking at is Azure Logic Apps / Microsoft Flow.

As far as I'm concerned - these are the same products, the differences boils down to:

Logic Apps

  • UX, with JSON editor is targetted for developers
  • Consumption based pricing - per actions used, perfect for multiple small requests 
    • So we end up compressing multiple actions into unreadable mess to save costs
  • Integration Services (biztalk scenarios)
  • Better for multi-tenant solutions.

Microsoft Flow

  • UX tries really hard to remain Power User friendly and hide JSON complexity
  • Per Flow execution pricing, with free buckets per tier
    • So we end up putting way too many steps inside a single Flow to save costs
  • Premium connectors as part of higher tier plans
  • Free licenses as part of Office 365 / Dynamic 365 plans making this cheaper for single-tenant solutions.

What can you do with Logic Apps/Flow?

  • Leverage connectors - (remember Embrace third-party services is a Serverless principle), these are hundreds of connectors implemented by the various product teams themselves directly.  So they know what they are doing* (most of the time)
  • You can do delay and wait easily in Flow
  • You can do loops easily in Flow (in Functions it's tricky without potentially hitting timeout).
  • You can do for-each loops in Flow and easily turn it into parallel execution (fan-out) with fan-in just part of the package
  • You can define repeat/retry policies with gradual fall back in Flow
  • You can define follow next token in Flow HTTP Request for REST paging
  • You can handle fallback behaviour as a scoped set, so if any actions fail you can orchestrate that
  • You can include human workflows with human approvals and send nice templated emails with attachments from Flow
  • Function shouldn't do more than one thing.  Use Flow to chain them.

 

Serverless API end points

As we build out a constellation (I stole this word from https://www.slideshare.net/HeitorLessa1/serverless-best-practices-plus-design-principles-20m-version) of functions.  We need to clean up all the microservices APIs with a unified API front.  There are two products for this:

Azure Functions Proxy

  • Simpler - can transform query/post messages

Azure API Management Service

  • More extensive - can transform REST to XML
  • Better Open API definitions

 

Serverless Websites / thicker Front-Ends

A serverless website is basically a CDN plus FaaS.  You don't scale Azure VM or even Azure WebJobs.  Build your entire website with your favourite JavaScript library (I like and recommend Angular - but you should use what your team uses), then bundle with Webpack into a couple of minified JS file for CDN.

Do your compute in the client.  And do your server compute with Azure Functions.

I'll even add here that a low-code solution such as PowerApps is extremely good at getting a proof of concept up and running quickly.  PowerApps supports offline capabilities and will happily call your Serverless APIs via a Swagger/OpenAPI file and treats them all as first class functions.

Wish

As part of the Azure services upgrade email (what a peculiar way to announce new features), the upgrade to the latest Windows Server means that Azure Functions, as part of Azure App Services, will gain ability to work with HTTP/2.

It means - we can get our entire HTTP website in one HTTP Get request, with our Function (or possibly our LogicApp/Flow) sending multiple resources in one response.

 

Serverless Database

Let me first define what is a Serverless Database.  Essentially, you have a database in the cloud. 

  • You want to pay for storage.
  • You want to pay for compute.  On consumption based plans
  • Pay nothing if it's not doing anything, automatically scale as necessary
  • The problem we are trying to fix is simple.  We want to start an application, pick a database, and have it scale with us.  We don't want to put SQL Azure on the cheapest free VM and have it run like crap.

This is an area where Azure is somewhat lacking.  My choices are:

  • Azure Storage Table
  • SharePoint Lists (only because I'm a Office 365 person and I've got office 365 tenants everywhere I look)

Wish

I predict boldly that Azure will bring out a Serverless CosmosDB solution in 2018 and it will be what everyone in the Microsoft ecosystem uses from there onwards.

Otherwise, look towards the competition:

  • Google Cloud Platform has Firebase - event driven, consumption based database, linked to Google Cloud Functions
  • Amazone Web Services has Aurora Serverless - in late 2017, AWS announced they've separated Aurora's cost model down to Compute and Storage.

 

Serverless Event Aggregator

The Azure Event Grid is a very interesting service.  I see the possibility that we'll see a unified way to manage all events in a system.

This is best explained with a parallel analogy.  In browser applications, we catch and handle events in the DOM all the time.

In the beginning, we do:

$(element).click(func)

This has all sorts of problems - how do you route.  How do you de-allocate.  How do you attach new events as new resources come online.  A few years later, we end up with this:

$(global).on("click", ".filter", func)

We attach events via one top level resource, ALL our event handlers are attached there.  And then we let events bubble to the root, apply the filter, then call the handler.

The Azure Event Grid has the potential to be this solution.  In 2019, if we are attaching event handling directly to a resource or a container, then we have stuffed up.  We should attach all our events to Event Grid, then filter within the event grid, and only then invoke the functions that fits the filter.

Wish

I'm hopeful if we can map Microsoft Graph events into the Azure Event Grid - then we'd have something super magical.

 

Serverless Visualization

I want to end on this one because I don't have a great solution, but I think we need a great solution.

Wish

As we built out our constellation of functions and orchestration, there's a need to visualize that design so we can both review the designs, and specifically see where the bottle necks are.

If a set of microservices are buggy, this would be a place to pintpoint this and switch the Functions back to the previous deployment slots.

With Azure Insights - we can get detailed logging for Functions and Flow/LogicApps, so perhaps this is something that needs to be layered on top of the logging.

 

References