Run any PnP-PowerShell in one AzureFunction from Microsoft Flow

pexels-photo-tool.jpg

Are you missing a SharePoint connector in Flow?  Do you need that one extra thing that's only in PnP-PowerShell (because there are 300 cmdlets in PnP-PowerShell)?  Frustrated with having to write another AzureFunction just for this one thing?

 

(Trumpet please) Behold!  I present the last AzureFunction (for PnP-PowerShell) you'll ever need to write.

Run-Any-PnP-PowerShell

And then, we'll call it from Flow.  Hmm wait, I think we can do better with the title...
(Trumpet please) Behold!

Run-Any-PowerShell in Flow

Wait, wait, no, why stop there with the title...
(Trumpet please) Behold!

Run Any Compute You Want in Flow

 

Plan

  • Create our Run-Any-PnP-PowerShell script
  • Call it from Flow
  • Call it to do some random compute for us

Create our Run-Any-PnP-PowerShell script

First - create an AzureFunctions app.  Install PnP-PowerShell in it. 

https://docs.microsoft.com/en-us/sharepoint/dev/declarative-customization/site-design-pnp-provisioning#upload-the-pnp-powershell-module-for-your-azure-function

I have an old post in 2016 that has similar steps, but the ones from SharePoint PnP Team is much more current.

If you have one already - great, just add another method. 

Select PowerShell from Experimental.  I call it Run-Any-PnP-PowerShell

Write the method.

# we need to import PnP-PowerShell cmdlets
Import-Module "D:\home\site\wwwroot\modules\SharePointPnPPowerShellOnline.psd1"

# POST method: $req
$script = Get-Content $req -Raw

# prepare credentials from environment USER/PW
$securepw = ConvertTo-SecureString $env:PW -AsPlainText -Force
$credentials = New-Object System.Management.Automation.PSCredential ($env:USER, $securepw)

# run the script from POST body
Invoke-Expression $script -OutVariable out | Tee-Object -Variable out

# catch output and send it back
Out-File -Encoding Ascii -FilePath $res -inputObject $out

6 lines of magic.

Call this from Flow

This works because Azure Functions is pretty clever about guessing the output of the Function, and changes the content type to JSON

Call it to do some random compute

Here's the same Function doing complex maths.  It'll also handle XML, JSON, .NET Framework and run any of the existing cmdlets or any extra ones we decide to import.

I think this ASCII module isn't very good.

 

Security [updated]

The feedback I got immediately was we'll need to secure that.  That's fair.  Here's a bunch of pretty important links to secure such a crazy "weapon of mass scripting".

Guard your AzureFunction activation key

Because you are calling AzureFunction from within a Flow - technically, only people that use the Flow can see the URL.

IP Filtering

Turn on IP Address Filtering for Azure Functions https://docs.microsoft.com/en-us/azure/app-service/app-service-ip-restrictions 
And the list of IP Addresses for your Flow runs from https://docs.microsoft.com/en-us/flow/limits-and-config#ip-address-configuration 

Secure by Azure AD

You can secure the Function with Azure AD https://docs.microsoft.com/en-us/azure/app-service/app-service-mobile-how-to-configure-active-directory-authentication

Your Flow HTTP request will need to authenticate as well before it can call your AzureFunction.  https://twitter.com/johnnliu/status/943761628671090688

Implementing Azure Functions Managed Service Identity

https://docs.microsoft.com/en-us/azure/app-service/app-service-managed-service-identity

This prevents the username/password being stored in the environment which can be read by the script.  With MSI, the credentials are stored in KeyVault.

Switch to Azure Automation

Azure Automation's native integration with KeyVault may offer the extra security you want.  I personally find it slower to start up, and the connector to retrieve the output is one extra step.  But this is definitely possible.

Add Persistent Logging

Because all the scripts has to come through your Flow - add a logging step that ensures scripts that are run are stored away with a log.

 

Finally

I do apologize I published this initially without talking about some security options.  Over the next day it became pretty clear that we need to secure such a crazy technique.

On the other hand, there was some pretty good discussions - so may be leaving out the security section for one day was a good idea after all.

Test yourself. A MicrosoftFlow Expressions Quiz!

I had an idea a while ago that a Quiz built with Microsoft Form would be a great way to learn and improve our knowledge of Microsoft Flow expressions.  For a long time there was just two questions on date time expressions.

I saw some entries since my previous post, and this sunny Saturday morning I decided to add more questions and ramp it up to 10.

board-chalk-chalkboard-459793.jpg

So, test yourself! The test is completely anonymous, and it gets really hard at the end, so I suggest unless you are PhD #DoctorFlow - treat this as an open book quiz. 

Reference

https://docs.microsoft.com/en-us/azure/logic-apps/logic-apps-workflow-definition-language

The Quiz

https://forms.office.com/Pages/ResponsePage.aspx?id=IFLmJmFV70aXg85fIEiSQVa44zYuPAVGrF-ZBk0inDpUMk1FT1hGOFo3N1ZEN0tVWk8wR1hRNDNINS4u

Special thanks to many of the early testers (Paul CulmseeFaustoPieterSandy UssiaDaniel Christian) that gave me a lot of feedback - I try to put in lots of comments regarding why a certain answer isn't correct, and clarify what I mean in many of the questions with examples of the result we wanted.

I'll leave this up indefinitely - use it to test your knowledge, and/or use it as a learning tool for some of the examples.  

Two complementary MicrosoftFlow podcasts in March, and Two Plateaus of MicrosoftFlow

Two complementary MicrosoftFlow podcasts in March, and Two Plateaus of MicrosoftFlow

John, you just recommended people learn Expressions and you cheered when Jon says Expressions is going away?!  Why are you so inconsistent?!

I finished listening to two parallel podcasts and there's some great contrasts between them that I wanted to point out, and write them down.

Read More

Design a "Delay until SharePoint File Changed" HTTP+Webhook for MicrosoftFlow

I love challenges - I love hearing people say "We need a Delay until File Changed" action in Microsoft Flow.  I agree.  But those are the same questions that sets one inquisitive mind to wander.

Plan

  • The Special Puzzle Piece
  • The generic nested Flow
  • The parent Flow
  • Results
  • Extend the technique

The Special Puzzle Piece "HTTP + Webhook"

Flows are triggered based on events - we can't easily have "a second trigger" in the middle of a Flow.  But there's a magical action in Flow that does do a 'wait' - and that's the HTTP + Webhook action.

"HTTP + Webhook" sounds like we plan to call someone else' webhook.  But we can use that to call our own webmethods.  So the idea then is that in the parent Flow, where we are building an approval, we'd delegate a 'wait' to a child nested flow and then have a generic Nested Flow that would perform any sort of 'wait'.  When the wait is over, the child Flow calls the callback URL and returns data back to the parent.

I first read about HTTP Webhook can be used this way reading @sp_muser blog post on Actionable Messages with Azure Functions. 
https://spwondering.wordpress.com/2018/01/17/actionable-messages-part-1-add-the-card-to-microsoft-flow/

HTTP Webhook left a very deep impression with me after reading that blog post.  Like a new lightbulb that won't go away.
Sometimes, we adapt new techniques.  This opens doors to new designs.  I hope this will help more people to be creative.

The nested child Flow

This is the child Flow - it uses a HTTP Request trigger and we need two arguments - a callback url, and a filepath to watch for in SharePoint.

We immediately check the file in SharePoint and remember the modified datetime (store this in a variable- modified).

Next, enter a do-until loop, and delay every 5 minutes and then grab the file's metadata again and check if the modified time has changed.  When it has changed, we exit the do-util loop, and call the original callback url with a plain HTTP action.

In your real scenario, think about whether 5 minute wait is too unnecessary, it may be better to check once an hour, or only a few times a day, if you don't want to loop too much.

The Parent Flow

In the parent workflow, where we need to delay until a File has changed - use the HTTP Webhook action, call the URL of the child Nested Flow, pass in the URL of the SharePoint file, as well as the listcallback() from the HTTP Webhook action.

Notice when the parent workflow is running and it enters the HTTP Webhook, the parent Flow enters a long-waiting asynchronous wait, magically.

Result

I go and modify the file that we are watching.

Notice the child Nested Flow wakes up after the next delay and detects the file's modified time has changed.

This exits the do-until loop, and call the callback URL.

Calling the callback URL wakes up and collapses the parent Flow

See the parent "waiting" action completes in some time in the future.  In my example, it was 10minutes.  But in real world this can be days or weeks.  Flow's timeout is 30 days.  So we can build quite complex, long-waiting, asynchronous processes.

Extend this technique

We can use multiple parallel HTTP Webhooks if we are waiting for multiple files or process updates to finish.

The nested Flow may not just check a file - it can be any long running process.

There's also a consideration that a long running HTTP Webhook works very similar to an Approval process and the Flow will happily wait for all these steps to return before collapsing and fan-in.

 

How to automatically enter MVP timesheets with Microsoft Flow

The Microsoft MVP Summit is next week.  I'm here at 6AM slaving away cracking on this Swagger API file so that we can all have the awesomeness of automatically submitting MVP timesheets with Microsoft Flow.

But really, who doesn't want automatic?! 

Flow, Make It So

Flow, Make It So

Plan

  • The MVP Contributions "timesheet"
  • MVP Production API
  • Custom Connection via Swagger
  • Set up the Flow
  • Future ideas

The MVP Contributions "timesheet"

The Microsoft MVP award is a recognition of our various activities throughout the previous year, and it is measured with both "reach" and "impact".  So, at a minimum, we have to do timesheets.  I really really don't like doing timesheets.

We really should be able to do this automatically.

Write a blog?  Made a podcast?  RSS -> Automatic.
Wrote a tweet?  Automatic.  (your MVP lead will probably have a chat with you about this)
IoT senses temperature change?  What better time to log an entry!
 

MVP Production API and the MVP PowerShell module

The MVP program with contribution from several MVPs created a set of APIs for querying and posting our profiles, contributions and details.  The starting point is here:

https://mvp.microsoft.com/en-us/Opportunities/my-opportunities-api-getting-started

To call this API, we need two pieces of authentication - we need OAuth to Windows Live Account, and we need an API key from the MVP API.  Follow the steps in the post and you'll end up with:

  • an MVP Production API subscription, which gives you a primary and secondary Api-Key
  • a MSA application with ClientID and ClientSecret

I want to note that there are existing work in a MVP PowerShell module

https://github.com/lazywinadmin/MVP

It works the same way, but because it doesn't remember your tokens, every time you run the PowerShell you need to login via a Live, get a token, then make the submissions and then when the PowerShell session finishes you lose everything.  So while this helps with entering the details, but doesn't help you manage your OAuth token, it certainly isn't "hands free / automatic".

So we will do this, automatically, with Flow

Custom Connection via Swagger

I frequently sing praises for Jan Vidar Elven's blog post on custom connector.

https://gotoguy.blog/2017/12/17/access-microsoft-graph-api-using-custom-connector-in-powerapps-and-flows/

I'm connecting a Custom Connection to Flow to help me manage the MSA account.  This is an extension of his detailed blog post.

To be able to call the MVP API we will need a Swagger (OpenAPI file) to create a custom connection.  After some struggling - I've got a working version of the swagger file here:

https://github.com/johnnliu/flow/blob/master/MVP%20Production.swagger.json

You can read this, but to use it - you need to replace line 35: 

"default": "ae2edf7-YOURKEYHERE",

With your real subscription key from the API.  You can use either the primary or the secondary.
Save the swagger file.  We go into Flow.

Set up the Flow

Start in Flow - create a custom connector by Importing an OpenAPI file

flow-security-1.png

For OAuth to work - the redirect URL from Flow must be allowed by this App

Return back to Flow Custom Connection

Create a connection

Setup the Flow to make your MVP Lead happy because now all your contributions are going to be automatically entered.

Results

I'm triggering this but clicking a button - you can hook this up to HTTP Request, Schedule Timer, RSS Feed... etc etc

The entry in the MVP tool.

Disclaimer

Running this Flow does not guarantee an MVP award.  But it will keep your lead happy.

 

Future Ideas

1. The Swagger File is generated from the MVP API tool, but underwent heavy modification.  For the curious you can compare the original vs my modified version.

2. As far as I can tell, the Swagger file defines two security definitions (for MSA and ApiKey), but Flow's Custom Connection UI can only handle 1 security setup.

Which is why I moved the ApiKey into an internal parameter within the Swagger File.

If Flow Custom Connection can handle multiple Authentication settings, then we can improve this part of the Swagger.

3. In Posting new Contributions - there are several settings are are ref objects.  ContributionType, ContributionTechnology (ContributionArea), and Visibility.  These should be connected to a dynamic lookup value, so within Flow UI, we will see a friendly dropdown menu that allows us to select one of the friendly names.

There's always more to do, but there's also a time to stop, and publish this blog post.