Azure Functions and Punch Cards

/

This is apparently a computer story, but is actually also very much a human story.  In which we explore the relationship between Azure Functions and Punch Cards.

My mother

It's been very warm in Sydney in this blistering summer.  My mother comes over to visit and relax and enjoy my air cond.  I'm preparing for Microsoft Ignite Australia on Azure Functions, and I was thinking about a human story.

I looked up at mum, this is the frail, little Chinese grandma that plants funny herbs in the garden.  We had a chat.  I then went online and found this picture.

By ArnoldReinhold - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=16041053

By ArnoldReinhold - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=16041053

Our story

I sometimes talk of my parents who met as comp science students in the 70s.  My mother, who sometimes can't remember how to restart her iPad.  Well, she cut her teeth on these punch cards. She tells me how big they are - gesturing with her hands - the box of cards would be nearly the size of A4 - much wider and longer than I thought.

I ask, so you need to pass the cards to an operator to run them and wait.

Of course.

What if you have an error in your code?

She pretends she spends a lot of time checking before submitting the cards.  But later admitted actually, the code would crash.  I think she figures out that it's a funnier story to tell the truth.

And since the output is a printout (the cards are input, there is no keyboard/screen) she'd get pages and pages of invalid instructions handed to her by the operator.

Imagine Developers, if your OPs is responsible to read to you all your compile errors.  Every time.

She will have to resubmit her program.

It'd be quite embarrassing she says.

I sat back, and tell her of @AzureFunctions and how I now submit code to be run in the cloud.

And how first few runs, my code would spit back rows of errors. we laughed.

Some things never change.

And then, some things do change

 

By NARA - http://www.archives.gov/75th/photos/1950s/20.html[dead link], Public Domain, https://commons.wikimedia.org/w/index.php?curid=17413686

I end this story with this picture.  This warehouse stores punch card programs for US National Archives 1959. Equivalent to 4Gb.

While my mother will have a second chance to resubmit her program in the afternoon, I can repeat as many execution as I'd like in the cloud.

The mainframe computer my mother used when she worked in Statistics Bureau of Taiwan is worth hundreds of thousands of dollars, and she has to wait with other programmers to use it.

The Azure Functions that I can write and deploy today, cost me less than $0.01 and most of the time the compute is free.

Future

Bonus: I told her future datacenters will be underwater and completely unmanned #ProjectNatick

Her eyes are full of wonder

 

 

 

Interact with Graph and make O365 Groups with AzureFunctions PowerShell

/

In this post, we talk about how to get an access_token in PowerShell to talk to the Microsoft Graph, so we can run automated non-interactive scripts in Azure Functions.

 

I'm so sorry for butchering the title every time.  But this blog post is cool.

I've written previously about creating SharePoint sites via PnP-PowerShell.  Which is very powerful and very popular - thank you for all the nice discussions and comments.  Soon, people asked - can we do the same for O365 Groups?

The PnP-PowerShell cmdlet for Connect-PnPMicrosoftGraph works, but it raises an login dialog, which makes it a show stopper for non-interactive scripts or Azure Functions.  I took it upon myself to find a workaround over my December holidays but alas the solution didn't come until January. 

What is the Trick?

The problem is to authenticate with Azure AD and get an access token that we can use to talk to the Office 365 Graph.  The trick, is a little not-well known thing called Resource Owner grant.

grant_type=password

I have a few links about the Resource Owner grant type at the end.  Basically, this grant_type lets you use username/password to obtain an access token.

Setting up Azure App Registration

You are probably familiar with these steps by now.  Today's screenshots came from the new Azure AD Portal.

Navigate to the Azure Active Directory portal, before we go further, grab the Directory ID - this is the tenant ID you'll need.  The tenant ID isn't a private value - it appears in URLs often when you are signing into Office 365.  But it's handy, right here, so take a copy.

Create our app.

Add Delegate Permission to Read and Write all groups.  Note this requires admin permissions.

Click the awesome Grant Permissions button at the top of the permissions registration, this grants it for users in your Active Directory.

You will need some clientsecrets - create them from Keys.  I like them not expiring for a long time.  So I pick the Never expires option.  It'll expire when I'm long gone.

Copy down your ClientSecret

One final check.  Copy the Application ID.  This is your app's Client ID.

To Azure Functions!

 

First, we create another HttpTrigger-PowerShell.

Because we aren't using PnP-PowerShell this time, there is no extra modules dependency.  This example is going to be simply Invoke-WebRequest and Invoke-RestMethod

Start with this code:

$requestBody = Get-Content $req -Raw | ConvertFrom-Json

$username = "[email protected]";
$password = $env:PW;
$client_id = "9308c103-8208-40cd-85e2-37a994b3578d";
$client_secret = $env:CS;
$tenant_id = "26e65220-5561-46ef-9783-ce5f20489241";
$resource = "https://graph.microsoft.com";

# grant_type = password

$authority = "https://login.microsoftonline.com/$tenant_id";
$tokenEndpointUri = "$authority/oauth2/token";
$content = "grant_type=password&username=$username&password=$password&client_id=$client_id&client_secret=$client_secret&resource=$resource";

$response = Invoke-WebRequest -Uri $tokenEndpointUri -Body $content -Method Post -UseBasicParsing
$responseBody = $response.Content | ConvertFrom-JSON
$responseBody
$responseBody.access_token

Out-File -Encoding Ascii -FilePath $res -inputObject $responseBody

Change to your own username/password/clientid/clientsecret/tenantid - note my password and clientsecret are stored in Function App Settings.

When this Azure Function runs -

You'll see the $responseBody and specifically, the $responseBody.access_token

We now have a key.

Calling Graph - Get Groups

$access_token = $responseBody.access_token

# GET https://graph.microsoft.io/en-us/docs/api-reference/v1.0/api/group_list
$body = Invoke-RestMethod -Uri "https://graph.microsoft.com/v1.0/groups" -Headers @{"Authorization" = "Bearer $access_token"}
$body | ConvertTo-JSON

Out-File -Encoding Ascii -FilePath $res -inputObject $body

Call graph, attach access_token in Header.

My Office 365 groups in JSON.  Notice the function took about 1 second between start, obtain access_token, and retrieve graph.

Calling Graph - Post Group

The Graph is not a read-only datasource, and our App has Group.ReadWrite.All

# POST - this creates groups https://graph.microsoft.io/en-us/docs/api-reference/v1.0/api/group_post_groups
$body = @{"displayName"="ps-blog"; "mailEnabled"=$false; "groupTypes"=@("Unified"); "securityEnabled"=$false; "mailNickname"="ps1" } | ConvertTo-Json 
$body = Invoke-RestMethod `
    -Uri "https://graph.microsoft.com/v1.0/groups" `
    -Headers @{"Authorization" = "Bearer $access_token"} `
    -Body $body `
    -ContentType "application/json" `
    -Method POST
$body | ConvertTo-JSON

Out-File -Encoding Ascii -FilePath $res -inputObject $body

Here we go.  A new O365 group "ps-blog" is created.  Note the function took 3 seconds.

Why I like PowerShell to demo Functions

  1. Code looks simple.  Cmdlets compresses a lot of functionality. 
    Take for example ConvertTo-Json or ConvertFrom-Json.  In C# we'd have to call a JsonSerializer and in NodeJS we'd need double promises just to get a json from fetch()
  2. PowerShell wraps C# and .NET libraries so they can be fairly powerful without a lot of code
  3. Not everyone understand JavaScript, or NodeJS JavaScript
  4. Suitable for IT Pros that may only know PowerShell
  5. Developers should learn more languages - and you can convert this in your head to C# or NodeJS anyway.

Bonus

This part is an exercise for the reader.  Make an PowerApps app that will call this function - passing in a parameter for the group name.  Hook up the Azure Function as a connection and bind it to a textbox and a button press.

Ignite Australia

Come and see me present all these cool techniques and more at Ignite Australia!

https://msftignite.com.au/sessions/session-details/1988/serverless-in-office-365-build-services-with-azure-functions-prod324

We'll talk about the Azure Functions story for today's Office 365 Developer, IT Pro and Power User.

Azure Functions has an amazing place from microservices to Swagger to the BotFramework!

 

Summary

  • Use PowerShell
  • Get access_token via username/password
  • Get our graph
  • Make O365 Groups

References

 

If you liked this post - please leave a comment below.  Please also try the code and let me know how it goes for you.  I love reading about people taking my brief demo code further to solve real problems.

Running OfficeDev PnP cmdlets in 32bit AzureFunctions

/

Some recent changes with AzureFunction created a small problem - x64 functions are not available in the starter-consumption plans.

The situation is not game over.

Since all the assemblies are not compiled for x64 only.  So just open the psd1 file and change 'Amd64' to 'None'.  They actually run just fine.

 

Build your PnP Site Provisioning with PowerShell in Azure Functions and run it from Flow

/

What if I tell you - you can build your own Azure Function and use PnP to provision SharePoint Online sites without firing up Visual Studio?

What if I tell you - we will follow the best practices from PnP.  And You can do this all within 10 minutes (OK I timed it - if you don't take any breaks it'll be 10 minutes), and we'll do it with PnP PowerShell.

You will think - this is magic.  You will probably say, this is all crazy.

What if I tell you - you can connect this to the newly released Microsoft Flow.  So you don't even need to learn WebHooks (which is awesome and you should learn it).

What if I tell you - the whole thing will cost you a little bit.  It's not totally free.  It will cost you may be two cents.  ($0.02)

You will say, get out.  I need to lie down.

Learning Software pieces like LEGO blocks

I see software pieces as I see LEGO blocks.  PnP Provisioning is a block (a .NET library written in C#).  PowerShell cmdlets are a block.  And Azure Functions is a block.  Flow is a block.

What would happen if you connect them together?

 

Setting up PowerShell from PnP

First, this is how we set up in our local machine:

Install-Module SharePointPnPPowerShellOnline

Why PowerShell? 

Why PowerShell, you ask - John you are a developer.  You are a .NET developer before you became a JavaScript developer.  Why PowerShell suddenly?

The answer is simple.  It is because everybody understand this:

# connect to the SPWeb context
Connect-SPOnline -url "https://sharepointgurus365.sharepoint.com/sites/projects/template"
# export (copy) a template
Get-SPOProvisioningTemplate -force -out project-template.xml

# connect to new blank SPWeb
Connect-SPOnline -url "https://sharepointgurus365.sharepoint.com/sites/projects/tea"
# paste the template
Apply-SPOProvisioningTemplate -path project-template.xml

Four lines - you have just copied a template from /projects/template and stamp it over to /projects/tea - this is the power and the hardwork of the PnP provisioning team.  They make this magic look simple.

PnP was updated about a week after the blogpost and several PnP commands have been renamed to avoid confusion with SharePoint admin commands. See PnP-PowerShell documentations.

Connect-SPOnline is now Connect-PnPOnline
Get-SPOProvisioningTemplate is Get-PnPProvisioningTemplate 
etc

A way to verify the powershell is to run it first on your local machine and make sure it works first, before uploading the same DLL modules and script to Azure Function.

Special thanks to @pskelly

 

But you can do this in C#

Of course, you can also do this in C#, it looks like this:

This shouldn't be too scary to a developer.  But everyone sees the bubbles and the flow chart and just stop trying.  It "looks" hard.

Additionally, in CSOM - everything is asynchronous.  C# makes this simpler with the await syntax, but in the PowerShell cmdlets, every request is essentially serialized - so they are easy to understand, even if they aren't the fastest way. 

If you want to see what this looks like in NodeJS - I have several blogs.  It's great to learn.  If you don't like JavaScript to begin with - then you'll probably run to PowerShell with open arms.

OK enough chatter - let's go back to the demo.

 

Setting up Azure Functions

Create a new HTTP Request function - select PowerShell as the language.

It says Experimental - what they really should say is "The World's Not Ready For This".

PowerShell Azure Functions looks like this.  The request is passed in as raw JSON content - which is parsed into an object.

Output from the function by writing out to the $res file.

 

Install-Module

install-module doesn't work.  But we still can have modules.

http://stackoverflow.com/questions/37724769/how-to-install-a-powershell-module-in-an-azure-function/39985646#39985646  - The steps are explained by Azure Function team's @tohling

Essentially, we can take a PowerShell Module, find all its component script and DLL, and copy them to a modules/ subfolder within the Azure Function.  It will be loaded when the function needs to run.

In Functions - go to Kudu

Navigate to your new function's directory, and add a modules subfolder

In your PC:

Navigate to

C:\Program Files\WindowsPowerShell\Modules\SharePointPnPPowerShellOnline\2.8.1610.1\

(if you don't have this folder - you missed Install-Module SharePointPnPPowerShellOnline from earlier)

You will also need Microsoft.IdentityModel

C:\Windows\assembly\GAC_MSIL\Microsoft.IdentityModel
C:\Windows\Microsoft.NET\assembly\GAC_MSIL\Microsoft.IdentityModel.Extensions

Grab all these DLLs and PowerShell script and throw them all inside modules.

 

Change Azure Function to x64

One more thing.  The PowerShell cmdlets prefer x64 architecture.  Azure Functions provide 32 bit by default.  Switch this in function settings > application settings > Platform

 

Setup Done!

Now we can write fun functions!  Go back to our site provisioning function.

$requestBody = Get-Content $req -Raw | ConvertFrom-Json

# {
#     "source": "https://sharepointgurus365.sharepoint.com/sites/demo02/projects/pst",
#     "destination": "https://sharepointgurus365.sharepoint.com/sites/demo02/projects/tea"
# }

$source = $requestBody.source
$destination = $requestBody.destination

$secpasswd = ConvertTo-SecureString $env:SPO_P -AsPlainText -Force
$mycreds = New-Object System.Management.Automation.PSCredential ($env:SPO_U, $secpasswd)

Connect-SPOnline -url $source -Credentials $mycreds 
Get-SPOProvisioningTemplate -force -out "D:\home\site\wwwroot\project-template.xml"

Connect-SPOnline -url $destination -Credentials $mycreds 
Apply-SPOProvisioningTemplate -path "D:\home\site\wwwroot\project-template.xml"

Out-File -Encoding Ascii -FilePath $res -inputObject "Done $destination"

Note: PnP was updated about a week after the blogpost and several PnP commands have been renamed to avoid confusion with SharePoint admin commands. See PnP-PowerShell documentations.

Connect-SPOnline is now Connect-PnPOnline
Get-SPOProvisioningTemplate is Get-PnPProvisioningTemplate 
etc

A way to verify the powershell is to run it first on your local machine and make sure it works first, before uploading the same DLL modules and script to Azure Function.

Special thanks to @pskelly

The script will run unattended.  So you will need to provide your username/password stored in Function Settings and read from $env variables.

Run success!

What do we have here?!  PowerShell cmdlets that wrapped the PnP C# library and talks to SharePoint Online and everything running inside an Azure Function.

You have a service.

If you are counting lines - there is 10 lines of PowerShell - including the function wrapper.

 

Flow

Let's top off this demo with a Microsoft Flow - this was GA'ed just this week.

Create a list "Copy-Site" - with two fields "ToSite" and "FromSite"

Have the flow trigger off ItemAdded, and call our Azure Function via the Http Request.

Pass in source and destination.

 

Test Flow... workflow

Flow tells you what was sent to the HTTP Request, and what Response body it got back.  Use this for debugging your functions.

 

We always talk Costs because Azure Function is cheap

The 11cents total is over several days, and the breakdown shows it's almost entirely bandwidth costs - which doesn't have a free bucket.  My AzureFunctions don't exceed the allowed free tier, so I'm not paying for any compute time.

 

Where do we go from here?

What can you do with PowerShell?  Desired State Configuration?  Download lists to CSV?  Synchronize and fix metadata?  Locate orphaned or misplaced documents and fix that?  Improve and merge user profile properties?  Provision users and sites and what else do you fancy?

Does this give you wild ideas?

 

Azure Functions is Serverless

When we say Azure Functions is Serverless - it always seems we are saying we don't need servers (and thus don't need IT Pros).  This isn't strictly the truth - which is that the server is maintained by professionals, and developers could focus on writing code.

But after working on these PowerShell for the last two weeks - actually, I'm beginning to wonder whether there will be a day we don't need Developers.

An analyst says to a Bot - I want this System A to connect to this System B and put result in this System C.  The bot arranges three connections and suggest how the parameters will be connected.  Offers to perform a trial run.  Analyst checks it and approves it.  The code goes into production.

Think about this.  Everyone needs to evolve.  IT Pros - you have a huge opportunity here to use your PowerShell skills to provide huge business value.  Developers - you should use the best tools to enable your users and do more, this is really easy to get into.

 

Tell me what you think

I really want to know what you think - if you can spend a bit of time and leave a comment below.  Where are we going as developers, where are we going as IT Pros?  What kind of PowerShell-AzureFunction are you going to build?  Does PnP need a repository for PowerShell recipes?

It's so nice to see the pieces fit together well.

Working with SharePoint WebHooks with JavaScript using an Azure Function

/

This blog post covers additional explanation on how to subscribe to SharePoint WebHooks and have it running with only JavaScript in Azure Functions.  The entire code is in one single JavaScript file.

The SharePoint team delivered on the promise to ship SharePoint WebHooks, and made an reference example in C#.  Be sure to watch the PnP webcast as well.

What is Azure Functions?

Azure Functions, in the simplest sense, is a Azure WebJob that lets you run a JavaScript function in a file (it can do C# too), and it will run it for you when you trigger it.  Because it is a Serverless platform - you don't pay for the WebJob unless your function is running.  This makes it really economical (you also get like a million free runs per month...)

Azure Function comes with a super user-friendly UI and you can paste or upload your JavaScript directly in the browser.  You don't need any tools installed.

Running in the browser

 

Of course, this is server-JavaScript.  So think NodeJS.  We can talk to lots of REST APIs (Graph, SPO) but there will be no browser DOM.

What can you do with this?

You can trigger it on a timer.   Hey that sounds like a SharePoint Timer Job.

You can trigger it on a web request.  This is super useful if you have a client-side UX and you need a button to do some high level elevated permission action.  You call the Azure Function to do it for you, using an App-Only permission elevation.

Why is SharePoint WebHooks important?

A SharePoint WebHook is a REST endpoint that you can attach a remote End Point to.  Right now, there is only a List endpoint.  So any updates to the list items will trigger an event, and SharePoint will call your function.

Hey.  Wait that sounds like a SharePoint Remote Event Receiver.  You are right!

Design

So the idea of our function is this - when triggered:

It will Auth and then talk to SharePoint REST and do one of the following.

  • Check if it has a request parameter "subs" - then it will list the current subscriptions on our target list
  • Check if it has a request parameter "sub" - it will try to attach itself to the target list
    SharePoint will immediately call the function with a validationtoken parameter so…
  • Check if the request has a validationtoken parameter - it will immediately reply with that token as text/plain.
  • Skipping all three conditions, it will run the default action
    The default action is that it will add an list item in a different destination list.  Because the function is running on its own App-Only permission, it can update a list that the original user doesn't have access to.

 

GET Subscriptions

options = {
    method: 'GET',
    uri: "https://johnliu365.sharepoint.com/_api/web/lists/getbytitle('subscribe-this')/subscriptions",
    headers: headers
};
request(options, function (error, res, body) {
    context.log(error);
    context.log(body);
    context.res = { body: body || '' };
    context.done();
});

GET subscriptions.  Array result is [] empty by default.  It has subscriptions after you attach hooks successfully.

 

POST Subscription (to add itself)

options = {
    method: 'POST',
    uri: "https://johnliu365.sharepoint.com/_api/web/lists/getbytitle('subscribe-this')/subscriptions",
    body: JSON.stringify({
        "resource": "https://johnliu365.sharepoint.com/_api/web/lists/getbytitle('subscribe-this')",
        "notificationUrl": "https://johnno-funks.azurewebsites.net/api/poke-spo2?code=q8sq9wxm62asd-YOURTRIGGERCODE",
        "expirationDateTime": "2017-01-01T16:17:57+00:00",
        "clientState": "jonnofunks"
    }),
    headers: headers
};
request(options, function (error, res, body) {
    context.log(error);
    context.log(body);
    context.res = { body: body || '' };
    context.done();
});

Sending POST subscription without handling validation token.  The request fails.

 

Handle Validation Token

// if validationtoken is specified in query
// immediately return token as text/plain
context.log(req.query);
context.log(req.query.validationtoken);
context.res = { "content-type": "text/plain", body: req.query.validationtoken };
context.done();

The function is called twice.  Second time by SharePoint to validate the subscription.

Result Video

The Poked list item was created by "SharePoint App" not "John Liu" the user.

Notes

This demo builds on top of the code from Azure Functions, JS and App-Only Updates to SharePoint Online that covers authentication with certificate, and running AppOnly permissions. 

Additionally, I've moved ClientID and Certifate ThumbPrint to Azure Function App Settings.  This means they are no longer part of the code.

App Settings

var clientId = process.env['MyClientId'];
var thumbprint = process.env['MyThumbPrint'];
Function App Settings > Configure App Settings

Function App Settings > Configure App Settings

 

Source Code

I'm making an effort to put all my demo source code on GitHub going forward.  This is part of the "Upgrade Your JS" demos.

https://github.com/johnnliu/demo-upgrade-your-js/tree/master/azure-function-web-hook

Let me know what you think about SharePoint WebHooks, Azure Functions and JavaScript that will rule everything ;-)

If you spot a bug or want to update the code - send me a Pull Request.