ApolloServer, AzureFunctions and local debugging

/

I wanted to jog down some notes that’ll help with local debugging of Apollo Server with Azure Functions.

Firstly, due to changes to Azure API routing - specifically x-ms-privatelink-id having strange invalid characters - we need to expand the function handler and blank out the x-ms-privatelink-id property.

https://github.com/Azure/azure-functions-host/issues/6013

Now to get local debug happening - we must configure CORS.

Add the cors settings - origin can be ‘true’ for every url, or ‘http://localhost:4200’ if only for your local build.

Additionally, copy the access-control-request-headers to Access-Control-Request-Headers because ApolloServer applyMiddleWare is case sensitive.

https://github.com/apollographql/apollo-server/issues/4178

const graphqlHandler = server.createHandler({
    cors:{
        origin: true,
        credentials: true,
    }
});
export default (context: Context, req: HttpRequest) => {
    // https://github.com/Azure/azure-functions-host/issues/6013
    req.headers['x-ms-privatelink-id'] = '';
    // apollo-server only reads this specific string
    req.headers['Access-Control-Request-Headers'] = req.headers['Access-Control-Request-Headers'] || req.headers['access-control-request-headers'];
    return graphqlHandler(context, req);
}

These changes by itself isn’t enough, we also need to make sure the Azure Functions runtime is forwarding the options method correctly to our Apollo Server. Make sure ‘methods’ contains ‘options’ otherwise the Azure Functions local runtime will not accept the browser pre-flight request.

(this second one was the one that got me for a while - why wasn’t Apollo Server options being called?!)

{
  "bindings": [
    {
      "authLevel": "function",
      "type": "httpTrigger",
      "direction": "in",
      "name": "req",
      "methods": [
        "get",
        "post",
        "options"
      ]
    },
    {
      "type": "http",
      "direction": "out",
      "name": "$return"
    }
  ],
  "scriptFile": "../dist/graphql/index.js"
}

Running Serverless Apollo GraphQL on AzureFunctions with cheap Azure Blob Table database(s)

/
leonardo-ramos-CJ4mbwSK3EY-unsplash.jpg

Merry Christmas holidays and happy new years.

This is a bit of a holiday reading, I wanted to start by give you the sense of what I’ve been tinkering as an experiment, and it looks increasingly like this is going into production before 2019 ends.


Plan

  • Have

    • I have a bunch of Azure storage Tables (about 70k rows of cheap database - at $1 / month storage) - they are spread out across several containers but for the basic purposes - I think of them as independent tables that lack proper relationships defined between them.
      I think that’s appropriate to describe most no-sql databases.

    • Azure Functions based CRUD REST APIs wraps around the Azure Blob Table - problem - no good caching or relationship mechanism. I wasn’t a big fan of keep rolling out CRUD REST endpoints, feeling that I should try find a different way.

  • Idea

    • Run a GraphQL server on Azure Functions

  • Learn:

    • If you want to run GraphQL on Azure Functions - there’s a dotnet way and there’s a NodeJS way

    • The dotnet version

      • https://www.tpeczek.com/2019/05/serverless-graphql-with-azure-functions.html

      • https://medium.com/@seekdavidlee/eklee-azure-functions-graphql-a-serverless-graphql-implementation-on-azure-484611c3680b

      • I’ve personally made a stack decision to stick to NodeJS/TypeScript/JavaScript, so I won’t be exploring the dotnet server.

    • The NodeJS version

      • A quick look around will send you to Apollo Server which has done 3 videos with Microsoft Azure/Channel9 on getting started, running on AppService, and running on Azure Functions (on consumption).

      • Part 1 https://www.youtube.com/watch?v=7R33hGFV4f0

      • Part 2 https://www.youtube.com/watch?v=Mt4bBpOdwyE

      • Part 3 https://www.youtube.com/watch?v=unUeFApHeT0

  • Write

    • Steps after Apollo + AzureFunctions 3-part videos.

    • Resolving GraphQL entities to Azure Blob Table

    • Resolving Relationships

    • Securing with Azure AD

    • Deploying to AzureFunctions

    • Reroute via Azure Function proxy

  • Future

    • Switch to Apollo RESTDataSource

    • Apollo Client with Angular and Observable bindings

    • Apollo Server with Redis cache



Learn

After the three videos, particularly the first and the third - to understand GraphQL server, and hosting it on AzureFunctions, you’ll end up with a working graphql server running on localhost showing you books array from memory. Getting here after watching three videos was extremely fast ~30minutes. Very surprising how much shortcut we took to get here this fast.

const typeDefs = gql`
    type Book {
        title: String
        author: String
    }
    type Query {
        books: [Book]
    }
`;

const resolvers = {
    Query: {
        books: () => books,
    },
};

let books = [
    {
        title: "A",
        author: "Mark B"
    },
    {
        title: "C",
        author: "John D"
    }
];

const playgroundSettings: any = {
    "schema.polling.enable": false
};
const server = new ApolloServer({ 
    typeDefs, 
    resolvers, 
    playground: {
        settings: playgroundSettings
    },
    debug: false 
});
export default server.createHandler();


A really fancy thing Apollo did was to put the graphql endpoint on POST, and run the playground test environment on GET. Pointing a browser to the same endpoint will show the playground.

Here is the playground running the sample books query. Debug works wonderfully running in localhost.

By default, the playground will do continuous polling on the endpoint. That’s going to keep the Function awake and incur a lot of costs. It might also keep bumping into any local debug breakpoints. So let’s turn that off. I also set debug to false.


Write - Resolving Azure Table

Next, we need to extend from here. My original AzureFunctions has REST endpoints that calls Azure storage Tables via the Azure.Storage SDK. So bring those in.

import { ApolloServer, gql } from "apollo-server-azure-functions"; 
import * as azure from "azure-storage";

And start to switch out the code

const typeDefs = gql`
    type Flow {
        RowKey: String
        name: String
        displayName: String
        critical: Boolean
        environmentName: String
        state: String
    }
    type Query {
        flows: [Flow]
    }
`;

const resolvers = {
    Query: {
        //books: () => books,
        flows: (parent,args,context) => azureTableQuery('flows'),
    }
};

const tableService = azure.createTableService();
const partition = "xxxx-partition-key";

const azureTableQuery = (table) => {
    let pRows = new Promise((resolve, reject)=>{

        let query = new azure.TableQuery().where('PartitionKey eq ?', partition);
        tableService.queryEntities(table, query, null, (error, result, response)=>{
            if(error){
                reject(error);
            }
            else {
                resolve(response.body['value']);
            }
        });
    });
    return pRows;
}

Notes: Azure.Storage does old style callbacks, so I’m wrapping them in a Promise that I can resolve. When I resolve - I want the raw JSON from the service, not the ‘tweaked’ JSON in result (it does weird things to your JSON entities, that’s a blog for another time).

A really nice thing about Apollo Server resolve function is that it knows how to handle promises naturally. So this tweak basically is me saying - hey, go fetch from Azure Storage Table from the ‘flows’ container.

To run this locally, we’ll need to store an Azure storage key in local.settings.json

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "",
    "FUNCTIONS_WORKER_RUNTIME": "node",
    "MyStore": "DefaultEndpointsProtocol=https;AccountName=mystore;AccountKey=FOOxxx"
  }
}

We can reference this like so:

const tableService = azure.createTableService(process.env["MyStore"]);

Write - Resolving relationships

const azureTableRetrieve = (table, key) => {
    return new Promise((resolve, reject)=>{
        tableService.retrieveEntity(table, partition, key, null,
        (error, result, response)=>{
            if(error){
                reject(error);
            }
            else {
                let entity: any = response.body;
                resolve(entity);
            }
        });
    });
}

const typeDefs = gql`
    type Environment {
        RowKey: String
        name: String
        displayName: String
        isDefault: Boolean
    }
    type Flow {
        RowKey: String
        id: String
        name: String
        displayName: String
        critical: Boolean

        environmentName: String
        ref_environment: Environment
        state: String
    }
}`;

const resolvers = {
    Query: {
        environment: (parent,{name},context) => azureTableRetrieve('environments', name),
        flows: (parent,args,context) => azureTableQuery('flows'),
        flow: (parent,{name},context) => azureBlobTableRetrieve('flows', name)
    },
    Flow: {
        ref_environment: (parent,args,context) =>
          azureTableRetrieve('environments', parent.environmentName),
    }
};

Notes:

I call “ref_” because I’m terrible at making property names. Lots of fields like “environmentName” or “environment” are already taken. It got too hard picking new names, I decided to call all entity references ref_

Deploy to Azure Functions

Need to deploy two parts:

Friends don’t let friends right click to deploy unless another friend put it in the right click menu. In that case, please don’t make me choose my friends.

Also, right click and upload local settings file to send our MyStore setting into Azure functions.

That’s two right clicks and deployment done!

Connect Azure Function proxy

My existing API routes are already covered with Azure Functions proxy. So I added a new route that takes graphql over to this separate Azure Functions. I also bury the functionkey in the route so users can’t see it.

Future

There’s a bunch more ideas for the future.

  • Explore federated authentication and user level security.

  • Figure out why Azure Storage SDK next generation doesn’t do Tables

  • Switch from using SDK to Azure Storage REST and use Apollo’s excellent RESTDataSource implementation instead (it already implemented HTTP caching, pagination, top, etc)

  • implement mutations, especially e-tag handling.

  • implement server side Redis cache

  • Implement Apollo Client to fetch data for my existing Angular front-end

  • Implement client side cache and observable pattern

Let me know if you find this interesting and want to follow along. I wrote this mostly as documentation for what I’m doing and the incremental changes I’m making as I go.

Presentations at the Digital Workplace Conference Australia 2019 I’m looking forward to

/

It’s really exciting for me to be back at the Digital Workplace Conference again in 2019, this is one of the first major conferences that I was very fortunate to be able to present at, and I have always looked forward to attend, to learn and to give back and present the latest happenings in SharePoint, Office 365, Azure and now also Power Platform.

The Digital Workplace Conference will be on August 6-7 in Sydney.

DWCAU-temp-dates.png


This year I’m presenting a topic (and sorry for the slightly concatenated topic name):

A quick introduction to Microsoft Flow, but rapidly takes audience deeper to see the possibilities of the types of solutions possible


There probably should be a line-break in there somewhere. You have to use your imagination for this title for that missing character, sorry!

Lets try that again!

An introduction to Microsoft Flow: Office 365 - Automated

This is an introduction session on Microsoft Flow, but my aim isn’t to show you a simple alert email. My aim is to explain that we have far transcended “Workflows”. It’s not just workflow. We are way beyond that - what we really have is a full Automation Engine. It is Office 365 - Automated.

Be very careful, writing Microsoft Flow is completely addictive.


The Sessions I want to see!

  • Become The Expert of You – Power Skills for Personal Development - Heather Newman

  • Trust-Based Corporate Culture: How to Kick Fear and Toxicity Out of the Workplace - Heather Newman

  • The Fruit Salad Formula for Implementing Ofice 365 - Debbie Ireland

  • Microsoft Teams Deep Dive – 29 Practical Tips and Tricks - Lee Stephens

  • Making Teams work without understanding Information Architecture - Alistair Pugin

  • A Zero-Hype Introduction to Artifical Intelligence and Machine Learning - Kilash Awati

  • Adding voice interaction to your apps - Brian Farnhill

  • Give your PowerApps and Flows some Vision, Language and Feeling - Rich Burdes

  • How to build a Project Hub with Hubsites and Sitedesign and Sitescripts - Knut Relbe-Moe

  • Success Factors in a Thriving Yammer Network - Rebecca Jackson

  • How to Govern PowerApps and Flow - Paul Culmsee

  • Governance & Adoption for Microsoft 365: Making the Marriage work - Megan Strant

But that’s not all

I’m at the conference both days and plan to hang around for the workshop day too. So if you have any questions on

  • Microsoft Flow, PowerApps, Power Platform Governance solutions

  • Azure LogicApps, Azure Functions

  • SharePoint, PnP, Microsoft Graph, SPFx

  • Flow Studio startup journey

  • The latest imaginations in the mind of a mad man

And whatever in between…
I’m available for a chat!

The value of a Physical Conference

I always feel the top value of a physical conference is being able to stash away two or three days to focus and immerse in an environment with like-minded people, we talk tech, have the same challenges, but may have solved some of them with our own personal takes of that experience.

We are very much alike - when we are together in the same conference - we are introduced to new ideas that gives us a much broader perspective than we would usually see when we are so focused on our immediate day to day business problems.

I treat it as a relaxed edu-vacation. Seek out the sessions that broadens my capacities. And grab hold of the presenters and get a deeper discussion to take that back with me.

I would love to see you (see you again) at the Digital Workplace Conference 2019. If you see me you should say Hi John - I want to ask / I want to tell you about <THIS THING>

Decode InfoPath attachments with a bit of JS AzureFunctions

/

Serge, April and me were discussing a problem with pulling out InfoPath Attachment from InfoPath form XML and writing them into a SharePoint document library.

This is a problem I tried to tackle before, but came to realization that I would need an AzureFunction. The main reason is that the InfoPath attachment is a base 64 byte array but the byte array has a variable length header that includes the attachment file name. Flow doesn’t have amazing byte manipulation or left-shift abilities. So we need to write an AzureFunction to help.

As I brood over the problem I also thought it might be easier to handle the byte array with JavaScript. So I gave it a go.

This blog is my version of the answer.

The original decoder code in C#

There is a pretty old MSDN article on the C# code

private void DecodeAttachment(BinaryReader theReader)
{
  //Position the reader to get the file size.
  byte[] headerData = new byte[FIXED_HEADER];
  headerData = theReader.ReadBytes(headerData.Length);

  fileSize = (int)theReader.ReadUInt32();
  attachmentNameLength = (int)theReader.ReadUInt32() * 2;

  byte[] fileNameBytes = theReader.ReadBytes(attachmentNameLength);
  //InfoPath uses UTF8 encoding.
  Encoding enc = Encoding.Unicode;
  attachmentName = enc.GetString(fileNameBytes, 0, attachmentNameLength - 2);
  decodedAttachment = theReader.ReadBytes(fileSize);
}

The updated code in JS AzureFunctions

module.exports = function (context, req) {
    context.log('JavaScript HTTP trigger function processed a request.');
    if (req.body && req.body.file) {
        // https://support.microsoft.com/en-us/help/892730/how-to-encode-and-decode-a-file-attachment-programmatically-by-using-v
        var buffer = Buffer.from(req.body.file, 'base64')
        //var header = buffer.slice(0, 16);  // unused header
        var fileSize = buffer.readUInt32LE(16);  // test is 5923 bytes
        var fileNameLength = buffer.readUInt32LE(20);  // test is 13 chars
        // article lies - it's utf16 now
        var fileName = buffer.toString('utf16le', 24, (fileNameLength-1)*4 -1);  
        var binary = buffer.slice(24 + fileNameLength * 2);
        context.res = {
            // status: 200, /* Defaults to 200 */            
            body: {
                fileName: fileName,
                fileNameLength: fileNameLength,
                fileSize: fileSize,
                fileContent: binary.toString('base64')
            }
        };
    }
    else {
        context.res = {
            status: 400,
            body: "Please pass a base64 file in the request body"
        };
    }
    context.done();
};


The InfoPath form


The Microsoft Flow that coordinates the work


Results

  • Need Azure Function here

  • JavaScript buffer is pretty good at doing byte decoding, easy to read too

  • Debugging and tweaking the byte offset is quite a bit of trial and error, was not expecting that. May be that MSDN article is too old, it is from 2003.

  • You may think - John 2018 is not the right year, or decade to be writing about InfoPath. But hear me out. As companies move their form technology forward, they will need to consider how to migrate the data and attachments in their current InfoPath forms somewhere - having this blog post as a reference is important for that eventual migration. Good luck!

From Office 365 to Azure Event Grid, the events must Flow

/
Photo by Archana More on Unsplash

Photo by Archana More on Unsplash

In this blog post, we capture all the events across an Office 365 Tenant from multiple event sources, gather them, and send them through an Azure Event Grid.

We then listen, filter and handle our events in a central, unified way.

The events must Flow.

This is also the full write up of this microblog posted to Twitter #FlowNinja earlier this month.

Plan

  1. What benefit do we get from this?

  2. Listen to every event across an Office 365 Tenant

  3. Construct a uniform event message

  4. Send them into a Serverless Event Solution - Azure Event Grid

  5. Filter and catch our events

We want to build 3 Flows

flow-event-grid-1.jpg

What benefit do we get from this?

First, we see the increasing availability of event hooks - we have subscriptions, delta queries, or webhooks, across various different products in Office 365. Some products like SharePoint is getting a SocketIO webhook. There will be more events, and our event handling design must evolve.

Second, we see the cost-effective solutions to handle these ever increasing flood of events in the form of Serverless compute. This is true with Azure Functions, Microsoft Flow or Azure Logic Apps.

We end up with a lot of individual event sources and a lot of individual event receivers. This is a common event handling problem. As number of events we handle increases, the worse the event management problem becomes.

Consider you have a “handle a document uploaded to a library” event - a very typical SharePoint Workflow. Now consider this library is cloned to a hundred project sites.

If we clone the event handler a hundred times, we have a problem.

If you have already done this with Flow, then try https://FlowStudio.app to help you manage them.
Ooh inline product placement!

If you are a developer, then consider this scenario.
Consider a typical event handling in the browser. A decade ago we used jQuery like this:

// 2008
$('button').click(handler);

// 2018
$(global).on('click', 'button', handler);

And gradually we find that unacceptable, because we have buttons, events everywhere, and managing individual event hooks was tedious and error prone. Eventually, we moved to a global handler model, and we filter just prior to event being raised.

The headache-less way to handle events is to set the hooks all at the global root level, and then filter by the event source and event type.

That is the exact reason we need Azure Event Grid.

  • Centrally manage our events

  • Decouple the event source from event handlers

  • Stay sane, with a hard problem

Listen to every event across an Office 365 Tenant

I had previously wrote about listening to Office 365 Management API via the HTTP action and app-only permissions. What I did not realize was that the Office 365 Management API also have fantastic webhooks.

I read about the webhooks from Kent Weare’s post, where he uses this event to get a trigger when Flows are created in the tenant.

https://flow.microsoft.com/en-us/blog/automate-flow-governance/

These are grouped into several categories: AzureAD, Exchange, SharePoint and General (other).

This is the subscriber. One picture of 4 blocks. I’m subscribing to three webhooks at once.



The Office 365 Management API is a fantastic general event source. The downside is that it’s not instant - event handler is called between 10-20minutes after the actual event. So it is great as an audit webhook, or for scheduling files or search or to signal for a bot to re-scan a document. But it’s not an instant webhook.

Of course, if our goal is to send events into an Event Grid - we can work with multiple event sources at the same time. We can add Microsoft Graph events or subscribe to SharePoint list webhooks directly.

This is the top of the Listener

Construct a uniform event message

The event grid has a event JSON structure, it also supports a CloudEvent structure.

When I built my implementation, the Event Grid connector is still in preview and I had troubles publishing a Cloud Event structure. I assume this wouldn’t be a problem anymore as the connector evolves.

This is the final loop design - all done.

Remember, we are running Serverless so abuse/utilize every opportunity to use as many Azure Servers as you can - if you can fan-out to parallelism you must.

Don’t talk to each individual HTTP action one at a time. Do (up to 50) all at the same time.


Send them into a Serverless Event solution - Azure Event Grid

The Azure Event Grid is a serverless event processing pipeline. It decouples our event source(s) from our event handlers.

Here is our first handler.

This catches every event on the Event Grid - in Event Grid, we see we have our first webhook attached - it appears as a LogicApps webhook.

Here are three examples of what it caught:

  • Flow created Event

  • Site Collection created Event

  • File uploaded Event

flow-event-grid-3.jpg

Filter and catch our events

We see the very specific webhook now registered on the event grid - and the filters are also listed

PPTX filter only runs when the file I’ve uploaded is a PowerPoint file.


Summary

I have been talking a lot about Serverless and how our tools and design must evolve. Having a unique Office 365 to Event Grid solution is something I talked about as far back as 2017. I’m glad a year later I’ve finally got a great prototype going.

  • Office 365 Management API is a great webhook source that catches all sorts of events. The downside is that it is an audit webhook, so the delay may not be acceptable to your needs.

  • Using Azure Event Grid to perform filtering and subscription gives us the unique ability to see EVERYTHING that’s going on in our tenant. That has tremendous value.

  • Because event source and event handling is now decoupled - we can add new event sources to push to the same Azure Event Grid. We can do this from Microsoft Graph, we can do this from SharePoint, or we can do this from a whole myriad of triggers available in Microsoft Flow

  • We can write Azure Functions to trigger off the Event Grid, and it would be visible as well.

  • I was reading and appreciating sending events to the new Azure SignalR service from Azure Functions - that would be pretty amazing to convert an event grid message into a websocket event.
    https://twitter.com/nthonyChu/status/1044427579460145152

The possibilities are endless. Our tools and our design must evolve.