A group_by filter for liquid-node

/

Here’s a group_by filter for liquid-node

// https://stackoverflow.com/a/34890276/40857
const groupBy = function(xs, key) {
    return xs.reduce(function(rv, x) {
        (rv[x[key]] = rv[x[key]] || []).push(x);
        return rv;
    }, {});
};
// https://github.com/docs/liquid#registering-new-filters
engine.registerFilters({
    group_by: (input, key) => {
        const result = groupBy(input, key);
        // map dictionary back to arry
        return Object.keys(result).map(key1=>{
            return {
                key: key1,
                items: result[key1]
            };
        });
    }
});

Use this like so

{% assign grouped = data | group_by:'user' %}
{% for group in grouped %}
    <ul>{{group.key}}
        {% for item in group.items %}
        <li>{{item.name}}</li>
        {% endfor %}
    </ul>
{% endfor %}

Implementing a fast sort with Microsoft Flow using Parallel Compute

/

This is #FlowNinja hack 112. Parallel Compute Sort.

I had written about how to sort with a variable (this is insertion sort) back in 2018 How to implement sort with Microsoft Flow-in-3-actions-within-a-loop

But in this previous method, the use of variable means we can’t run apply to each in parallel, so this method was always slow when array is large. Today, while chatting with Hiro - I had a sudden idea to revisit the pattern and see if I can make this quicker.

Photo by&nbsp;Amy Shamblen&nbsp;on&nbsp;Unsplash

Photo by Amy Shamblen on Unsplash

The problem

To create a sort, using parallel apply to each, and get a sorted array at the end.



In 2018, I didn’t have many of the patterns I need to make this new 2020 sort method. Firstly, to get results from parallel apply to each, we need Pieter’s Method (Compose apply to each inner output) to fan-in after parallel fan-in.

Second, we need to sort the actual array, and I came up with a pretty interesting method.



How this works

Consider array [ “d”, “e”, “c”, “b”, “a” ]
If we say for each character, filter array for items that are < than the current item, we’d get:

3:d, 4:e, 2:c, 1:b, 0:a

Then, if we consider, hey, we have 5 items

[0, 1, 2, 3, 4] => map to this dictionary, we’d get [ “a”, “b”, “c”, “d”, “e” ]



Side Story

I actually was thinking about this pattern while driving home, once it clicked I had to pull over the side, park the car, take out my laptop and write this Flow, and after I saw it work I drove home.



Steps

Some additional considerations


If the original array has duplicates

[ “a”, “b”, “b”, “c” ]
0:a, 1:b, 3:c

We’ll see 2 is missing. This is not end of the world, but when we do the final map of

[0,1,2,3] => [ “a”, “b”, null, “c” ]

Observation

I was worried that JSON('{ "0": "a", "0":"a" }') would give an error, but it seems like the duplicate key is ignored. This could be an interesting way to detect duplicates in the future by building a dictionary.





Talks and Workshop at Difinity 2020 conference Auckland

/
henry-mcintosh-xbo-Hz_Gs74-unsplash.jpg

Hello, Kia Ora Auckland.

I’m glad to be returning to New Zealand on February 17-20 for the Difinity Conference in Auckland. I try to tell people that I’m actually 1/3 kiwi - I grew up in Auckland in my youth. So it’s always good to see you all again.

I’m presenting three topics:

February 17 Workshop: Master your inner Flow

This is an updated - all day workshop covering beginner and advanced Power Automate (Microsoft Flow). With several enhanced lab materials that you can choose to work through in the afternoon:

  • Complex Business Approvals

  • Power Apps integration

  • Data collection of Excel

  • Automating Teams

February 18 Talk: Take control of Flow and Power Apps Governance

Talk: Take control of Flow and PowerApps Governance

This is a multi-faceted look at Power Platform governance - what’s available within the Admin platform, extended with Microsoft’s Center of Excellence kit, practical guides on how to deploy CoE, and ultimately, technical ideas and examples of how we can extend CoE to provide an all encompassing governance solution.

Ultimately, there is a techincal part to governance, and there is a much much bigger human part. We will cover the technical, but leave you with ideas you must form regarding where and how you will utilize the technical components to guide your organization into the best use of this fantastic platform.

February 19 Talk: How to make everything with Microsoft Flow in 2020

This is a talk that makes you look at Power Automate (Microsoft Flow) with a totally different angle, and realize that it is not merely workflows, some email alerts, or an rule to copy a file.

Power Automate is the key to connect and enrich your entire platform investment - within Power Platform, extended to Dynamics, Office 365 or Data platform, and beyond even the Microsoft ecosystem.

You can make and automate everything with Power Automate (Microsoft Flow).


Make sure to say hello

I don’t travel a lot, so when I do visit New Zealand - come say hello, ask me anything about Flow (Automate), Power Apps, Power Governance, Office 365, Microsoft Graph, Azure. I have twenty years of consulting experience to back me up and I know solutions to many of your problems.

I hope to see you in Auckland on February 17-20 at the Difinity Conference.

Difinity Conference website

Difinity 2020 agenda



Flow - Format Number advanced tips and tricks

/

I put together a video to celebrate the new update in Microsoft Flow (Power Automate) and Logic Apps - formatNumber()

The video explains 3 tips and 2 gotchas.

  • Tip 1: Use action in prod right now

  • Tip 2: Use ### and 000 patterns in formatNumber() expressions

  • Gotcha 2.1: May be don't use of $### - use $##0

  • Gotcha 2.2: May be don't use $ - use C or C2

  • Tip 3: Use formatNumber in collections with Select/Create HTML

If you want the clipboard paste of the Format Number action to use, use this.

{
    "id": "c1fa8a84-e2c0-4be0-823e-3a45-1e6834aa",
    "brandColor": "#a098e6",
    "connectionReferences": {},
    "icon": "https://psuxaustralia.azureedge.net/Content/Images/DesignerOperations/numberfunctions.png",
    "isTrigger": false,
    "operationName": "Format_number",
    "operationDefinition": {
        "type": "Expression",
        "kind": "FormatNumber",
        "inputs": {
            "number": 12345,
            "format": "C4"
        },
        "runAfter": {
            "Compose": [
                "Succeeded"
            ]
        }
    }
}

Power Platform Governance with Power Clarity

/
Clear the FOG - let there be Clarity

Clear the FOG - let there be Clarity

Some asked me recently how does my SaaS product Power Clarity work with Microsoft’s Center of Excellence kit.

We can look at this from two directions:

  • Turnkey Governance with Power Clarity (running without CoE)

  • Enhanced Governance with Power Clarity (running with CoE)

  • ALM with Power Clarity

Center of Excellence kit (CoE)

I’m a big fan of CoE built by Microsoft and the community. I love that Microsoft acknowledges there is a need for governance, and the community rises to build tools to help. CoE starter kit is a collection of resources that will be a template to create your own Center of Excellence kit.

Power Clarity is a turnkey solution - you grant your credentials, it starts scanning. You get dashboards, forms and reports and workflows. I want to stress that Power Clarity is an enhanced version of Flow Studio - something I’ve been building since 2018, so it was developed in parallel to CoE, there are similarities, which leads to how they can be used together.


Here’s a list of roadmap on our first year (2020).

Turnkey Governance with Power Clarity (running without CoE)

If you are a company using the power platform and has not developed or deployed CoE - then Power Clarity can be used as a turnkey starting point.

If you don’t know where to start, or how to build one, we will get you started quickly to help you uncover the platform.

If you don’t have a power governance solution, and need one in 15 minutes, we are your… service.

  • No install - we manage it and keep improving it.
    Power Clarity is a SaaS service so the constant improvements are baked into the subscription. (In my opinion, we aren’t very expensive either).

  • Add one or many concurrent scanning accounts

  • Use accounts that are not administrator accounts - Power Clarity can scan with or without admin permissions, or operating in mix mode between different environments. If you have environment admin - you can see everything. If you only has environment maker - you can still monitor assets that you own.

  • Power Clarity performs hourly scanning, because it’s an incremental scanner and very light on API access. We understand at high API throughput - both source and storage can get upset. Power Clarity incremental scanning of a tenant of 1000 (flows/apps) is typically less than 3 minutes.

  • Power Clarity does not require premium accounts

Enhanced Governance with Power Clarity

If you have CoE deployed and already has CDS entities - then Power Clarity can be used to greatly enhance that data collection exercise.

You no longer need to worry about ‘how the data is collected’, but can keep customizing your reporting and utilization of that data to drive your governance outcomes.
In fact, we give you more events, signals, and policy actions you can perform. We want you to have clarity.

  • Writes to Azure Storage Table and CDS Entities (use Power Clarity to collect data instead of CoE’s admin sync flows)

    (Note: CDS write requires the scanning account to have Premium license for CDS access)

  • Deep API reads into Flows - runs, run actions, run errors and remediation

  • Recall run metadata beyond 30 days

  • When events happen - you can get an email report, or get a webhook call so you can handle it yourself. Power Clarity provides complete tenant flow error reporting without try/catch pattern. Since it’s inception, Power Clarity is a Flow monitoring service as well as a Power Platform governance service.

  • Advanced detection of cross tenant resources

  • Advanced detection of flows utilizing premium licenses

  • Advanced count of flows utilizing API rate limits

  • White and black list filtering of resources (domains, URLs) and connectors

  • Clash detection of flows and DLP policies

  • Clash detection of flows using deprecated actions

Application Lifecycle Management with Power Clarity

Because Power Clarity maintains offline granted access, it can perform advanced active actions to the Power Platform on your behalf, actions that don’t currently have an connector.

  • Flow Studio’s migrate feature (which lets us copy, switch connections and deploy flows), can be invoked from Clarity API call

  • Call Power Platform management cmdlets via Power Clarity API.

  • Continuous backup of flow definitions and Power Apps snapshots

  • Power Clarity provides an advanced GraphQL view of the entire data set available within its stores.

  • Read and reports on managed and unmanaged resources in Solutions

  • History of changes across the entity tracking

See it for yourself

Send me an email john.liu at flowstudio.app to get started.