On using Microsoft Flow as a pre-ETL step for Power BI

/
Photo by WeRoad on Unsplash

Photo by WeRoad on Unsplash

A topic I’ve presented a few times to an Power BI crew is the concept that Microsoft Flow makes a great pre-ETL step for Power BI.

I talked to many attendees at Difinity Power BI Conference Auckland about this. It’s probably a time to write this up to summarize my thoughts.

There’s just something nice about having a product called Flow gathering data into your lakes.

Power BI

Power BI - Power Query and DAX is extremely good at crunching numbers.

Flow has variables, branching and loop logic, but they are slow for number crunching. Loop limitations like 5k rows or total number of action steps are also very limiting.

Don’t use Flow to crunch numbers.

Flow

Power BI connects to about 70 sources - Flow connects to about 250 connections.

Flow’s HTTP action and Custom Connector framework is better at gathering data. Automatic paging, more authentication options, retry policy, async HTTP request.

Flow can gather data on Trigger, copy them all to a simple location (e.g. SharePoint library or Azure Blog Storage) for Power BI to start refresh. So data processing can be done in a secure way.

Flow can call Power BI API to start a refresh. Turning the ETL into a trigger-push system. We get the best of both worlds - Flow to gather data, and Power BI to crunch them.

Power BI Scheduled Refresh with Flow

  1. Schedule Refresh requires Flow to call Power BI API with a HTTP Request. But until very recently – this requires a delegate permission (so Flow has to call Power BI as a user) to perform delegate permission call requires a Custom Connector – which is what this blog post from Konstantinos goes into detail about creating.

    https://medium.com/@Konstantinos_Ioannou/refresh-powerbi-dataset-with-microsoft-flow-73836c727c33

  2. In February, Power BI has started releasing application permission to call Power BI API – this work is still in preview and requires quite a lot of set up, it also requires the administrator to approve the permission. App-only read/write everything is a pretty high level permission, non-admin can't grant this.

    https://powerbi.microsoft.com/en-us/blog/use-power-bi-api-with-service-principal-preview/

    pros: easier to call with Service Identity, and HTTP request
    cons: may be impossible to get your admin to approve a new AAD APP permission

  3. The April roadmap says Refresh is an action coming to Flow’s Power BI Connector – this will use delegate permissions and would be much simpler to use. So my thinking is that wait for this to drop after April

    https://docs.microsoft.com/en-us/business-applications-release-notes/April19/microsoft-flow/improved-power-bi-connector

#FlowNinja hack 78 - modifying Modified By and Modified time with Microsoft Flow

/
rodion-kutsaev-59544-unsplash.jpg

In this quick blog post - we talk about how we can tweak the “Modified By” user field and “Modified” datetime field in SharePoint list and document libraries using Microsoft Flow.

This technique follows the very detailed article by MVP Andrew Koltyakov on LinkedIn - he describes how to call it with C#, JSOM and REST. He also has detailed examples of every possible field and what format you’ll need to run this with.

I’m calling it with Flow. So to see the full explanation you’ll have to read Andrew’s article ;-)

Also, before you read any further - please repeat the title of the blog post loudly, really fast 3 times.

Plan

  • Requirements - why do we want this?

  • Call ValidateUpdateListItem method with Flow


Requirements - why do we want this?

  • You have updated a list item - you don’t really want the items to all say “Updated by System Account”.

  • You’ve copied a bunch of files but you don’t want it to all say “Modified (now)”.

  • You want to modify the item but you don’t want to create a new version.

  • You uploaded a file but you don’t want to have two versions when you tweak the metadata.

  • You wanted the update to tell you if the validation fails some SharePoint list item rule (this method will return a validation array object)

I need this. You also need this.

I don’t think we need to explain why you really really need this.


Flow

Here we have an item - modified a few seconds ago by me.

We want to update two values - the Modified By and the Modified (date time) fields.

Note the User is a JSON object that needs to be serialized by string() inside another JSON object.

The modified date is a date value, formatted to the “g” format. This format is usually MM/dd/yyyy HH:MM AA in EN locales. It is DIFFERENT if your environment has different locales. You can use “g” as date time string format to get the correct one for your locale.

Updates: 2020-06

Instead of “g”, we can also use 'yyyy-MM-dd hh:mm:ss' which seems to work for all locales. More reading on this issue.

Also, Mikael pointed out we should use bNewDocumentUpdate: true to avoid creating a new version.

Here is when it runs. Notice the Editor field is escaped JSON string.

Result

The modified date is now Feb 18, 2018!

And the user is someone else - my minion Gandalf.

Extra Date Time Note

Here is an example of using addDays(utcNow(), -365, ‘g’)
So this date is a year before right now, with the ‘g’ date formatting string.

https___pbs.twimg.com_media_Dzr4RL3U8AAAEa2.png

Difinity Conference 2019 Auckland - Hackathon Workshop, Flow and PowerApps

/

I’ll be presenting at Difinity Conference 2019 Auckland on several presentations.

Difinity is the largest Microsoft Data Platform conference in New Zealand. I’ll be presenting two talks - one on Microsoft Flow and one on PowerApps.

dfinity-conf.jpg

Pre-Conf Hackathon/Workshop: Master your inner Flow and PowerApps

This is a whole day workshop covering PowerApps and Microsoft Flow at a beginner to intermediate level, for the Difinity conference - the exercises are tuned for PowerBI integration.

http://difinity.co.nz/pre-conference-workshops-difinity-2019/#CustomConnector

Introduction to PowerApps and Power BI

This session is about two things – it is firstly a thorough introduction about PowerApps, where it came from and where is it going.

And it is also about how the sum is more than the individual parts – when we combine PowerApps and Power BI we can build some truly amazing more interactive reports & dashboards.

How to make everything with Microsoft Flow (advanced)

This is a intermediate-advance level Microsoft Flow session that looks at the very different types of Flow automations that we can do. Whether it is personal automation, enterprise workflows, or developer webservices. Flow is that flexible tool.

Chat!

Those that have met me will know I’m a very chatty person!

I’ll be hanging out at the community booths, so come find me, or tweet me and ask your Flow, PowerApps (or SharePoint) questions!

MS Ignite the Tour 2019 Sydney - MS Flow x2

/

I’ll be presenting at Microsoft Ignite the Tour 2019, Sydney on two Microsoft Flow presentations.

Microsoft Ignite The Tour - Community Breakout Social Image Template.png

Advancing the Flow - understand expressions in Microsoft Flow

Is a short 15 minute theatre session on expressions. In a short 15 minutes I’d like to cover why you might want to use expressions, and how it opens the entire Flow engine to your command.

Flow for Developers - insane low-code Serverless automation

This is a full hour breakout session where I wanted to talk about what we can do with Flow, at the intermediate to advanced end and the power it opens up to every platform it touches. Whether it is SharePoint Online, Dynamics CRM, PowerApps or Power BI, from Microsoft 365 to Dynamics 365 to Azure. We have 250+ connectors and it is completely bananas for developer productivity.

Chat!

Those that have met me will know I’m a very chatty person!

I’ll be hanging out at the community booths or at the SharePoint Gurus + Valo booth, so come find me, or tweet me and ask your Flow (or SharePoint) questions!

How to implement Sort with Microsoft Flow in 3 actions within a loop

/
Photo by Sophie Elvis on Unsplash

Photo by Sophie Elvis on Unsplash

For some reason, Flow (and LogicApps) doesn’t have a built in sort() method.

So after much nudging from Paul Culmsee, I brooded about this and sat down and built one. There’s three actions within a loop, packing quite a lot of tricks, this is how it all works.

Plan

How does it work conceptually?

Build your own Sort in Microsoft Flow

Simplify it to 3 actions within a loop

Extend it to sort objects and beyond.


Defining what we need


Let’s set up some input and outputs - we want to have our initial variable within the variable “initial-array” and we want to have our final sorted result within the variable “sorted-array”

We want to do a simple insertion sort - for the academic - this is an O(n^2) sort. Not the fastest, but simple enough to understand and express with logic expressions.

How does it work conceptually?

For each element in the initial-array, we will loop add it to a final ‘sorted-array’ in a sorted position.

In the first loop, the sorted array would be empty - but as we progress, for each new value we are considering - we would split the sorted-array into two sets of sub-arrays, ones that are smaller than the current value, and ones that are larger than the current value.

Finally, we combine everything back together in a union, set that to be the new “sorted-array” value, and proceed with the next element in the initial-array.

Build your own Sort in Microsoft Flow

So extending our Flow - add a loop over the initial-array.

Within the loop, we want to put the current item into a compose action. (going with the example above, this would be the element ‘4’ )

Add a compose at the end to see the final output of sorted-array.

Cutting sorted-array into lesser half, greater half

We do this in a parallel branch - these two don’t affect each other - add two Filter Array

Left, set the item() of the sorted-array to be <= the current value of the initial-array
Right, set the item() of the sorted-array to be > the current value of the initial-array

Union the arrays together. That is, the left-array, the right array, and in the middle, the current value, but put that inside an array of 1 element with createArray( <current-value> )

Before the apply to each ends, write the union result back into the “sorted-array” that concludes this loop, getting us ready for the next value.

See how it runs:

The result is a sorted-array.

See for example, loop 6, when the value is ‘4’

Clean up and simplify

We can remove some of the blocks to make the sort ‘smaller’

The final result

3 actions within one loop

Sorting complex (JSON) Objects

It is best to build an sort-array like this with numbers to understand how it works, before attempting to tweak it to sort complex object arrays.

Say for example - if we want to sort by the Date of a complex object array (our scenario was an RSS feed) - within the apply to each, we want to isolate the property we want to compare.

Because the current item’s property is used several times - we add the top compose back into the apply to each loop, we can consider the Compose is a kind of temporary variable used to hold the value that we are using to compare with.

We need to compare against the date in the two Filter-Arrays, but we don’t actually use it for the final Union - because we want to union the arrays together, not union the dates together.

And that’s Sort. Complex explanations, but very elegant with 3 actions inside a loop.