John Liu .NET

View Original

Bulk-copy files across site collection in MicrosoftGraph with MicrosoftFlow, in parallel and in batch

John Liu

MicrosoftGraph represents SharePoint Document Libraries as "Drives" and Folders and Files as "DriveItems".  It has ability to copy these objects.

I did several tests with MicrosoftGraph and MicrosoftFlow, here are the notes.

  • Copying Files with MicrosoftFlow parallel for-each
  • Copying Files with MicrosoftGraph's $batch
  • Copying Folders when copying with MicrosoftGraph

 

Copying Files with Flow parallel for-each

  • ForEach is running in parallel mode 
  • There are 23 items in the library
  • Because Flow's HTTP Action automatically follows 202 redirect location header - it will check when the MSGraph copy is completed (this is out of box default behaviour).
  • 23 files are copied in 12 seconds
  • No bytes were downloaded to Flow during the copy
  • HTTP Action's retry mechanism would deal with errors or request throttling

 

Copying Files with MicrosoftGraph's $batch

  • MicrosoftGraph's $batch handles 20 request so to handle 23 items, we still have to use foreach and execute two batch requests.
  • Building $batch request needs a Flow for-each loop with incremental index (alas, in Flow expression we have item() for the current item but no index() for the current index).
  • $batch executes quickly, and returns individual responses (including any success, redirect or errors).  BUT it doesn't follow 202 location header so it isn't guaranteed if the copy has finished.  If we want proper fan-in, we will need to track it manually with a second for-each loop querying the response headers.
  • The files are still copied successfully.  So if you don't need a fan-in scenario this may not be a problem.

 

Copying Folders via MicrosoftGraph

  • In MicrosoftGraph, a folder is a DriveItem
  • So copying folder structures is native, and nested
  • We NEVER had this in SharePoint
  • The 202 redirect took longer as it has to copy child items

 

 

References