Flow Auto Backup

Post Reply
User avatar
gabrielp
Advanced member
Posts: 645
Joined: Fri Aug 08, 2014 4:31 pm
Location: Boston
Contact:

Flow Auto Backup

Post by gabrielp » Thu Jun 23, 2016 7:39 pm

Introduction
Continuing with my experiments relating to injecting Switch backing files, I now have been able to automatically backup flows. I have created a flow group which automatically backs up every flow on a Switch system, creating .sflow files which can be easily restored. By default, the flow is triggered every hour and archives the resulted flows to a network location, but since I used the Portal Callback pattern, you can change that without compromising the core functions.

Usage
Image

Image

You do not need to modify any of the 'FAB Functions' flows. The flow 'Auto Backup Program' is yours to mess around with.

You control the frequency of backups by means of the Dummy Job Clock which by default is set to 1 hour (3600 seconds).

You then are required to use the Flow Auto Backup (FAB) dataset which consists of a few Private Data keys:
  • FAB Portal Callback - The Portal channel where the packed flows will be delivered once the core functions are complete. You do not have to change this.
  • FAB Flows Path - The absolute path to your Switch Server/flows directory
  • FAB Manifest Path - The absolute path to your generic manifest.xml file. See below for more information.
Out of the Callback, .sflows will start flowing in. Each of these .sflows is ready for use and contains the following new FAB dataset fields:
  • FAB Flow Name - The name of the flow
  • FAB Flow Version - The current version of the flow
  • FAB Flow ID - The numerical ID of the flow
If any of your flows are password protected, these values will say 'Encrypted'.

By default, the 'Auto Backup Program' flow archives the flows into folders for each flow name. The version number is part of the .sflow name, so it will overwrite the same version, but if it's a new version, it will save a new copy in that folder.

Here is the result (Yes, this flow backs up itself. How meta ;) ):
Image

And encrypted flows all go together in the same folder:
Image

Manifest Setup
To reverse-engineer the .sflow filetype and reproduce its behavior within Switch, we need to introduce a simple XML file called a manifest. Luckily, Switch doesn't require there to be very much information in it, so we can simply use a static file. The best way to handle this is to export any flow, rename the .sflow to .zip, unzip it, and copy the manifest to a static location and use that absolute URL as the FAB Manifest Path.

For reference, here are the contents of my manifest.xml:

Code: Select all

<Manifest>
    <ProductInfo>Switch Version 13</ProductInfo>
    <ExportFormatVersion>1.0</ExportFormatVersion>
    <FlowFile>flow.xml</FlowFile>
    <SwitchFlavour>Switch</SwitchFlavour>
    <SwitchReleaseVersionNumber>13</SwitchReleaseVersionNumber>
    <SwitchReleaseType></SwitchReleaseType>
    <SwitchReleaseTypeNumber>0</SwitchReleaseTypeNumber>
    <SwitchUpdateVersionNumber>100</SwitchUpdateVersionNumber>
    <OperatingSystem>Mac OS X</OperatingSystem>
</Manifest>
Detailed Explanation
So now that you know how to use it, here is how it works. Everything is triggered from the Auto Backup Program flow. This flow is invoking FAB Get Flow Function via the Portal Callback pattern. The advantage of this is that you don't have to modify any of the core FAB flows which means if I ever update the FAB Functions, you can simply swap them out and everything will keep working. Said another way, the core flows are acting as a "black box": you don't know and don't need to know what's going on inside because it's abstracted away from you.

Within the FAB Get Flow Function, the triggering job is immediately asserted to ensure the required FAB dataset is set. If so, it then uses those values to inject a copy of your flows folder (being sure not to delete it after injection). The flows folder is split into individual files, where it then injects a copy of the manifest and the flow XML itself is consumed as a dataset. The flow and manifest are assembled into one, and then it writes the remaining fields of the FAB dataset before handing the job off to FAB Pack Flow Function.

Within FAB Pack Flow Function, the job goes through another assertion to verify the callback is set. The job is ungrouped, then renamed to match the .sflow file structure (flow.xml, manifest.xml), and then merged again. The job folder is now archived, but it can't sit within a top-level folder as the Archive configurator would do, so I built a script to archive the files at the top level called Flat Archive. A .zip comes out, which has its suffix replaced with .sflow and then sent to the callback.

Back in the Auto Backup Program flow, the .sflows stream out and the user can do what they want with them. In the default case, they get backed up onto the network separated by folders for each flow name.

What was used
Private data script: https://github.com/open-automation/Swit ... rivateData
Dummy job clock script: https://github.com/open-automation/DummyJobClock
Portals: https://github.com/open-automation/SwitchPortals
Variable Assert: https://github.com/open-automation/SwitchVariableAssert
Flat Archive: https://github.com/open-automation/switch-flat-archive

Download
All Flows: https://drive.google.com/file/d/0B9ciRz ... sp=sharing
FAB Get Flow Function Documentation: https://www.googledrive.com/host/0B9ciR ... mZRdExyekE
FAB Pack Function Documentation: https://www.googledrive.com/host/0B9ciR ... 3o5S0ZLUGc
Auto Backup Program Documentation: https://www.googledrive.com/host/0B9ciR ... jFwQ2YzWnc

Update 9/16/16
This has been updated. Download and new documentation can be found at this repo:
https://github.com/open-automation/swit ... uto-backup
Last edited by gabrielp on Fri Sep 16, 2016 5:23 pm, edited 2 times in total.
Free Switch scripts: open-automation @ GitHub
Free Switch apps: open-automation @ Enfocus appstore

If I've helped you and you'd like to say thanks, consider adding me on LinkedIn and leaving me an endorsement or recommendation. It helps a lot!

lombert
Member
Posts: 167
Joined: Fri Feb 04, 2011 2:31 pm
Location: Sweden

Re: Flow Auto Backup

Post by lombert » Thu Jun 23, 2016 9:22 pm

I'll be the first to say wow, and thanks to you to share it. I'll defently try it..

I just download the portal script today, and that is also great idéa..
Between jobs!

User avatar
gabrielp
Advanced member
Posts: 645
Joined: Fri Aug 08, 2014 4:31 pm
Location: Boston
Contact:

Re: Flow Auto Backup

Post by gabrielp » Tue Jun 28, 2016 1:15 am

Updated the flow group to utilize Portals v2 (with the Program scope).
Free Switch scripts: open-automation @ GitHub
Free Switch apps: open-automation @ Enfocus appstore

If I've helped you and you'd like to say thanks, consider adding me on LinkedIn and leaving me an endorsement or recommendation. It helps a lot!

User avatar
gabrielp
Advanced member
Posts: 645
Joined: Fri Aug 08, 2014 4:31 pm
Location: Boston
Contact:

Re: Flow Auto Backup

Post by gabrielp » Tue Jul 05, 2016 11:56 pm

I highly recommend everyone do this. It's saved me twice already.
Free Switch scripts: open-automation @ GitHub
Free Switch apps: open-automation @ Enfocus appstore

If I've helped you and you'd like to say thanks, consider adding me on LinkedIn and leaving me an endorsement or recommendation. It helps a lot!

colins
Newbie
Posts: 5
Joined: Tue Jul 05, 2016 8:52 pm

Re: Flow Auto Backup

Post by colins » Fri Sep 16, 2016 5:18 pm

We updated the flow to utilize Inject lite, Portals v3, Assert Not Blank, Private Data Write 25, and Flow Group Discovery scripts as well as added two new features. The first is a better way to structure the backup flows based on the group hierarchy. The second is a failure callback feature that passes all failure private data as well as the failed files out of the FAB functions.

We have setup a repository on Github for this as well: https://github.com/open-automation/swit ... uto-backup

Dave23
Member
Posts: 37
Joined: Thu Oct 12, 2017 4:42 pm

Re: Flow Auto Backup

Post by Dave23 » Thu Jun 27, 2019 1:39 pm

Hi,

I've been trying to get this Auto Backup Flow to work however I have hit a bit of a snag. I am getting a script error on the Get Flow Function portal in the Auto Backup Program flow. Have you seen this error before?

AutoBackupProgramError.PNG
AutoBackupProgramError.PNG (31.99 KiB) Viewed 2040 times

Thanks

Dave

User avatar
gabrielp
Advanced member
Posts: 645
Joined: Fri Aug 08, 2014 4:31 pm
Location: Boston
Contact:

Re: Flow Auto Backup

Post by gabrielp » Fri Jun 28, 2019 3:12 pm

Hey Dave,

Try remaking the portals with the latest version of the scripts or the Portals app from the app store. It may be a bit tedious but if you bring in new configurators and configure them manually you can be sure that nothing is corrupt. I bet one problem is that the versions of the Portals script in the example flows are very old.

You can grab the latest scripts from here https://github.com/open-automation/switch-portals and the app version from the appstore here https://www.enfocus.com/en/appstore/product/portals

Let me know if that works out.
Free Switch scripts: open-automation @ GitHub
Free Switch apps: open-automation @ Enfocus appstore

If I've helped you and you'd like to say thanks, consider adding me on LinkedIn and leaving me an endorsement or recommendation. It helps a lot!

EDO
Newbie
Posts: 1
Joined: Wed Aug 14, 2019 11:03 am

Re: Flow Auto Backup

Post by EDO » Wed Aug 14, 2019 11:30 am

Thank you for this Flow Auto Backup, I added it to my Flows.

Also I expanded it a bit. Thought it could be useful to backup the ActionLists and Variable Sets as well.

Just route the Trigger to another flow. Use the tool to pick up jobs for example C:/Users/Druckserver/AppData/Roaming/Enfocus Prefs Folder/Variable Sets give it a hierarchy and put it to your Archiv.

Image
Attachments
Flow.png
Flow.png (12.68 KiB) Viewed 1531 times

Post Reply