Versioning scripts & Builds

Post Reply
stargazerllc
Newbie
Posts: 12
Joined: Mon Jan 25, 2021 8:24 pm

Versioning scripts & Builds

Post by stargazerllc »

I understand the concept of script folders vs built/packaged .sscript files.

What I don't get is the development process they want people to follow. Let's say I have a project that holds all of the script folders that I use in a project. This is fairly large 200+ scripts.

1. Each has its own development folder.
2. When developing you point to the .sscript file in the folder.
3. when done you build it and tell the flow to look at the newly built script? This seems clunky to have to go back and repoint the script in the flow to the newest built script.
* Do you build it in the same as the script folder? Then you don't have to update where it's pointed? seems like an anti-pattern.
4. how do you go about automating this and removing human error?
patej
Member
Posts: 79
Joined: Sun Nov 25, 2012 12:15 pm

Re: Versioning scripts & Builds

Post by patej »

Hi,

Good question. We have ended up working in the following way, for now at least:
  • each script has its own directory (i.e. script folder) and then we have a separate build directory for the packaged .sscript files. Building into the script folder doesn't make sense because you lose the "magical 0 KB .sscript" the script folder idea relies upon; probably someone from Enfocus can give a more scientific answer... (I actually tested just now and SwitchScriptTool does pack into the same directory if you ask it to, which is interesting...)
  • all scripts are version controlled in a single git repository with develop and final branches. To keep things clear, we merge into "final" branch only those scripts that are ready to use in production.
  • our Switch servers have two separate directories for scripts: one cloning "develop" branch and another one cloning "final" branch
––> this way we can e.g. use two script elements in the same flow and we can route test files to the test script (which is using script folder and, therefore, often does not need a flow reactivation).
––> In the future, we are probably going to move into using only script folders and switch only between using scripts from the directory cloning "final" branch, but because currently packaged scripts are faster to execute, we have to point the element first to the .sscript in the script folder and then in production to the packaged .sscript. The paths to local repositories are the same on all servers so it is not a problem for us that script folders aren't packaged into flow exports.

So our current process goes like this:
  1. develop script either on your own computer or on a Switch server ––> push changes to git "develop" branch
  2. pull from git to Switch development installation followed by "npm i" if needed and point flow element to script folder script-name/script-name.sscript
  3. if more lengthy tests than the 4 h maximum the development license allows are needed, "git pull" changes on a production installation and point element to the develop repository/script-name/script-name.sscript
  4. when testing finished, --pack to /build/script-name.sscript and merge it to "final" branch
  5. git pull "final" branch to production installation(s) and point flow elements to /build/script-name.sscript
  6. ––> on subsequent updates, only a git pull followed by a flow reactivation is required
––> points 4.–6. will naturally be more straight-forward when we can use script folders in production
––> with regards to automation, in my opinion it is important to have a manual step in the end to triple check everything before deploying a script update because unfortunately Switch doesn't yet provide tools to e.g. load balance and sync flows between several Switch installations or create automated backups of individual flows to provide quick recovery in case of e.g. a failed deployment. Hopefully those will be provided later on, now that development has more time on their hands and we can move into more automated deployment, too.

Here is a quick flow chart about the above:
Image

In a more simple environment with only one Switch installation, you can just switch between using the script folder and the packaged script or even just continue using the script folder, depending on the case. In any case, and especially if using script folders, it is important to plan the process so that it is very clear which script is used in production and which can be developed further without affecting production. Have you noticed these points Enfocus mentions regarding deployment: https://www.enfocus.com/manuals/Develop ... cript.html?

Looking forward to your thoughts and hopefully others share their processes, too!

--patrik
stargazerllc
Newbie
Posts: 12
Joined: Mon Jan 25, 2021 8:24 pm

Re: Versioning scripts & Builds

Post by stargazerllc »

Patrik,

Thanks for the great write up. I think this is similar to what I was thinking was going to have to be done but I really hoped I was missing something.

"unfortunately Switch doesn't yet provide tools to e.g. load balance and sync flows between several Switch installations"
It seems to me like you could use version control on the flows folder but I think there are some outside dependencies that need to be worked in with it as well?

What I would like to be able to do is set up more like a CI/CD where: |dev-box| -> clones to -> |test-box| -> clones to -> |prod-box|
I could see the testing/packing being done on the test-box. The more I understand about Switch is that this is probably not going to be possible in its current state.

The information I am getting (second hand) is that you have to pay for a dev license and a production license for switch? is that the case across the board? I am just getting involved with this project and it seems odd to me that they don't automatically provide a dev env with the production version.
patej
Member
Posts: 79
Joined: Sun Nov 25, 2012 12:15 pm

Re: Versioning scripts & Builds

Post by patej »

stargazerllc wrote: Tue Feb 02, 2021 3:38 pm "unfortunately Switch doesn't yet provide tools to e.g. load balance and sync flows between several Switch installations"
It seems to me like you could use version control on the flows folder but I think there are some outside dependencies that need to be worked in with it as well?
I wouldn't dare to try that... ;) First of all, the flows are named numerically flow_1, flow_2, ... so the same flows imported to different servers get different file names and only the Switch Server\settings\flowsPaneLayout.xml connects the dots. Same goes with Property Sets, which are stored in numeric folders e.g. Switch Server\PropertySets\3 and importing to Switch reserves the next available number. So it is quite complicated to get it to work manually... We've made some changes to flows in batch by searching and replacing e.g. network paths in the xml's and that has worked, but it requires a full restart of the server to register the changes. However, if you really don't make any changes to the flows in your prod box, maybe you could just clone the whole thing, but I don't promise anything and getting Switch up and running after that is another thing... The Application Data Root Tool Enfocus provides with Switch is their approach to backup/restore, but that is only for a whole installation and in my experience is useless in big installations; at least in our installation it didn't even finish creating the backup over night. Also the flows' update process Switch requires after upgrade take hours so we try to avoid such things as much as possible and is not a feasible way to handle HA/DR...
stargazerllc wrote: Tue Feb 02, 2021 3:38 pm What I would like to be able to do is set up more like a CI/CD where: |dev-box| -> clones to -> |test-box| -> clones to -> |prod-box|
I could see the testing/packing being done on the test-box. The more I understand about Switch is that this is probably not going to be possible in its current state.
In my opinion you can't compare a Switch setup to a software development project because, at least in most cases, within one Switch installation most flows aren't dependent on each other (some of course might) so you should be able to make "flow-specific releases", which currently cannot be automated.. Coupled with the problem of slow restore I described above. However, your description of your use case "I have a project that holds all of the script folders that I use in a project. This is fairly large 200+ scripts." is quite different than a normal Switch use case... I'd be interested to hear more if you can elaborate a bit what your project is about..? :)
stargazerllc wrote: Tue Feb 02, 2021 3:38 pm The information I am getting (second hand) is that you have to pay for a dev license and a production license for switch? is that the case across the board? I am just getting involved with this project and it seems odd to me that they don't automatically provide a dev env with the production version.
Yes that's true. Same goes with e.g. PitStop Server if you need that in your flows. Also if you need a paid license from the Enfocus App Store, you have to buy it also for dev server, unless the 30 day trial is enough. Not ideal, but it seems to be quite common within the graphic industry, I don't know why...
Padawan
Advanced member
Posts: 358
Joined: Mon Jun 12, 2017 8:48 pm
Location: Belgium
Contact:

Re: Versioning scripts & Builds

Post by Padawan »

Is it necessary to test all your scripts in Switch?

From your explanation I suspect that you are the main maintainer of a Switch server and you continuously develop further on your sets of scripts. Personally, I work for an Enfocus reseller. Therefore I write scripts which are used in different Switch installations on very different use-cases. Once they are in production at the customer they don't change often anymore. As a result of that I don't have such a big need for a continuous automated process. (I also came from a graphic arts background and grew into development, so my background might be different).

I personally first write and test my main code in VS Code, using the available frameworks for that. By example:

I recently wrote a Switch script which needed to transform an xml into another xml format. In my main.ts I wrote a main function which accepts the input and returns the output path of the translated xml. Then I have another typescript file with a big bunch of test cases which I use to develop the main function. I do all tests in VS Code. When the main function is fully functional I add jobarrived in main.ts which sends the path of the incoming job to the main function and sends the path of the translated xml to the outgoing connection.

I recently also had to write a set of scripts where one downloads from a DAM system and another uploads to a DAM system. These have script logic in common, so I could not write a single main function for both if I didn't want to have some duplicate logic in both of them. So I wrote a typescript module to handle all communication to the DAM system. I tested the typescript module completely in VS Code and when that was done I installed the module in the project folders of the Switch dam upload and the Switch dam download script.


I think the gist is that Switch is very much rooted in the graphic arts industry and as a result of that it hasn't the toolchain that you might expect if you are used to some well known development stacks. That's why I test as much of my scripts as possible in VS Code, there are lots of development tools available to do that since lots of other node.js developers work like this. Is this something which could help your way of working?


Also, I think that Switch has been evolving more and more the direction of a development tool in recent years. We've received node.js support, typescript support, Web services API, Remote Processing API, ... We are currently able to activate and deactivate flows from an API, hopefully we will be able to build one from an API in the future.
stargazerllc
Newbie
Posts: 12
Joined: Mon Jan 25, 2021 8:24 pm

Re: Versioning scripts & Builds

Post by stargazerllc »

I apologize for the late reply but I have been testing some devFlow stuff and trying to understand switch more.

So to put it in perspective I'm working with a print house that has a large set of workflows and multiple maintainers. This leads to inconsistencies in many different areas. I am working on cleaning up some of those inconsistencies.

> Is it necessary to test all your scripts in Switch?
Ideally yes, is it required probably not. Without some kind of automated End to end testing how do I know if a change I made somewhere else didn't have an effect downstream? From my experience in how switch bundles its packages/script folders I'm not sure this is as much of an issue but might be.

>I wouldn't dare to try that... ;)
After looking into things more I think that I am going to try to avoid any versioning on the xml stuff. But I am going to use git to control my script package folders and the common utilities they share. They will all get built and then someone will have to shuttle them off to production. I'm still a little foggy on how I am going to accomplish the deploy process but I think I have the dev side down.

I have this folder structure

workflows (main .git)
- builtFlows
- .build (builds from the script packages folder)
- packages
- - mySwitchScriptFolder (that the switch script creator tool builds)
-- utilities (a package with a collection of shared utilities)

I figure that the dev-> PR process would be
- build each changed script folder and dump it into the .build folder
- dev points workflow to the built script file
- dev exports flow to builtFlows folder

* what I am not sure of yet.
after PR the testing machine can import all the modified flows (seems like all takes a while so maybe not worth it) * might use the PR/Git to generate a report of the changed functions. be nice to do it programatically.
the testing machine can run E2E tests
when they clear we can use beyond compare to "clone" to the production machines.
bsbertolani
Newbie
Posts: 4
Joined: Thu May 19, 2022 10:08 am
Location: Germany
Contact:

Re: Versioning scripts & Builds

Post by bsbertolani »

Hey guys, nice discussion. Quick question have you made any upgrades since this last talk?
patej
Member
Posts: 79
Joined: Sun Nov 25, 2012 12:15 pm

Re: Versioning scripts & Builds

Post by patej »

patej wrote: Tue Feb 02, 2021 2:25 pm
Since my last comment above, we've made a couple of changes:
  • after Switch upgrades, the performance of script folders is mostly on par with packaged scripts so we are using script folders exclusively to be able to update scripts without restarting flows (naturally only if there are no changes to .sscript)
  • we use VS Code remote-SSH plugin which makes it possible to develop directly on our switch (dev) server in its develop branch so we can do the testing without going through github with every change. Only after we're ready for a commit and even release, we go through github to pull changes to main branch. Only issue is that VS Code on mac cannot relay ssh keys to a Windows server so github interaction has to be done on the Switch server itself, but that's a minor drawback compared to the improvement in the overall process.
Here is an updated chart:
Image

A note/warning, however: one major issue we're having, and what Enfocus has hard time verifying is how Switch behaves when a script's properties are updated and the script has to be updated in several flows. We have scripts that we use in dozens, sometimes in over 100 flows in all our Switches. After pulling the changes, even restarting Switch Server isn't enough to make Switch update all instances of the script across the whole server. We have tried both restarting the server and using the Pause&Start processing, but only reliable way is to one-by-one deactivate and activate the flows. Even trying to activate a flow group at once (and Switch informs that several scripts will be updated), sometimes doesn't update all scripts in all of the flows in the group; the flows are all activated just fine, but only when a job goes through the (not)updated script, we see that it hasn't actually been activated. When such a flow is deactivated and activated, Switch once again informs that there are scripts that need updating. This is quite frustrating and good to be careful if you have scripts in several flows. We would be interested in hearing if anyone else has experienced similar problems.

--patrik
Post Reply