Beta testers requested for new "Flow Links" app

[Padawan]

Hello Everyone,

I have been working on a new App which I want to make available for free on the Appstore. The App will be called "Flow Links" and will have functionality very similar to the Portals app.

I've created an alternative for the Portals App because I had issues with it in the past and I've been planning to creating an alternative for a long time already. Now I've finally got the time to do so. I am aware that several people already switched from the Portals App to the Switch2Switch App. I do think my Flow Links app has benefits over the Switch2Switch App because it is specifically built to move jobs between flows on the same system, unlike the Switch2Switch App which was originally built to move jobs between Switch flows. It can also be used to move jobs within the same Switch instance but uses technology built for communication over different instances to do so. I've built "Flow Apps" to only work on the same Switch instance, which means that it has less settings compared to Switch2Switch and it is faster (for larger files).

I did some speed tests and these are the results
Moving a 42 KB file over one flow link/portal/switch2switch (average of 3 tests)
Portals (scanning interval set to 2 seconds): 1.8 seconds
Switch2Switch:1.4 seconds
Flow Links: 1.4 seconds

Moving a 10.6 MB file over one flow link/portal/switch2switch (average of 3 tests)
Portals (scanning interval set to 2 seconds): 2.2 seconds
Switch2Switch: 2 seconds
Flow Links: 1.4 seconds

Moving a 114.7 MB file over one flow link/portal/switch2switch (average of 3 tests)
Portals (scanning interval set to 2 seconds): 3.6 seconds
Switch2Switch: 2.6 seconds
Flow Links: 1.6 seconds

Moving a 435 MB file over one flow link/portal/switch2switch (average of 3 tests)
Portals (scanning interval set to 2 seconds): 6.7 seconds
Switch2Switch: 4.2 seconds
Flow Links: 1.6 seconds

Tests are done on a 2017 macbook Pro.

As you can see the time to move the files grows for Portals and Switch2Switch. The biggest effect is for Portals since that App zips and unzips each jobs which passes. The second biggest effect is for Switch2Switch since a job files goes from disk to a http request back to disk. For Flow Links the filesize has no effect since only a move operation is done (on the same disk).

You can also find the current documentation here:
https://drive.google.com/open?id=1rJJQz ... 8NEfXFsT0R

At the moment the App is in review by Enfocus. I am looking for volunteers to do a beta period afterwards. It would be really good to have people test the App in (a copy of) production flows. My stability tests didn't find any issues, but it would be nice if other people with other setups could confirm this.

If you would like to beta test, please send me your Enfocus ID by PM.

[essexmate]

I see the Switch2Switch app is now at version 6.0 as of today. The new feature is that “The channel name is now URL encoded and when the transfer is local, the job is copied and not transferred via http.”

[ThomasDeschamps]

Hello,
I switched all my portal app to flow-link app, and it work almost perfectly fine. it's clearly faster than the portal app.

Just one note, on some situation (that i did'nt had time to investigate yet) it seems that there is a lot if file "stuck" just before a flow-link send.
Some time it's dummy file (few Ko) who juste disapear if i stop the flow and reactivate it.
But rarely it's the real file, who is stuck, and we need to do the same thing : restart the flow.

Any idea?

[Padawan]

The dummy file is a technical necessity because I work with webhooks for messaging.

My testing showed that when the system is under stress the arrival of webhook events can be delayed or not happen at all. Therefore there is a retry mechanism in the Flow Links Apps. The app should retry to cleanup dummy files and retry to process the real files if the relevant webhook did not arrive the first time. By stopping and starting the flow you force such a retry, but it should not be necessary since the problem should resolve itself in time.

If the real file also appears to be stuck in the input folder:
- Firstly, make sure that the flow element before the Flow Links Send is not holding the file in use, which can stop the Flow Links App from moving it.
- Secondly, is there anything in the messages related to the job which is stuck?

[ThomasDeschamps]

Ok, so i think i'm under the "stress" option.
i can have hundred of small file within seconds.

The dummy file is a technical necessity because I work with webhooks for messaging.

My testing showed that when the system is under stress the arrival of webhook events can be delayed or not happen at all. Therefore there is a retry mechanism in the Flow Links Apps. The app should retry to cleanup dummy files and retry to process the real files if the relevant webhook did not arrive the first time. By stopping and starting the flow you force such a retry, but it should not be necessary since the problem should resolve itself in time.

At wich rate does the retry happend? is there a way to handle this? (via a property in a future version maybe?)
Because i have some "ungroup job" in a first flow, and an re-assemble job in the next one. In order to set the orphan time-out properly it's important for me.

If the real file also appears to be stuck in the input folder:
- Firstly, make sure that the flow element before the Flow Links Send is not holding the file in use, which can stop the Flow Links App from moving it.

Do you have an exemple for this? by construction it must be a folder, but I can test with two folder to be sure that the file is not hold.

- Secondly, is there anything in the messages related to the job which is stuck?

I have no exemple yet, but i'll try to catch one as soon as possible.

Thanks for your rapid answer

[Padawan]

At wich rate does the retry happend? is there a way to handle this? (via a property in a future version maybe?)
Because i have some "ungroup job" in a first flow, and an re-assemble job in the next one. In order to set the orphan time-out properly it's important for me.

The retry is a timerfired which runs every 60 seconds. This might seem slow, but when the issue happens the system is already under stress and frequent retries would increase the stress and hence possible worsen the issue. It is currently not possible to change this timeout using settings.

Each time Flow Links Send sends a webhook notification to the Receive there should be a message in the Switch messages to indicate this.

Do you have an exemple for this? by construction it must be a folder, but I can test with two folder to be sure that the file is not hold.

If the file is being in use by another process, then there should be Switch messages indicating this.

Switch messages of the issue should help

[ThomasDeschamps]

Ok, now that i have a better understanding of how it work it seem's more clear.

I have an exemple right now :

One file submitted trough flow link send at 9:56 this morning, wich worked perfectly fine, but the original file is still in the entry point at 14:34.
No special message on the log with the job name, nor the element/flow name just :
"Notification sent to Receive Flow Link."
If i stop the flow, and relaunch it disapear as intended.

I gathered the data and achieved to get this graph:

X is time, and y is number of flowlink send grouped by minute with peak at 250/minute.
Is this considered "under stress"?
If yes, i'll try to add "hold job" in order to limit the max number of file at a given moment.

[Padawan]

What is considered "under stress" will be different for each system. However, I think the hold job has a big chance to help.

Which Switch version are you using? I had a lot less issues with 19 fall. ( I didn't do stress tests with 20 spring yet)

If you have the issue in Switch 19 fall or higher, then I'll have to report the underlying webhooks issue to Enfocus. Would you be willing to test a small reproducible sample in that case?

[ThomasDeschamps]

What is considered "under stress" will be different for each system. However, I think the hold job has a big chance to help.

Which Switch version are you using? I had a lot less issues with 19 fall. ( I didn't do stress tests with 20 spring yet)

If you have the issue in Switch 19 fall or higher, then I'll have to report the underlying webhooks issue to Enfocus. Would you be willing to test a small reproducible sample in that case?

I'm actually in 19 fall on windows 10, and will be testing in 2020 in few day/weeks.

Do you want a specific methodology for the test (sample file, number of file, frequency etc..)?

[Padawan]

Can you try the following when the issue happens again?

1) Check if it is not a refresh issue in Switch Designer.
When you have a job before the Flow Links Send element which is not moving, right click the folder element while holding the shift key and choose "Open inExplorer". Do you also see the file in Explorer?

If not, then it is a refresh issue in Switch Designer and not an issue in the Flow Links Apps.
If it is, continue with the next steps

2) Send me the Switch messages from the moment the job arrived to an hour afterwards.
Please also let me know the name of the job and the name of the flow links send element so I can find the actual job in the messages.

This way I should be able to check the behavior of the retry mechanism.

You can send it to luno.tools@gmail.com

I'm building a test flow with a script which will make it easier to do stress tests for this. If you confirm it is not a refresh issue in Switch Designer I'll finish it and provide it to you.

[ThomasDeschamps]

Can you try the following when the issue happens again?

1) Check if it is not a refresh issue in Switch Designer.
When you have a job before the Flow Links Send element which is not moving, right click the folder element while holding the shift key and choose "Open inExplorer". Do you also see the file in Explorer?

If not, then it is a refresh issue in Switch Designer and not an issue in the Flow Links Apps.
If it is, continue with the next steps

2) Send me the Switch messages from the moment the job arrived to an hour afterwards.
Please also let me know the name of the job and the name of the flow links send element so I can find the actual job in the messages.

This way I should be able to check the behavior of the retry mechanism.

You can send it to luno.tools@gmail.com

I'm building a test flow with a script which will make it easier to do stress tests for this. If you confirm it is not a refresh issue in Switch Designer I'll finish it and provide it to you.

It's not a refresh issue, the file are still in the explorer
As soon as i have logs i will send them via mail

[ThomasDeschamps]

Little update, but with the recent covid 19 and hollidays begining our flows is'nt stressed as usually and i didn't noticed the bug since.

I'll keep you updated in september

[Padawan]

No problem, no stress (in flows and in life) is always a good thing

[PdFUser5000]

Sample flows not available in appstore, although the documentation says they are available?

[Padawan]

That is fixed now