Problems with HTTP Request

[Soul Forge]

Hello!

So, I sent an e-mail for the support team already, but decided to post here either way in case someone is having the same problem.

Since the first 2024 update I've been having problems with Node's Socket Hangup error. It's said that the most recent update solves this issue, I still get it sometimes, but at least it's not multiple times a day like it was before.

Now my issue is the HTTP Request app, that under heavy loads is saying the "theUrl" parameter for some files has a problem and stops every other file process on that app. What's weird is that, if I stop then start the flow again, the same files that were stuck can go through as if nothing happened, even the file that "started" the problem.



Here's a link to a video of me stopping and strating the flow:
https://imgur.com/9p8vNPy

[sander]

I am facing the same issue with the same error message. Both at 2023 Fall and 2024 Spring.

Now my issue is the HTTP Request app, that under heavy loads is saying the "theUrl" parameter for some files has a problem and stops every other file process on that app. What's weird is that, if I stop then start the flow again, the same files that were stuck can go through as if nothing happened, even the file that "started" the problem.

100% the same issue. It indeed only happens on heavy load.

Edit:

jobpath.png (127.84 KiB) Viewed 12739 times

Do you also face this problem? 'Cannot upload job_ provided path does not exist' also happens now and then, under heavy load too. Stop/start flow and it will run without the same error again.

Also, talking about heavy load: the HTTP request will download 1500-2000 PDF's within a couple of minutes. What is your load?

[Soul Forge]

Also, talking about heavy load: the HTTP request will download 1500-2000 PDF's within a couple of minutes. What is your load?

Sure, but the problem starts with only a couple hundred files being downloaded. The problem with this is that I manually need to stop and start the flow again, but I'm not always available to do that. When I'm not available the factory simply stops working thanks to those Switch errors.

My bosses want to scale the company, so thousands of files in a couple minutes will be a reality. My fear is that Switch won't be able to handle it and then we will have to look for alternatives.

Do you also face this problem? 'Cannot upload job_ provided path does not exist' also happens now and then, under heavy load too. Stop/start flow and it will run without the same error again.

I noticed that in the 2024 updates Switch stopped waiting for the files to be transferred between one folder to another.
I had to slow down hotfolders I had created for Switch, otherwise it would grab the folder the operators were still copying to the hotfolder and then this path problem would happen.
Tried to change hotfolder for SubmitPoints, but then browsers won't accept transfering folders with a couple gigabytes within them, for some reason.

The same happened to every HTTP Request I have. Had to add hold jobs with 4 secs between each job, so these erros would happen less often.
Now everything works a lot slower, since it has to wait for the downloads to happen.

[freddyp]

How many concurrent processing channels do you have? If you have 20, or even more, channels all busy doing HTTP downloads I can imagine this may have side effects. And it is not efficient anyhow if you are eating up all your bandwidth, because you have more concurrent downloads but they take longer. You are then better off doing less at the same time, but quicker. In the end it is the throughput that counts.

I suggest to do the following: place a Hold job in front of the input folder to HTTP request and on the outgoing connection set "Folder constraint" to "Yes" and the maximum number of jobs to a number that gives the best throughput. Some experimenting will be necessary.

[Soul Forge]

How many concurrent processing channels do you have? If you have 20, or even more, channels all busy doing HTTP downloads I can imagine this may have side effects. And it is not efficient anyhow if you are eating up all your bandwidth, because you have more concurrent downloads but they take longer. You are then better off doing less at the same time, but quicker. In the end it is the throughput that counts.

I suggest to do the following: place a Hold job in front of the input folder to HTTP request and on the outgoing connection set "Folder constraint" to "Yes" and the maximum number of jobs to a number that gives the best throughput. Some experimenting will be necessary.

I already did that, it makes the issue better but don't solve it.
We receive a couple hundred files every hour at exactly xx:00. Without that fix every hour I had to restart the flow. After I added we get that problem 2 or 3 times a day, usually when I'm not on my shift.

But to get to that I only allow 2 files with a delay of 5 secs between each. This delay keeps adding up the more we sell. We wouldn't be able to scale if it's gotten to thousands of files every hour.

I've been thinking on alternatives to make it better, but it involves us creating our own solution, which defeats the purpose of using Switch in the first place.

[tdeschampsBluewest]

One other fix to consider could be to set the flow in advanced mode, and limit the number of concurrent task within the element itself.

If the problem is tied to a single element handling multiple download at once, and you still want to use multiple thread, it's worth the try to add a hold job beforehand making a dispatch to X HTTP request elements, + the max number of concurrency task for those elements.

Alternatively, since i think you have the Scripting module, you could developp an alternative with your own tool, focusing on the essential for your needs? With axios this should not be a big deal.

[Soul Forge]

So, I received a response from the development team. The problem I was having started when I started using nested variables.

Code: Select all

[Job.PrivateData:Key="wtpCorrigir[Job.PrivateData:Key="jobTipo"]"]

But in a user perspective, it didn't appear like it would cause some trouble, since Switch lets us nest variables in the first place. I also use nested variables in other apps, HTTP request is the only one giving me trouble.

So, for anyone having the same problems, maybe this information could help.