EMFILE: too many open files ..main.js

[magnussandstrom]

Hi, after updating a few default apps last week I've had a recurring issue that random flow elements crashes (stops with the red gear-icon) and I get the message below. I have reverted most of the apps to older versions and restarted our server but the problem comes back.

WARNING:
EMFILE: too many open files, open 'C:/***/**/main.js'

Any suggestion how to fix this?

I'm thinking of restoring my whole VM from an image backup if I can't find a solution.

I'm running Switch 2023 fall on Windows Server 2022.

[magnussandstrom]

And for anyone at Enfocus reading this: Could you please add time stamps and mandatory release notes for the different app-versions in the Appstore?

[r.zegwaard]

Hi Magnus,

I've had these errors also. These were caused by scripts that created tmp-files that the script did not properly clean up.
Rebooting the whole system helped me get Switch running again, but after a while, the errors reappeared.
So check you're script(s) for unclosed (tmp) files.

Another solution might be to use 'fs-extra' instead of 'fs' in your scripts.
FS-extra claims to handle EMFILE errors better. I haven't tried it in this case sofar, so I can't tell if it helps.



Robert

[magnussandstrom]

Hi Magnus,

I've had these errors also. These were caused by scripts that created tmp-files that the script did not properly clean up.
Rebooting the whole system helped me get Switch running again, but after a while, the errors reappeared.
So check you're script(s) for unclosed (tmp) files.

Another solution might be to use 'fs-extra' instead of 'fs' in your scripts.
FS-extra claims to handle EMFILE errors better. I haven't tried it in this case sofar, so I can't tell if it helps.



Robert

Thanks Robert, I experienced the exact same thing. But with the little caveat that I don't even have the scripting module.

Last night I did another attempt revert some of the generic apps (to pre-nodejs version) and deleted a few recently installed apps and now the issue is gone. I suspect that some app is badly written and caused this issue (maybe Docparser?). It's very time consuming to test exactly what app causing the issue otherwise I could use the method of exlusion to pinpoint the exact app and version.

[JimmyHartington]

I do not know if this is related.

But after updating an app and installing another, I have begun receiving very weird errors and failed jobs.
When retrying the work.

I have reported this to support.

The errors are from different flows. Which have not changed or do not use the apps in question.

It is errors like this:

Code: Select all

01.03.2024 08:21:04.679 error Ungroup job
    Message: 99999 En lille ny test 2024-03-01 : Creating job error: Internal server error

Code: Select all

28.02.2024 09:36:13.150 error JSON pickup
    Message: Method is not allowed at this time. Have you called any sendTo method already?

Code: Select all

28.02.2024 10:24:42.441 error Merge PDF
    Message: read ECONNRESET

[r.zegwaard]

@JimmyHartington: found this on StackOverflow about the ECONNRESET error.

"ECONNRESET" means the other side of the TCP conversation abruptly closed its end of the connection. This is most probably due to one or more application protocol errors. You could look at the API server logs to see if it complains about something.

So I don't think that this is related.

Which app did you update?

Robert

[JimmyHartington]

Which app did you update?

I updated the Photoshop app and installed the Inject-XML app from the appstore.
Both are uninstalled now.

But I can still replicate the one error.

Code: Select all

04.03.2024 13:45:45.173 error JSON pickup
    Message: Method is not allowed at this time. Have you called any sendTo method already?

[davyverstaen]

Generally, The “Too many open files” error typically occurs when a process reaches the maximum number of open file descriptors allowed by the operating system (It depends upon resources and configuration of the system).

In our investigation we noticed that some apps are not closing the tmp files correctly, when these are not properly closed Switch is also not able to delete the tmp files after processing. We are currently in the process of identifying these default element apps and updating the code to avoid this.

we are currently updating all the apps that use tmp files so they get closed properly. If you are writing your own node.js scripts we recommend to discard the descriptor for that use tmp.file({descardDescriptor: true}) creates a temporary file with discardDescriptor: true, which means the file descriptor is closed immediately after the file is created.

[r.zegwaard]

@davyverstaen:

Could you provide a piece of sample code on how to create, open, and close tmp files?
So we have some kind of "recommended" way of working with tmp-files.

Robert

[magnussandstrom]

Generally, The “Too many open files” error typically occurs when a process reaches the maximum number of open file descriptors allowed by the operating system (It depends upon resources and configuration of the system).

In our investigation we noticed that some apps are not closing the tmp files correctly, when these are not properly closed Switch is also not able to delete the tmp files after processing. We are currently in the process of identifying these default element apps and updating the code to avoid this.

we are currently updating all the apps that use tmp files so they get closed properly. If you are writing your own node.js scripts we recommend to discard the descriptor for that use tmp.file({descardDescriptor: true}) creates a temporary file with discardDescriptor: true, which means the file descriptor is closed immediately after the file is created.

Great to hear that you are working on the problem Davy, I'm looking forward to update the apps again once the issue is solved.

[pjb]

Good morning. I updated a few apps on Friday (8/03/2024) including 'Unpack Job', 'Ungroup Job', and 'Inject Job', all seemed to be fine. However today Monday morning) have 102 errors in the dashboard all of them are 'EMFILE: too many open files etc'.
What is the immediate solution to get everything running as normal?
Do all the apps I've updated need reinstalling with the old version and a reboot?
Thanks
Peter

[pjb]

Good morning. I updated a few apps on Friday (8/03/2024) including 'Unpack Job', 'Ungroup Job', and 'Inject Job', all seemed to be fine. However today Monday morning) have 102 errors in the dashboard all of them are 'EMFILE: too many open files etc'.
What is the immediate solution to get everything running as normal?
Do all the apps I've updated need reinstalling with the old version and a reboot?
Thanks
Peter

Update . . .
I have re-installed all the default apps with the older versions that we had installed before I updated on Friday afternoon.
Done a controlled re-boot and restarted all the flows and resubmitted all the jobs that were in the dashboard. All working OK.
The Apps I had updated were; Inject Job, Ungroup Job, Unpack Job, Unarchive, Export Meta Data, HTTP Request, and Sort by Preflight.
Is any of these know to be an issue?
Thanks
Peter

[magnussandstrom]

Very similar to my experience and solved it in the same way.

[tdeschampsBluewest]

Hi,
I had a talk with Enfocus support about it, and they confirmed that it's tied to something not closing the file properly.

For tmp, you "should" use the optional "discard descriptor" propertie :

Code: Select all

async function createTmpFile(content: string | Buffer | Uint8Array, prefix: string, extension: string, encoding: fs.EncodingOption = "utf-8"): Promise<tmp.FileResultNoFd> {
    tmp.setGracefulCleanup();
    const tempPath = tmp.fileSync({ mode: 0o644, discardDescriptor: true, prefix: prefix, postfix: extension });
    fs.writeFileSync(tempPath.name, content, encoding);
    return tempPath;
}

And not forget to cleanup your tmp anyway :

Code: Select all

const tmpFile = await createTmpFile("some content", "switch_appname", "xml")
// do some stuff here
tmpFile.removeCallback()

For us, it was strongly tied to the app "PDF review reporter" with multiple customer for which we created a flow with loop inside switch.
By reducing the frequency of the loop + planned restart, we have diminished the frequency of the problem with our customer.

[JaDe0051]

In our case it was the default app inject job. We reinstalled the older Version of the app. It solved the problem.