High CPU spikes

[matthew.hearne]

Hi

We have an ongoing problem with our server that runs Switch.

Out of nowhere, we can get CPU spikes up to 100% which then stop everything from working.

At this point, Switch will be taking up a high portion of the CPU and many other processes seem to then be accounting for the rest.

In normal working times, the CPU will sit anywhere in the 5-15% area with Switch taking a small part of that.

We then have to pause the processing (140 flows), stop Switch and reboot the server. Sometimes this works. Other times, we have to do this and then leave the machine to just settle down which it will do after a while.

Other times we have to stop all flows, and start then individually until all is back up and running.

I can't see any logic as to what is going on, or what causes it. It happens randomly by the look of it.

I am not sure if / how we can tell if it is actually Switch causing the problem OR is there something else happening on the server that's causing the slow down, and then causing Switch to slow down i.e. network speed?

Any help/suggestions as to the best way to approach resolving this would be greatly appreciated.

Our IT guys have suggested upping the spec of the server, but considering it will be running fine for days / weeks with no problem and then just spikes out of nowhere, I can't see this will help? If it was a server resource issue, I would assume it would be slow all the time and not just a sudden jump?!

Thanks for any help with this. It's getting very frustrating at the moment as we are wasting hours of time each instance of this happening.

[sander]

What are the hardware specifications and OS version you are running?

[magnussandstrom]

Could you see any warnings or errors in Switch messages when this happens? Could it be related to this issue (see link below)?

viewtopic.php?t=4929&hilit=emfile

[tdeschampsBluewest]

I could suggest you using a "deadman switch" to be alerted if there is a problem happening.

Just in case you are not familiar with this technic :

A dead man's switch is a switch that is designed to be activated or deactivated if the human operator becomes incapacitated, such as through death, loss of consciousness, or being bodily removed from control. Originally applied to switches on a vehicle or machine, it has since come to be used to describe other intangible uses, as in computer software.

https://en.wikipedia.org/wiki/Dead_man%27s_switch

How to do it with switch and uptimeKuma :
Install uptimeKuma on another server : https://github.com/louislam/uptime-kuma
For myself, i have installed it on a simple raspeberry pi, no needs to have a big setup for this

It can be installed with docker with a minimalist Yaml config :

Code: Select all

version: "3"
services:
    uptime-kuma:
        container_name: uptimekuma
        image: louislam/uptime-kuma
        restart: unless-stopped
        ports:
            - 3001:3001
        volumes:
            - ./data:/app/data
            - /var/run/docker.sock:/var/run/docker.sock

You'll then be able to create a "Push" monitor type with an URL. If this URL is not called every X s, you will be able to get notified by Email, slack/teams or whatever you want to.

Now within switch create a simple flow, with a flowTrigger app + HTTP request, that will call this URL.

It won't solve your problem, but at least you'll be notified when it's down! (and uptime kuma could help you to do this for every service you'll have.