How to stop the node gracefully to avoid penalties?

Node Running
1

Context

A SIGINT (sent to it with the appropriate kill command or ctrl-c) should do it
@cassie

I have also added KillSignal=SIGINT to my services
by default KillMode is set to control-group and control-group first terminates processes with SIGTERM

You can set KillSignal=SIGINT which overrides that default
@Tyga

systemd.kill — Process killing procedure configuration
https://www.freedesktop.org/software/systemd/man/latest/systemd.kill.html

Question

Does this work for people running the release_autorun.sh via their service file?
And if not what is the solution?

Service file example

[Unit]
Description=Ceremony Client Go App Service

[Service]
Type=simple
Restart=always
RestartSec=5s
WorkingDirectory=/root/ceremonyclient/node
ExecStart=/root/ceremonyclient/node/release_autorun.sh

[Install]
WantedBy=multi-user.target

Service file with kill (suggestion by Claude AI)

[Unit]
Description=Ceremony Client Go App Service

[Service]
Type=simple
Restart=always
RestartSec=5s
WorkingDirectory=/root/ceremonyclient/node
ExecStart=/bin/bash -c '/root/ceremonyclient/node/release_autorun.sh & wait $!'
ExecStop=/bin/kill -SIGINT $MAINPID
KillMode=mixed
KillSignal=SIGTERM
TimeoutStopSec=70s

[Install]
WantedBy=multi-user.target

The above service file:

  1. Runs the script in a way that systemd can track the main process.
  2. Explicitly sends SIGINT when stopping, which should be caught by the bash process and propagated to the node process.
  3. Uses KillMode=mixed to ensure that if the main process doesn’t exit, systemd will still attempt to kill all processes in the control group.
  4. Provides a 70-second timeout for the stop process.

I have no clue if the above could work. Bigger brains needed. Thanks

1 Like
2

nice, thanks for sharing. would be nice to integrate the ‘graceful’ stop into systemd directly. i imagine many of us run ceremonyclient as a service and could effectively keep stopping and starting nodes the same way if this does indeed work.

(waiting for bigger brains to confirm the above, lol)

3

I gathered some more opinions and this is probably the easier way to solve the issue for now

[Unit]
Description=Ceremony Client Go App Service

[Service]
Type=simple
Restart=always
RestartSec=5s
WorkingDirectory=/root/ceremonyclient/node
ExecStart=/root/ceremonyclient/node/node-1.4.21.1-linux-amd64
KillSignal=SIGINT
TimeoutStopSec=30s

[Install]
WantedBy=multi-user.target

at least unless we understand if the new release_autorun.sh will be able to handle this “graceful stop” well even if it’s the service executing the script

More also here: Announcements | Quilibrium.one

1 Like
4

Screenshot 2024-10-11 230114
Screenshot 2024-10-11 2301141914×211 26.1 KB

Screenshot 2024-10-11 230124
Screenshot 2024-10-11 2301241906×155 14.7 KB

Is this how successful graceful stop looks like?

edit: this is cluster running para service

5

I am copying some more input from a convo I had with a more experienced user on TG (I don’t know their forum handle)

Ceremony Client Service Configuration

Chris, [12/10/2024 11:01]
So this is my ceremony.service file:

Chris, [12/10/2024 11:01]

Description=Ceremonyclient
After=network.target
Wants=network-online.target
StartLimitIntervalSec=0
[Service]
Type=simple
WorkingDirectory=/root/ceremonyclient/node/
Restart=on-failure
RestartSec=5
StartLimitBurst=5
User=ubuntu
ExecStart=/root/ceremonyclient/node/node-1.4.21.1-linux-arm64
ExecStop=/bin/kill -s SIGINT $MAINPID
ExecReload=/bin/kill -s SIGINT $MAINPID && /home/ubuntu/ceremonyclient/node/node-1.4.21.1-linux-arm64
KillSignal=SIGINT
RestartKillSignal=SIGINT
FinalKillSignal=SIGINT
TimeoutStopSec=30
PIDFile=/var/run/ceremonyclient.pid
[Install]
WantedBy=multi-user.target

Chris, [12/10/2024 11:01]
What you would need to do is having a script that gets called via ExecStop

Chris, [12/10/2024 11:01]
This script needs to do two things:

  1. Kill the autorun process
  2. Properly kill (SIGINT) the main node process

LaMat, [12/10/2024 11:04]
but this should work correct?

[Unit]
Description=Ceremony Client Go App Service
StartLimitIntervalSec=0
StartLimitBurst=0
[Service]
Type=simple
Restart=always
RestartSec=5s
WorkingDirectory=/root/ceremonyclient/node
ExecStart=/root/ceremonyclient/node/node-1.4.21.1-linux-amd64
KillSignal=SIGINT
TimeoutStopSec=30s
[Install]
WantedBy=multi-user.target

Chris, [12/10/2024 11:04]
Yes.

Chris, [12/10/2024 11:04]
Because it starts the node process directly.

LaMat, [12/10/2024 11:06]
so AFAIU the only way to exec the release_autorun.sh via servcie file and also be able to stop the node gracefully is to have a custom script to handle that, like

ExecStop=/path/to/graceful_shutdown.sh

Chris, [12/10/2024 11:07]
Correct. You could even create a script that checks whether the node is running via autorun or directly and properly kill the node process and any autorun process. This way it wouldn’t matter but just work.

2 Likes