Hi, I have a problem with 2 nodes. After transferring to a new AMD RYZEN 7950X3D 128GB RAM node. I have the following errors. The new node works without any problems, after stopping and pasting the new .config I get this error ;/ what can I do about it?
files that I pasted in .config
It looks like some files in the ‘store’ folder are corrupt. When the main process starts and discovers it, it terminates while the spawned child processes keep trying to connect to it.
Based on my experience, I would go back to the original node, stop the ‘ceremonyclient’ service, with the service stopped back up the files in the ‘store’ folder and then try to put them on this new node.
I had this issue when I was backing up my servers wrong. Depending on the command you’re using this could very well be your issue.
I was using rsync but using improper flags. This meant that while it would save new files, it wouldn’t update those files already existing in the backups if there were any changes.
Files in the store that get consolidated were left in their original state and never updated.
Only found out that this was a thing when going to restore a node after decommissioning close to a dozen smaller nodes.
Probably lost around 300-400 QUIL that way.
HIGHLY SUGGEST YOU TEST YOUR BACKUP METHODS BEFORE YOU NEED THEM!
2 Likes
Unfortunately, I no longer have access to the old server, backups were made automatically to the cloud, out of 20 servers I only have a problem with 2 ;/ There’s nothing I can do about it?
there isn’t much that I’m aware about.
There is a slight possibility that if your files are synced that there just may be old files that shouldn’t be there (but the ones that are are correct). If that’s the case you can attempt to delete some of the older MANIFEST and .log files but it would be a PIA to figure out which combo works.
My experience also suggests no way to fix the corrupt files as they’re binary and we do not know the format as well as a safe way to edit them.
I do not aim to increase the frustration, but, as Tyga implies, it may not be possible to use the corrupt store to claim the already earned reward.
It feels like the store backup topic is not fully studied/tested or maybe I just never came across a decent guide that would prove to work in 99% of the practical use cases.
1 Like
hmm, the 2 backups that don’t work are 1GB in size, the rest that work are 250-400MB. I found old backups of these nodes, I will be there for 2-3 weeks without rewards and I will verify the files between these copies. Old backup have 75 files (in store folder) corrupted have 275 files in store folder
Since you’ve changed hardware, have you deleted SELF_TEST before running your node?
no, i move all files
I see, it’s mandatory to remove SELF_TEST when up/downgrading hardware since SELF_TEST is the node’s attestation to the hardware it’s working with.
@cassie I was under the impression that the node automatically will self test itself when put on new hardware.
That SELF_TEST is mostly optional (deleting allows you to skip the tests on startup).
SELF_TEST will detect if the core count or the memory changed, but if you move to a new node with same memory and core count, but different clock speed, then you need to delete it.
1 Like