Node Setup

THORChain Bare-Metal Validator — Copy Chain Data

There is different reason we may want to spin up an additional daemon pod, such as: migrating to a new server, or simply having two…

There is different reason we may want to spin up an additional daemon pod, such as: migrating to a new server, or simply having two instance of a daemon in case a data corruption happen.

For sure we can simply start the new daemon fresh and wait for it to download a snapshot and sync to the tip. But we can also take advantage of a daemon that is already synced to the tip if we already have one.

In this article, I will go over copying Chain Data from an instance to another.

I will use BSC (Binance-Smart-Chain) as an example, because there was alot of issues downloading a working snapshot in the past few weeks, but the same method could probably be used with any other chains, it could also be used for the data subfolder of a THORChain validator, more on this later.

First we will want to install our new pod. In this example, c0 will be the namespace of our existing daemon, and c1 will be the namespace of the new daemon we are creating.

Install Daemon

First, we Install the new daemon from the git as usual.

// Get git
cd ~
mkdir c1
cd c1
git clone https://gitlab.com/thorchain/devops/node-launcher.git
cd node-launcher

// Edit Daemons Configs to enable just desired one
nano thornode-stack/mainnet.yaml

// Install Daemon
NAME=c1 TYPE=daemons NET=mainnet make install

Stop Daemon

Once the pod and its PVC is created, we will stop it so we can populate the chain data ourself.

kubectl scale --replicas=0 deploy/binance-smart-daemon -n c1

Find target location

Then we will find the PVC storage folder and clear its content. With a microk8s setup like the one in my previous guide, all our pv are located inside /data as per defined in our default StorageClass.

cd /data/
ls
cd c1-binance-smart-daemon-pvc-680f0848-8ccb-4bb1-8e8f-222222bbb222/

rm -fr geth
sudo rm -fr geth.full
rm -fr keystore/

Initial Chain Data Copy

Then we will use rsync to copy the Chain Data content from our healthy chain. Note that at this point I didn’t scaled down the source chain yet to minimise downtime.

// Copy from a chain on the same server
rsync -azPHh --delete /data/c0-binance-smart-daemon-pvc-fb176913-9649-43df-9e39-111111aaa111/ /data/c1-binance-smart-daemon-pvc-680f0848-8ccb-4bb1-8e8f-222222bbb222/

// Copy from a chain on another Server
rsync -azPHh --delete [email protected]:/data/c0-binance-smart-daemon-pvc-fb176913-9649-43df-9e39-111111aaa111/ /data/c1-binance-smart-daemon-pvc-680f0848-8ccb-4bb1-8e8f-222222bbb222/

Important: The Trailing Slash in the source folder path is Required so we copy the content of the folder instead of the folder itself. Trailing Slash in the destination folder path is optional.

This step may take a while, for BSC it toke me about 12h between two server over a Gigabit Ethernet connection. This is why we may want to do an initial copy without stopping the source daemon, if any active node is using it.

Once this step is completed, we can run the rsync command a few more times to catchup what happen during the initial copy, until the rsync execution is almost instant, this will ensure that our Final Copy is very quick, minimizing downtime on our source daemon.

Final Chain Data Copy

For the final copy, we will scale down the source daemon before running the rsync command one last time. This copy should be fairly quick, minimizing downtime on the source daemon instance.

// Stop Daemon on Source Host
kubectl scale --replicas=0 deploy/binance-smart-daemon -n c0

// Run Rsync Command from the previous Step

// Once Rsync completed, Start Both Daemon
kubectl scale --replicas=1 deploy/binance-smart-daemon -n c0
kubectl scale --replicas=1 deploy/binance-smart-daemon -n c1

Both Pods should now start and catch up the tip in no time and will be ready to use.

THORChain or MAYAChain Data SubFolder

We can use this approach to sync additional Validator in no time. This can be a lot quicker than downloading the latest snapshot and synching to the tip, which sometime take several hours.

Note that we don't want to change anything else than the content of the .thornode/data subfolder, which only contains the Chain Data.

Copying MAYAnode Data

Scale down both daemon (m1 is the source and m2 is the target)

kubectl -n m1 scale deployments --replicas=0 --all
kubectl -n m2 scale deployments --replicas=0 --all

Delete Target Chain Data

We would think that the --delete parameter of rsync would take care of everything in the directory but it doesn't always, save yourself the trouble and clear it prior to copying.

sudo rm -r /data1/m2-mayanode-pvc-984746bb-2303-456a-9b10-lalalalalala/.mayanode/data/

Copy Maya Chain Data

sudo rsync -azPHh --delete /data1/m1-mayanode-pvc-a8733e09-f0ab-4586-bf55-lalalalalala/.mayanode/data/ /data1/m2-mayanode-pvc-984746bb-2303-456a-9b10-lalalalalala/.mayanode/data/

Clear the Validator State of the target node (priv_validator_state.json)
(only required if copied from active node)

nano /data1/m2-mayanode-pvc-984746bb-2303–456a-9b10–lalalalala/.mayanode/data/priv_validator_state.json

Replace priv_validator_state.json with this content:

{
  "height": "0",
  "round": 0,
  "step": 0
}

Scale up both daemon

kubectl -n m1 scale deployments --replicas=1 --all
kubectl -n m2 scale deployments --replicas=1 --all

Look at validator health

k9s
make status

Copying Thornode Data:

Same apply to THORChain

sudo cp -r /data/n1-thornode-pvc-5519e83b-8585-4d1a-bd40-1a1a1a1a1a1a/.thornode/data/. /data/n2-thornode-pvc-332f9efe-9a53-44d0-a04c-2b2b2b2b2b2b/.thornode/data/

In this example I used the cp command instead of the rsync. Both would work, but I would recommend using rsync, specially if we want to do an initial copy before scaling down the source pod.

This could also be useful for a migration or when recovering multiple nodes instead of re-downloading everything multiple times from the Internet.

Note that it may still be a good thing to Sync from a pruned snapshot if our source data was online for a long time, to minimise storage usage that will naturally increase over time.