Difference between revisions of "Set up a Cardano staking pool"

From Organic Design wiki
m (Sending Tip to PoolTool)
(Withdraw Rewards: Update docs.)
 
(50 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
{{stub}}
 
{{stub}}
  
The [https://docs.cardano.org/projects/cardano-node/en/latest/getting-started/install.html official documentation] for setting up a node and configuring it as a stake pool on the mainnet is mostly very clear and complete. I was able to get my node compiled, installed and running pretty easily. But configuring the node to run as a stake pool is more complicated and the documentation seems to have some missing bits, so I'm documenting here just the specific parts that I had trouble with.
+
The [https://docs.cardano.org/getting-started/installing-the-cardano-node official documentation] for setting up a node and configuring it as a stake pool on the mainnet is mostly very clear and complete. I was able to get my node compiled, installed and running pretty easily. But configuring the node to run as a stake pool is more complicated and the documentation seems to have some missing bits, so I'm documenting here just the specific parts that I had trouble with.
  
 
== Prerequisites ==
 
== Prerequisites ==
Line 248: Line 248:
 
The Cardano node should always be stoped with the SIGINT signal (e.g. '''kill -SIGINT <PROCESS>''') otherwise it might not shutdown cleanly, it is designed to recover after a messy shutdown but next startup might be 30-40 minutes instead of the usual 5 minutes.
 
The Cardano node should always be stoped with the SIGINT signal (e.g. '''kill -SIGINT <PROCESS>''') otherwise it might not shutdown cleanly, it is designed to recover after a messy shutdown but next startup might be 30-40 minutes instead of the usual 5 minutes.
  
== Updating Pool Registration ==
+
== Pool Registration ==
 +
 
 +
You will need the network protocol parameters, if you don't already have them.
 +
 
 +
<source lang="bash">
 +
cardano-cli query protocol-parameters \
 +
  --mainnet \
 +
  --out-file protocol.json
 +
</source>
 +
 
 +
Then you need to get information about the wallet you are going to make the transaction from:
 +
 
 +
<source lang="bash">
 +
cardano-cli query utxo --mainnet --address <Your Payment Address (payment.addr)>
 +
</source>
 +
 
 +
Then prepare a blank transaction to calculate fees:
 +
 
 +
<source lang="bash">
 +
cardano-cli transaction build-raw \
 +
--tx-in <TxHash>#<TxIx> \
 +
--tx-out <Your Payment Address>+0 \
 +
--invalid-hereafter 0 \
 +
--fee 0 \
 +
--out-file tx.draft \
 +
--certificate-file pool-registration.cert
 +
</source>
 +
 
 +
Then calculate fees:
 +
 
 +
<source lang="bash">
 +
cardano-cli transaction calculate-min-fee \
 +
--tx-body-file tx.draft \
 +
--tx-in-count 1 \
 +
--tx-out-count 1 \
 +
--witness-count 3 \ # This is the number of keys that will sign the transaction - in our case: 3.
 +
--byron-witness-count 0 \
 +
--mainnet \
 +
--protocol-params-file protocol.json
 +
</source>
 +
 
 +
Calculate the change after paying the fee:
 +
<source lang="bash">
 +
expr <UTxO> - <FEE>
 +
</source>
 +
 
 +
Query the tip can copy the slot number:
 +
<source lang="bash">
 +
cardano-cli query tip --mainnet
 +
</source>
 +
 
 +
Create the transaction for real:
 +
<source lang="bash">
 +
cardano-cli transaction build-raw \
 +
--tx-in <TxHash>#<TxIx> \
 +
--tx-out <Your Payment Address>+<Change> \
 +
--invalid-hereafter <Slot number + time in seconds. E.g. Add 300 to the last step to give yourself 5 mins> \
 +
--fee <FEE> \
 +
--out-file tx.raw \
 +
--certificate-file pool-registration.cert
 +
</source>
 +
 
 +
Sign the transaction:
 +
<source lang="bash">
 +
cardano-cli transaction sign \
 +
  --tx-body-file tx.raw \
 +
  --signing-key-file payment.skey \
 +
  --signing-key-file stake.skey \
 +
  --signing-key-file cold.skey \
 +
  --mainnet \
 +
  --out-file tx.signed
 +
</source>
 +
 
 +
Submit the transaction:
 +
<source lang="bash">
 +
cardano-cli transaction submit \
 +
  --tx-file tx.signed \
 +
  --mainnet
 +
</source>
 +
=== Updating ===
 
Pool registrations can be updated by simply sending a new pool registration certificate - no need to de-register! Just follow the same process but don't worry about the pool deposit or adding a delegation certificate.
 
Pool registrations can be updated by simply sending a new pool registration certificate - no need to de-register! Just follow the same process but don't worry about the pool deposit or adding a delegation certificate.
  
Line 309: Line 388:
 
--tx-file tx.signed \
 
--tx-file tx.signed \
 
--mainnet
 
--mainnet
 +
</source>
 +
 +
== Withdraw Rewards ==
 +
Withdrawing rewards requires the recipient wallet to be able to sign the transaction and pay for the fee. So a using CLI may be necessary.
 +
 +
Take note of the rewards balance:
 +
<source lang="bash">
 +
cardano-cli query stake-address-info \
 +
--mainnet \
 +
--address <Your Stake Address (stake.addr)>
 +
</source>
 +
 +
Then you need to get information about the wallet you are going to make the transaction from:
 +
<source lang="bash">
 +
cardano-cli query utxo --mainnet --address <Your Payment Address (payment.addr)>
 +
</source>
 +
 +
Create the transaction:
 +
<source lang="bash">
 +
cardano-cli conway transaction build --mainnet \
 +
--tx-in <TxHash>#<TxIx> \
 +
--withdrawal <Your Stake Address>+<REWARD BALANCE> \
 +
--change-address <Your Payment Address> \
 +
--out-file withdrawal-tx.raw
 +
</source>
 +
 +
Sign the transaction:
 +
<source lang="bash">
 +
cardano-cli transaction sign \
 +
  --tx-body-file withdrawal-tx.raw  \
 +
  --signing-key-file payment.skey \
 +
  --signing-key-file stake.skey \
 +
  --out-file withdrawal-tx.signed
 +
</source>
 +
 +
Submit the transaction:
 +
<source lang="bash">
 +
cardano-cli transaction submit \
 +
  --tx-file withdrawal-tx.signed \
 +
  --mainnet
 
</source>
 
</source>
  
Line 349: Line 468:
 
</source>
 
</source>
  
== Sending Tip to PoolTool ==
+
== Updating ==
 +
If you want to update your nodes, first check if any additional steps are listed on the cardano-node github release.
 +
 
 +
Then follow the steps on the [https://github.com/input-output-hk/cardano-node/blob/master/doc/getting-started/install.md/ official documentation] but if there are any issues what I have done is listed below.
 +
 
 +
Enter the repository you build the node from.
 +
<source lang="bash">
 +
cd cardano-node
 +
git checkout master
 +
git pull
 +
git fetch --tags --all # add '-f' if you get 'clobber' errors)
 +
git checkout <VERSION> # e.g. 1.29.0
 +
git status
 +
</source>
 +
 
 +
That last command should say something like: '''HEAD detached at <VERSION>'''
 +
 
 +
Then you can build the new version:
 +
<source lang="bash">
 +
cabal clean
 +
cabal update
 +
cabal build all
 +
</source>
 +
 
 +
Then you can find the paths to the build executables with:
 +
(Be sure to run inside the cardano-node directory.)
 +
<source lang="bash">
 +
find . -regex './dist-newstyle/build/[^/]*/[^/]*/cardano-\(node\|cli\)-[0-9\.]*/x/cardano-\(node\|cli\)/build/cardano-\(node\|cli\)/cardano-\(node\|cli\)'
 +
</source>
 +
 
 +
=== Cabal/GHC Issues ===
 +
Check you library versions still match the recommended ones on the [https://docs.cardano.org/projects/cardano-node/en/latest/getting-started/install.html page].
 +
 
 +
 
 +
You can update Cabal without reinstall like so:
 +
<source lang="bash">
 +
cabal install Cabal cabal-install
 +
</source>
 +
 
 +
 
 +
The new documentation says to use '''ghcup''' and sometimes the build may fail due to cabal/ghc version so to install and set a specific version use:
 +
<source lang="bash">
 +
ghcup install cabal <VERSION>
 +
ghcup set cabal <VERSION>
 +
ghcup install ghc <VERSION>
 +
ghcup set ghc <VERSION>
 +
</source>
 +
 
 +
=== "cardano-crypto-class" Missing ===
 +
<source>
 +
cabal: Could not resolve dependencies:
 +
[__0] trying: cardano-crypto-class-2.0.0 (user goal)
 +
[__1] rejecting: cardano-crypto-class:+secp256k1-support (conflict: pkg-config
 +
package libsecp256k1-any, not found in the pkg-config database)
 +
[__1] rejecting: cardano-crypto-class:-secp256k1-support (manual flag can only
 +
be changed explicitly)
 +
[__1] fail (backjumping, conflict set: cardano-crypto-class,
 +
cardano-crypto-class:secp256k1-support)
 +
After searching the rest of the dependency tree exhaustively, these were the
 +
goals I've had most trouble fulfilling: cardano-crypto-class,
 +
cardano-crypto-class:secp256k1-support
 +
</source>
 +
 
 +
Related issue: https://github.com/input-output-hk/cardano-node/issues/3844.
 +
 
 +
To fix install libsecp256k1:
 +
<source lang="bash">
 +
git clone https://github.com/bitcoin-core/secp256k1.git
 +
cd secp256k1
 +
./autogen.sh
 +
./configure --enable-module-schnorrsig --enable-experimental
 +
make
 +
make check
 +
make install
 +
</source>
 +
 
 +
=== "katip" Issues ===
 +
I have run into katip having version issues to fix add the following line to the '''cabal.project''' file at the end of the constraints section (near the end of the file).
 +
<source>
 +
, katip <= 0.8.4.0
 +
</source>
 +
 
 +
== Update KES ==
 +
 
 +
'''IMPORTANT''': Each time you generate a KES the counter increments if you increment any more than what the next number should be the blocks will not be minted - this can be caused by generating new kes keys more than once.
 +
 
 +
Currently KES needs to be updated every 90 days, gLiveView will tell you how long is left. You can update the KES from a cold machine with Daedalus synced, you need to generate '''kes.vkey''', '''kes.skey''' and '''node.cert''' to update. Of these 3, only the '''kes.skey''' and '''node.cert''' need to be uploaded to the server. You need access to '''cold.skey''' and '''cold.counter'''.
 +
 
 +
 
 +
This script will output the needed files in the current working directory, after running save '''kes.vkey''', '''kes.skey''' and '''node.cert''' to a secure location then copy only '''kes.skey''' and '''node.cert''' to the block producing node and restart.
 +
 
 +
https://code.organicdesign.nz/organicdesign/pudim/src/branch/master/update-kes.sh
 +
 
 +
=== Using a Remote Node ===
 +
 
 +
If you don't have Daedalus running locally or is not up to date you can mount a remote one over ssh using the following command:
 +
 
 +
<source lang="bash">
 +
sshfs root@server:/remote/path/data /local/path
 +
</source>
 +
 
 +
=== Modifying the KES Counter ===
 +
 
 +
To view the current KES period you can run:
 +
 
 +
<source lang="bash">
 +
cardano-cli text-view decode-cbor --in-file node.cert  | grep int | head -1
 +
</source>
 +
 
 +
You can generate a new KES Counter (cold.counter) by running:
 +
<source lang="bash">
 +
cardano-cli node new-counter \
 +
--cold-verification-key-file cold.vkey \
 +
--counter-value <KES PERIOD> \
 +
--operational-certificate-issue-counter-file node.counter
 +
</source>
 +
 
 +
Then you can rotate KES using this counter and it will produce a KES for that period and update the file for the next one.
 +
 
 +
== Calculate Leader Logs ==
 +
They have finally added a way to calculate leader logs using cardano-cli:
 +
 
 +
https://code.organicdesign.nz/organicdesign/pudim/src/branch/master/calculate-leader-logs.sh
 +
 
 +
== PoolTool ==
 +
 
 +
=== Sending Tip ===
 
If you have setup the node without all the extra tools sending your tip is easiest done through the shell/systemd script found [https://github.com/papacarp/pooltool.io/tree/master/sendmytip/shell/systemd here]. You can send this info from your relays.
 
If you have setup the node without all the extra tools sending your tip is easiest done through the shell/systemd script found [https://github.com/papacarp/pooltool.io/tree/master/sendmytip/shell/systemd here]. You can send this info from your relays.
  
Line 357: Line 602:
  
 
You may also need to modify the '''cadano-node''' and '''cardano-cli''' if the script is running from systemd:
 
You may also need to modify the '''cadano-node''' and '''cardano-cli''' if the script is running from systemd:
<source lang="shell">
+
<source lang="bash">
 
# CARDANO BINARIES
 
# CARDANO BINARIES
 
CNODE=$(command -v cardano-node)
 
CNODE=$(command -v cardano-node)
 
CCLI=$(command -v cardano-cli)
 
CCLI=$(command -v cardano-cli)
 +
</source>
 +
 +
=== Sending Slots ===
 +
The easiest way to send slots to pooltool is to use the cncli sendSlots command, but that does not allow you to use a proxy - the below script is how you can send it manually while allowing specification of proxy. You will need run this with the [https://github.com/AndrewWestberg/cncli/blob/develop/scripts/cncli-fivedays.sh cncli-fivedays.sh] script as shown in the [https://github.com/AndrewWestberg/cncli/blob/develop/INSTALL.md installation guide].
 +
 +
 +
Manual sending of slots involve sending your slot count with the hash (Using [https://github.com/oconnor663/blake2_simd blake2] NOT sha256!) of the compressed array of slot numbers (e.g. "[123, 456, 789]") and the array of the last epoch's slot numbers.
 +
 +
 +
 +
'''Ensure that the OUTPUT_DIR exists and you install [https://github.com/oconnor663/blake2_simd blake2]!'''
 +
 +
https://code.organicdesign.nz/organicdesign/pudim/src/branch/master/send-slots.sh
 +
 +
=== Proxy ===
 +
If you wish to send stats from your block producing node I recommend writing a script on a relay or other server that listens for the request and makes an identical one to pooltool so that the identity of your block producing node is not compromised. An example of such a script can be found [https://code.organicdesign.nz/organicdesign/tools/src/branch/master/crypto/pudim/pooltool.php here].
 +
 +
== CNCLI ==
 +
The prebuilt [https://github.com/AndrewWestberg/cncli CNCLI] release does not work with Debian 10 so you need to build from source.
 +
 +
 +
If cncli is missing after build:
 +
<source lang="bash">
 +
cp /root/.cargo/bin/cncli /usr/local/bin/cncli
 
</source>
 
</source>
  
 
== Debugging ==
 
== Debugging ==
 +
=== Missing bin-path script (cardano-node build location) ===
 +
I found the second time I build cardano-node the script that outputs the build was missing, so if you need to find it manually run:
 +
<source lang="bash">
 +
find ./dist-newstyle -type f -name cardano-node
 +
find ./dist-newstyle -type f -name cardano-cli
 +
</source>
 +
 +
The locations when I ran it was:
 +
<source>
 +
./dist-newstyle/build/x86_64-linux/ghc-8.10.2/cardano-node-1.26.2/x/cardano-node/build/cardano-node/cardano-node
 +
./dist-newstyle/build/x86_64-linux/ghc-8.10.2/cardano-cli-1.26.2/x/cardano-cli/build/cardano-cli/cardano-cli
 +
</source>
 +
 +
 
=== Era Mismatch ===
 
=== Era Mismatch ===
 
If you get errors running some commands saying there is an era mismatch like the following:
 
If you get errors running some commands saying there is an era mismatch like the following:
Line 496: Line 779:
 
cardano-cli shelley transaction submit --tx-file tx.signed --mainnet
 
cardano-cli shelley transaction submit --tx-file tx.signed --mainnet
 
</source>
 
</source>
 +
 +
== Admin ==
 +
If you want to quickly assess the state of the node run '''/root/scripts/gLiveView.sh''' (make sure to answer no to updating anything) and check that:
 +
* Tip is up to date
 +
* It has a peer for every relay server it is connected to.
 +
* KES shows and hasn't expired.
 +
* Blocks are not showing as invalid.
 +
 +
=== Services ===
 +
Most of the operation is handled by a couple systemd services - if anything has gone wrong you should check these are running.
 +
 +
* '''cardano.service''' - This service handles running the cardano node, it logs to the file '''/root/debug.log''', if this is not running the node is probably not running and you will not be minting blocks.
 +
* '''sendmytip.service''' - This service just runs the '''/root/scripts/sendmytip.sh''' script for sending tip to pooltool.
 +
* '''cncli-sync.service''' - This service just keeps cncli in sync so leader logs can be requested - if this is not running or has not fully synchronized then leaderlogs will fail.
 +
* '''prometheus.service''' - This runs the prometheus monitor for debugging.
 +
* '''node-exporter.service''' - This is just for providing system details for prometheus.
 +
 +
=== Scripts ===
 +
 +
==== Startup ====
 +
All startup scripts are located in the '''/root''' directory.
 +
 +
* '''start.sh''' - This starts the cardano block producing node and is run by the 'cardano' service.
 +
* '''start-prometheus.sh''' - This just starts the prometheus debugger and is run by the 'prometheus' service.
 +
* '''start-node-exporter.sh''' - This runs the node exporter to provide system details for prometheus and is rung by the 'node-exporter' service.
 +
* '''start-as-node.sh''' - This script starts the cardano block producing node as a normal (relay) node, so it shouldn't be used outside debugging.
 +
 +
==== Status ====
 +
All scripts related to monitoring the status and health of the cardano node is found in the '''/root/scripts''' directory.
 +
 +
* '''cncli-fivedays.sh''' - This is run by a cron job to figure out if it is the epoch border or not.
 +
* '''sendSlots.sh''' - This obtains the leaderlogs, saves them for debugging and sends them to pooltool. It is run by the cron job if it passes the check to detect the epoch border. This script will fail if the cncli-sync.service has failed or not fully synchronized.
 +
* '''sendmytip.sh''' - This just obtains the current block tip from the cardano log and sends it to pooltool on regular intervals. It is run by the sendmytip.service.
 +
* '''gLiveView.sh''' - This script provides an interface to view the current state of the cardano node. It will produce a few errors about cardano node not running or being out of date - ignore and don't update (new versions don't work for some reason) and you should be able to see an interface for the node's current state.
 +
 +
=== Leaderlogs ===
 +
The leaderlogs get saved to files in '''/root/leaderlogs''' for inspection at a later date. Each epoch creates 3 logs:
 +
* '''XXX_logs_full.json''' - This is the one that's most useful to inspect as it provides the count and time of blocks for this and earlier epochs
 +
* '''XXX_logs.json''' - This is an array of the slots that the blocks are assigned to that gets sent to pooltool after it is one epoch old.
 +
* '''XXX_logs.hash''' - This is the hash of the above array that gets sent to pooltool within an hour of the epoch starting. (This and the above info provides a way of proving you are honest about assigned blocks.)
 +
 +
If these files are missing for the current epoch it means the leaderlog generation failed for some reason and '''/root/scripts/sendSlots.sh''' should be run manually.
  
 
== Resources ==
 
== Resources ==
 
*[https://www.coincashew.com/coins/overview-ada/guide-how-to-build-a-haskell-stakepool-node Another very descriptive setup guide]
 
*[https://www.coincashew.com/coins/overview-ada/guide-how-to-build-a-haskell-stakepool-node Another very descriptive setup guide]
 
*[https://forum.cardano.org/t/lets-talk-about-keys/33379/7 Forum thread about all the key types]
 
*[https://forum.cardano.org/t/lets-talk-about-keys/33379/7 Forum thread about all the key types]
 +
*[https://github.com/DamjanOstrelic/cardano-leader-logs Leader Logs] ''- check the leader logs for the epoch''
  
 
== See also ==
 
== See also ==

Latest revision as of 00:40, 23 October 2024

Cone.png This article or section is a stub. Stubs are articles that have not yet received substantial attention from the authors. They are short or insufficient pieces of information and require additions to further increase the article's usefulness. The project values stubs as useful first steps toward complete articles.


The official documentation for setting up a node and configuring it as a stake pool on the mainnet is mostly very clear and complete. I was able to get my node compiled, installed and running pretty easily. But configuring the node to run as a stake pool is more complicated and the documentation seems to have some missing bits, so I'm documenting here just the specific parts that I had trouble with.

Prerequisites

You will need to have two servers (but three service is ideal so you can run two relays so one can be restarted to update topology without interrupting the service) and one offline machine, all running the node, so the first part of the procedure for compiling, installing and running is done on all three of the machines.

Procedure

First ensure that cardano-node is installed on the pool server, the relay server and a local machine (didn't have much luck with docker images).

Install cardano-addresses on the local machine.

Create a file called generate-keys.sh on the local machine with these contents (Note, the first line generates a phrase - remove it if you want to use an existing one.):

# Remove this line if you want to use an existing phrase stored in phrase.prv
cardano-address recovery-phrase generate > phrase.prv

# generate root keys
cat phrase.prv | cardano-address key from-recovery-phrase Shelley > rootkey.prv

cat rootkey.prv | cardano-address key public --with-chain-code > rootkey.pub
cat rootkey.prv | cardano-address key child 1852H/1815H/0H/0/0 > addr.prv
cat addr.prv | cardano-address key public --with-chain-code | cardano-address address payment --network-tag mainnet > payment.addr
cat rootkey.prv | cardano-address key child 1852H/1815H/0H/2/0 > stake.prv
cat stake.prv | cardano-address key public --with-chain-code | cardano-address address stake --network-tag mainnet > stake.addr


# Create payment and stake vkey/skey files

cardano-cli key convert-cardano-address-key --signing-key-file stake.prv --shelley-stake-key --out-file ShelleyStake.skey
cardano-cli key convert-cardano-address-key --signing-key-file addr.prv --shelley-payment-key --out-file ShelleyPayment.skey

cardano-cli key verification-key --signing-key-file ShelleyStake.skey --verification-key-file Ext_ShelleyStake.vkey
cardano-cli key verification-key --signing-key-file ShelleyPayment.skey --verification-key-file Ext_ShelleyPayment.vkey

cardano-cli key non-extended-key --extended-verification-key-file Ext_ShelleyStake.vkey --verification-key-file ShelleyStake.vkey
cardano-cli key non-extended-key --extended-verification-key-file Ext_ShelleyPayment.vkey --verification-key-file ShelleyPayment.vkey

cardano-cli stake-address build --stake-verification-key-file ShelleyStake.vkey --out-file ShelleyStake.addr --mainnet
cardano-cli address build --payment-verification-key-file ShelleyPayment.vkey  --stake-verification-key-file ShelleyStake.vkey --out-file ShelleyPayment.addr --mainnet

# Nicer formatting
mkdir out

cp ShelleyPayment.addr out/payment.addr
cp ShelleyPayment.skey out/payment.skey
cp ShelleyPayment.vkey out/payment.vkey

cp ShelleyStake.addr out/stake.addr
cp ShelleyStake.skey out/stake.skey
cp ShelleyStake.vkey out/stake.vkey

Once you have that script ready disconnect from the internet and insert a fresh usb create a directory on it called wallet-keys and go into that directory in the terminal and run the script (script will work in your current working directory).
The script will have generated the keys you want int the out/ directory, copy the two .addr files onto your machine since these are not secret, and remove the drive.

On the server run:

cardano-cli query protocol-parameters \
  --mainnet \
  --out-file protocol.json


Send the keyDeposit amount (2 Ada at time of writing) as defined in the protocol.json and 1.5 Ada (1 for minimum wallet amount and 0.5 for transaction fees) to the payment address contained in payment.addr that you copied of the usb.

You should then continue as per the documentation page 4 but on step 5 generate the pool keys offline on the usb. Note: you will need to run any queries on the server (which has the node running) and formulate the transactions locally - disconnecting from the net, generating certs and signing transactions - then copy the transaction to the server to get fee size and actually send the transaction.

Skip step 6. Key Evolving Signature and KES period as it is redundant because it is just reiterating over the commands done in step 5.

Note on step 8 you can add the pledge balance to the address at a later time but your node will not produce blocks until the pledge is met. Note: for some reason the pre-calculated fee is different than what it should be - you just have to fail and use the fee that it says you should have used. If you want to use DNS use --single-host-pool-relay instead of --pool-relay-ipv4.

Once you hit step 9. Start your nodes copy kes.skey and vrf.skey from the offline usb to the computer and upload them to the server when you reconnect.

I recommend to write some startup scripts (for the block producing server, the relay and Prometheus) that log to a file like this:

#!/bin/sh
/root/.local/bin/cardano-node run \
--topology /root/config/mainnet-topology.json \
--database-path /root/data \
--socket-path /root/data/node.socket \
--host-addr <IP> \
--port <PORT> \
--config /root/config/mainnet-config.json \
--shelley-kes-key /root/keys/kes.skey \
--shelley-vrf-key /root/keys/vrf.skey \
>> /root/debug.log

And a corresponding systemd service:

# /etc/systemd/system/cardano.service
########################################
[Unit]
Description=Starts the cardano node.
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
ExecStart=/root/start.sh
Restart=on-failure
RestartSec=5
KillSignal=SIGINT

[Install]
WantedBy=multi-user.target

Start with service cardano start
Start on boot with systemctl enable cardano
I would recommend changing minSeverity in mainnet-config.json to Warning for efficiency and log readability.

Testnet Procedure

Everything went smoothly until I got to Register stake address on the blockchain. The main issue is that you need to have some balance in your payment.addr address, and the previous step of creating a sample transaction is not optional, things in that step such as protocol.json and the output hash are referred to in this step.

The Key Evolving Signature and KES period section is redundant because it is done already in the Generate your stake pool keys section, but it is explained better in the later one.

The Configure topology files for block-producing and relay nodes section requires that we have a relay node IP address.

Relay Quickstart

Copy the binaries to the relay in /root/.local/bin/.

cd ~

# packages
apt-get update
apt-get install -y jq curl libtool git automake build-essential bc man-db -y

# Libsodium
mkdir src
cd src
git clone https://github.com/input-output-hk/libsodium
cd libsodium
git checkout 66f017f1
./autogen.sh
./configure
make
make install
cd ~

# setup paths
echo "export PATH=\"/root/.local/bin:/root/.local/bin/cardano-node:/root/.local/bin/cardano-cli:$PATH\"" >> .bashrc
echo "export CARDANO_NODE_SOCKET_PATH=/root/data/node.socket" >> .bashrc
echo "export LD_LIBRARY_PATH=\"/usr/local/lib:$LD_LIBRARY_PATH\"" >> .bashrc
echo "export PKG_CONFIG_PATH=\"/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH\"" >> .bashrc
source .bashrc

# get config
mkdir config
cd config
wget https://hydra.iohk.io/job/Cardano/cardano-node/cardano-deployment/latest-finished/download/1/mainnet-config.json
wget https://hydra.iohk.io/job/Cardano/cardano-node/cardano-deployment/latest-finished/download/1/mainnet-byron-genesis.json
wget https://hydra.iohk.io/job/Cardano/cardano-node/cardano-deployment/latest-finished/download/1/mainnet-shelley-genesis.json
wget https://hydra.iohk.io/job/Cardano/cardano-node/cardano-deployment/latest-finished/download/1/mainnet-topology.json
cd ~

# topology updater
mkdir scripts
cd scripts
wget https://raw.githubusercontent.com/cardano-community/guild-operators/alpha/scripts/cnode-helper-scripts/env
wget https://raw.githubusercontent.com/cardano-community/guild-operators/alpha/scripts/cnode-helper-scripts/topologyUpdater.sh
wget https://raw.githubusercontent.com/cardano-community/guild-operators/master/scripts/cnode-helper-scripts/gLiveView.sh

chmod +x topologyUpdater.sh
chmod +x gLiveView.sh
cd ~

# startup script
echo "#!/bin/sh

export PATH=\"/root/.local/bin:/root/.local/bin/cardano-node:/root/.local/bin/cardano-cli:$PATH\"
export CARDANO_NODE_SOCKET_PATH=/root/data/node.socket
export LD_LIBRARY_PATH=\"/usr/local/lib:$LD_LIBRARY_PATH\"
export PKG_CONFIG_PATH=\"/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH\"

cardano-node run \
--topology /root/config/mainnet-topology.json \
--database-path /root/data \
--socket-path /root/data/node.socket \
--host-addr 0.0.0.0 \
--port 3002 \
--config /root/config/mainnet-config.json \
>> /root/debug.log" > /root/start.sh

chmod +x start.sh

# Restart script
echo "#!/bin/sh

/bin/systemctl restart cardano.service" > /root/reboot-relay.sh

chmod +x /root/reboot-relay.sh

# systemd script
echo "
# /etc/systemd/system/cardano.service
########################################
[Unit]
Description=Starts the cardano node.
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
ExecStart=/root/start.sh
Restart=on-failure
RestartSec=5
KillSignal=SIGINT

[Install]
WantedBy=multi-user.target" >> /etc/systemd/system/cardano.service

# Set to start on boot.
systemctl enable cardano

Remember to edit the Prometheus IP in /root/config/mainnet-config.json to 0.0.0.0
Add relays to /root/config/mainet-topology.json
Edit your static IP address in /root/start.sh

Edit the /root/scripts/env file to have the following configuration:

CCLI="/root/.local/bin/cardano-cli"
CNODE_PORT=3002
CONFIG="/root/config/mainnet-config.json"
TOPOLOGY="/root/config/mainnet-topology.json"

Edit the /root/scripts/topologyUpdater.sh file to include your block producing node's ip in CUSTOM_PEERS (Don't change the CNODE_HOSTNAME variable to an ip address.) Also set BATCH_AUTO_UPDATE=Y
Test the topology updater script runs by running it: /root/scripts/topologyUpdater.sh
Add it as a cron job crontab -e

0 * * * * /root/scripts/topologyUpdater.sh >> /root/debug-topology-updater.log
30 0 * * * /root/reboot-relay.sh >> /root/debug.log

Note change the second 0 on the reboot-relay script to what hour of the day you want it to run - best space out the relay restarts as far as possible from the other relays, we run it on the half hour so that topology updater does not try to query it when it is restarting.
You will need wait until the second time it can execute the topology updater (1 hour after the first time) for it to update the providers.


Run /root/scripts/gLiveView.sh to get information on how the relay is running.


Once you are ready run service cardano start to start the relay.

Stopping

The Cardano node should always be stoped with the SIGINT signal (e.g. kill -SIGINT <PROCESS>) otherwise it might not shutdown cleanly, it is designed to recover after a messy shutdown but next startup might be 30-40 minutes instead of the usual 5 minutes.

Pool Registration

You will need the network protocol parameters, if you don't already have them.

cardano-cli query protocol-parameters \
  --mainnet \
  --out-file protocol.json

Then you need to get information about the wallet you are going to make the transaction from:

cardano-cli query utxo --mainnet --address <Your Payment Address (payment.addr)>

Then prepare a blank transaction to calculate fees:

cardano-cli transaction build-raw \
--tx-in <TxHash>#<TxIx> \
--tx-out <Your Payment Address>+0 \
--invalid-hereafter 0 \
--fee 0 \
--out-file tx.draft \
--certificate-file pool-registration.cert

Then calculate fees:

cardano-cli transaction calculate-min-fee \
--tx-body-file tx.draft \
--tx-in-count 1 \
--tx-out-count 1 \
--witness-count 3 \ # This is the number of keys that will sign the transaction - in our case: 3.
--byron-witness-count 0 \
--mainnet \
--protocol-params-file protocol.json

Calculate the change after paying the fee:

expr <UTxO> - <FEE>

Query the tip can copy the slot number:

cardano-cli query tip --mainnet

Create the transaction for real:

cardano-cli transaction build-raw \
--tx-in <TxHash>#<TxIx> \
--tx-out <Your Payment Address>+<Change> \
--invalid-hereafter <Slot number + time in seconds. E.g. Add 300 to the last step to give yourself 5 mins> \
--fee <FEE> \
--out-file tx.raw \
--certificate-file pool-registration.cert

Sign the transaction:

cardano-cli transaction sign \
  --tx-body-file tx.raw \
  --signing-key-file payment.skey \
  --signing-key-file stake.skey \
  --signing-key-file cold.skey \
  --mainnet \
  --out-file tx.signed

Submit the transaction:

cardano-cli transaction submit \
  --tx-file tx.signed \
  --mainnet

Updating

Pool registrations can be updated by simply sending a new pool registration certificate - no need to de-register! Just follow the same process but don't worry about the pool deposit or adding a delegation certificate.

Delegation (Offline)

If you want to delegate to a pool without being required to be online first ensure that you have access to your payment/stake keys (see script under procedure).

Generate your stake address registration certificate and your delegation certificate:

cardano-cli stake-address registration-certificate \
	--stake-verification-key-file stake.vkey \
	--out-file stake.cert

# Delegate to PUDIM! (or use the pool id of the pool you wish to delegate to)
cardano-cli stake-address delegation-certificate \
	--stake-verification-key-file stake.vkey \
	--stake-pool-id pool1ujcu3myfg9wwvdyh2ks653954lamtufy3lefjs73jrr327q53j4 \
	--out-file delegation.cert

Then generate a raw transaction to calculate fees:

cardano-cli transaction build-raw \
	--tx-in <TxHash>#<TxIx> \
	--tx-out <PAYMENT ADDRESS>+0 \
	--invalid-hereafter 0 \
	--fee 0 \
	--out-file tx.raw \
	--certificate-file stake.cert \
	--certificate-file delegation.cert

Upload tx.raw to an online machine that has cardano-node running to calculate fees.
Once you know how much fees will be calculate your change using BALANCE - KEYDEPOSIT - FEES (see procedure if you don't know what pool deposit is - probably 2 Ada).
The build the transaction for real - note the order of certificate is important!

cardano-cli transaction build-raw \
	--tx-in <TxHash>#<TxIx> \
	--tx-out <PAYMENT ADDRESS>+<CHANGE> \
	--invalid-hereafter <SLOT NO> \
	--fee <FEE> \
	--out-file tx.raw \
	--certificate-file stake.cert \
	--certificate-file delegation.cert

Sign it:

cardano-cli transaction sign \
	--tx-body-file tx.raw \
	--signing-key-file payment.skey \
	--signing-key-file stake.skey \
	--mainnet \
	--out-file tx.signed

Upload the signed transaction tx.signed to your online machine and send it!

cardano-cli transaction submit \
	--tx-file tx.signed \
	--mainnet

Withdraw Rewards

Withdrawing rewards requires the recipient wallet to be able to sign the transaction and pay for the fee. So a using CLI may be necessary.

Take note of the rewards balance:

cardano-cli query stake-address-info \
--mainnet \
--address <Your Stake Address (stake.addr)>

Then you need to get information about the wallet you are going to make the transaction from:

cardano-cli query utxo --mainnet --address <Your Payment Address (payment.addr)>

Create the transaction:

cardano-cli conway transaction build --mainnet \
--tx-in <TxHash>#<TxIx> \
--withdrawal <Your Stake Address>+<REWARD BALANCE> \
--change-address <Your Payment Address> \
--out-file withdrawal-tx.raw

Sign the transaction:

cardano-cli transaction sign \
  --tx-body-file withdrawal-tx.raw  \
  --signing-key-file payment.skey \
  --signing-key-file stake.skey \
  --out-file withdrawal-tx.signed

Submit the transaction:

cardano-cli transaction submit \
  --tx-file withdrawal-tx.signed \
  --mainnet

De-registering Certificates

To register the stake key you create a registration certificate which requires a key deposit, to get that deposit back follow the same steps as registering a certificate but use deregistration-certificate instead of registration-certificate and add the key deposit to the change address.

Prometheus

The documentation has pretty good information on how to get this running but I had to change the ip (under Prometheus) in mainnet-config.json to 0.0.0.0.
You may want to setup Prometheus on two different servers for redundancy so if something goes wrong with one you can still get information from the other.

You need to set the data directory otherwise it defaults to ~/data what cardano-node is using --storage.tsdb.path /root/prometheus/data

My prometheus.yml config uses this config:

scrape_configs:
  - job_name: 'cardano' # To scrape data from the block producing node
    scrape_interval: 5s
    static_configs:
      - targets: ['<Block Producing Node IP>:12798']

  - job_name: 'relay' # To scrape data from the relay node
    scrape_interval: 5s
    static_configs:
      - targets: ['<Relay Node IP>:12798']

  - job_name: 'node' # To scrape data from a node exporter to monitor your linux host metrics.
    scrape_interval: 5s
    static_configs:
      - targets: ['<Block Producing Node IP>:9100']

  - job_name: 'prometheus monitor (external)' # To watch relay monitor for problems
    scrape_interval: 5s
    static_configs:
      - targets: ['<External Monitor IP>:9090']

  - job_name: 'prometheus monitor (self)' # To watch self for completeness
    scrape_interval: 5s
    static_configs:
      - targets: ['<Block Producing Node IP>:9090']

Updating

If you want to update your nodes, first check if any additional steps are listed on the cardano-node github release.

Then follow the steps on the official documentation but if there are any issues what I have done is listed below.

Enter the repository you build the node from.

cd cardano-node
git checkout master
git pull
git fetch --tags --all # add '-f' if you get 'clobber' errors)
git checkout <VERSION> # e.g. 1.29.0
git status

That last command should say something like: HEAD detached at <VERSION>

Then you can build the new version:

cabal clean
cabal update
cabal build all

Then you can find the paths to the build executables with: (Be sure to run inside the cardano-node directory.)

find . -regex './dist-newstyle/build/[^/]*/[^/]*/cardano-\(node\|cli\)-[0-9\.]*/x/cardano-\(node\|cli\)/build/cardano-\(node\|cli\)/cardano-\(node\|cli\)'

Cabal/GHC Issues

Check you library versions still match the recommended ones on the page.


You can update Cabal without reinstall like so:

cabal install Cabal cabal-install


The new documentation says to use ghcup and sometimes the build may fail due to cabal/ghc version so to install and set a specific version use:

ghcup install cabal <VERSION>
ghcup set cabal <VERSION>
ghcup install ghc <VERSION>
ghcup set ghc <VERSION>

"cardano-crypto-class" Missing

cabal: Could not resolve dependencies:
[__0] trying: cardano-crypto-class-2.0.0 (user goal)
[__1] rejecting: cardano-crypto-class:+secp256k1-support (conflict: pkg-config
package libsecp256k1-any, not found in the pkg-config database)
[__1] rejecting: cardano-crypto-class:-secp256k1-support (manual flag can only
be changed explicitly)
[__1] fail (backjumping, conflict set: cardano-crypto-class,
cardano-crypto-class:secp256k1-support)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: cardano-crypto-class,
cardano-crypto-class:secp256k1-support

Related issue: https://github.com/input-output-hk/cardano-node/issues/3844.

To fix install libsecp256k1:

git clone https://github.com/bitcoin-core/secp256k1.git
cd secp256k1
./autogen.sh
./configure --enable-module-schnorrsig --enable-experimental
make
make check
make install

"katip" Issues

I have run into katip having version issues to fix add the following line to the cabal.project file at the end of the constraints section (near the end of the file).

, katip <= 0.8.4.0

Update KES

IMPORTANT: Each time you generate a KES the counter increments if you increment any more than what the next number should be the blocks will not be minted - this can be caused by generating new kes keys more than once.

Currently KES needs to be updated every 90 days, gLiveView will tell you how long is left. You can update the KES from a cold machine with Daedalus synced, you need to generate kes.vkey, kes.skey and node.cert to update. Of these 3, only the kes.skey and node.cert need to be uploaded to the server. You need access to cold.skey and cold.counter.


This script will output the needed files in the current working directory, after running save kes.vkey, kes.skey and node.cert to a secure location then copy only kes.skey and node.cert to the block producing node and restart.

https://code.organicdesign.nz/organicdesign/pudim/src/branch/master/update-kes.sh

Using a Remote Node

If you don't have Daedalus running locally or is not up to date you can mount a remote one over ssh using the following command:

sshfs root@server:/remote/path/data /local/path

Modifying the KES Counter

To view the current KES period you can run:

cardano-cli text-view decode-cbor --in-file node.cert  | grep int | head -1

You can generate a new KES Counter (cold.counter) by running:

cardano-cli node new-counter \
--cold-verification-key-file cold.vkey \
--counter-value <KES PERIOD> \
--operational-certificate-issue-counter-file node.counter

Then you can rotate KES using this counter and it will produce a KES for that period and update the file for the next one.

Calculate Leader Logs

They have finally added a way to calculate leader logs using cardano-cli:

https://code.organicdesign.nz/organicdesign/pudim/src/branch/master/calculate-leader-logs.sh

PoolTool

Sending Tip

If you have setup the node without all the extra tools sending your tip is easiest done through the shell/systemd script found here. You can send this info from your relays.


You need to ensure minSeverity logging is set to Notice!


You may also need to modify the cadano-node and cardano-cli if the script is running from systemd:

# CARDANO BINARIES
CNODE=$(command -v cardano-node)
CCLI=$(command -v cardano-cli)

Sending Slots

The easiest way to send slots to pooltool is to use the cncli sendSlots command, but that does not allow you to use a proxy - the below script is how you can send it manually while allowing specification of proxy. You will need run this with the cncli-fivedays.sh script as shown in the installation guide.


Manual sending of slots involve sending your slot count with the hash (Using blake2 NOT sha256!) of the compressed array of slot numbers (e.g. "[123, 456, 789]") and the array of the last epoch's slot numbers.


Ensure that the OUTPUT_DIR exists and you install blake2!

https://code.organicdesign.nz/organicdesign/pudim/src/branch/master/send-slots.sh

Proxy

If you wish to send stats from your block producing node I recommend writing a script on a relay or other server that listens for the request and makes an identical one to pooltool so that the identity of your block producing node is not compromised. An example of such a script can be found here.

CNCLI

The prebuilt CNCLI release does not work with Debian 10 so you need to build from source.


If cncli is missing after build:

cp /root/.cargo/bin/cncli /usr/local/bin/cncli

Debugging

Missing bin-path script (cardano-node build location)

I found the second time I build cardano-node the script that outputs the build was missing, so if you need to find it manually run:

find ./dist-newstyle -type f -name cardano-node
find ./dist-newstyle -type f -name cardano-cli

The locations when I ran it was:

./dist-newstyle/build/x86_64-linux/ghc-8.10.2/cardano-node-1.26.2/x/cardano-node/build/cardano-node/cardano-node
./dist-newstyle/build/x86_64-linux/ghc-8.10.2/cardano-cli-1.26.2/x/cardano-cli/build/cardano-cli/cardano-cli


Era Mismatch

If you get errors running some commands saying there is an era mismatch like the following:

Shelley command failed: query utxo  Error: A query from a certain era was applied to a ledger from a different era: EraMismatch {ledgerEraName = "Allegra", otherEraName = "Shelley"}

This is because the ledger is on the "Allegra" soft fork while the cli is running "Shelley", you need to either update the cli and/or add a parameter to the command specifying the era like so:

cardano-cli query utxo \
	--address $(cat payment.addr) \
	--mainnet \
	--allegra-era

Cardano Node Socket

If you have issues with cli commands referencing the cardano node socket, first ensure that the node is running (it can take a minute to generate the node.socket file in the database, then ensure the environment variable has been set.

export CARDANO_NODE_SOCKET_PATH=path/to/db/node.socket

Transactions

If your transaction fails due to a input-output mismatch, you need to check that the inputs (--tx-in) values sum to the output values (--tx-out) plus the fee's (--fee).

If your transaction fails due to output too small, you need to check that all the outputs (including change addresses) have the minimum value as set in the "minUTxOValue" parameter in the genesis block config (1000000 Lovelace or 1 Ada at the time of writing).

Floating IP/DNS

Floating IP's for some reason will not work with cardano-node so if you want the flexibility of floating IP's use DNS instead (with --single-host-pool-relay).

Guild Live View

I had an issue where the gLiveView script suddenly failed to connect to a running instance with the following error message:

COULD NOT CONNECT TO A RUNNING INSTANCE, 3 FAILED ATTEMPTS IN A ROW!

This problem appears to be with how the monitor records uptime, this should be fixed when cardano-node version 1.26 is released. A temporary fix is to prevent it from throwing the error, to do this change line 468 from:

if [[ ${uptimens} -le 0 ]]; then

To:

if false; then

Staking from a paper wallet

todo... I'll document this properly after I've done it myself

The idea is to first temporarily install a Daedalus on an offline live booted Linux and then restore a paper wallet into it with the 24 word backup phrase. Then we should be able to interact with the running wallet's associated node from the CLI to create and sign a delegation transaction. That transaction can then be broadcast to the network using any online Cardano node. This Shelley exercise explains the process of CLI delegation.

First you'll need to find the port that your Daedalus is running on:

ps x | grep cardano-node

Then you'll need to create the key files:

cardano-cli shelley address key-gen \
--verification-key-file payment.vkey \
--signing-key-file payment.skey \
--host-addr 127.0.0.1 \
--port PORT

Todo: We need to use a registered stake address that we have the keys for.

Create a delegation certificate, here the pool is the characters after the 5820 in the cborHex of the target pool's cold.vkey file.

cardano-cli shelley stake-address delegation-certificate \
--stake-verification-key-file stake.vkey \
--cold-verification-key-hash POOL \
--out-file delegation.cert \
--host-addr 127.0.0.1 \
--port PORT


Draft the transaction:

cardano-cli shelley transaction build-raw \
--tx-in <UTXO>#<TxIx> \
--tx-out $(cat payment.addr)+0 \
--ttl 0 \
--fee 0 \
--out-file tx.draft \
--certificate-file delegation.cert


Calculate the fees:

cardano-cli shelley transaction calculate-min-fee \
--tx-body-file tx.draft \
--tx-in-count 1 \
--tx-out-count 1 \
--mainnet \
--witness-count 1 \
--byron-witness-count 0 \
--protocol-params-file protocol.json


Build the transaction:

cardano-cli shelley transaction build-raw \
--tx-in <UTXO>#<TxIx> \
--tx-out $(cat payment.addr)+<CHANGE IN LOVELACE> \
--ttl <TTL> \
--fee <FEE> \
--out-file tx.raw \
--certificate-file pool-registration.cert \
--certificate-file delegation.cert


Sign the transaction:

cardano-cli shelley transaction sign \
--tx-body-file tx.raw \
--signing-key-file payment.skey \
--signing-key-file stake.skey \
--signing-key-file cold.skey \
--mainnet \
--out-file tx.signed


Submit the transaction:

cardano-cli shelley transaction submit --tx-file tx.signed --mainnet

Admin

If you want to quickly assess the state of the node run /root/scripts/gLiveView.sh (make sure to answer no to updating anything) and check that:

  • Tip is up to date
  • It has a peer for every relay server it is connected to.
  • KES shows and hasn't expired.
  • Blocks are not showing as invalid.

Services

Most of the operation is handled by a couple systemd services - if anything has gone wrong you should check these are running.

  • cardano.service - This service handles running the cardano node, it logs to the file /root/debug.log, if this is not running the node is probably not running and you will not be minting blocks.
  • sendmytip.service - This service just runs the /root/scripts/sendmytip.sh script for sending tip to pooltool.
  • cncli-sync.service - This service just keeps cncli in sync so leader logs can be requested - if this is not running or has not fully synchronized then leaderlogs will fail.
  • prometheus.service - This runs the prometheus monitor for debugging.
  • node-exporter.service - This is just for providing system details for prometheus.

Scripts

Startup

All startup scripts are located in the /root directory.

  • start.sh - This starts the cardano block producing node and is run by the 'cardano' service.
  • start-prometheus.sh - This just starts the prometheus debugger and is run by the 'prometheus' service.
  • start-node-exporter.sh - This runs the node exporter to provide system details for prometheus and is rung by the 'node-exporter' service.
  • start-as-node.sh - This script starts the cardano block producing node as a normal (relay) node, so it shouldn't be used outside debugging.

Status

All scripts related to monitoring the status and health of the cardano node is found in the /root/scripts directory.

  • cncli-fivedays.sh - This is run by a cron job to figure out if it is the epoch border or not.
  • sendSlots.sh - This obtains the leaderlogs, saves them for debugging and sends them to pooltool. It is run by the cron job if it passes the check to detect the epoch border. This script will fail if the cncli-sync.service has failed or not fully synchronized.
  • sendmytip.sh - This just obtains the current block tip from the cardano log and sends it to pooltool on regular intervals. It is run by the sendmytip.service.
  • gLiveView.sh - This script provides an interface to view the current state of the cardano node. It will produce a few errors about cardano node not running or being out of date - ignore and don't update (new versions don't work for some reason) and you should be able to see an interface for the node's current state.

Leaderlogs

The leaderlogs get saved to files in /root/leaderlogs for inspection at a later date. Each epoch creates 3 logs:

  • XXX_logs_full.json - This is the one that's most useful to inspect as it provides the count and time of blocks for this and earlier epochs
  • XXX_logs.json - This is an array of the slots that the blocks are assigned to that gets sent to pooltool after it is one epoch old.
  • XXX_logs.hash - This is the hash of the above array that gets sent to pooltool within an hour of the epoch starting. (This and the above info provides a way of proving you are honest about assigned blocks.)

If these files are missing for the current epoch it means the leaderlog generation failed for some reason and /root/scripts/sendSlots.sh should be run manually.

Resources

See also