Troubleshooting

Detailed guide for troubleshooting node problems.

Knowing common errors and their solutions helps keep your node healthy.

Common error codes

These are the most frequent errors and their solutions:

Consensus errors

If you get a consensus error, act quickly and appropriately:

Error: "Consensus failure - height halted"
Solution: Check for network upgrades or chain halts
Command: seid status

Error: "Private validator file not found"
Solution: Restore validator key or check file permissions
Location: $HOME/.sei/config/priv_validator_key.json

Error: "Duplicate signature"
Solution: IMMEDIATELY STOP NODE - potential double signing risk
Action: Check validator operation on other machines

Network errors

Network errors can prevent your node from participating in consensus:

Error: "Dial tcp connection refused"
Solution: Check network connectivity and firewall rules
Commands:
  - netstat -tulpn | grep seid
  - ufw status

Error: "No peers available"
Solution: Verify peer connections and network config
Commands:

- curl localhost:26657/net_info

Database errors

Database corruption can require immediate attention:

Error: "Database is corrupted"
Solution: Reset database or restore from backup
Commands:
  - seid tendermint unsafe-reset-all
  - cp -r backup/data $HOME/.sei/

Diagnostic commands

These commands help you investigate problems and monitor your node:

# Check node synchronization
seid status

# Check validator status
seid query staking validator $(seid tendermint show-validator)

# Monitor real-time logs
journalctl -fu seid -o cat

# View system resource usage
top -p $(pgrep seid)

AppHash mismatch errors

If you get an AppHash mismatch, capture the state to compare it with a known-good version:

# For SeiDB (all supported nodes):
git clone https://github.com/sei-protocol/sei-db.git
cd sei-db/tools
make install
systemctl stop seid
seidb dump-iavl -d $HOME/.sei/data/committer.db -o /home/ubuntu/iavl-dump
systemctl restart seid

As with FlatKV, the memIAVL store has two possible locations. Nodes created before the layout change keep the legacy $HOME/.sei/data/committer.db path shown above (it takes precedence when present). New nodes use $HOME/.sei/data/state_commit/memiavl. Point -d at whichever path exists on your node.

On Giga Storage nodes, EVM state lives in a FlatKV store instead of the memIAVL trees. An AppHash comparison there also requires a FlatKV state dump. Use the dump-flatkv command to iterate and dump the physical (key, value) pairs into per-bucket files. The files match the dump-iavl format, so the same diff tooling works on both:

# For FlatKV (Giga Storage nodes hold EVM state in FlatKV):
systemctl stop seid
seidb dump-flatkv --db-dir $HOME/.sei/data/state_commit/flatkv --output-dir /home/ubuntu/flatkv-dump
systemctl restart seid

Nodes created before the storage layout change keep the FlatKV store at the legacy path $HOME/.sei/data/flatkv instead of $HOME/.sei/data/state_commit/flatkv. Pass whichever directory exists on your node to --db-dir.

The dump-flatkv command accepts these flags:

• --db-dir (-d): The FlatKV database directory.
• --output-dir (-o): The output directory, with one file for each bucket. Required unless --lthash-only is set.
• --height: The FlatKV target version. The default, 0, selects the latest available version.
• --bucket (-b): Restrict the dump to a single bucket (account, code, storage, or legacy). The default is all buckets. This only filters which files are written; the full keyspace is always scanned, and the LtHash always covers all four buckets.
• --lthash: Also compute the per-bucket and total LtHash (lattice hash) over the scanned state, and verify the total against the committed root recorded in snapshot metadata. The default is true. A mismatch exits non-zero.
• --lthash-only: Compute and verify the LtHash without writing any bucket dump files. Requires --lthash=true, and does not require --output-dir. The default is false.
• --read-limit-mb: Throttle the scan to at most this many MiB/s of (key+value) bytes read, so a dump against a running node does not starve the chain of disk bandwidth. The default is 64. Set 0 for unlimited. Keep it at the default or lower on a shared/live node; raise it only for offline runs on idle disks.

Because dump-flatkv opens an independent read-only clone of the store (the snapshot is hard-linked and the changelog WAL is replayed into a temp directory), it is safe to run against a live, block-producing node, and the --read-limit-mb throttle prevents it from starving the node of disk bandwidth.

For example, to dump only the storage bucket at a specific version:

seidb dump-flatkv --db-dir $HOME/.sei/data/state_commit/flatkv --output-dir /home/ubuntu/flatkv-dump --height 12345678 --bucket storage

To verify the FlatKV lattice hash against the committed snapshot metadata without writing any key/value dump files:

seidb dump-flatkv --db-dir $HOME/.sei/data/state_commit/flatkv --lthash-only

For an offline run on an idle disk, go full speed and skip the LtHash:

seidb dump-flatkv --db-dir $HOME/.sei/data/state_commit/flatkv --output-dir /home/ubuntu/flatkv-dump --read-limit-mb 0 --lthash=false

With --lthash, the output prints a per-bucket count and checksum plus a TOTAL LtHash, followed by a LtHash verification vs snapshot metadata PASS/FAIL line. Verification is skipped (rather than failing) when the selected snapshot predates LtHash metadata, since the committed hash then covers only replayed WAL deltas rather than full state.

Comparing EVM state between memIAVL and FlatKV

When you debug an AppHash mismatch that involves EVM state, a byte-for-byte physical dump can diverge between backends, even when the underlying state is identical. This happens because every FlatKV value embeds a per-key block-height stamp (the height at which the key was last written or migrated). On a freshly migrated node, this stamp differs from the memIAVL leaf versions.

The evm-logical-digest command works around this with a backend-independent digest of the EVM logical state. On both sides, it strips the serialization-version and block-height header. Then it digests only the logical payload: account balance, nonce, and code hash, plus bytecode and storage words. With this digest, you can compare a memIAVL node and a FlatKV node at the same chain height:

# FlatKV digest at a height (WAL-replays to it):
seidb evm-logical-digest --backend flatkv \
    --db-dir $HOME/.sei/data/state_commit/flatkv --height 213200000

# memIAVL digest at the same height (0 = current symlink):
seidb evm-logical-digest --backend memiavl \
    --db-dir $HOME/.sei/data/state_commit/memiavl --height 213200000

The --backend flag accepts flatkv, memiavl, and composite.

For memIAVL, --memiavl-open-mode controls how leaves are read:

• snapshot (the default) sequentially scans the completed snapshot kvs file at snapshot-<height>/evm. This is the fast path and requires an on-disk snapshot at that exact height (or --height 0 for the current symlink). Prefer it whenever your target height matches an existing snapshot boundary.
• replay opens a read-only DB, replays the changelog up to --height, then walks the in-memory/mmap tree. It is roughly an order of magnitude slower than snapshot, so use it only when no snapshot exists at the target height (for example, when a node’s snapshot rewrite lags the tip):

# memIAVL digest for a height with no on-disk snapshot (slower):
seidb evm-logical-digest --backend memiavl --memiavl-open-mode replay \
    --db-dir $HOME/.sei/data/state_commit/memiavl --height 213205000

On a node that is mid-migration, EVM state is split between the two backends: FlatKV holds the rows already migrated, while memIAVL still holds the rows not yet past the migration boundary. Use --backend composite to digest the union of both, so a migrating node can be compared against a memIAVL-only node at the same height. In composite mode, --db-dir is not required; instead provide the two backend directories with --flatkv-dir and --memiavl-dir. Because a live migrating node usually keeps memIAVL snapshots at heights outside FlatKV’s retained window, pass --memiavl-open-mode replay:

# Mid-migration node: digest the flatkv + memiavl union at one height:
seidb evm-logical-digest --backend composite --memiavl-open-mode replay \
    --flatkv-dir $HOME/.sei/data/state_commit/flatkv \
    --memiavl-dir $HOME/.sei/data/state_commit/memiavl --height 213200000

Each run prints per-bucket bucket_digest values and a single FINAL_DIGEST line that covers the account, code, storage, and legacy buckets. Compare the FINAL_DIGEST lines from both backends at the same height. They should match. FlatKV can contain FlatKV-only migration marker rows that a memIAVL-only node never owns: the migration-version marker (present once a migration completes) and the migration-boundary cursor (present only while a migration is in flight). The command automatically omits both rows from the final result, so memIAVL, mid-migration, and completed nodes all produce comparable digests.

For targeted debugging, the command can also inspect a single normalized bucket instead of printing the global digest. The seidb tooling section of the technical reference has the full flag reference, including inspect mode, sharding, and --find-hash. These examples list the first 50 account rows with version metadata, and shard the storage bucket under a key prefix by the next 2 bytes:

seidb evm-logical-digest --backend flatkv -d $HOME/.sei/data/state_commit/flatkv --height 213200000 \
    --inspect-bucket account --list --list-limit 50 --details

seidb evm-logical-digest --backend flatkv -d $HOME/.sei/data/state_commit/flatkv --height 213200000 \
    --inspect-bucket storage --key-prefix 03 --shard-next-bytes 2

Enable SeiDB State Commit (SC). Set sc-enable = true in the [state-commit] section of app.toml. The legacy IAVL backend was fully removed, and SC is now mandatory. If SC is not enabled, the node no longer falls back to IAVL. It panics at startup with this error:

SeiDB state-commit (SC) must be enabled; IAVL backend has been fully deprecated

The seid debug dump-iavl command was also removed with the IAVL backend. To inspect state, use the seidb dump-iavl tool shown above.

When you report a problem, always include the app hash, commit hash, and block height from your logs.

Identifying AppHash errors

In logs, AppHash errors usually look like this:

ERR wrong Block.Header.AppHash. Expected [EXPECTED_HASH], got [ACTUAL_HASH]
block_id={"hash":"...","parts":{"hash":"...","total":1}} height=[HEIGHT]

Common causes:

• Using an incorrect node version during sync (make sure that you run the latest version)
• Corrupted or incorrectly applied snapshots
• Database inconsistencies from improper shutdowns
• Syncing with outdated or incompatible peers

Resolution steps:

1. Stop the node immediately.

2. Try a node rollback first. See Node rollback.

3. If the rollback fails, restore from a fresh snapshot:

   • Download a recent snapshot from trusted providers (Polkachu, PublicNode)
   • Make sure that you use the correct node version
   • Verify that peer configurations are up to date

4. Restart the node and monitor the logs for continued errors.

Peer connection issues as AppHash red herrings

Important: Peer connection failures are often symptoms of underlying AppHash errors, not the root cause.

If you see many peer connection errors like these:

ERR failed to handshake with peer
ERR failed to send request for peers
ERR peer handshake failed endpoint={} err=EOF

Do not focus only on fixing peer connections first. Instead:

1. Scan your logs carefully for AppHash errors that may appear intermittently
2. Look for the actual error pattern:
   ERR wrong Block.Header.AppHash. Expected [HASH], got [HASH]
   
3. Check whether your node is stuck at a specific height despite peer connection attempts

Why this happens:

• AppHash mismatches prevent proper block validation
• The node cannot advance to new blocks because of a state inconsistency
• Peers may reject connections from nodes with corrupted state
• The network appears to be the problem, but the cause is a local state issue

Debugging approach:

1. First, check for AppHash errors in your logs (search for “wrong Block.Header.AppHash”)
2. If you find AppHash errors, treat them as the primary issue
3. Focus on peer connection fixes only if no AppHash errors exist

This approach targets the root cause, not the symptoms, and can save hours of debugging time.

Peer connection and handshake issues

Identifying peer issues:

Look for these error patterns in your logs:

ERR failed to handshake with peer err="expected to connect with peer \"[EXPECTED_ID]\", got \"[ACTUAL_ID]\""
ERR failed to send request for peers err="no available peers to send a PEX request to (retrying)"
ERR peer handshake failed endpoint={} err=EOF module=p2p

Common causes:

• Outdated peer configurations with mismatched node IDs
• Network infrastructure changes on the peer side
• A firewall that blocks connections on port 26656
• DNS resolution issues

Resolution steps:

1. Update peer configurations with current node IDs:

2. Verify network connectivity:

   # Test connection to peer endpoints
   nc -zv p2p.state-sync-0.pacific-1.seinetwork.io 26656
   
   # Check if port 26656 is open for inbound connections
   netstat -tulpn | grep :26656
   

3. Check the current peer status:

   curl http://localhost:26657/net_info | jq '.result.peers | length'
   curl http://localhost:26657/lag_status | jq .
   

Sync performance issues

Identifying sync problems:

Monitor these indicators:

# Check sync status and lag
curl http://localhost:26657/lag_status | jq .

# Monitor if height is progressing
curl http://localhost:26657/status | jq '.result.sync_info'

Common solutions:

1. Increase the packet payload size for large block processing:

   # In config.toml [p2p] section
   max-packet-msg-payload-size = 1024000  # Increase from default 102400
   

2. Optimize the mempool settings in config.toml:

   # In [mempool] section
   keep-invalid-txs-in-cache = true
   ttl-duration = "5s"
   ttl-num-blocks = 5
   

3. If the node gets stuck at a specific height:

   • Try restarting the node
   • If that does not help, perform a rollback
   • Consider taking a fresh snapshot

Warning signs to watch for:

• The current height does not increase over time
• Increasing lag between the current height and the max peer height
• Repeated timeout errors in the logs
• Mempool size that consistently reaches its limits

Crash and panic debugging

For crashes, panics, or nil pointer exceptions:

• Capture at least 1,000 lines of logs before the crash or 15 minutes of log data, whichever gives more context
• Include the full stack trace, if it is available

Logging configuration

Proper logging configuration is essential for debugging and monitoring:

# In config.toml
# Set appropriate log level
log_level = "debug"  # Use "trace" for maximum detail

# Choose log format
log_format = "json"  # Use "plain" for human-readable logs

Configure log rotation to manage storage:

# Example logrotate configuration
sudo tee /etc/logrotate.d/seid << EOF
/var/log/seid/*.log {
    daily
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 sei sei
    sharedscripts
    postrotate
        systemctl reload seid
    endscript
}
EOF

Enable core dumps for crash analysis:

# Set unlimited core dump size
ulimit -c unlimited

# Configure core dump location
echo "/tmp/core.%e.%p" > /proc/sys/kernel/core_pattern

Other common issues and fixes

1. Sync problems

   • Check available disk space (df -h)
   • Make sure that peer connections work (curl http://localhost:26657/net_info)
   • Check that the firewall allows port 26656

2. Performance issues

   • Monitor system resources (htop or iotop)
   • Check disk I/O performance (iostat)
   • Analyze network traffic (iftop)

3. Database issues

   • Run database integrity checks:

     seid debug dump-db | grep -i error
     

     If you find errors, consider restoring from a recent backup.

   • To keep less historical data, lower ss-keep-recent in app.toml.

   • To rebuild the node with a smaller database, reset it. Then resync from a snapshot or with state sync. The reset deletes all chain data, so it is not a pruning tool. Before you reset, back up priv_validator_key.json and priv_validator_state.json, as described in Clean up:

     seid tendermint unsafe-reset-all --home $HOME/.sei --keep-addr-book
     

     Alternatively, remove old state snapshots manually to free disk space:

     rm -rf $HOME/.sei/data/snapshots/*
     

Node rollback

To roll back a node from an AppHash mismatch, first stop the node in your preferred way.

Next, roll back the node:

seid rollback

Then, restart the node.

If you see this error when you try to roll back:

failed to initialize database: resource temporarily unavailable

This means that you did not shut down the node properly. In that case, try to shut down or kill the seid process directly. If this does not help, restart your machine.

Then try the rollback steps again.

HashVault app-hash equivocation panic

Autobahn validators and fullnodes run an app-hash equivocation guard called HashVault, enabled by default. It stores the app hashes the node has committed in a durable database under <PersistentStateDir>/hashvault. If the node is ever about to commit a different app hash for a height it has already finalized, HashVault halts the node to prevent it from “changing its mind” about a finalized block.

When the guard trips, you will see a panic in the logs similar to:

Hashvault detected app hash mismatch; node attempted to change its mind. DO NOT RESTART WITHOUT HUMAN INVESTIGATION. ...
blockHeight=[HEIGHT] existingHex=[...] incomingHex=[...] hashVaultDir=[/path/to/hashvault]

Do not restart the node without human investigation. A HashVault mismatch can indicate a genuine equivocation. Restarting blindly, or removing the guard, can cause your validator to externalize a conflicting hash for an already-finalized height, which risks slashing.

Investigate first. Determine whether the mismatch reflects a real equivocation (for example, the same validator key running on more than one machine, or committed state that diverged from what the vault recorded) or a benign situation such as an out-of-band rollback or restore that left the committed app state inconsistent with the vault’s history.

Recovery (only if you are certain there is no real equivocation)

If, and only if, you are certain the stored hashes are wrong and there is no real equivocation, you can bypass the guard:

1. Stop the node.
2. Delete the HashVault data directory shown as hashVaultDir in the panic message (by default <PersistentStateDir>/hashvault).
3. Restart the node. It starts with an empty equivocation history and re-commits hashes as it re-executes blocks.

Deleting this directory removes equivocation protection for the heights it covered. If the node then commits a conflicting hash for a height it has already finalized, the validator may be slashed.

Last-resort: disabling HashVault

If you repeatedly hit the same panic on new blocks and are very sure the stored hashes are totally wrong, you can run the node with the guard disabled by setting the top-level hash-vault-disabled-unsafe field in config.toml:

# config.toml (top-level, before any [section] header)
hash-vault-disabled-unsafe = true

This runs the node without any app-hash equivocation protection and logs error-level warnings on every startup. The default is false, and you should re-enable the guard (set it back to false or remove the line) as soon as the underlying issue is resolved.