Two Failures After a Shiden Node Downgrade: RocksDB Lock Contention and a litep2p Panic

This incident started with an ordinary EC2 instance downgrade.

The Shiden archive RPC node was at the chain head before the change. Its peers, RPC responses, container state, and host checks all looked healthy. After the host restarted, two errors appeared in sequence:

IO error: While lock file: /data/chains/shiden/db/full/LOCK:
Resource temporarily unavailable

After the lock was released, the client failed again:

Thread 'tokio-runtime-worker' panicked at
'SelectNextSome polled after terminated'

The recovery did not delete LOCK, reset the database, or rebuild terabytes of chain data. It identified the live lock owner, stopped an unintended backup container, and then recreated the primary container without replacing its bind-mounted datadir.

This is evidence from one incident, not a claim that every Astar client version or deployment fails in the same way.

The node was healthy before the downgrade

I captured four groups of checks before stopping the instance:

EC2 system, instance, and EBS status checks were healthy.
The primary container had no restarts and had not been OOM-killed.
system_health reported peers and isSyncing: false.
system_syncState showed currentBlock equal to highestBlock, while logs kept importing new blocks.

That baseline mattered. Without it, the post-reboot lock error could easily be mistaken for an already-corrupt database.

An instance downgrade interrupts processes, the Docker daemon, and network sessions. It does not intentionally rewrite a bind-mounted chain database, but it does expose every retained container, restart policy, and startup-order assumption.

Failure one: the backup container acquired the RocksDB lock

The primary repeatedly exited with:

Error: Service(Client(Backend(
  "IO error: While lock file: /data/chains/shiden/db/full/LOCK:
   Resource temporarily unavailable"
)))

RocksDB allows a database directory to be opened by one process at a time and uses an operating-system lock to prevent unsafe concurrent access ^[1].

The useful question was not “How do I remove LOCK?” It was “Which process owns it?”

I first listed containers and collator processes:

docker ps -a --format 'Name={{.Names}} Status={{.Status}}'
pgrep -af astar-collator

Then I asked the operating system for the lock owner:

lsof /data/chains/shiden/db/full/LOCK

# Use fuser when lsof is unavailable
fuser -v /data/chains/shiden/db/full/LOCK

The PID belonged to node-backup, not the expected node-primary. The backup was an artifact from an earlier recovery attempt, but it still retained a restart policy. After the host came back, it was running and had opened the same /data first.

Docker restart policies have distinct exit and daemon-lifecycle behavior. A name containing backup carries no operational meaning; the live policy and state are what matter ^[2].

Why the node still looked partly healthy

The confusing part was that RPC had not disappeared completely.

system_health still returned peers and isSyncing: false. system_syncState showed a node at the chain head, and the backup logs continued to import Shiden parachain and Kusama relaychain blocks.

That could have ended the investigation too early.

An EVM call told a different story:

curl -sS -H 'Content-Type: application/json' \
  --data-binary '{"jsonrpc":"2.0","id":1,"method":"eth_chainId","params":[]}' \
  http://127.0.0.1:9944

It returned:

{
  "error": {
    "code": -32601,
    "message": "Method not found"
  }
}

The backup served Substrate RPC but lacked the primary’s --enable-evm-rpc flag. Astar documents that EVM RPC is disabled by default and must be enabled explicitly ^[3].

One healthy RPC method does not prove that every required RPC namespace on the same port is available.

Resolve lock contention without touching chain data

The safe order was to neutralize the backup’s restart behavior, stop it gracefully, and verify that the lock had been released.

Inspect the live state first:

docker inspect node-backup \
  --format 'Running={{.State.Running}} Policy={{.HostConfig.RestartPolicy.Name}}'

Disable restart and stop the container:

docker update --restart=no node-backup
docker stop --timeout 120 node-backup

Then check the lock again:

lsof /data/chains/shiden/db/full/LOCK

Only after no process owns the lock is it safe to start another container against the same datadir.

Deleting LOCK is not a repair. If another RocksDB process is alive, removing the file does not make concurrent writers safe; it removes a valuable diagnostic signal.

Failure two: a litep2p panic after the lock was released

The lock error disappeared, but the old primary still failed during litep2p startup:

[Relaychain] Running litep2p network backend
[Relaychain] Essential task `basic-block-import-worker` failed.
Thread 'tokio-runtime-worker' panicked at
'SelectNextSome polled after terminated'

At this point, calling the incident “the same lock problem” would have been inaccurate. lsof showed no competing owner, and the original error signature was gone.

I reduced the test to three comparisons:

Disable automatic retries, wait, and cold-start the old primary once.
Keep the same image, launch arguments, and bind-mounted /data.
Replace only the container metadata and Docker network state.

The old container reproduced the panic after a cooldown. A newly created container with the same image, arguments, and datadir started successfully.

The defensible conclusion is limited: the evidence points to bad runtime or network state associated with the old container, and recreating it removed that state. This incident alone does not prove the exact internal litep2p cause.

Recreate the container, not the database

This boundary was safe because the chain data lived in a host bind mount rather than the container writable layer. Docker bind mounts expose a host path inside the container, allowing a new container to mount the same persistent data ^[4].

Verify the mount before removing anything:

docker inspect node-primary --format '{{json .Mounts}}'

After confirming that the host /data is mounted at container /data, remove only the stopped container:

docker rm node-primary

This is a sanitized version of the recovery command. The 96g memory limit came from this deployment’s downgraded host and should be recalculated for other systems:

docker run -d \
  --name node-primary \
  --restart=on-failure:3 \
  --memory=96g \
  --memory-swap=96g \
  -p 30333:30333 \
  -p 9944:9944 \
  --mount type=bind,src=/data,dst=/data \
  staketechnologies/astar-collator:latest \
  astar-collator \
  --pruning archive \
  --blocks-pruning archive \
  --enable-evm-rpc \
  --name shiden-rpc \
  --chain shiden \
  --base-path /data \
  --rpc-external \
  --rpc-methods Safe \
  -- \
  --database paritydb \
  --sync warp \
  --state-pruning 256 \
  --blocks-pruning 256

If the new container fails, the rollback still should not mutate /data: stop the new container, verify lock release, and start the last known working container.

Prove the node is actually recovered

I used the original error signatures as the regression check rather than treating a successful docker start as proof.

Check container state:

docker inspect node-primary \
  --format 'Running={{.State.Running}} Restarts={{.RestartCount}} OOM={{.State.OOMKilled}}'

Confirm that the primary collator owns the lock:

lsof /data/chains/shiden/db/full/LOCK

Scan for both failures:

docker logs --since 10m node-primary 2>&1 | \
  grep -E 'While lock file|SelectNextSome polled after terminated|Essential task.*failed'

Verify Substrate RPC:

curl -sS -H 'Content-Type: application/json' \
  --data-binary '{"jsonrpc":"2.0","id":1,"method":"system_health","params":[]}' \
  http://127.0.0.1:9944

curl -sS -H 'Content-Type: application/json' \
  --data-binary '{"jsonrpc":"2.0","id":1,"method":"system_syncState","params":[]}' \
  http://127.0.0.1:9944

Then verify EVM RPC:

curl -sS -H 'Content-Type: application/json' \
  --data-binary '{"jsonrpc":"2.0","id":1,"method":"eth_chainId","params":[]}' \
  http://127.0.0.1:9944

curl -sS -H 'Content-Type: application/json' \
  --data-binary '{"jsonrpc":"2.0","id":1,"method":"eth_blockNumber","params":[]}' \
  http://127.0.0.1:9944

The final green signals were:

The primary stayed running with zero restarts and no OOM kill.
Every backup was stopped with restart policy set to no.
The primary process owned the RocksDB lock.
isSyncing was false and currentBlock equaled highestBlock.
Shiden and Kusama logs kept importing blocks.
Both eth_chainId and eth_blockNumber returned valid results.

Reusable downgrade and recovery checklist

Before the downgrade

Record instance type, container ID, image digest, launch arguments, and mounts.
Capture system_health, system_syncState, and EVM RPC responses.
List every container that references /data, not just the expected primary.
Set backup containers to restart=no.
Confirm that stop/start will not replace or delete the data volume.

When a LOCK error appears

Use lsof or fuser to identify the real owner.
Correlate pgrep, docker ps -a, and docker inspect.
Do not delete LOCK or run two writable datadir users.
Stop the unintended process and verify release before continuing.

When the primary still fails

Treat the new signature separately from the resolved lock error.
Disable automatic retry and run one controlled cold start.
Inspect OOM evidence, exit code, Docker events, and the first fatal or essential-task log line.
Once the mount is proven, prefer recreating container state over rebuilding chain data.

After recovery

Verify both Substrate and EVM RPC.
Watch block imports over time instead of sampling one height.
Check restart count, OOM state, resources, disk, and lock ownership.
Keep rollback containers stopped and configured with restart=no.

What this incident exposed

The downgrade did not corrupt the database. It exposed a lifecycle gap: a backup container had not been made operationally inert.

A renamed container retains its runtime configuration. Any process that unexpectedly mounts the same writable datadir can change the primary’s startup result. Worse, a backup can serve enough RPC to create a false healthy signal.

I now treat recovery as four independent layers:

Host and disk health.
Exactly one process owning the database lock.
Parachain and relaychain progress.
Every RPC namespace required by consumers.

The node is recovered only when all four layers pass.