cloud-sre
When Lighthouse Says InvalidToken but Erigon Returns Forbidden
A field note on fixing a Gnosis Erigon and Lighthouse Engine API connection that looked like a JWT problem but was actually a vhost mismatch.
A Gnosis archive node was running Erigon as the execution layer and Lighthouse as the consensus layer. Erigon kept printing a warning like this:
flag --externalcl was provided, but no CL requests to engine-api in 22h...
Lighthouse was running, but its logs repeated:
Failed jwt authorization error: InvalidToken
Execution engine call failed error: Auth(InvalidToken)
HTTP status client error (403 Forbidden) for url (http://gnosis:8551/)
At first glance this looked like a bad JWT secret. It was not. The final fix was to allow the Docker service name in Erigon’s Engine API vhost list:
--authrpc.vhosts=gnosis,localhost,127.0.0.1
Terms used here
| Term | Meaning |
|---|---|
| Execution layer | The client that stores and executes blocks. In this setup it was Erigon. |
| Consensus layer | The beacon client that follows the proof-of-stake chain and drives fork choice. In this setup it was Lighthouse. |
| Engine API | The authenticated API between consensus and execution clients. It commonly listens on port 8551. |
| JWT secret | A shared secret file used by the consensus client to authenticate to the Engine API. |
| authrpc vhost | Erigon’s allowed host list for authenticated RPC requests. A valid JWT is not enough if the request host is rejected. |
| Checkpoint sync | A Lighthouse fast-start path that loads a recent finalized checkpoint instead of syncing the beacon chain from genesis. |
What the logs showed
Erigon was started with an external consensus layer:
--externalcl
--authrpc.addr=0.0.0.0
--authrpc.port=8551
--authrpc.jwtsecret=/jwt/jwt.hex
That matches the Erigon model: Erigon can run with its own embedded consensus layer, but --externalcl tells it to use an external consensus client. Erigon’s README also says Gnosis needs a consensus layer client alongside Erigon, and if the consensus client is not local you should expose authrpc and configure --authrpc.vhosts for the consensus host [1].
Lighthouse was pointed at the Docker service name:
--execution-endpoint http://gnosis:8551
--execution-jwt /jwt/jwt.hex
So the request had two parts: the JWT and the HTTP host. In this case the host was gnosis:8551.
The JWT was checked first
The obvious failure mode was a token mismatch, so the first checks were around the shared secret:
sha256sum /data/jwt/jwt.hex
docker exec gnosis sha256sum /jwt/jwt.hex
docker exec gnosis-lighthouse sha256sum /jwt/jwt.hex
The host file and both container mounts matched.
I also sent a direct Engine API request using a short-lived JWT generated from the same file. That returned HTTP 200 for engine_exchangeCapabilities. That check narrowed the problem: Erigon could validate the token, and the Engine API was reachable. The repeated InvalidToken in Lighthouse was hiding a 403 rejection path.
Updating Lighthouse exposed two separate issues
The running Lighthouse image was old, so I pinned the image instead of relying on a floating tag:
sigp/lighthouse:v8.2.0
The v8.2.0 release was current at the time and marked high priority by the Lighthouse project [2].
That upgrade uncovered an old database problem:
Unable to open database: LoadConfig(ConfigError(InvalidVersionByte(Some(0))))
The execution data lived under Erigon’s datadir and was not touched. The old Lighthouse datadir was moved aside, and Lighthouse was restarted with Gnosis checkpoint sync:
--checkpoint-sync-url https://checkpoint.gnosischain.com
Gnosis documents that checkpoint server for Lighthouse, and Lighthouse’s own docs recommend checkpoint sync because it is much faster than syncing from genesis while providing the same node functionality [3][4].
One more cleanup was needed. Newer Lighthouse rejects this combination:
--checkpoint-sync-url ...
--allow-insecure-genesis-sync
With checkpoint sync enabled, the old genesis-sync flag had to be removed.
The actual connection fix
After Lighthouse v8.2.0 started cleanly, it still returned:
InvalidToken("HTTP status client error (403 Forbidden) for url (http://gnosis:8551/)")
That was the useful part: 403 Forbidden, not a connection timeout.
The execution endpoint used the Docker service name gnosis. Erigon was listening on all interfaces, but the authenticated RPC vhost list did not include that host. The Erigon command was changed to:
--authrpc.addr=0.0.0.0
--authrpc.port=8551
--authrpc.vhosts=gnosis,localhost,127.0.0.1
--authrpc.jwtsecret=/jwt/jwt.hex
Then only the Erigon container was restarted, using the same /data/erigon and the same JWT file.
The working shape
The execution side:
erigon
--chain=gnosis
--datadir=/data
--externalcl
--authrpc.addr=0.0.0.0
--authrpc.port=8551
--authrpc.vhosts=gnosis,localhost,127.0.0.1
--authrpc.jwtsecret=/jwt/jwt.hex
--prune.mode=archive
The consensus side:
lighthouse bn
--network gnosis
--datadir /data
--execution-endpoint http://gnosis:8551
--execution-jwt /jwt/jwt.hex
--checkpoint-sync-url https://checkpoint.gnosischain.com
Verification
After the Erigon restart, the important Erigon log line appeared:
[GetClientVersionV1] Received request fromClientCode: LH, Lighthouse-v8.2.0...
Lighthouse also changed from JWT errors to normal Engine API flow:
Execution engine online
Issuing forkchoiceUpdated
Synced ... exec_hash: ... (unverified)
Head is optimistic
The recent error count was clean:
docker logs --since 2m gnosis-lighthouse 2>&1 | grep -c 'InvalidToken'
# 0
Head is optimistic was expected at that point. It meant Lighthouse had the consensus head, but the execution layer still had to validate the corresponding payloads. The Erigon logs showed the execution stage still applying blocks, so the remaining work was execution catch-up rather than Engine API authentication.
The small operational lesson: if Lighthouse says InvalidToken, read the embedded HTTP status. A 403 from Erigon can be an authrpc host allowlist problem, even when the JWT file itself is correct.
References
[1] Erigon README: Beacon Chain / Consensus Layer
[2] Lighthouse v8.2.0 release notes