Skip to content
Minotaur
+ 02 · Validator

Validator Troubleshooting

Common issues and solutions for running the Minotaur validator — Anvil, subtensor, leader election, consensus, peer discovery, scoring, weights, and Docker.

Common issues and solutions for running the Minotaur validator. Each section starts with the symptom you’ll see, then walks you through the checks that resolve it.

Anvil Connection Issues

Symptom: Simulation failures, connection refused errors, or plans that never score.

Verify ANVIL_RPC_URL is set and reachable:

terminal SH
curl -X POST -H "Content-Type: application/json" \
--data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
$ANVIL_RPC_URL

If using Alchemy or Infura on the upstream (ETH_UPSTREAM_RPC_URL / BASE_UPSTREAM_RPC_URL), confirm your API key is valid and has not exceeded rate limits.

For local testnet, ensure the Anvil container is healthy:

terminal SH
docker compose ps anvil

Check that each Anvil fork is on the expected chain (ETH = 1, Base = 8453, BT EVM = 964, local Anvil = 31337).

Symptom: Simulations hang or time out.

Anvil has a 2-second block time by default. The simulator calls evm_mine after sending transactions. If the upstream RPC is slow, simulations may time out. Increase logging to DEBUG to see simulation details:

terminal SH
export LOG_LEVEL=DEBUG

Subtensor Sync Issues

Symptom: Validator cannot read metagraph, weight emission fails, or connection refused to subtensor.

Verify SUBTENSOR_URL is correct:

  • Mainnet: wss://entrypoint-finney.opentensor.ai:443
  • Local testnet: ws://localhost:9944

Test the connection:

terminal SH
# Using wscat
wscat -c "$SUBTENSOR_URL" -x '{"id":1,"jsonrpc":"2.0","method":"system_health","params":[]}'

For local testnet, confirm the subtensor container is running and healthy:

terminal SH
docker compose ps subtensor

Leader Election

Symptom: Validator is running but not processing orders (BlockLoop idle).

Only the leader runs the BlockLoop. Check if this validator is the leader:

terminal SH
curl http://localhost:9100/leader

The leader is the validator with the highest TAO stake on subnet 112 (ties broken by hotkey lexicographic order). If you have less stake than other validators you will be a follower. For local testnet or development, set FORCE_LEADER=1 to bypass stake-based election.

Consensus Failures

Symptom: Plans are scored but never relayed on-chain. Quorum not reached in logs.

Check consensus configuration and discovered peers:

terminal SH
curl http://localhost:9100/consensus/info

The peers field should be non-empty — if it’s empty, see No peers discovered below. Then verify:

  • Discovered peer axon URLs are reachable from this validator (network / firewall rules)
  • Check the live on-chain quorum: make get-quorum-base (or cast call $VALIDATOR_REGISTRY 'quorumBps()(uint256)' --rpc-url $BASE_RPC_URL). The default 6666 is 2-of-3 BFT. See Quorum Management for changing it.
  • VALIDATOR_PRIVATE_KEY is set and valid — used to sign EIP-712 consensus messages
  • Followers independently re-simulate and re-score; if a follower’s scores don’t both exceed threshold, it won’t sign

For local testnet only: set QUORUM_BPS_OVERRIDE to force a local value without going through the registry. Production deployments should leave it unset.

No peers discovered

Symptom: curl http://localhost:9100/consensus/info returns an empty peers list, or the daemon log says ProtocolConfig: peer discovery: probed 0 candidates → 0 verified.

Discovery requires four things to line up. Check each in order:

  1. Your VALIDATOR_AXON_URL is set and reachable

    From any other host:

    terminal SH
    curl $VALIDATOR_AXON_URL/identity

    Should return a JSON payload, not a 503. If 503, the daemon is missing one of: bittensor wallet (no my_hotkey), VALIDATOR_AXON_URL env, or a signing key.

  2. Your hotkey is on the metagraph with the correct axon URL

    terminal SH
    btcli subnet metagraph --netuid 112 --subtensor.network finney

    Find your hotkey. The axon column must match VALIDATOR_AXON_URL. If wrong, re-run btcli to update it (or your bittensor wallet’s serve_axon runner).

  3. Your EVM signing address is in the on-chain ValidatorRegistry

    terminal SH
    cast call $VALIDATOR_REGISTRY 'isValidator(address)(bool)' 0xYourEvmAddress --rpc-url $RPC_URL

    Must return true on every chain you operate on. If false, see validator quickstart Step 4 for the handshake with the registry owner.

  4. Other validators' /identity endpoints are reachable from your host

    terminal SH
    curl <their-axon-url>/identity

    If unreachable, it’s a network issue (firewall, NAT).

Symptom: identity probe returns valid JSON but discovery still rejects it. Check the daemon logs for Identity probe ... recovered EVM X but it is not in ValidatorRegistry.getValidators() — rejecting. That’s the on-chain handshake (step 3) for the other validator — they need to be added to the registry too.

JS Scoring Engine Issues

Symptom: JS scores are always 0.0, NaN, or scoring errors in logs.

Verify Node.js 20.x is installed:

terminal SH
node --version  # Should be v20.x

The JS engine runs app scoring code in a Node.js sandbox. Check that the app’s JS code exports the required functions:

app/scoring.js JS
module.exports = { config, manifest, score };

The score(plan, state, context) function receives:

  • planExecutionPlan dict (with metadata, interactions, etc.)
  • state — Structured IntentState export with raw_params, control, and typed_context (plus compatibility extra / rawParams aliases)
  • context — Full context with context.simulation (token transfers, gas, state changes), context.state, and context.oracle

Check the validator logs for JS execution errors. Enable debug logging:

terminal SH
export LOG_LEVEL=DEBUG

BlockLoop Not Processing Orders

Symptom: Orders are submitted but never processed.

  1. Confirm you are the leader

    See Leader Election above.

  2. Check BlockLoop status

    terminal SH
    curl http://localhost:9100/blockloop/status
  3. Verify orders exist and are OPEN

    terminal SH
    curl http://localhost:9100/orders
  4. Check that app definitions are loaded

    terminal SH
    curl http://localhost:9100/health

    The response should show the number of loaded intents.

  5. Verify store path

    If using --store-path, verify the file exists and is readable.

  6. Review logs

    Look for errors during plan generation, simulation, or scoring.

Weight Emission Not Working

Symptom: Validator runs but never emits weights on-chain.

Weights are emitted once per epoch (default: 60 seconds, production: 1200s, configurable via --epoch-seconds). A champion miner must exist — weights are only emitted when a solver has been submitted and accepted:

terminal SH
curl http://localhost:9100/weights
curl http://localhost:9100/weights/history

Verify Bittensor wallet configuration:

  • WALLET_NAME and HOTKEY_NAME must match a wallet with a registered hotkey on subnet 112.
  • The wallet directory must be accessible (default: ~/.bittensor/wallets/).

Miner Submissions Rejected (leader-only)

Symptom: Miner solver submissions fail or are not adopted.

This section only applies if you are running the optional API service and are currently the leader (highest-stake validator). Third-party validators running only the canonical platform/validator/ stack don’t accept submissions — the leader does. If you’re not the leader, miners shouldn’t be hitting your box for submissions.

If you are the leader:

terminal SH
curl http://localhost:8080/v1/submissions

Then verify:

  • Solver code goes through three screening stages before adoption. Check logs for rejection reasons.
  • The miner is registered on the subnet and its hotkey appears in the metagraph.
  • The miner is pointing at the correct API URL for /v1/submissions*.

Port Conflicts

Symptom: Address already in use on startup.

Check what is using the port:

terminal SH
lsof -i :9100

Change the port with --port:

terminal SH
python -m minotaur_subnet.validator.main --port 9101

Docker / Local Testnet Issues

Symptom: Containers fail to start or are unhealthy.

Check container status and logs:

terminal SH
docker compose ps
docker compose logs validator
docker compose logs anvil

The validator daemon waits for its three Anvil forks to report healthy before starting (see depends_on in platform/validator/docker-compose.yml). On a first cold start this can take 60–90 seconds — anvil-btevm in particular waits on a public RPC. If you see “dependency failed to start”, wait a bit and docker compose up -d again; the start_period on each anvil healthcheck gives them grace time on subsequent retries.

For the local-testnet path, ensure the .env file in platform/local_testnet/ has valid ALCHEMY_RPC_URL and BASE_ALCHEMY_RPC_URL values. The canonical validator stack reads from platform/validator/.env.

If containers are stuck, do a clean restart:

terminal SH
make testnet-down
make testnet-up

The init container runs once on startup (registers subnet, deploys contracts). Check its logs if other services fail:

terminal SH
docker compose logs init

Insufficient TAO Balance

Symptom: Registration or weight emission fails with balance errors.

Check your balance:

terminal SH
btcli wallet balance --wallet.name my-validator --subtensor.network finney

Subnet 112 registration requires a burn fee. Ensure you have enough TAO. For local testnet, the init container handles registration and funding automatically.

See also