Three Times the Testing, Four Times Faster: THORNode CI, One Month Later

Over the last month, the THORNode team rebuilt its continuous integration (CI), the automated checks that run on every proposed code change before it can be merged. Every merge request (MR) now runs more checks, on more machines, and gets an answer in about 24 minutes instead of an hour and a half. Here is what changed, measured from 38,401 GitLab CI jobs.

"Before" covers September 7 to 20, before most of the CI work landed. "Now" covers October 12 to 21, after the main changes were merged.

More Work, Less Waiting

CI now runs three times as many jobs per day, and each pipeline runs 19 jobs where it used to run 11. Pipelines still finish faster than they did over the summer. In early September, the new bond-yield feature added a second validator churn (a rotation of the active node set) to the network simulation, which pushed pipelines past 90 minutes. That extra coverage is still in place.

Every Stage Got Faster

Every job in the pipeline now runs faster than it did in September. The heaviest one is the network simulation, which spins up THORChain together with 16 real L1 chains and runs swaps, churns, and refunds end to end. It now finishes in 16 minutes, and it starts five minutes into the pipeline instead of 21.

Reliable Enough to Trust

A flaky CI teaches developers to hit retry without reading the failure. The team removed most of the infrastructure noise, so a red pipeline now usually points to a real problem in the code.

New Kinds of Tests

The time saved went into new tests. Ordinary tests check that the code does what its author expected. The new ones attack it with random input, break it on purpose, replay it on separate nodes, restart it mid-flight, and check THORNode and Bifrost (the component that watches and signs on external chains) against each other.

Over the month, the codebase gained 1,088 Go test functions (from 5,796 to 6,884), about 60,000 lines of Go test code in 208 new test files, five new test jobs on merge requests, and seven new regression suites (from 185 to 192).

Fuzz Testing, on Every MR

A fuzzer generates thousands of malformed, random, and edge-case inputs and feeds them to the code that parses untrusted data. It looks for crashes and for inputs that get through when they should be rejected. The inputs it collects are kept and replayed on later runs. The repository's first five fuzz targets cover the memo parser, the address parser, the FROST evidence decoder, FROST signed payloads, and the Solana RPC version. They have already found two decoder bugs, both fixed in the same change. See !5190.

Mutation Testing, When Security Code Changes

Mutation testing tests the tests. CI plants a known bug in security-critical code, such as letting a quorum pass with one signer too few, and runs the test suite against it. If the tests still pass, they would miss that bug in real life, and the job fails. Five curated mutants run on merge requests: the two-thirds supermajority quorum boundary, signing halt boundaries, an attestation signature bypass, attestation replay, and the direction of a safe subtraction. Two more run in a deeper scheduled pass. Source files are never modified, as the bugs are applied through Go build overlays. See !5185.

Consensus Determinism, on Every MR

Every node has to compute exactly the same state from the same blocks, or the chain halts. Five tests in a dedicated job replay the same blocks on separate node instances and compare the results, prove that the check catches a divergence, restart and export mid-chain, and run the store migration from version 18 to 19. See !5186.

Adversarial Quorum Tests, on Every MR

Validators sign and vote on what they observed on other chains. These tests change one field at a time in a signed payload (chain, amount, address, memo, height, and more) and check that the signature no longer verifies. They also check that votes one short of the threshold, duplicate signers, and reordered signers never pass, and who owns the quorum cache. See !5188 and !5182.

Crash and Restart Recovery, on Every MR

Nodes restart, sometimes in the middle of signing. These tests inject crashes and retries into the Bifrost signer and observer, including during vault retirement, and check that both recover their saved state and broadcast each transaction only once. They also cover state-sync timeouts. See !5189 and !5233.

THORNode and Bifrost Contract Tests, on Every MR

Many past chain bugs sat in the gap between THORNode and Bifrost, where each side's tests mocked the other. These tests check the real bytes and units that pass between the two, including keysign byte parity and gas units, and run the EVM client against the real V6 router contract on an in-process chain. See !5239 and !5240.

Stricter Simulation, on Every MR

The full network simulation now fails when it cannot check its invariants, instead of passing quietly. Its own unit tests run in CI, Monero has new regression coverage, and a separate job runs the mainnet-only Monero checks. See !5184, !5183, and !5150.

Gates That Can Say No, on Every MR

Known Go vulnerabilities now block a merge (govulncheck), starting from a cleared baseline. Four existing checks that could never fail, because of how they were wired, now report problems. Mainnet-only code paths get their own static analysis pass. See !5137, !5087, and !5205.

In Review

• Accounting conservation campaigns: seeded random runs that check no value is created or lost across pools and vaults (!5187).
• More mutation coverage, with assertion checks (!5191).
• A test-integrity guard that blocks changes editing a test to agree with a bug (!5242).
• 80% unit test coverage for every file and package (!5179).

What Changed

The main merge requests behind these numbers, all merged to the develop branch between September 16 and October 20:

• !5087, checks that can fail: four CI checks that always passed now report problems.
• !5142, less infrastructure noise: retries for network and registry failures, image mirrors, and a prebuilt tools image.
• !5147, pipeline graph: jobs start as soon as their inputs are ready instead of waiting stage by stage, plus shallow clones, faster proto checks, and faster simulated churn.
• !5151, right-sized runners: short jobs run on small slots, which keeps the large ones free for the simulation.
• !5166, regression tests on demand: blocks are created when a test needs one instead of on a timer. Together with the native runner from !5147, regression tests went from 25 minutes to six.
• !5167, simulation in half the time: shorter intervals, parallel setup and funding, and eight suites at once.
• !5173, fast lint and unit tests: one linter pass over the whole module, fixes for sleep-bound tests, and cached test results.
• !5139, no wasted runs: a new push cancels the pipeline it replaces.
• !5137, vulnerability gate: govulncheck is now blocking.
• !5182 to !5186 and !5188 to !5190, test-quality series: fuzzing, mutation testing, consensus replay, adversarial quorum payloads, signer and observer recovery, and fail-closed invariants.
• !5197, flaky-test sweep: a review of the week's CI failures found and fixed the flaky tests behind them. It is set up to run weekly.
• !5238 to !5240, contract tests: THORNode and Bifrost checked against each other, and the EVM client against the real router.

How It Was Measured

• The data covers every job in the thornode GitLab project from July 20 to October 21, 2026: 38,401 jobs in 2,586 pipelines, pulled from the GitLab API with retried jobs included.
• "Before" is September 7 to 20, before most of the CI work landed. "Now" is October 12 to 21, after the main changes were merged.
• A pipeline is green when the final attempt of every required job passed. This matches GitLab's own pipeline status for all 873 pipelines in the two windows. Pipeline time runs from creation to the last job finishing, so it includes queueing and retries. Only pipelines that run the full network simulation are counted.
• Job counts include jobs that started on a runner. Skipped jobs and jobs canceled before they started are left out. Over the month, that is 7,387 jobs from August 21 to September 20 and 22,841 from September 21 to October 21.
• "Pipelines that hit any failure" counts a pipeline once if any required job attempt failed, whatever the cause. Per-job failure rates are not shown, because automatic cancellation now stops many jobs before they can fail.
• Part of the gain comes from new hardware: three self-hosted runner machines joined between September 29 and October 4. Splitting jobs by size is what lets the pipeline use them well.

Source: gitlab.com/thorchain/thornode CI, data pulled October 21, 2026.