How to validate a blockchain dataset against a second source

1. What you can check from inside the data

Validation of a dataset comes in two kinds, and only one of them can be done from inside the dataset itself. The first kind is structural and internal consistency: are addresses and hashes well-formed, are the keys unique, are block numbers contiguous, do row counts between related tables agree, do value sums add up. These checks run against the data on hand and they are essential. They are also blind to one thing: a row that was never written.

The second kind is completeness, and it cannot be established from one copy of the data. An omission leaves no internal trace. The only way to detect it is to compare against something that reached the chain by a different path and ask where the two disagree. This guide walks the full procedure on a real dataset: structural checks first, then a count reconciliation against an independent source, then an escalation from a coarse disagreement down to the exact missing record.

2. Structural checks: necessary, not sufficient

Start with the cheap checks, because they catch the common failures: a truncated import, a duplicated batch, a gap in the block range. Run them on a window of Polygon logs in Dune, the source we use for the comparison in this guide:

Both pass. Every block in the window is present, and all 15,825,038 logs have a unique key. Here is the catch: this is the exact window in which the dataset is, in fact, missing records. The checks pass over data that is incomplete, because contiguity and uniqueness describe the rows that are there and say nothing about the rows that are not. Structural validation is necessary and it is not sufficient. To go further you need a second opinion.

3. Reconcile counts against a second source

Reconciliation compares aggregate counts between two datasets, bucket by bucket, and flags the buckets that disagree. The single rule that makes it work is escalating granularity: start coarse, then split only the buckets that disagree into finer ones, so the search narrows by an order of magnitude at each step instead of scanning everything. Reconciling a full table count would bury a handful of missing rows in tens of millions; reconciling a targeted slice by day surfaces them.

Take one slice, the logs emitted by the Polygon state-receiver contract at 0x0000…1001, and count it per day on each side. On Dune that is one grouped query:

The same count from Portal comes off the stream API, filtered to that contract and event over the day's block range. The stream returns matches in chunks, each response ending at a block number you pass back as the next fromBlock until you reach toBlock:

Side by side, the daily counts settle the question that neither source answers alone. Dune's own numbers swing from 125 to 345 a day on natural volume, so a low day looks like nothing in particular. Against Portal, two days separate from the rest:

2025-07-29 agrees to the row, which is the result you want most days and the evidence the two counts are measuring the same thing. The next two days do not: Dune is short 55 commit logs on 2025-07-30 and 25 on 2025-07-31, 80 across the two days, spread over 42 state-sync transactions. The reconciliation has told you where to look. It has not yet told you what is missing.

4. Escalate to the record

Keep splitting the disagreeing bucket. Within 2025-07-31, the same count by block lands on individual blocks, and one of them is 74,614,768. Counting all logs in that block, not just the slice, Portal returns 856 and Dune returns 848. The eight-row difference is a single transaction: a synthetic state-sync receipt that Dune's polygon.transactions omits entirely, holding 114 records where Portal holds 115.

That is the bottom of the escalation: a coarse daily disagreement, narrowed in two steps to two specific log records that one source has and the other does not. The records turn out to be Polygon state-sync logs, a class the chain's block header never commits to, which is why a structurally valid dataset can drop them without any internal check noticing. The mechanism, and why no Merkle proof would have caught this either, is the subject of the companion guide on the onchain data block proofs leave out.

5. A source that does not share the blind spot

Reconciliation is only as good as the independence of the two sources. Two datasets built from the same upstream, or on the same assumption about what counts as a transaction, share the same gaps and agree with each other while both being wrong. The state-sync case is exactly that trap: any pipeline that reconstructs logs from the canonical transaction list drops the same records, so a second source built the same way would have confirmed the wrong answer.

So a disagreement is a lead, not a verdict, and the record gets confirmed on a third source that reaches the chain differently: an archive node's eth_getLogs, which returns the state-sync logs directly, or the public explorer, which renders the synthetic transaction in full. When three sources with different ingestion paths agree on a record and one dataset lacks it, the gap is real. This is why a validation setup worth the name cross-checks against several sources, RPC nodes and independent warehouses among them, rather than a single mirror.

6. Reconciliation with SQD

The procedure above is the one SQD runs against its own data on the way in: structural checks, then count reconciliation against independent sources at escalating granularity, then record-level spot checks, repeated as new blocks arrive rather than once at import. It is the part of data quality that proofs and structural checks cannot reach, and it is why Portal returns records like the state-sync logs above that a transaction-list pipeline silently omits.

The same queries are the reader's to run. Portal serves logs, transactions, traces, and state diffs as a stream you can reconcile any third-party dataset against, and the Squid and Pipes SDKs pull the same data into your own store for continuous checks. For where complete, reconciled data is load-bearing, see the analytics and compliance solutions.