Skip to content

feat(bitcore-node): Implement multithreaded sync for UTXO chains (#3600)#4135

Open
bittoby wants to merge 2 commits intobitpay:masterfrom
bittoby:fix/issue-#3600
Open

feat(bitcore-node): Implement multithreaded sync for UTXO chains (#3600)#4135
bittoby wants to merge 2 commits intobitpay:masterfrom
bittoby:fix/issue-#3600

Conversation

@bittoby
Copy link
Copy Markdown

@bittoby bittoby commented Apr 3, 2026

Description

Resolves #3600

UTXO chain sync is currently bottlenecked by strictly sequential block fetching — each block must be fully fetched and processed before the next is even requested. This PR parallelizes the fetch step while keeping the store step sequential (required for UTXO ordering, nextBlockHash pointers, and height-from-parent validation).

The implementation follows the existing EVM multi-thread sync pattern (evm/p2p/sync.ts, evm/p2p/syncWorker.ts, evm/p2p/p2p.ts) with two complementary approaches:

Phase 1 — Sliding-window prefetch (single-thread mode):
Prefetches up to prefetchSize blocks ahead via P2P while the current block is being processed. Used for near-tip sync and post-initial-sync catch-up. Configurable via prefetchSize config option (default: 10, set to 0 to disable).

Phase 2 — Worker thread sync (multi-thread mode):
Worker threads fetch raw blocks via Bitcoin Core RPC (getblock <hash> 0), while the coordinator buffers out-of-order responses and processes them in strict height order. Used during initial sync only, then automatically switches to single-thread P2P mode. Configurable via threads config option (default: CPU cores - 1, set to 0 to disable).

Dual-mode pattern matches EVM: the in-memory initialSyncComplete flag is checked before the DB read to avoid overwriting the flag set by the INITIALSYNCDONE event.

Changelog

  • Add UtxoMultiThreadSync coordinator class that manages worker threads, distributes block hashes, and processes blocks in height order using an ordered buffer (src/modules/bitcoin/sync.ts)
  • Add UtxoSyncWorker worker thread that fetches raw block hex via Bitcoin Core RPC using plain http/https — no external dependencies in worker context (src/modules/bitcoin/syncWorker.ts)
  • Refactor BitcoinP2PWorker.sync() to support dual-mode sync: multi-thread during initial sync, single-thread with sliding-window prefetch near tip (src/modules/bitcoin/p2p.ts)
  • Extract getHeadersForSync() and handleGenesisBlock() as shared helper methods in BitcoinP2PWorker
  • Add useMultiThread() method matching the existing EVM pattern
  • Add prefetchSize and threads config options to IUtxoNetworkConfig (src/types/Config.ts)
  • Add 13 unit tests covering prefetch behavior, sequential processing order, dual-mode switching, ordered block processing, thread management, and INITIALSYNCDONE emission

Testing Notes

  • All 238 unit tests pass (npm run test:unit), including 13 new tests
  • TypeScript compiles cleanly under strict mode
  • ESLint passes with zero warnings
  • No new external dependencies added — worker threads use Node.js built-in http/https for RPC calls
  • The worker thread path resolves correctly in compiled output (build/src/modules/bitcoin/syncWorker.js via __dirname)
  • Integration tests are unaffected — UtxoMultiThreadSync constructor is inert (no threads spawned until sync() is called), and stop() is safe when called before sync()
  • To test configuration, add to bitcore.config.json under a UTXO chain network:
    {
      "threads": 4,
      "prefetchSize": 10
    }
    

Set threads: 0 to disable multi-thread sync entirely. Set prefetchSize: 0 to disable prefetching in single-thread mode.


Checklist

  • I have read CONTRIBUTING.md and verified that this PR follows the guidelines and requirements outlined in it.

bittoby added 2 commits April 3, 2026 05:41
Add parallel block fetching for UTXO chain sync with two complementary
approaches, following the existing EVM multi-thread sync pattern:

Phase 1 - Sliding-window prefetch (single-thread mode):
  - Prefetch up to `prefetchSize` blocks ahead via P2P while processing
  - Blocks still processed sequentially to preserve UTXO ordering
  - Used for near-tip sync and post-initial-sync catch-up
  - Configurable via `prefetchSize` (default: 10, 0 to disable)

Phase 2 - Worker thread sync (multi-thread mode):
  - Worker threads fetch raw blocks via Bitcoin RPC (`getblock <hash> 0`)
  - Coordinator buffers out-of-order responses, processes in height order
  - Used during initial sync only, then switches to single-thread P2P
  - Configurable via `threads` (default: CPU cores - 1, 0 to disable)

Dual-mode sync matches EVM pattern: multi-thread check happens before
DB read to avoid overwriting the in-memory initialSyncComplete flag.

Resolves bitpay#3600
- Add explicit type annotations on class properties with defaults
- Use block syntax for single-line if-returns
- Use type cast instead of non-null assertion for parentPort
- Match EVM worker entry point style (async function vs arrow)
- Replace JSDoc blocks with inline comments
- Remove explicit type annotations on handler params
@bittoby bittoby changed the title Fix/issue #3600 feat(bitcore-node): Implement multithreaded sync for UTXO chains (#3600) Apr 3, 2026
@bittoby
Copy link
Copy Markdown
Author

bittoby commented Apr 6, 2026

@kajoseph could you please review this PR? Welcome to any feedback.
thank you

@kajoseph kajoseph added the BCN This pull request modifies the bitcore-node package label Apr 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

BCN This pull request modifies the bitcore-node package

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement multithreaded sync for UTXO chains

2 participants