feat(bitcore-node): Implement multithreaded sync for UTXO chains (#3600)#4135
Open
bittoby wants to merge 2 commits intobitpay:masterfrom
Open
feat(bitcore-node): Implement multithreaded sync for UTXO chains (#3600)#4135bittoby wants to merge 2 commits intobitpay:masterfrom
bittoby wants to merge 2 commits intobitpay:masterfrom
Conversation
Add parallel block fetching for UTXO chain sync with two complementary approaches, following the existing EVM multi-thread sync pattern: Phase 1 - Sliding-window prefetch (single-thread mode): - Prefetch up to `prefetchSize` blocks ahead via P2P while processing - Blocks still processed sequentially to preserve UTXO ordering - Used for near-tip sync and post-initial-sync catch-up - Configurable via `prefetchSize` (default: 10, 0 to disable) Phase 2 - Worker thread sync (multi-thread mode): - Worker threads fetch raw blocks via Bitcoin RPC (`getblock <hash> 0`) - Coordinator buffers out-of-order responses, processes in height order - Used during initial sync only, then switches to single-thread P2P - Configurable via `threads` (default: CPU cores - 1, 0 to disable) Dual-mode sync matches EVM pattern: multi-thread check happens before DB read to avoid overwriting the in-memory initialSyncComplete flag. Resolves bitpay#3600
- Add explicit type annotations on class properties with defaults - Use block syntax for single-line if-returns - Use type cast instead of non-null assertion for parentPort - Match EVM worker entry point style (async function vs arrow) - Replace JSDoc blocks with inline comments - Remove explicit type annotations on handler params
Author
|
@kajoseph could you please review this PR? Welcome to any feedback. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Resolves #3600
UTXO chain sync is currently bottlenecked by strictly sequential block fetching — each block must be fully fetched and processed before the next is even requested. This PR parallelizes the fetch step while keeping the store step sequential (required for UTXO ordering,
nextBlockHashpointers, and height-from-parent validation).The implementation follows the existing EVM multi-thread sync pattern (
evm/p2p/sync.ts,evm/p2p/syncWorker.ts,evm/p2p/p2p.ts) with two complementary approaches:Phase 1 — Sliding-window prefetch (single-thread mode):
Prefetches up to
prefetchSizeblocks ahead via P2P while the current block is being processed. Used for near-tip sync and post-initial-sync catch-up. Configurable viaprefetchSizeconfig option (default: 10, set to 0 to disable).Phase 2 — Worker thread sync (multi-thread mode):
Worker threads fetch raw blocks via Bitcoin Core RPC (
getblock <hash> 0), while the coordinator buffers out-of-order responses and processes them in strict height order. Used during initial sync only, then automatically switches to single-thread P2P mode. Configurable viathreadsconfig option (default: CPU cores - 1, set to 0 to disable).Dual-mode pattern matches EVM: the in-memory
initialSyncCompleteflag is checked before the DB read to avoid overwriting the flag set by theINITIALSYNCDONEevent.Changelog
UtxoMultiThreadSynccoordinator class that manages worker threads, distributes block hashes, and processes blocks in height order using an ordered buffer (src/modules/bitcoin/sync.ts)UtxoSyncWorkerworker thread that fetches raw block hex via Bitcoin Core RPC using plainhttp/https— no external dependencies in worker context (src/modules/bitcoin/syncWorker.ts)BitcoinP2PWorker.sync()to support dual-mode sync: multi-thread during initial sync, single-thread with sliding-window prefetch near tip (src/modules/bitcoin/p2p.ts)getHeadersForSync()andhandleGenesisBlock()as shared helper methods inBitcoinP2PWorkeruseMultiThread()method matching the existing EVM patternprefetchSizeandthreadsconfig options toIUtxoNetworkConfig(src/types/Config.ts)INITIALSYNCDONEemissionTesting Notes
npm run test:unit), including 13 new testshttp/httpsfor RPC callsbuild/src/modules/bitcoin/syncWorker.jsvia__dirname)UtxoMultiThreadSyncconstructor is inert (no threads spawned untilsync()is called), andstop()is safe when called beforesync()bitcore.config.jsonunder a UTXO chain network:{ "threads": 4, "prefetchSize": 10 }Set threads: 0 to disable multi-thread sync entirely. Set prefetchSize: 0 to disable prefetching in single-thread mode.
Checklist