feat: lazy per-bundle loading for vault, driver, datastore, and report registries

[stack72]

Summary

• Extends the lazy per-bundle loading pattern (from Lazy per-bundle loading — import individual extension bundles on demand #1053) to all remaining extension registries: vault, driver, datastore, and report
• Each registry now supports LazyEntry types, setTypeLoader/ensureTypeLoaded for on-demand single-bundle imports, and registerLazy/promoteFromLazy lifecycle
• Each loader gains buildIndex() (mtime-based catalog population without importing) and loadSingleType() (on-demand single bundle import)
• Shared ExtensionCatalogStore (SQLite at .swamp/_extension_catalog.db) passed from CLI startup to all loaders
• Vault type search returns lazy entries from catalog without importing bundles

Performance

Before/after on swamp vault type search --json with @swamp/aws-sm pulled:

Run	Old binary	New binary
First run (bootstrap)	1.56s	3.67s
Second run (cached)	1.56s	0.70s (55% faster)

Test Plan

• Added lazy loading tests to all 4 registry test files (registerLazy, ensureTypeLoaded, concurrent dedup, retry after failure)
• All 4159 existing tests pass
• Before/after validation with real pulled extensions:
  • @swamp/s3-datastore (datastore), @swamp/aws-sm (vault), @john/tensorlake (driver), @webframp/aws/cost-report (report)
  • Verified catalog entries populated correctly across all extension kinds
  • Verified search commands return results from catalog without importing bundles

Closes #1062

🤖 Generated with Claude Code

[stack72]

Heads up: merge conflicts with #1103 + #1104

PRs #1103 and #1104 (fixing #1100 — S3 datastore bundle path resolution) landed on main and touch the same files this PR modifies. Here's what changed and what you'll need to account for when rebasing:

What changed

All 4 extension loaders (UserVaultLoader, UserDriverLoader, UserDatastoreLoader, UserReportLoader) now accept an optional DatastorePathResolver parameter and have a resolveBundlePath() helper method. The constructors changed from:

constructor(denoRuntime: DenoRuntime, repoDir: string | null = null)

to:

constructor(
  denoRuntime: DenoRuntime,
  repoDir: string | null = null,
  datastoreResolver?: DatastorePathResolver,
)

Except UserDatastoreLoader — it must NOT receive the resolver due to bootstrap ordering (it loads datastore extensions that configure the resolver). Its constructor is unchanged.

What this PR needs to do after rebasing

1. New buildIndex() and loadSingleType() methods added by this PR will construct bundle paths. These should use this.resolveBundlePath(...) instead of join(this.repoDir, SWAMP_DATA_DIR, SWAMP_SUBDIRS.*, ...) — for vault, driver, and report loaders. The datastore loader's new methods should keep using the hardcoded local path (no resolver).

2. mod.ts wiring — the loader instantiation sites changed. Model, vault, driver, and report loaders now receive a lazy DatastorePathResolver factory. The datastore loader is explicitly excluded. Your new buildIndex calls and catalog wiring need to work with this pattern.

3. auto_resolver_adapters.ts — InstallerAdapterConfig now has an optional datastoreResolver field. Hot-load functions pass it to model and vault loaders.

Key constraint

datastore-bundles is intentionally excluded from DEFAULT_DATASTORE_SUBDIRS (#1104). The datastore loader always reads from local .swamp/datastore-bundles/. Including it in the datastore tier causes the S3 setup migration to delete the bundles, breaking subsequent extension loading.

[github-actions]

Code Review

Well-structured PR that extends the lazy per-bundle loading pattern from models to the remaining four extension registries (vault, driver, datastore, report). The shared ExtensionCatalogStore is correctly hoisted to CLI startup and passed down, the sourceDirsFingerprint utility is properly extracted for reuse, and all four registries follow the same consistent pattern.

Blocking Issues

None.

Suggestions

1. High code duplication across loaders: The buildIndex, findStaleFiles, rebundleAndUpdateCatalog, populateCatalogFromRegistry, populateCatalogFromDir, and discoverFilesSync methods are nearly identical across all four loaders (vault, driver, datastore, report), differing only in the kind string, export name, schema, and registry calls. This is a significant amount of copy-paste (~400 lines × 4). A shared base class or a generic helper parameterized by kind/schema/registry would reduce the maintenance surface considerably. Not blocking since the pattern is proven from the model loader and all four copies are consistent.

2. Report loader bundleWithCache refactor: The report loader's bundleWithCache method received a significant rewrite in this PR (the "try rebundle, fall back to cache" logic with isExpectedBundleFailure). This looks like a separate improvement that could have been its own commit for easier bisection, but the logic itself is sound.

3. vault_type_search.ts imports from internal path: src/cli/commands/vault_type_search.ts:33 imports getVaultTypes from ../../domain/vaults/vault_types.ts rather than from ../../libswamp/mod.ts. This predates this PR, so not blocking here, but worth noting for future cleanup per the CLAUDE.md libswamp import boundary rule.

DDD Assessment

The changes correctly keep domain logic (registries, lazy entries, type promotion) in the domain layer and infrastructure concerns (catalog store, SQLite) in the infrastructure layer. The LazyEntry types are proper value objects (immutable metadata with no identity). The registries continue to function as domain services. The ExtensionCatalogStore remains a clean infrastructure concern. Good separation.

Test Coverage

All four registries have comprehensive lazy loading tests covering: registerLazy, ensureTypeLoaded (happy path, already loaded, unknown type), concurrent dedup, and retry after failure. The vault registry has additional tests for promoteFromLazy, getAllLazy, and registerLazy skip-if-already-loaded. Good coverage.

[github-actions]

Adversarial Review

Critical / High

None found.

Medium

1. ReportRegistry case normalization mismatch — src/domain/reports/report_registry.ts:65-68 vs user_report_loader.ts:363

   registerLazyFromCatalog stores lazy entries keyed by entry.type_normalized (lowercased from populateCatalogFromDir). But promoteFromLazy(parsed.data.name, ...) (line 363 of the loader) uses the original-case report name. Since promoteFromLazy does this.lazyTypes.delete(name) with the original-case key, it will miss the lowercased lazy entry if the name contains uppercase characters.

   Breaking example: A report named "Cost-Report" is cataloged as "cost-report" (lowercased). registerLazy stores it under key "cost-report". After loading, promoteFromLazy("Cost-Report", ...) tries this.lazyTypes.delete("Cost-Report") — no-op, since the key is "cost-report". The lazy entry is orphaned. If ensureTypeLoaded("cost-report") is later called, it sees the orphaned lazy, triggers re-import, and importAndRegisterBundle checks reportRegistry.get(entry.type_normalized) = get("cost-report") which misses "Cost-Report" in reports — causing a redundant bundle import.

   Not currently triggered since no production code calls reportRegistry.ensureTypeLoaded() yet, but this is a latent bug waiting to surface.

   Suggested fix: Either normalize the name in promoteFromLazy and ensureTypeLoaded (using .toLowerCase() like all other registries), or store lazy entries with the original-case key. The former is more consistent with the other registries.

2. listAvailableTypes doesn't include lazy types — src/libswamp/vaults/create.ts:88

   listAvailableTypes() calls vaultTypeRegistry.getAll() which only returns fully loaded types, not lazy ones. When vault creation fails with "Unknown vault type", the error message listing available types will omit extension vault types that are indexed but not yet imported. Not a correctness issue — the error message is just incomplete.

Low

1. getVaultBundlePath / getDriverBundlePath / getDatastoreBundlePath / getReportBundlePath return empty string when repoDir is null — e.g. user_vault_loader.ts around the getVaultBundlePath method. The empty string is then passed to catalog.upsert as bundle_path. If loadSingleType later reads this entry, Deno.readTextFile("") would throw. In practice, repoDir is always set when buildIndex is called (the code path requires a repo), so this is theoretical.

2. discoverFilesSync has unbounded recursion — All four loader implementations recursively descend directories with no depth limit. A symlink loop (even accidental) could cause a stack overflow. The async discoverFiles has the same pattern, so this is pre-existing.

Verdict

PASS — Well-structured extension of the existing lazy-loading pattern. The four registries and loaders follow a consistent template with good test coverage (lazy registration, concurrent dedup, retry-after-failure). The report registry case mismatch (Medium #1) is latent and not yet triggered in production code. The catalog store's per-kind fingerprint approach is backward-compatible with the existing model loader. The bundleWithCache refactor in the report loader improves resilience by attempting a rebundle before falling back to cache.