Skip to content

[lake/lance] Implement Lance LakeSource read path with pushdown and enhanced split semantics#3013

Open
fhan688 wants to merge 1 commit intoapache:mainfrom
fhan688:fluss-lake-lance-source
Open

[lake/lance] Implement Lance LakeSource read path with pushdown and enhanced split semantics#3013
fhan688 wants to merge 1 commit intoapache:mainfrom
fhan688:fluss-lake-lance-source

Conversation

@fhan688
Copy link
Copy Markdown

@fhan688 fhan688 commented Apr 7, 2026

Purpose

This PR implements the Lance read-path foundation for Fluss LakeSource by completing LakeSource -> Planner -> RecordReader, integrating pushdown parameters, and enriching split semantics for snapshot/fragment-aware planning and reading.

Linked issue: #2715

Brief change log

Implemented and wired Lance read-path core classes:

  • LanceLakeSource
  • LanceSplitPlanner
  • LanceRecordReader
  • LanceSplit
  • LanceSplitSerializer

Completed storage-layer integration:

  • LanceLakeStorage#createLakeSource now returns the Lance source implementation

Added pushdown integration:

  • projection / filter / limit

  • introduced LancePredicatePushDown for common predicate conversion

Enhanced split semantics:

  • added fragmentId / snapshotId / fragmentRows / scanLimit / bucket / partition

Tests

  • LanceSplitSerializerTest
  • LanceSplitPlannerTest
  • LanceRecordReaderTest
  • LanceSourceTestBase

Fixed Java 8 test compatibility for lambda capture (effectively-final variables)
Added JNI availability guard in test base to skip JNI-dependent tests in unsupported environments

API and Format

  • No external API breaking changes
  • LanceSplit serialization now includes snapshot/fragment/limit/bucket/partition-related fields with versioned serializer handling
  • No incompatible external data format change introduced

Documentation

None

@fhan688 fhan688 changed the title [lance] Implement Lance LakeSource read path with pushdown and enhanced split semantics [lake/lance] Implement Lance LakeSource read path with pushdown and enhanced split semantics Apr 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant