feat: (Python) Add async context manager#487
Open
qzyu999 wants to merge 2 commits intoapache:mainfrom
Open
Conversation
…, and scanners in Python bindings
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #456
To implement asynchronous context managers (
async with) forAppendWriter,UpsertWriter, andLogScannerin the Python bindings, ensuring proper resource lifecycle management and automated, non-blocking flushing of records.Brief change log
AppendWriter,UpsertWriter): Implemented__aenter__and__aexit__protocols.flush()on normal exit, guaranteeing data delivery before releasing the context.flush()to instantly free the Pythonasyncioevent loop (fail-fast semantics). Note: Because theRecordAccumulatoris a shared resource on the connection, this relies on a "best-effort" non-blocking design. It avoids callingclose()orabort()to prevent permanently bricking the sharedMemoryLimiter, meaning records appended prior to the exception may still be transmitted by the background Tokio thread.LogScanner,RecordBatchLogScanner): Implemented__aenter__and__aexit__to establish the API contract for asynchronous resource reclamation.__init__.pyidocstrings to clearly document the best-effort transactional semantics so developers understand the limits of client-side atomicity.test_log_table.py::test_list_offsetsby adding an explicitasyncio.sleep(0.1)delay, ensuring strict chronological ordering when resolving timestamp-based offsets.Tests
Added comprehensive coverage in a new
bindings/python/test/test_context_manager.pysuite:test_append_writer_success_flushandtest_upsert_writer_context_managerverify that data is automatically flushed and available to scanners/lookupers without explicitflush()calls.test_append_writer_exception_no_flushusestime.perf_counter()to assert that exiting a failed context block takes< 0.1s, successfully proving that the speed-of-light network RTT wait is bypassed without destroying the connection for subsequent tests.create_log_scanner()andcreate_record_batch_log_scanner()resource lifecycle bounds, including exception propagation.API and Format
__aenter__and__aexit__magic methods, enablingasync withsyntax.Documentation
Yes. This introduces a new feature for the Python client. Python type hints (
__init__.pyi) and docstrings have been updated to reflect the new syntax and explicitly document the behavior of the exception fault-path.