Skip to content

feat: (Python) Add async context manager#487

Open
qzyu999 wants to merge 2 commits intoapache:mainfrom
qzyu999:feat/456-python-async-context-manager
Open

feat: (Python) Add async context manager#487
qzyu999 wants to merge 2 commits intoapache:mainfrom
qzyu999:feat/456-python-async-context-manager

Conversation

@qzyu999
Copy link
Copy Markdown
Contributor

@qzyu999 qzyu999 commented Apr 10, 2026

Purpose

Linked issue: close #456

To implement asynchronous context managers (async with) for AppendWriter, UpsertWriter, and LogScanner in the Python bindings, ensuring proper resource lifecycle management and automated, non-blocking flushing of records.

Brief change log

  • Writer Context Managers (AppendWriter, UpsertWriter): Implemented __aenter__ and __aexit__ protocols.
    • Happy Path: Automatically awaits flush() on normal exit, guaranteeing data delivery before releasing the context.
    • Exception Path: Bypasses flush() to instantly free the Python asyncio event loop (fail-fast semantics). Note: Because the RecordAccumulator is a shared resource on the connection, this relies on a "best-effort" non-blocking design. It avoids calling close() or abort() to prevent permanently bricking the shared MemoryLimiter, meaning records appended prior to the exception may still be transmitted by the background Tokio thread.
  • Scanner Context Managers (LogScanner, RecordBatchLogScanner): Implemented __aenter__ and __aexit__ to establish the API contract for asynchronous resource reclamation.
  • Documentation & Type Hints: Updated __init__.pyi docstrings to clearly document the best-effort transactional semantics so developers understand the limits of client-side atomicity.
  • Test Stability: Fixed a micro-race condition in test_log_table.py::test_list_offsets by adding an explicit asyncio.sleep(0.1) delay, ensuring strict chronological ordering when resolving timestamp-based offsets.

Tests

Added comprehensive coverage in a new bindings/python/test/test_context_manager.py suite:

  • Happy Path Verification: test_append_writer_success_flush and test_upsert_writer_context_manager verify that data is automatically flushed and available to scanners/lookupers without explicit flush() calls.
  • Non-Blocking Exception Path: test_append_writer_exception_no_flush uses time.perf_counter() to assert that exiting a failed context block takes < 0.1s, successfully proving that the speed-of-light network RTT wait is bypassed without destroying the connection for subsequent tests.
  • Scanner Coverage: Explicit tests added for both create_log_scanner() and create_record_batch_log_scanner() resource lifecycle bounds, including exception propagation.

API and Format

  • API: Yes. This introduces new API surface to the Python bindings by adding the __aenter__ and __aexit__ magic methods, enabling async with syntax.
  • Format: No changes to the underlying storage format or the core Rust RPC protocols.

Documentation

Yes. This introduces a new feature for the Python client. Python type hints (__init__.pyi) and docstrings have been updated to reflect the new syntax and explicitly document the behavior of the exception fault-path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add async context manager support in python

1 participant