Skip to content

perf: Defer query-time imports in conversation_base#236

Open
KRRT7 wants to merge 3 commits intomicrosoft:mainfrom
KRRT7:perf/defer-query-imports
Open

perf: Defer query-time imports in conversation_base#236
KRRT7 wants to merge 3 commits intomicrosoft:mainfrom
KRRT7:perf/defer-query-imports

Conversation

@KRRT7
Copy link
Copy Markdown
Contributor

@KRRT7 KRRT7 commented Apr 10, 2026

Stack: depends on #235. Merge #235 first, then this PR.


conversation_base.py eagerly imported four query-time modules at module level:

  • answers (20 ms cumulative — pulls in search + query)
  • search_query_schema (10 ms)
  • searchlang (3 ms)
  • answer_response_schema (1 ms)

These are only used in the query() method. Moved the imports inside query() and used TYPE_CHECKING + from __future__ import annotations for the type hints.

Benchmark

Azure Standard_D2s_v5 — 2 vCPU, 8 GiB RAM, Python 3.13

Import Time (hyperfine, warmup 5, min-runs 30)

Benchmark Before (#235) After Speedup
import typeagent 734 ms ± 13 ms 683 ms ± 15 ms 1.07x

Cumulative since baseline (main):

Benchmark main This PR Speedup
import typeagent 791 ms ± 11 ms 683 ms ± 15 ms 1.16x

Upstream opportunity: pydantic_ai transitive imports

pydantic_ai contributes ~161 ms cumulative to import typeagent through its transitive dependency chain:

Module Cumulative (ms) Notes
logfire 43 Observability SDK, imported eagerly
griffe 34 Code introspection for tool schemas
pydantic_ai.messages 22 Pydantic model definitions
genai_prices 9 Pricing data snapshot

This is the single largest remaining import cost and would need to be addressed upstream in pydantic-ai.


Generated by codeflash optimization agent

KRRT7 and others added 3 commits April 10, 2026 16:38
black is only used at runtime in two cold formatting paths:
- create_context_prompt() in answers.py (LLM debug context)
- format_code()/pretty_print() in utils.py (developer terminal output)

Both format Python data structures, which is exactly what pprint does.
Replace black.format_str with pprint.pformat + ast.literal_eval,
eliminating the runtime dependency entirely.

Move black from dependencies to dev dependency-group — it remains
available for make format/check but is no longer required by library
consumers.
answers, search_query_schema, searchlang, and answer_response_schema
are only used in the query() method. Move their imports from module
level into query() and use TYPE_CHECKING + __future__.annotations for
the type hints.

These modules pull in search, query, and schema initialization that
isn't needed when creating or indexing conversations.
Copy link
Copy Markdown
Collaborator

@gvanrossum gvanrossum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't follow precisely what games you are playing with from __future__ import annotations, but I don't want it in my code -- while it's not yet deprecated, eventually it will be, and I don't want to add a future burden. If that means the whole optimization can't happen, so be it -- as I wrote before, these optimizations aren't all that valuable given the typical LLM roundtrip.

Copy link
Copy Markdown
Contributor Author

@KRRT7 KRRT7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point — I can drop from __future__ import annotations and use string literals for the four annotations that reference the deferred modules instead:

_query_translator: "typechat.TypeChatJsonTranslator[search_query_schema.SearchQuery] | None" = None
_answer_translator: "typechat.TypeChatJsonTranslator[answer_response_schema.AnswerResponse] | None" = None

and in the query() signature:

async def query(
    self,
    question: str,
    search_options: "searchlang.LanguageSearchOptions | None" = None,
    answer_options: "answers.AnswerContextOptions | None" = None,
) -> str:

The TYPE_CHECKING block + runtime import inside query() stay the same — only the future import goes away. Type checkers still resolve the string annotations against the TYPE_CHECKING imports. I'll push that update.

On the value question — I was honestly hesitant to open import-time optimizations at all, which is why I asked first on #229 before going further. That said, if you'd rather keep the code straightforward and not defer these, I'm happy to close this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants