Skip to content

chore: regenerate api-client, fix local docker stack, environment-agnostic VCR cassettes#1507

Open
tychtjan wants to merge 4 commits intomasterfrom
regen-test
Open

chore: regenerate api-client, fix local docker stack, environment-agnostic VCR cassettes#1507
tychtjan wants to merge 4 commits intomasterfrom
regen-test

Conversation

@tychtjan
Copy link
Copy Markdown
Contributor

@tychtjan tychtjan commented Mar 31, 2026

Summary

Brings the SDK in sync with latest staging: regenerated api-client, added a new feature field, fixed the local docker-compose stack, fixed data source credential handling, and made VCR cassettes fully environment-agnostic so they produce identical files whether recorded from local or staging.

1. Remove accidental Pydantic v2 model files

Cleans up Pydantic v2 model files that were accidentally committed and don't belong in the codebase.

2. Regenerate api-client from latest staging OpenAPI spec

Re-generates gooddata-api-client from the latest staging OpenAPI spec, picking up new endpoints and schema changes.

3. Add parameters field to CatalogDeclarativeAnalyticsLayer

Exposes the new parameters field from the API in the high-level SDK model.

4. Fix docker-compose api-gw cache duration format

The latest api-gw ECR image switched to Kotlin duration parsing — bare numbers like 60 are no longer accepted. Changed CACHE_EXPIRE_AFTER_WRITE and USER_CACHE_EXPIRE_AFTER_WRITE from 60 to 1m. Without this fix, api-gw crashes on startup.

5. Persist DS password after layout upload + fix local credential patching

  • Layout PUT doesn't persist passwords: Added an entities PATCH call after layout PUT to explicitly set passwords.
  • Local credential patching: Changed _patch_ds_credentials fixture to fall back to test_config["ds_password"] when the env var is absent.

6. Environment-agnostic VCR cassette normalization

Core changevcrpy_utils.py now normalizes all environment-specific values during cassette recording so that cassettes recorded from staging or local docker-compose produce identical files.

What configure_normalization() does

Called at session start from conftest.py, it reads the active test config and builds a list of string replacements (longest-first to prevent partial matches):

Staging value Canonical (local) value
https://python-sdk-dex.dev-latest.stg11.panther.intgdc.com http://localhost:3000
python-sdk-dex.dev-latest.stg11.panther.intgdc.com localhost
Python SDK Dex Default Organization
python-sdk-dex default

When running against local, the replacements list is empty (no-op).

Where normalization is applied

  • Request URIs (custom_before_request): staging host → http://localhost:3000
  • Request bodies (custom_before_request): org_id, org_name in JSON payloads
  • Response bodies (custom_before_response): handles both bytes and str body types (VCR passes bytes with decode_compressed_response=True)
  • Response headers (custom_before_response): Location headers containing org_id paths

Key implementation details

  • VCR passes response bodies as bytes, not str — the normalization decodes, replaces, and re-encodes
  • Response header values (like Location: /api/v1/entities/admin/organizations/python-sdk-dex) are normalized alongside the existing header sorting/placeholder logic
  • The existing match_on, custom serializer, and body matcher are unchanged
  • Normalization only runs during recording (before_record_* callbacks); playback is unaffected

7. Re-record all cassettes from staging

All gooddata-sdk cassettes re-recorded from staging with normalization active. The cassettes contain only canonical localhost:3000 / default values.

⚠️ Cache clearing when modifying tests-support

The tests-support package is installed as a wheel in tox environments. When modifying vcrpy_utils.py (or any file in tests-support), you must clear the caches before running tests, otherwise tox will use the stale installed version:

rm -rf packages/gooddata-sdk/.tox
uv cache clean tests-support --force

This is also documented in the configure_normalization() docstring.

Files changed

  • packages/tests-support/src/tests_support/vcrpy_utils.pyconfigure_normalization(), request/response/header normalization
  • packages/gooddata-sdk/tests/conftest.py — calls configure_normalization(config) at session start
  • packages/gooddata-sdk/tests/**/*.yaml — all cassettes re-recorded with normalized values

Test plan

  • Local docker-compose stack starts successfully
  • make test passes all tests against local stack (normalization is no-op)
  • make test-staging passes all 364 tests against staging (cassettes recorded with normalization)
  • CI pipeline passes — cassettes replay correctly without a live server
  • Zero occurrences of python-sdk-dex, stg11.panther, or Python SDK Dex in cassettes (excluding gd_test_config.yaml)

JIRA: TRIVIAL
risk: low

@tychtjan tychtjan force-pushed the regen-test branch 4 times, most recently from e81ff59 to c5db746 Compare April 1, 2026 08:51
@tychtjan tychtjan changed the title chore: regenerate api-client and update test fixtures for latest staging API chore: regenerate api-client, fix local docker stack, re-record cassettes Apr 1, 2026
@codecov
Copy link
Copy Markdown

codecov bot commented Apr 1, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.13%. Comparing base (35605f6) to head (985422f).
⚠️ Report is 6 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1507      +/-   ##
==========================================
+ Coverage   77.32%   78.13%   +0.80%     
==========================================
  Files         227      228       +1     
  Lines       14768    14928     +160     
==========================================
+ Hits        11420    11664     +244     
+ Misses       3348     3264      -84     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tychtjan tychtjan changed the title chore: regenerate api-client, fix local docker stack, re-record cassettes chore: regenerate api-client, fix local docker stack, environment-agnostic VCR cassettes Apr 1, 2026
tychtjan added 4 commits April 1, 2026 17:20
Regenerate gooddata-api-client from latest staging OpenAPI spec.
Add parameters field to CatalogDeclarativeAnalyticsLayer.

risk: low
- Use duration strings for api-gw cache config (60 -> 1m)
- Persist DS password after layout upload via entities PATCH
- Fix _patch_ds_credentials to fall back to test_config ds_password

risk: low
Add configure_normalization() to vcrpy_utils that rewrites all
environment-specific values to canonical local equivalents during
cassette recording. Normalization covers:

- Request URIs: staging host -> http://localhost:3000
- Request bodies: org_id, org_name in JSON payloads (bytes + str)
- Response bodies: all env-specific values (bytes + str)
- Response headers: Location headers with org_id paths

When running against local, the replacements list is empty (no-op).

IMPORTANT: When modifying tests-support, clear caches first:
  rm -rf packages/gooddata-sdk/.tox
  uv cache clean tests-support --force

risk: low
Comment on lines +20 to +23
# Fields stripped from request bodies before VCR body matching.
# These differ between local and staging environments but don't affect
# the logical identity of a request.
_ENV_SPECIFIC_BODY_FIELDS = {"password", "token", "url", "username", "privateKey"}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There might be some missing fields.

  • client_secret (used by Databricks data sources)
  • private_key_passphrase (used by Snowflake connections)

if not body:
return body or ""
try:
data = json.loads(body)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider using orjson instead of json. Should be faster.

# Populated by configure_normalization() before tests run.
# Each entry is (source_string, replacement_string).
# Ordered longest-first so more specific patterns match before substrings.
_normalization_replacements: list[tuple[str, str]] = []
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_normalization_replacements is a module-level empty list populated by configure_normalization(). If any test or import triggers cassette recording before the
session-scoped test_config fixture runs, normalization is silently a no-op. Currently safe due to fixture ordering, but brittle.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants