chore: regenerate api-client, fix local docker stack, environment-agnostic VCR cassettes#1507
chore: regenerate api-client, fix local docker stack, environment-agnostic VCR cassettes#1507
Conversation
e81ff59 to
c5db746
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1507 +/- ##
==========================================
+ Coverage 77.32% 78.13% +0.80%
==========================================
Files 227 228 +1
Lines 14768 14928 +160
==========================================
+ Hits 11420 11664 +244
+ Misses 3348 3264 -84 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Regenerate gooddata-api-client from latest staging OpenAPI spec. Add parameters field to CatalogDeclarativeAnalyticsLayer. risk: low
- Use duration strings for api-gw cache config (60 -> 1m) - Persist DS password after layout upload via entities PATCH - Fix _patch_ds_credentials to fall back to test_config ds_password risk: low
Add configure_normalization() to vcrpy_utils that rewrites all environment-specific values to canonical local equivalents during cassette recording. Normalization covers: - Request URIs: staging host -> http://localhost:3000 - Request bodies: org_id, org_name in JSON payloads (bytes + str) - Response bodies: all env-specific values (bytes + str) - Response headers: Location headers with org_id paths When running against local, the replacements list is empty (no-op). IMPORTANT: When modifying tests-support, clear caches first: rm -rf packages/gooddata-sdk/.tox uv cache clean tests-support --force risk: low
| # Fields stripped from request bodies before VCR body matching. | ||
| # These differ between local and staging environments but don't affect | ||
| # the logical identity of a request. | ||
| _ENV_SPECIFIC_BODY_FIELDS = {"password", "token", "url", "username", "privateKey"} |
There was a problem hiding this comment.
There might be some missing fields.
- client_secret (used by Databricks data sources)
- private_key_passphrase (used by Snowflake connections)
| if not body: | ||
| return body or "" | ||
| try: | ||
| data = json.loads(body) |
There was a problem hiding this comment.
Consider using orjson instead of json. Should be faster.
| # Populated by configure_normalization() before tests run. | ||
| # Each entry is (source_string, replacement_string). | ||
| # Ordered longest-first so more specific patterns match before substrings. | ||
| _normalization_replacements: list[tuple[str, str]] = [] |
There was a problem hiding this comment.
_normalization_replacements is a module-level empty list populated by configure_normalization(). If any test or import triggers cassette recording before the
session-scoped test_config fixture runs, normalization is silently a no-op. Currently safe due to fixture ordering, but brittle.
Summary
Brings the SDK in sync with latest staging: regenerated api-client, added a new feature field, fixed the local docker-compose stack, fixed data source credential handling, and made VCR cassettes fully environment-agnostic so they produce identical files whether recorded from local or staging.
1. Remove accidental Pydantic v2 model files
Cleans up Pydantic v2 model files that were accidentally committed and don't belong in the codebase.
2. Regenerate api-client from latest staging OpenAPI spec
Re-generates
gooddata-api-clientfrom the latest staging OpenAPI spec, picking up new endpoints and schema changes.3. Add
parametersfield toCatalogDeclarativeAnalyticsLayerExposes the new
parametersfield from the API in the high-level SDK model.4. Fix docker-compose api-gw cache duration format
The latest
api-gwECR image switched to Kotlin duration parsing — bare numbers like60are no longer accepted. ChangedCACHE_EXPIRE_AFTER_WRITEandUSER_CACHE_EXPIRE_AFTER_WRITEfrom60to1m. Without this fix,api-gwcrashes on startup.5. Persist DS password after layout upload + fix local credential patching
_patch_ds_credentialsfixture to fall back totest_config["ds_password"]when the env var is absent.6. Environment-agnostic VCR cassette normalization
Core change —
vcrpy_utils.pynow normalizes all environment-specific values during cassette recording so that cassettes recorded from staging or local docker-compose produce identical files.What
configure_normalization()doesCalled at session start from
conftest.py, it reads the active test config and builds a list of string replacements (longest-first to prevent partial matches):https://python-sdk-dex.dev-latest.stg11.panther.intgdc.comhttp://localhost:3000python-sdk-dex.dev-latest.stg11.panther.intgdc.comlocalhostPython SDK DexDefault Organizationpython-sdk-dexdefaultWhen running against local, the replacements list is empty (no-op).
Where normalization is applied
custom_before_request): staging host →http://localhost:3000custom_before_request): org_id, org_name in JSON payloadscustom_before_response): handles bothbytesandstrbody types (VCR passesbyteswithdecode_compressed_response=True)custom_before_response):Locationheaders containing org_id pathsKey implementation details
bytes, notstr— the normalization decodes, replaces, and re-encodesLocation: /api/v1/entities/admin/organizations/python-sdk-dex) are normalized alongside the existing header sorting/placeholder logicmatch_on, custom serializer, and body matcher are unchangedbefore_record_*callbacks); playback is unaffected7. Re-record all cassettes from staging
All gooddata-sdk cassettes re-recorded from staging with normalization active. The cassettes contain only canonical
localhost:3000/defaultvalues.tests-supportThe
tests-supportpackage is installed as a wheel in tox environments. When modifyingvcrpy_utils.py(or any file intests-support), you must clear the caches before running tests, otherwise tox will use the stale installed version:This is also documented in the
configure_normalization()docstring.Files changed
packages/tests-support/src/tests_support/vcrpy_utils.py—configure_normalization(), request/response/header normalizationpackages/gooddata-sdk/tests/conftest.py— callsconfigure_normalization(config)at session startpackages/gooddata-sdk/tests/**/*.yaml— all cassettes re-recorded with normalized valuesTest plan
make testpasses all tests against local stack (normalization is no-op)make test-stagingpasses all 364 tests against staging (cassettes recorded with normalization)python-sdk-dex,stg11.panther, orPython SDK Dexin cassettes (excludinggd_test_config.yaml)JIRA: TRIVIAL
risk: low