How to detect all OPAL_DATA_CONFIG_SOURCES have been fully loaded at startup

[goncalo-silva-prt]

We are using OPAL_OPA_HEALTH_CHECK_POLICY_ENABLED to try to detect when all initial data entries have been fully loaded into OPA.

Our setup uses OPAL_DATA_CONFIG_SOURCES with external_source_url. The external endpoint is responsible for doing partitioning of th data and returning the multiple entries(for example, http://our-api.com/api/users?partition=0, http://our-api.com/api/users?partition=1, etc).

We observed the following behaviors when calling /v1/data/system/opal/:

• The healthy becomes true immediately after the first external data source is successfully loaded, even though several other partitions are still being fetched.
• The ready remains false permanently.

We are not using the policy updater (OPAL_POLICY_UPDATER_ENABLED=false) and the server has OPAL_REPO_WATCHER_ENABLED=false.

Is there a built-in or recommended way to know when all entries from the external_source_url are fully loaded?
We need that to identify when an instance is ready to receive traffic.

[orweis]

Hi, Welcome 🤗

If ready remains false, it means something failed in loading either the policy or data.
Check your OPAL client logs, and OPA logs

Note: You must have some policy and some data.

[orweis]

nope.
You got my answer.

[goncalo-silva-prt]

Well, I definitely have policies and data.

I'll try to explain the question more clearly.
In our setup:

• Policies are loaded using OPA bundles via the OPA config.yaml configuration, not through OPAL policy updater (OPAL_POLICY_UPDATER_ENABLED=false).
• Data is loaded via external_source_url, which returns multiple entries (/api/users?partition=0, /api/users?partition=1).

In a setup like this, should /v1/data/system/opal/ready become true after all external data sources are successfully loaded? Or since OPAL_POLICY_UPDATER_ENABLED is false, ready will always be false?

[orweis]

Your setup is not a supported one.
OPAL has no way of tracking your externally loaded policy.

[goncalo-silva-prt]

Maybe the policy updater could be ignored when OPAL_POLICY_UPDATER_ENABLED is false. It looks like something similar is already done for the data updater here .

I also ran some local tests enabling OPAL_POLICY_UPDATER_ENABLED. In that case, after the policy is loaded and the first chunk of data (/api/users?partition=0) is loaded, ready becomes true immediately, and it doesn’t wait for all external data sources to finish loading.

This is unexpected behavior, I would say. If there is a way to load multiple data sources, there should be a way to start serving requests only when all entries are loaded.

[orweis]

If you'd like to work on this feature, we'd be happy to accept contributions