Best approach for handling large initial datasets in OPAL_DATA_CONFIG_SOURCES

[goncalo-silva-prt]

Hi,

My initial data is fetched through OPAL_DATA_CONFIG_SOURCES, where I call an API to retrieve users and roles. I’m wondering what OPAL’s recommended approach is for handling large datasets during the initial data load.

Specifically:

• Is there a way to paginate the HTTP request within OPAL_DATA_CONFIG_SOURCES?
• Or is designing a multi-tenant architecture (or another strategy) the recommended solution for scalability?

Any guidance, best practices, or pointers to relevant docs/examples would be much appreciated. Thanks!

[orweis]

1. Multiple Data Source Entries

You can break down your data loading into multiple entries within OPAL_DATA_CONFIG_SOURCES, where each entry fetches a different subset of data and stores it at different paths[(1)](https://docs.opal.ac/getting-started/tldr). Each entry can point to a different URL (e.g., different pages of your API) with unique dst_path values.

2. External API Server for Dynamic Configuration

OPAL supports redirecting to an external API server that can serve different DataSourceEntry configurations dynamically[(2)](https://docs.opal.ac/tutorials/configure_external_data_sources). This approach allows you to:

• Have your external API server handle the pagination logic
• Return multiple data source entries programmatically based on your data size
• Serve different configurations based on client identity using JWT claims[(2)](https://docs.opal.ac/tutorials/configure_external_data_sources)

The external API server would expose an endpoint that returns a DataSourceConfig with multiple entries, each handling a portion of your paginated data[(2)](https://docs.opal.ac/tutorials/configure_external_data_sources).

Multi-Tenant Architecture Consideration

For scalability with multiple tenants, OPAL supports a single-topic multi-tenant configuration where all tenants share one topic but each gets its own data source with different destination paths[(3)](https://docs.opal.ac/tutorials/single-topic-multi-tenant). This approach provides better resource utilization and doesn't require service restarts when adding new tenants[(3)](https://docs.opal.ac/tutorials/single-topic-multi-tenant).

For more details on configuring external data sources and the OPAL Scope API, check out the [Use External Data Source guide](https://docs.permit.io/how-to/manage-data/use-external-data-source)[[(4)](https://docs.permit.io/how-to/manage-data/use-external-data-source)](https://docs.permit.io/how-to/manage-data/use-external-data-source).

**