Using elasticsearch as visibility store breaks existing elasticsearch cluster

[jgoodall]

Expected Behavior

I expect that my elasticsearch instance - which is in the same Kubernetes workload where I am testing temporal - is going to work the way it always has, but adding temporal (installed via the helm chart) changes the default elasticsearch behavior in an unexpected way.

Actual Behavior

I believe that because of schema/elasticsearch/visibility/cluster_settings_v7.json, the elasticsearch default index setting action.auto_create_index: true is changed to false by the temporal server. Since this has the effect of breaking existing elasticsearch clusters, this seems like a bug. If it was scoped to just temporal indices that would be fine, but this is changing the behavior of existing indices by changing the default value. Even if I start elasticsearch using the ECK option action.auto_create_index: true, temporal overrides that.

Steps to Reproduce the Problem

1. Start temporal with helm chart setting elasticsearch as the visibility store: --set server.config.persistence.datastores.visibility.elasticsearch.version=v7

Specifications

• Version: temporal 1.30.3 / helm chart 1.0.0-rc3
• Platform: linux / kubernetes

[jgoodall]

A quick fix if this setting is needed is to scope the setting in schema/elasticsearch/visibility/cluster_settings_v7.json to only turn off auto_create_index for temporal visibility index, but leave it on otherwise (that might also mess with existing patterns since it is a dynamic setting it could override previously defined disallow rules).

{
  "persistent": {
    "action.auto_create_index": "-temporal_visibility_v1*,+*"
  }
}

This would follow the index pattern of the template, "index_patterns": ["temporal_visibility_v1*"] in schema/elasticsearch/visibility/versioned/v14/index_template_v7.json

But it would be preferable not to rely on this setting.

Elasticsearch Automatically create data streams and indices documentation