docker upgrade to 11.2.8 opensearch authentication error

[tsdogs]

Acknowledgements

• I have searched (https://github.com/utmstack/UTMStack/issues) for past instances of this issue
• I have verified that my UTMStack version is up-to-date

Describe the bug

System upgraded to latest version 11.2.8
backend docker keeps restarting as connection to opensearch container (elasticsearch) fails with error 401 (unauthorized)
I've checked the configured password (from utmstack.yml) and hash is correct inside the internal_users.yml for the admin user (seems in previous versions opensearch was not using authentication?)
Downgraded to 11.2.6 and UTM Stack started working again.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

container should start and correctly authenticate to elasticsearch

Current Behavior

container fails to connect to elasticsearch and keeps on beeing created

Reproduction Steps

Upgrade to 11.2.8 of dockerized UtmStack

Possible Solution

No response

Additional Information/Context

I've not tried a fresh install

UTMStack Version

11.2.8

Operating System and version

Ubuntu 24.04

Hypervisor and Version | Server Vendor and Model

docker-ce 29.5.0

Browser and version

Any

[tsdogs]

Now UTMStack auto upgraded to 11.2.9 and I still have the issue.

[tsdogs]

This is from the opensearch docker:

It seems it has to do with authentication. I've tried changing various configuration files with matching password (hashed)
Nothing seems to work beside downgrading.

[karlbrown87]

@tsdogs Thank you for reporting this issue.

We have analyzed the behavior described and were able to reproduce a similar authentication issue in our development environment.

Based on our findings, we recommend verifying that the OpenSearch password configured across all UTMStack components is consistent. In particular, please ensure that the password defined in /root/utmstack.yml under the opensearch_password: parameter matches the passwords currently configured and running in the OpenSearch and Backend services.

You can verify the configured values using the following commands:

cat /root/utmstack.yml | grep opensearch_password

cat /usr/local/bin/compose.yml | grep ELASTICSEARCH_PASSWORD

cat /usr/local/bin/compose.yml | grep OPENSEARCH_INITIAL_ADMIN_PASSWORD

cat /utmstack/events-engine-workdir/pipeline/opensearch_plugins.yaml 

cat /utmstack/events-engine-workdir/pipeline/utmstack_plugins.yaml

Additionally, please verify the environment variables currently applied to the running Docker services:

docker service inspect utmstack_node1 | grep OPENSEARCH_INITIAL_ADMIN_PASSWORD

docker service inspect utmstack_user-auditor | grep ELASTICSEARCH_PASSWORD

docker service inspect utmstack_backend | grep ELASTICSEARCH_PASSWORD

Please make sure that all services are using the same password defined in /root/utmstack.yml. We have observed that authentication failures resulting in HTTP 401 errors can occur when one or more services retain an outdated password after an upgrade, causing OpenSearch authentication to fail and the Backend service to continuously restart.

Once you have verified the configuration, please let us know the results and we will be happy to continue assisting with the investigation and the solution if necessary.

Regards,
Carlos

[tsdogs]

@karlbrown87 Thank you for your answer.
I have checked all the files indicated and the opensearch password is the same in all of them.

Could the problem be that the auto-generated password has a couple of # and a @ in it?