Multi broker setup hardening

[jeanouii]

The restart sometimes does not happen fast enough and we try to create connections
Also using ephemeral ports because client uses VM connector and the NOB is already using transportConnectors.get(0).getConnectUri() so there is really no reason not to

[cshannon]

Since I wrote this test initially if I recall I think the main thing to wait for is not that the broker can accept connections, but that the network bridges were created by the connector and are online. I can look more at it and see what would make sense but maybe we can just check activeBridges() on the network connector for the status to verify we are online. The NetworkBridge interface may need to be enhanced.

[jeanouii]

I can rework the condition, the error was initially more that the network connections were connecting before the port would be open. So I thought it was the right condition but you are correct, we could harden it a bit more by checking that all network connection are active.
Let me know what you think is best and I can update the PR.

[cshannon]

I can rework the condition, the error was initially more that the network connections were connecting before the port would be open. So I thought it was the right condition but you are correct, we could harden it a bit more by checking that all network connection are active. Let me know what you think is best and I can update the PR.

Yeah you can give it a shot to see if there is a good way to detect the bridges are started and online

[cshannon]

Looking at this test again, it could actually be the brokers not coming online fast enough as we are creating client connections as well. @jeanouii - do you have the failed test output for where the test failed and the line number? I was thinking it was the network bridges starting but I peaked again and we already have a bunch of waits in the code for waiting for the network durables to come online, so I think you may actually be right it's the client connections that are the issue and not the bridges after all.

[jeanouii]

@cshannon It took me a while to move back enough to find the test failures again. Here they are

Error:  Failures: 
Error:    AMQ4160Test>CombinationTestSupport.runBare:113->CombinationTestSupport.runBare:107->testInactiveBridgStillActive:236->JmsMultipleBrokersTestSupport.startAllBrokers:285 Broker broker1 transport connectors ready
Error:    TwoSecureBrokerRequestReplyTest>CombinationTestSupport.runBare:113->CombinationTestSupport.runBare:107->testRequestReply:46->JmsMultipleBrokersTestSupport.startAllBrokers:285 Broker receiver transport connectors ready

You can find the complete logs at https://github.com/apache/activemq/actions/runs/21259419563/job/61182838770

[cshannon]

@cshannon It took me a while to move back enough to find the test failures again. Here they are

Thanks it looks like Github actions can publish test results, so that would make it easier to view in the future.

[jeanouii]

I'm happy to poke at it.
I have a shell script locally to hit the REST endpoint and collect the builds failing and then doing a grep to extract test failures.

Fine for me because I'm all hands on, but when I'm done having failures published would be better