Forced tserver shutdown

[ivakegg]

The problem scenario we came across is where a node became inaccessible, however somehow tservers on that node were keeping the zookeeper locks alive. The master was still continually trying to contact that node for tablet assignments, various fate transactions (bulk loads, table deletes, compactions). All of the communications were timing out because sockets could not be established. The master got to the point where it was attempting to shutdown the tservers bug of course that was failing as well. After removing the node from the cluster.yaml and failing all of the fate transactions, the master still would not get past the issue. We finally has to issue an admin stop -f to force the lock to be removed and to get past the issue.

I would like the ability for the master to forcefully remove the zookeeper lock after a configurable number of attempts to stop the same tserver.

This was in accumulo 2.1.4

[dlmarion]

@ivakegg - so you are thinking that we should remove the tserver lock from the Manager in this code when halting via RPC throws an exception?

accumulo/server/manager/src/main/java/org/apache/accumulo/manager/Manager.java

Lines 1192 to 1202 in e557fd9

	log.warn("attempting to stop {}", server);
	try {
	TServerConnection connection2 = tserverSet.getConnection(server);
	if (connection2 != null) {
	connection2.halt(managerLock);
	}
	} catch (TTransportException e1) {
	// ignore: it's probably down
	} catch (Exception e2) {
	log.info("error talking to troublesome tablet server", e2);
	}

[ddanielr]

This has been completed in the 2.1 branch but was done as a no-opt merge into main.
A separate PR will be created for merging this feature into main due to the difference in lock removals.