IGNITE-28105 Ensure new leader clock is not lower than last applied cmd

[EgorKuts]

It's possible that new elected leader would have a lower clock's value than the last applied command(see ItNewLeaderClockTest).
Such behavior breaks state machine invariants and causes all nodes in the replication group to fail.

[ascherbakoff]

Propagating HLC in each AppendEntry request creates additinal contention on HLC.
For this reason it was removed earlier.
Could we instead do it in RequestVoteRequest ?

[EgorKuts]

No. If the leader has the most advanced clock, it is possible for the new leader to have a stale clock value. This scenario is shown in the reproducer (2 out of 3 nodes have a lower HLC).

If we need to address the contention, I propose creating a separate clock per partition. This clock would be updated on each appendEntry and merged with the main clock when the node becomes a leader.

[ascherbakoff]

Safe timestamp is already passed with each group command.
Can we just propagate it's value to a node's clock to ensure monotonic invariant, like this:

                    if (safeTimestamp != null) {
                        clock.update(safeTimestamp);

                        try {
                            safeTimeTracker.update(safeTimestamp, commandIndex, commandTerm, command);
                        } catch (TrackerClosedException ignored) {
                            // Ignored.
                        }
                    }