Rolling Upgrade: 10.4 node failed with Assertion `active() == false' failed

[ShahriyarR]

Opening this issue here because actual assertion happened in:
"/home/shako/Galera_Tests/MariaDB/wsrep-lib/src/transaction.cpp", line=159

Here is the scenario:

• Started 5 nodes 10.3
• Upgraded node5 to 10.4
• Run following SQLs on one of 10.3 nodes:

CREATE TABLE test.sbtest4(id int(10) NOT NULL, PRIMARY KEY (id))ENGINE=InnoDB DEFAULT CHARSET=latin1;
CREATE TABLE IF NOT EXISTS test.temp4 AS(SELECT * FROM test.sbtest4)
CREATE TABLE IF NOT EXISTS test.temp5 AS (SELECT * FROM test.sbtest4)

• Lose node5

Bt output:

(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007ff0d9c38801 in __GI_abort () at abort.c:79
#2  0x00007ff0d9c2839a in __assert_fail_base (fmt=0x7ff0d9daf7d8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x55ac37dc3457 "active() == false", 
    file=file@entry=0x55ac37dc3418 "/home/shako/Galera_Tests/MariaDB/wsrep-lib/src/transaction.cpp", line=line@entry=159, 
    function=function@entry=0x55ac37dc47e0 <wsrep::transaction::start_transaction(wsrep::ws_handle const&, wsrep::ws_meta const&)::__PRETTY_FUNCTION__> "int wsrep::transaction::start_transaction(const wsrep::ws_handle&, const wsrep::ws_meta&)") at assert.c:92
#3  0x00007ff0d9c28412 in __GI___assert_fail (assertion=0x55ac37dc3457 "active() == false", 
    file=0x55ac37dc3418 "/home/shako/Galera_Tests/MariaDB/wsrep-lib/src/transaction.cpp", line=159, 
    function=0x55ac37dc47e0 <wsrep::transaction::start_transaction(wsrep::ws_handle const&, wsrep::ws_meta const&)::__PRETTY_FUNCTION__> "int wsrep::transaction::start_transaction(const wsrep::ws_handle&, const wsrep::ws_meta&)") at assert.c:101
#4  0x000055ac3789a59c in wsrep::transaction::start_transaction (this=0x7ff0ac006c30, ws_handle=..., ws_meta=...)
    at /home/shako/Galera_Tests/MariaDB/wsrep-lib/src/transaction.cpp:159
#5  0x000055ac36f3dcac in wsrep::client_state::start_transaction (this=0x7ff0ac006bc0, wsh=..., meta=...)
    at /home/shako/Galera_Tests/MariaDB/wsrep-lib/include/wsrep/client_state.hpp:366
#6  0x000055ac36f3b4df in Wsrep_high_priority_service::start_transaction (this=0x7ff0d45395f0, ws_handle=..., ws_meta=...)
    at /home/shako/Galera_Tests/MariaDB/sql/wsrep_high_priority_service.cc:198
#7  0x000055ac378907f7 in apply_write_set (server_state=..., high_priority_service=..., ws_handle=..., ws_meta=..., data=...)
    at /home/shako/Galera_Tests/MariaDB/wsrep-lib/src/server_state.cpp:192
#8  0x000055ac37894056 in wsrep::server_state::on_apply (this=0x55ac39f1e830, high_priority_service=..., ws_handle=..., ws_meta=..., data=...)
    at /home/shako/Galera_Tests/MariaDB/wsrep-lib/src/server_state.cpp:944
#9  0x000055ac378aa045 in wsrep::high_priority_service::apply (this=0x7ff0d45395f0, ws_handle=..., ws_meta=..., data=...)
    at /home/shako/Galera_Tests/MariaDB/wsrep-lib/include/wsrep/high_priority_service.hpp:46
#10 0x000055ac378a773e in (anonymous namespace)::apply_cb (ctx=0x7ff0d45395f0, wsh=0x7ff0d4538860, flags=73, buf=0x7ff0d4538870, meta=0x7ff0d4538b20, 
    exit_loop=0x7ff0d4538abd) at /home/shako/Galera_Tests/MariaDB/wsrep-lib/src/wsrep_provider_v26.cpp:489
---Type <return> to continue, or q <return> to quit---
#11 0x00007ff0d7864492 in galera::TrxHandleSlave::apply (this=this@entry=0x7ff0ac05f980, recv_ctx=recv_ctx@entry=0x7ff0d45395f0, 
    apply_cb=0x55ac378a750f <(anonymous namespace)::apply_cb(void*, wsrep_ws_handle_t const*, uint32_t, wsrep_buf_t const*, wsrep_trx_meta_t const*, wsrep_bool_t*)>, 
    meta=..., exit_loop=exit_loop@entry=@0x7ff0d4538abd: false) at /home/shako/Galera_Tests/Galera/galera/src/trx_handle.cpp:418
#12 0x00007ff0d7898225 in galera::ReplicatorSMM::apply_trx (this=this@entry=0x55ac39f506c0, recv_ctx=recv_ctx@entry=0x7ff0d45395f0, ts=...)
    at /home/shako/Galera_Tests/Galera/galera/src/replicator_smm.cpp:489
#13 0x00007ff0d789c4ee in galera::ReplicatorSMM::process_trx (this=0x55ac39f506c0, recv_ctx=0x7ff0d45395f0, ts_ptr=...)
    at /home/shako/Galera_Tests/Galera/galera/src/replicator_smm.cpp:2114
#14 0x00007ff0d787ebf4 in galera::GcsActionSource::process_writeset (this=this@entry=0x55ac39f59db0, recv_ctx=0x7ff0d45395f0, act=..., exit_loop=@0x7ff0d453937e: false)
    at /home/shako/Galera_Tests/Galera/galera/src/gcs_action_source.cpp:62
#15 0x00007ff0d787ed85 in galera::GcsActionSource::dispatch (this=this@entry=0x55ac39f59db0, recv_ctx=recv_ctx@entry=0x7ff0d45395f0, act=..., 
    exit_loop=@0x7ff0d453937e: false) at /home/shako/Galera_Tests/Galera/galera/src/gcs_action_source.cpp:109
#16 0x00007ff0d787f159 in galera::GcsActionSource::process (this=0x55ac39f59db0, recv_ctx=0x7ff0d45395f0, exit_loop=@0x7ff0d453937e: false)
    at /home/shako/Galera_Tests/Galera/galera/src/gcs_action_source.cpp:182
#17 0x00007ff0d7895720 in galera::ReplicatorSMM::async_recv (this=0x55ac39f506c0, recv_ctx=0x7ff0d45395f0)
    at /home/shako/Galera_Tests/Galera/galera/src/replicator_smm.cpp:383
#18 0x00007ff0d78b16c8 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at /home/shako/Galera_Tests/Galera/galera/src/wsrep_provider.cpp:236
#19 0x000055ac378a8430 in wsrep::wsrep_provider_v26::run_applier (this=0x55ac39f1ece0, applier_ctx=0x7ff0d45395f0)
    at /home/shako/Galera_Tests/MariaDB/wsrep-lib/src/wsrep_provider_v26.cpp:690
#20 0x000055ac36f5dfb4 in wsrep_replication_process (thd=0x7ff0ac000d60, arg=0x0) at /home/shako/Galera_Tests/MariaDB/sql/wsrep_thd.cc:61
#21 0x000055ac36f4f766 in start_wsrep_THD (arg=0x55ac39fd7c60) at /home/shako/Galera_Tests/MariaDB/sql/wsrep_mysqld.cc:2768
#22 0x00007ff0dab336db in start_thread (arg=0x7ff0d453a700) at pthread_create.c:463
#23 0x00007ff0d9d1988f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

[ShahriyarR]

bt full:

(gdb) bt full
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
        set = {__val = {552967, 140671654933632, 140672331123488, 140672422643168, 140672991330304, 140671654933632, 140671654933733, 140671654933632, 140671654933632, 
            140671654933837, 140671654933932, 140671654933632, 140671654933932, 0, 0, 0}}
        pid = <optimized out>
        tid = <optimized out>
        ret = <optimized out>
#1  0x00007ff0d9c38801 in __GI_abort () at abort.c:79
        save_stage = 1
        act = {__sigaction_handler = {sa_handler = 0x7ff0ac05b480, sa_sigaction = 0x7ff0ac05b480}, sa_mask = {__val = {0, 140671654559776, 0, 0, 0, 140672331120936, 0, 
              140672331120768, 140672423899600, 21474836480, 140672423884760, 0, 10070624250478259456, 140672423869716, 0, 140672423884760}}, sa_flags = 937178136, 
          sa_restorer = 0x55ac37dc3457}
        sigs = {__val = {32, 0 <repeats 15 times>}}
        __cnt = <optimized out>
        __set = <optimized out>
        __cnt = <optimized out>
        __set = <optimized out>
#2  0x00007ff0d9c2839a in __assert_fail_base (fmt=0x7ff0d9daf7d8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x55ac37dc3457 "active() == false", 
    file=file@entry=0x55ac37dc3418 "/home/shako/Galera_Tests/MariaDB/wsrep-lib/src/transaction.cpp", line=line@entry=159, 
    function=function@entry=0x55ac37dc47e0 <wsrep::transaction::start_transaction(wsrep::ws_handle const&, wsrep::ws_meta const&)::__PRETTY_FUNCTION__> "int wsrep::transaction::start_transaction(const wsrep::ws_handle&, const wsrep::ws_meta&)") at assert.c:92
        str = 0x7ff0ac05b480 ""
        total = 4096
#3  0x00007ff0d9c28412 in __GI___assert_fail (assertion=0x55ac37dc3457 "active() == false", 
    file=0x55ac37dc3418 "/home/shako/Galera_Tests/MariaDB/wsrep-lib/src/transaction.cpp", line=159, 
---Type <return> to continue, or q <return> to quit---
    function=0x55ac37dc47e0 <wsrep::transaction::start_transaction(wsrep::ws_handle const&, wsrep::ws_meta const&)::__PRETTY_FUNCTION__> "int wsrep::transaction::start_transaction(const wsrep::ws_handle&, const wsrep::ws_meta&)") at assert.c:101
No locals.
#4  0x000055ac3789a59c in wsrep::transaction::start_transaction (this=0x7ff0ac006c30, ws_handle=..., ws_meta=...)
    at /home/shako/Galera_Tests/MariaDB/wsrep-lib/src/transaction.cpp:159
        __PRETTY_FUNCTION__ = "int wsrep::transaction::start_transaction(const wsrep::ws_handle&, const wsrep::ws_meta&)"
#5  0x000055ac36f3dcac in wsrep::client_state::start_transaction (this=0x7ff0ac006bc0, wsh=..., meta=...)
    at /home/shako/Galera_Tests/MariaDB/wsrep-lib/include/wsrep/client_state.hpp:366
        lock = {mutex_ = @0x7ff0ac006b88, locked_ = true}
        __PRETTY_FUNCTION__ = "int wsrep::client_state::start_transaction(const wsrep::ws_handle&, const wsrep::ws_meta&)"
#6  0x000055ac36f3b4df in Wsrep_high_priority_service::start_transaction (this=0x7ff0d45395f0, ws_handle=..., ws_meta=...)
    at /home/shako/Galera_Tests/MariaDB/sql/wsrep_high_priority_service.cc:198
        _db_stack_frame_ = {func = 0x55ac3795d0da "wsrep_replication_process", file = 0x55ac3795d0a8 "/home/shako/Galera_Tests/MariaDB/sql/wsrep_thd.cc", 
          level = 2147483650, line = 198, prev = 0x7ff0d45395d0}
#7  0x000055ac378907f7 in apply_write_set (server_state=..., high_priority_service=..., ws_handle=..., ws_meta=..., data=...)
    at /home/shako/Galera_Tests/MariaDB/wsrep-lib/src/server_state.cpp:192
        ret = 0
        __PRETTY_FUNCTION__ = "int apply_write_set(wsrep::server_state&, wsrep::high_priority_service&, const wsrep::ws_handle&, const wsrep::ws_meta&, const wsrep::const_buffer&)"
#8  0x000055ac37894056 in wsrep::server_state::on_apply (this=0x55ac39f1e830, high_priority_service=..., ws_handle=..., ws_meta=..., data=...)
    at /home/shako/Galera_Tests/MariaDB/wsrep-lib/src/server_state.cpp:944
        __PRETTY_FUNCTION__ = "int wsrep::server_state::on_apply(wsrep::high_priority_service&, const wsrep::ws_handle&, const wsrep::ws_meta&, const wsrep::const_buffer&)"
#9  0x000055ac378aa045 in wsrep::high_priority_service::apply (this=0x7ff0d45395f0, ws_handle=..., ws_meta=..., data=...)
    at /home/shako/Galera_Tests/MariaDB/wsrep-lib/include/wsrep/high_priority_service.hpp:46
---Type <return> to continue, or q <return> to quit---
No locals.
#10 0x000055ac378a773e in (anonymous namespace)::apply_cb (ctx=0x7ff0d45395f0, wsh=0x7ff0d4538860, flags=73, buf=0x7ff0d4538870, meta=0x7ff0d4538b20, 
    exit_loop=0x7ff0d4538abd) at /home/shako/Galera_Tests/MariaDB/wsrep-lib/src/wsrep_provider_v26.cpp:489
        high_priority_service = 0x7ff0d45395f0
        __PRETTY_FUNCTION__ = "wsrep_cb_status_t {anonymous}::apply_cb(void*, const wsrep_ws_handle_t*, uint32_t, const wsrep_buf_t*, const wsrep_trx_meta_t*, wsrep_bool_t*)"
        data = {ptr_ = 0x7ff0c8000050, size_ = 133}
        ws_handle = {transaction_id_ = {id_ = 46}, opaque_ = 0x7ff0ac05f980}
        ws_meta = {gtid_ = {static undefined_ = {static undefined_ = <same as static member of an already seen type>, id_ = {static undefined_ = {
                  static undefined_ = <same as static member of an already seen type>, data_ = '\000' <repeats 15 times>}, data_ = '\000' <repeats 15 times>}, seqno_ = {
                seqno_ = -1}}, id_ = {static undefined_ = {static undefined_ = <same as static member of an already seen type>, data_ = '\000' <repeats 15 times>}, 
              data_ = "\032z\235\270,@\021\351\260\036Oޓ\034\026P"}, seqno_ = {seqno_ = 6}}, stid_ = {server_id_ = {static undefined_ = {
                static undefined_ = <same as static member of an already seen type>, data_ = '\000' <repeats 15 times>}, 
              data_ = "\032y\370\316,@\021\351\247<\206\031\070\252\361{"}, transaction_id_ = {id_ = 46}, client_id_ = {id_ = 19}}, depends_on_ = {seqno_ = 5}, flags_ = 19}
#11 0x00007ff0d7864492 in galera::TrxHandleSlave::apply (this=this@entry=0x7ff0ac05f980, recv_ctx=recv_ctx@entry=0x7ff0d45395f0, 
    apply_cb=0x55ac378a750f <(anonymous namespace)::apply_cb(void*, wsrep_ws_handle_t const*, uint32_t, wsrep_buf_t const*, wsrep_trx_meta_t const*, wsrep_bool_t*)>, 
    meta=..., exit_loop=exit_loop@entry=@0x7ff0d4538abd: false) at /home/shako/Galera_Tests/Galera/galera/src/trx_handle.cpp:418
        buf = <optimized out>
        wb = {ptr = 0x7ff0c8000050, len = 133}
        i = 0
        wsrep_flags = 73
        err = 0
        ws = @0x7ff0ac05faa0: {<gu::RecordSetIn<galera::DataSet::RecordIn>> = {<gu::RecordSetInBase> = {<gu::RecordSet> = {static MAX_VERSION = gu::RecordSet::VER2, 
                static VER2_ALIGNMENT = 8, size_ = 157, count_ = 1, version_ = 2 '\002', check_type_ = 3 '\003', alignment_ = 8 '\b'}, head_ = 0x7ff0c8000038 "+", 
---Type <return> to continue, or q <return> to quit---
              next_ = 157, begin_ = 24}, <No data fields>}, version_ = galera::DataSet::VER1}
        wh = {trx_id = 46, opaque = 0x7ff0ac05f980}
        __FUNCTION__ = "apply"
#12 0x00007ff0d7898225 in galera::ReplicatorSMM::apply_trx (this=this@entry=0x55ac39f506c0, recv_ctx=recv_ctx@entry=0x7ff0d45395f0, ts=...)
    at /home/shako/Galera_Tests/Galera/galera/src/replicator_smm.cpp:489
        __PRETTY_FUNCTION__ = "void galera::ReplicatorSMM::apply_trx(void*, galera::TrxHandleSlave&)"

        ae = {<gu::Exception> = {<std::exception> = {<No data fields>}, msg_ = "", err_ = 0}, data_ = 0x0, const_data_ = 0x0, data_len_ = 0}
        ao = {global_seqno_ = 6, depends_seqno_ = 5, is_local_ = false, is_toi_ = false, trx_ = 0x7ff0ac05f980}
        co = {global_seqno_ = 6, mode_ = galera::ReplicatorSMM::CommitOrder::NO_OOOC, is_local_ = false, trx_ = 0x7ff0ac05f980}
        __FUNCTION__ = "apply_trx"
        meta = {gtid = {uuid = {data = "\032z\235\270,@\021\351\260\036Oޓ\034\026P", alignment = 16794275046261357082}, seqno = 6}, stid = {node = {
              data = "\032y\370\316,@\021\351\247<\206\031\070\252\361{", alignment = 16794275046636419354}, trx = 46, conn = 19}, depends_on = 5}
        exit_loop = false
        safe_to_discard = <optimized out>
#13 0x00007ff0d789c4ee in galera::ReplicatorSMM::process_trx (this=0x55ac39f506c0, recv_ctx=0x7ff0d45395f0, ts_ptr=...)
    at /home/shako/Galera_Tests/Galera/galera/src/replicator_smm.cpp:2114
        __PRETTY_FUNCTION__ = "virtual void galera::ReplicatorSMM::process_trx(void*, const TrxHandleSlavePtr&)"
        ts = @0x7ff0ac05f980: {<galera::TrxHandle> = {_vptr.TrxHandle = 0x7ff0d7b94078 <vtable for galera::TrxHandleSlave+16>, static TRXHANDLE_FLAGS_MASK = 33791, 
            static EXPLICIT_ROLLBACK_FLAGS = 10, static FLAGS_MATCH_API_FLAGS = true, static num_states_ = 12, state_ = {delete_ = false, 
              trans_map_ = 0x7ff0d7bb20a0 <galera::TrxHandleSlave::trans_map_>, state_ = {first = galera::TrxHandle::S_APPLYING, second = 467}, 
              state_hist_ = std::vector of length 2, capacity 2 = {{first = galera::TrxHandle::S_REPLICATING, second = 0}, {first = galera::TrxHandle::S_CERTIFYING, 
                  second = 3042}}}, source_id_ = {data = "\032y\370\316,@\021\351\247<\206\031\070\252\361{", alignment = 16794275046636419354}, conn_id_ = 19, 
            trx_id_ = 46, timestamp_ = 5302100489997, version_ = 4, write_set_flags_ = 73, local_ = false, master_ = false}, static trans_map_ = {
            impl_ = std::unordered_map with 11 elements = {[{from_ = galera::TrxHandle::S_ROLLING_BACK, to_ = galera::TrxHandle::S_ROLLED_BACK}] = {
---Type <return> to continue, or q <return> to quit---
                pre_guard_ = empty std::__cxx11::list, post_guard_ = empty std::__cxx11::list, pre_action_ = empty std::__cxx11::list, 
                post_action_ = empty std::__cxx11::list}, [{from_ = galera::TrxHandle::S_COMMITTING, to_ = galera::TrxHandle::S_COMMITTED}] = {
                pre_guard_ = empty std::__cxx11::list, post_guard_ = empty std::__cxx11::list, pre_action_ = empty std::__cxx11::list, 
                post_action_ = empty std::__cxx11::list}, [{from_ = galera::TrxHandle::S_COMMITTING, to_ = galera::TrxHandle::S_REPLAYING}] = {
                pre_guard_ = empty std::__cxx11::list, post_guard_ = empty std::__cxx11::list, pre_action_ = empty std::__cxx11::list, 
                post_action_ = empty std::__cxx11::list}, [{from_ = galera::TrxHandle::S_APPLYING, to_ = galera::TrxHandle::S_REPLAYING}] = {
                pre_guard_ = empty std::__cxx11::list, post_guard_ = empty std::__cxx11::list, pre_action_ = empty std::__cxx11::list, 
                post_action_ = empty std::__cxx11::list}, [{from_ = galera::TrxHandle::S_REPLICATING, to_ = galera::TrxHandle::S_ABORTING}] = {
                pre_guard_ = empty std::__cxx11::list, post_guard_ = empty std::__cxx11::list, pre_action_ = empty std::__cxx11::list, 
                post_action_ = empty std::__cxx11::list}, [{from_ = galera::TrxHandle::S_APPLYING, to_ = galera::TrxHandle::S_COMMITTING}] = {
                pre_guard_ = empty std::__cxx11::list, post_guard_ = empty std::__cxx11::list, pre_action_ = empty std::__cxx11::list, 
                post_action_ = empty std::__cxx11::list}, [{from_ = galera::TrxHandle::S_REPLICATING, to_ = galera::TrxHandle::S_CERTIFYING}] = {
                pre_guard_ = empty std::__cxx11::list, post_guard_ = empty std::__cxx11::list, pre_action_ = empty std::__cxx11::list, 
                post_action_ = empty std::__cxx11::list}, [{from_ = galera::TrxHandle::S_COMMITTING, to_ = galera::TrxHandle::S_ROLLED_BACK}] = {
                pre_guard_ = empty std::__cxx11::list, post_guard_ = empty std::__cxx11::list, pre_action_ = empty std::__cxx11::list, 
                post_action_ = empty std::__cxx11::list}, [{from_ = galera::TrxHandle::S_CERTIFYING, to_ = galera::TrxHandle::S_APPLYING}] = {
                pre_guard_ = empty std::__cxx11::list, post_guard_ = empty std::__cxx11::list, pre_action_ = empty std::__cxx11::list, 
                post_action_ = empty std::__cxx11::list}, [{from_ = galera::TrxHandle::S_CERTIFYING, to_ = galera::TrxHandle::S_ABORTING}] = {
                pre_guard_ = empty std::__cxx11::list, post_guard_ = empty std::__cxx11::list, pre_action_ = empty std::__cxx11::list, 
                post_action_ = empty std::__cxx11::list}, [{from_ = galera::TrxHandle::S_ABORTING, to_ = galera::TrxHandle::S_ROLLING_BACK}] = {
                pre_guard_ = empty std::__cxx11::list, post_guard_ = empty std::__cxx11::list, pre_action_ = empty std::__cxx11::list, 
                post_action_ = empty std::__cxx11::list}}}, local_seqno_ = 8, global_seqno_ = 6, last_seen_seqno_ = 5, depends_seqno_ = 5, ends_nbo_ = -1, 
          mem_pool_ = @0x55ac39f50f80, write_set_ = {header_ = {
static MAGIC_BYTE = 71 'G', static V3_CHECKSUM_SIZE = 8, static V3_ANNOT_FLAG = 1 '\001', 
              static V3_UNORD_FLAG = 2 '\002', static V3_MAGIC_OFF = 0, static V3_HEADER_VERS_OFF = 1, static V3_HEADER_SIZE_OFF = 2, static V3_SETS_OFF = 3, 
---Type <return> to continue, or q <return> to quit---
              static V3_FLAGS_OFF = 4, static V3_PA_RANGE_OFF = 6, static V3_LAST_SEEN_OFF = 8, static V3_SEQNO_OFF = 8, static V3_TIMESTAMP_OFF = 16, 
              static V3_SOURCE_ID_OFF = 24, static V3_CONN_ID_OFF = 40, static V3_TRX_ID_OFF = 48, static V3_CRC_OFF = 56, static V3_SIZE = 64, static V3 = {
                header_ver_ = 1, header_size_ = 2, sets_ = 3, flags_ = 4, pa_range_ = 6, last_seen_ = 8, seqno_ = 8, timestamp_ = 16, source_id_ = 24, conn_id_ = 40, 
                trx_id_ = 48, crc_ = 56}, static MAX_HEADER_SIZE = 64, local_ = '\000' <repeats 63 times>, ptr_ = 0x7ff0c7ffffd0 "GC@\024\t@\001", 
              ver_ = galera::WriteSetNG::VER4, size_ = 64 '@', chksm_ = {<No data fields>}}, size_ = 264, 
            keys_ = {<gu::RecordSetIn<galera::KeySet::KeyPart>> = {<gu::RecordSetInBase> = {<gu::RecordSet> = {static MAX_VERSION = gu::RecordSet::VER2, 
                    static VER2_ALIGNMENT = 8, size_ = 40, count_ = 2, version_ = 2 '\002', check_type_ = 3 '\003', alignment_ = 8 '\b'}, 
                  head_ = 0x7ff0c8000010 "+\001\234", next_ = 40, begin_ = 24}, <No data fields>}, version_ = galera::KeySet::FLAT8}, 
            data_ = {<gu::RecordSetIn<galera::DataSet::RecordIn>> = {<gu::RecordSetInBase> = {<gu::RecordSet> = {static MAX_VERSION = gu::RecordSet::VER2, 
                    static VER2_ALIGNMENT = 8, size_ = 157, count_ = 1, version_ = 2 '\002', check_type_ = 3 '\003', alignment_ = 8 '\b'}, head_ = 0x7ff0c8000038 "+", 
                  next_ = 157, begin_ = 24}, <No data fields>}, version_ = galera::DataSet::VER1}, 
            unrd_ = {<gu::RecordSetIn<galera::DataSet::RecordIn>> = {<gu::RecordSetInBase> = {<gu::RecordSet> = {static MAX_VERSION = gu::RecordSet::VER2, 
                    static VER2_ALIGNMENT = 8, size_ = 0, count_ = 0, version_ = 0 '\000', check_type_ = 0 '\000', alignment_ = 0 '\000'}, head_ = 0x0, next_ = 0, 
                  begin_ = 0}, <No data fields>}, version_ = galera::DataSet::EMPTY}, annt_ = 0x0, check_thr_id_ = 0, check_thr_ = false, check_ = true, 
            static SIZE_THRESHOLD = 4194304}, buf_ = 0x7ff0ac05f980, action_ = {first = 0x7ff0c7ffffd0, second = 264}, certified_ = true, committed_ = false, 
          exit_loop_ = false, cert_bypass_ = false, queued_ = false, explicit_rollback_ = false}
        retval = WSREP_OK
        __FUNCTION__ = "process_trx"
#14 0x00007ff0d787ebf4 in galera::GcsActionSource::process_writeset (this=this@entry=0x55ac39f59db0, recv_ctx=0x7ff0d45395f0, act=..., exit_loop=@0x7ff0d453937e: false)
    at /home/shako/Galera_Tests/Galera/galera/src/gcs_action_source.cpp:62
        __PRETTY_FUNCTION__ = "void galera::GcsActionSource::process_writeset(void*, const gcs_action&, bool&)"
        tsp = {px = 0x7ff0ac05f980, pn = {pi_ = 0x7ff0ac03aab0}}
        __FUNCTION__ = "process_writeset"
#15 0x00007ff0d787ed85 in galera::GcsActionSource::dispatch (this=this@entry=0x55ac39f59db0, recv_ctx=recv_ctx@entry=0x7ff0d45395f0, act=..., 
---Type <return> to continue, or q <return> to quit---
    exit_loop=@0x7ff0d453937e: false) at /home/shako/Galera_Tests/Galera/galera/src/gcs_action_source.cpp:109
        __PRETTY_FUNCTION__ = "void galera::GcsActionSource::dispatch(void*, const gcs_action&, bool&)"
        __FUNCTION__ = "dispatch"
#16 0x00007ff0d787f159 in galera::GcsActionSource::process (this=0x55ac39f59db0, recv_ctx=0x7ff0d45395f0, exit_loop=@0x7ff0d453937e: false)
    at /home/shako/Galera_Tests/Galera/galera/src/gcs_action_source.cpp:182
        release = {act_ = @0x7ff0d4539190, gcache_ = @0x55ac39f50af8}
        act = {seqno_g = 6, seqno_l = 8, buf = 0x7ff0c7ffffd0, size = 264, type = GCS_ACT_WRITESET}
        rc = 264
        skip = false
        __FUNCTION__ = "process"
        __PRETTY_FUNCTION__ = "virtual ssize_t galera::GcsActionSource::process(void*, bool&)"
#17 0x00007ff0d7895720 in galera::ReplicatorSMM::async_recv (this=0x55ac39f506c0, recv_ctx=0x7ff0d45395f0)
    at /home/shako/Galera_Tests/Galera/galera/src/replicator_smm.cpp:383
        rc = <optimized out>
        __FUNCTION__ = "async_recv"
        exit_loop = false
        retval = WSREP_OK
        __PRETTY_FUNCTION__ = "virtual wsrep_status_t galera::ReplicatorSMM::async_recv(void*)"
#18 0x00007ff0d78b16c8 in galera_recv (gh=<optimized out>, recv_ctx=<optimized out>) at /home/shako/Galera_Tests/Galera/galera/src/wsrep_provider.cpp:236
        __PRETTY_FUNCTION__ = "wsrep_status_t galera_recv(wsrep_t*, void*)"
        repl = <optimized out>
#19 0x000055ac378a8430 in wsrep::wsrep_provider_v26::run_applier (this=0x55ac39f1ece0, applier_ctx=0x7ff0d45395f0)
    at /home/shako/Galera_Tests/MariaDB/wsrep-lib/src/wsrep_provider_v26.cpp:690
No locals.
---Type <return> to continue, or q <return> to quit---
#20 0x000055ac36f5dfb4 in wsrep_replication_process (thd=0x7ff0ac000d60, arg=0x0) at /home/shako/Galera_Tests/MariaDB/sql/wsrep_thd.cc:61
        _db_stack_frame_ = {func = 0x55ac37c7bda5 "?func", file = 0x55ac37c7bdab "?file", level = 2147483649, line = -1, prev = 0x0}
        applier_service = {<Wsrep_high_priority_service> = {<wsrep::high_priority_service> = {
              _vptr.high_priority_service = 0x55ac382365d0 <vtable for Wsrep_applier_service+16>, server_state_ = @0x55ac39f1e830, 
              must_exit_ = false}, <wsrep::high_priority_context> = {_vptr.high_priority_context = 0x55ac38236678 <vtable for Wsrep_applier_service+184>, 
              client_ = @0x7ff0ac006bc0, orig_mode_ = wsrep::client_state::m_local}, m_thd = 0x7ff0ac000d60, m_rli = 0x7ff0ac0168b0, m_rgi = 0x7ff0ac021fb0, m_shadow = {
              option_bits = 2147748608, server_status = 2, vio = 0x0, tx_isolation = 2, db = 0x0, db_length = 0, user_time = {val = 0}, row_count_func = -1, 
              wsrep_applier = true}}, <No data fields>}
        ret = wsrep::provider::success
#21 0x000055ac36f4f766 in start_wsrep_THD (arg=0x55ac39fd7c60) at /home/shako/Galera_Tests/MariaDB/sql/wsrep_mysqld.cc:2768
        thd = 0x7ff0ac000d60
        thd_args = 0x55ac39fd7c60
#22 0x00007ff0dab336db in start_thread (arg=0x7ff0d453a700) at pthread_create.c:463
        pd = 0x7ff0d453a700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140672331130624, -5855974270309587128, 140672331128768, 0, 94198195649632, 94198167608544, 5862225233336772424, 
                5862229909192746824}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#23 0x00007ff0d9d1988f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
No locals.

[ShahriyarR]

Easy way to reproduce with 10.4 cluster - 3 nodes:

MariaDB [test]> XA BEGIN 'xid';
ERROR 2013 (HY000): Lost connection to MySQL server during query