protocol change: define order of mempool txs in status of scripthash

Note that this is a soft fork: the server can apply it even for past
protocol versions.

Previously, with the order being undefined, if an address had multiple
mempool transactions touching it, switching between different servers
could result in a change in address status simply as a result of these
servers ordering mempool txs differently. This would result in the client
re-requesting the whole history of the address.

-----

D/i | interface.[electrum.blockstream.info:60002] | <-- ('blockchain.scripthash.subscribe', ['660b44502503064f9d5feee48726287c0973e25bc531b4b8a072f57f143d5cd0']) {} (id: 12)
D/i | interface.[electrum.blockstream.info:60002] | --> 9da27f9df91e3f860212f65b736fa20a539ba6e3d509f6370367ee7f10a4d5b0 (id: 12)
D/i | interface.[electrum.blockstream.info:60002] | <-- ('blockchain.scripthash.get_history', ['660b44502503064f9d5feee48726287c0973e25bc531b4b8a072f57f143d5cd0']) {} (id: 13)
D/i | interface.[electrum.blockstream.info:60002] | --> [
{'fee': 200, 'height': 0, 'tx_hash': '3ee6d6e26291ce360127fe039b816470fce6eeea19b5c9d10829a1e4efc2d0c7'},
{'fee': 239, 'height': 0, 'tx_hash': '9e050f09b676b9b0ee26aa02ccee623fae585a85d6a5e24ecedd6f8d6d2d3b1d'},
{'fee': 178, 'height': 0, 'tx_hash': 'fb80adbf8274190418cb3fb0385d82fe9d47a844d9913684fa5fb3d48094b35a'},
{'fee': 200, 'height': 0, 'tx_hash': '713933c50b7c43f606dad5749ea46e3bc6622657e9b13ace9d639697da266e8b'}
] (id: 13)

D/i | interface.[testnet.hsmiths.com:53012] | <-- ('blockchain.scripthash.subscribe', ['660b44502503064f9d5feee48726287c0973e25bc531b4b8a072f57f143d5cd0']) {} (id: 12)
D/i | interface.[testnet.hsmiths.com:53012] | --> f7ef7237d2d62a3280acae05616200b96ad9dd85fd0473c29152a4a41e05686c (id: 12)
D/i | interface.[testnet.hsmiths.com:53012] | <-- ('blockchain.scripthash.get_history', ['660b44502503064f9d5feee48726287c0973e25bc531b4b8a072f57f143d5cd0']) {} (id: 13)
D/i | interface.[testnet.hsmiths.com:53012] | --> [
{'tx_hash': '9e050f09b676b9b0ee26aa02ccee623fae585a85d6a5e24ecedd6f8d6d2d3b1d', 'height': 0, 'fee': 239},
{'tx_hash': 'fb80adbf8274190418cb3fb0385d82fe9d47a844d9913684fa5fb3d48094b35a', 'height': 0, 'fee': 178},
{'tx_hash': '3ee6d6e26291ce360127fe039b816470fce6eeea19b5c9d10829a1e4efc2d0c7', 'height': 0, 'fee': 200},
{'tx_hash': '713933c50b7c43f606dad5749ea46e3bc6622657e9b13ace9d639697da266e8b', 'height': 0, 'fee': 200}
] (id: 13)

Author: SomberNight

docs/protocol-basics.rst

@@ -153,10 +153,11 @@ block)
 
   * ``height`` is the height of the block it is in.
 
-3. Next, with mempool transactions in any order, append a similar
-string for those transactions, but where **height** is ``-1`` if the
-transaction has at least one unconfirmed input, and ``0`` if all
-inputs are confirmed.
+3. Next, with mempool transactions ordered
+:ref:`as described here <mempool_tx_order>`,
+append a similar string for those transactions,
+but where **height** is ``-1`` if the transaction has at least
+one unconfirmed input, and ``0`` if all inputs are confirmed.
 
 4. The :dfn:`status` of the script hash is the :func:`sha256` hash of the
 full string expressed as a hexadecimal string, or :const:`null` if the

docs/protocol-changes.rst

@@ -178,6 +178,10 @@ Changes
   * Breaking change for the version negotiation: we now mandate that
     the :func:`server.version` message must be the first message sent.
     That is, version negotiation must happen before any other messages.
+  * The status of a scripthash, :func:`blockchain.scripthash.get_mempool`,
+    and :func:`blockchain.scripthash.get_history` previously did not define
+    an order for mempool transactions. We now mandate that these are sorted
+    by `(-height, tx_hash)`.
   * The previously required *height* argument for
     :func:`blockchain.transaction.get_merkle` is now optional.
   * Optional *mode* argument added to :func:`blockchain.estimatefee`.

docs/protocol-methods.rst

@@ -369,8 +369,9 @@ hashes>`.
 
 **Result**
 
-  A list of mempool transactions in arbitrary order.  Each mempool
-  transaction is a dictionary with the following keys:
+  A list of mempool transactions, ordered by `(-height, tx_hash)`
+  (see :ref:`below <mempool_tx_order>`).
+  Each mempool transaction is a dictionary with the following keys:
 
   * *height*
 
@@ -384,6 +385,14 @@ hashes>`.
 
     The transaction fee in minimum coin units (satoshis).
 
+
+  .. _mempool_tx_order:
+
+  Mempool txs are ordered by `(-height, tx_hash)`, that is,
+  ``0`` height txs come before ``-1`` height txs, and secondarily the
+  txid (with endianness same as displayed hex) is used to arrive at
+  a canonical ordering.
+
 **Result Example**
 
 ::

electrumx/server/mempool.py

@@ -469,12 +469,15 @@ async def potential_spends(self, hashX):
         return result
 
     async def transaction_summaries(self, hashX):
-        '''Return a list of MemPoolTxSummary objects for the hashX.'''
+        '''Return a list of MemPoolTxSummary objects for the hashX,
+        sorted as expected by protocol methods.
+        '''
         result = []
         for tx_hash in self.hashXs.get(hashX, ()):
             tx = self.txs[tx_hash]
             has_ui = any(hash in self.txs for hash, idx in tx.prevouts)
             result.append(MemPoolTxSummary(tx_hash, tx.fee, has_ui))
+        result.sort(key=lambda x: (x.has_unconfirmed_inputs, bytes(reversed(x.hash))))
         return result
 
     async def unordered_UTXOs(self, hashX):

electrumx/server/session.py

@@ -1143,7 +1143,7 @@ async def address_status(self, hashX: bytes) -> Optional[str]:
 
         Status is a hex string, but must be None if there is no history.
         '''
-        # Note history is ordered and mempool unordered in electrum-server
+        # Note both confirmed history and mempool history are ordered
         # For mempool, height is -1 if it has unconfirmed inputs, otherwise 0
         db_history, cost = await self.session_mgr.limited_history(hashX)
         mempool = await self.mempool.transaction_summaries(hashX)
@@ -1234,7 +1234,7 @@ async def scripthash_get_balance(self, scripthash):
         return await self.get_balance(hashX)
 
     async def unconfirmed_history(self, hashX):
-        # Note unconfirmed history is unordered in electrum-server
+        # Note both confirmed history and mempool history are ordered
         # height is -1 if it has unconfirmed inputs, otherwise 0
         result = [{'tx_hash': hash_to_hex_str(tx.hash),
                    'height': -tx.has_unconfirmed_inputs,
@@ -1244,7 +1244,7 @@ async def unconfirmed_history(self, hashX):
         return result
 
     async def confirmed_and_unconfirmed_history(self, hashX):
-        # Note history is ordered but unconfirmed is unordered in e-s
+        # Note both confirmed history and mempool history are ordered
         history, cost = await self.session_mgr.limited_history(hashX)
         self.bump_cost(cost)
         conf = [{'tx_hash': hash_to_hex_str(tx_hash), 'height': height}