network keeps disconnecting (reconnect loop) due to RPCError(1, 'history too large'). response too large (at least 1020631 bytes) when adding address with large history. MemoryError('dropping message over 1,000,000 bytes and re-synchronizing')

[circulosmeos]

This error is shown when running "electrum -v" v3.1.3 (Windows or linux)
after adding address 1Q7zx6cUVXDZpAJHgPQwbbZUfTKFFuEV6z:

[Synchronizer] response error: {'error': {'code': -32600, 'message': 'response too large (at least 1020631 bytes)'}, 'method': 'blockchain.scripthash.get_history', 'jsonrpc': '2.0', 'params': ['1Q7zx6cUVXDZpAJHgPQwbbZUfTKFFuEV6z'], 'id': 39}

Note that this address has 39374 transactions.

The balance shown is zero (and no transactions in History are shown):

but an invalid balance is shown when adding more addresses: for example after adding 1GK3btZr57VDEQvwpib6Qhj479Ckzt8z2n the balance is 10 bits, when it should be 0:

[SomberNight]

Yes, the current protocol is not suitable for addresses with long histories unfortunately.
Related kyuupichan/electrumx#348

[SomberNight]

Note that there is a max message size restriction enforced on BOTH client-side and server-side, by default of ~1 MB.

---

client-side:

electrum/electrum/interface.py

Line 70 in 72950bf

MAX_INCOMING_MSG_SIZE = 1_000_000 # in bytes

see config key network_max_incoming_msg_size (added in 5a7c3dc)

---

server-side: https://electrumx-spesmilo.readthedocs.io/en/latest/environment.html#envvar-MAX_SEND

---

If you bump into these errors that means there is an address in your wallet with over 10k transactions, so very heavy address reuse. The current electrum protocol is very inefficient for heavily reused addresses, so there is a max tx limit per address that is enforced as a DDOS-protection for servers. The default limit is around 10k txns for a given address.

I recommend creating a new wallet - and sending your coins from the existing wallet to that.
Further, in the future, you should not reuse addresses; or at least not this many times... Give out new addresses for incoming txs.

If you bump into these errors, and you have your own server, I suggest raising the limits, in both your local client and your server, as a temporary workaround so that you can send your coins out.
In case you don't run your own server, try fortress.qtornado.com:443:s, it has a somewhat higher limit than default -- you will still need to increase the limit in the client.

[SomberNight]

in case the server raises, the log in current code looks like this:
E/i | interface.[fortress.qtornado.com:443] | disconnecting due to GracefulDisconnect(RPCError(1, 'history too large'))

in case the server uses higher limits than the client, and it is only the client that raises, the exception is:
MemoryError('dropping message over 1,000,000 bytes and re-synchronizing',)

[johnzweng]

Hi @SomberNight ! 🙂

As I see that the limit on electrumX can be configured via ENV variable, would it be possible to also allow the limit on the client side to be configurable (either via ENV variable or config file)?

This would allow powerusers which have to work with addresses with large histories to tune their own setup (server + client) to work around this problem without needing to maintain their own electrum branch (where the limit is increased).

Note: I see this problem mostly as a legacy problem as thanks to HD wallets we hopefully won't see this large histories anymore in future, but for some non-technical reasons, we still need to be able to monitor old legacy addresses (which unfortunately have already large histories).

[SomberNight]

@johnzweng done in 5a7c3dc

[johnzweng]

This was quick! Thanks a lot! 🙂 👍

[SomberNight]

note: electrum protocol 1.5 would resolve this
spesmilo/electrumx#90