query: ip diversity filter

[guillaumemichel]

Summary

IP diversity checks have been added to the DHT implementation to enhance the DHT robustness (see implementation here and here). However, these checks are only effective for the routing table. These checks aren't helping clients to avoid malicious nodes during lookups.

Problem

When clients want to write content to the DHT (provider records, IPNS record), they will look for the k closest peers to the key they are trying to write. Since most DHT Servers implement the IP Diversity Filter, it means that these honest DHT Servers should never return more than 2 peers from the same IP block/ASN.

However, if the query hits a (malicious) peer not implementing the IP Diversity Filter, this peer may return plenty of closer peers sharing the same IP block/ASN, or even the same IP address. Since these peers are closer to the target, the client will continue the lookup by querying these potentially malicious peers, and the records will eventually be allocated to these peers.

Same for when clients want to read records from the DHT.

Solution

The query process on the client side can implement a check similar to the IP diversity filter check in the routing table. If a peer respond with more than e.g 3 closer peer sharing the same IP block/ASN, the client would drop all these addresses sharing the same address range, and only keep less represented IP addresses (if any).

Implications

This means that GetClosestPeers may not return the actual k closest peers in XOR distance to a key, if many of the actual closest peers share the same address range. GetClosestPeers may return a limited (e.g 3) number of peers sharing the same address range, but the other returned peers will be located in other IP blocks/ASNs.

Hence in the case of a many nodes on the same IP range in the same small keyspace region, the content is expected to be stored on some of the actual closest peers (in XOR distance) that are on the same IP range (potentially malicious), and on the closest peers (in XOR distance) to the key that are using other IP blocks/ASNs. Hence client should be able to retrieve the value from the nodes with more diverse IPs.

References

• Depends on peerdiversity: export IPGroupKey go-libp2p-kbucket#141

[guillaumemichel]

Added the IP diversity filter to accelerated DHT client GetClosestPeers to get feature parity.

[sukunrt]

TBH, I am confused about how the diversity filter works or why the queries do actually converge. It's been there for a while now, so I guess it's okay.

Is there any reference for understanding how this works?

[guillaumemichel]

Is there any reference for understanding how this works?

The best we have is the IPFS Kademlia DHT draft spec, see this section.