invalid memory address or nil pointer dereference

[dennis-tra]

Checklist

• This is a bug report, not a question. Ask questions on discuss.ipfs.tech.
• I have searched on the issue tracker for my bug.
• I am running the latest kubo version or have an issue updating.

Installation method

dist.ipfs.tech or ipfs-update

Version

Kubo version: 0.39.0-Homebrew
Repo version: 18
System version: arm64/darwin
Golang version: go1.25.4

Config

{
  "API": {
    "HTTPHeaders": {}
  },
  "Addresses": {
    "API": "/ip4/127.0.0.1/tcp/5001",
    "Announce": [],
    "AppendAnnounce": [],
    "Gateway": "/ip4/127.0.0.1/tcp/8080",
    "NoAnnounce": [],
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/::/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/webrtc-direct",
      "/ip4/0.0.0.0/udp/4001/quic-v1",
      "/ip4/0.0.0.0/udp/4001/quic-v1/webtransport",
      "/ip6/::/udp/4001/webrtc-direct",
      "/ip6/::/udp/4001/quic-v1",
      "/ip6/::/udp/4001/quic-v1/webtransport"
    ]
  },
  "AutoConf": {},
  "AutoNAT": {},
  "AutoTLS": {},
  "Bitswap": {},
  "Bootstrap": [
    "auto"
  ],
  "DNS": {
    "Resolvers": {
      ".": "auto"
    }
  },
  "Datastore": {
    "BlockKeyCacheSize": null,
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "mountpoint": "/blocks",
          "path": "blocks",
          "prefix": "flatfs.datastore",
          "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
          "sync": false,
          "type": "flatfs"
        },
        {
          "compression": "none",
          "mountpoint": "/",
          "path": "datastore",
          "prefix": "leveldb.datastore",
          "type": "levelds"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "10GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": true
    }
  },
  "Experimental": {
    "FilestoreEnabled": false,
    "Libp2pStreamMounting": false,
    "OptimisticProvide": false,
    "OptimisticProvideJobsPoolSize": 0,
    "P2pHttpProxy": false,
    "UrlstoreEnabled": false
  },
  "Gateway": {
    "DeserializedResponses": null,
    "DisableHTMLErrors": null,
    "ExposeRoutingAPI": null,
    "HTTPHeaders": {},
    "NoDNSLink": false,
    "NoFetch": false,
    "PublicGateways": null,
    "RootRedirect": ""
  },
  "HTTPRetrieval": {},
  "Identity": {
    "PeerID": "12D3KooWFUXuVPeAphFDefhRQ8PThxjSRWGuCWymooF3fzRBwjJq"
  },
  "Import": {
    "BatchMaxNodes": null,
    "BatchMaxSize": null,
    "CidVersion": null,
    "FastProvideRoot": null,
    "FastProvideWait": null,
    "HashFunction": null,
    "UnixFSChunker": null,
    "UnixFSDirectoryMaxLinks": null,
    "UnixFSFileMaxLinks": null,
    "UnixFSHAMTDirectoryMaxFanout": null,
    "UnixFSHAMTDirectorySizeThreshold": null,
    "UnixFSRawLeaves": null
  },
  "Internal": {},
  "Ipns": {
    "DelegatedPublishers": [
      "auto"
    ],
    "RecordLifetime": "",
    "RepublishPeriod": "",
    "ResolveCacheSize": 128
  },
  "Migration": {},
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns",
    "MFS": "/mfs"
  },
  "Peering": {
    "Peers": null
  },
  "Pinning": {
    "RemoteServices": {}
  },
  "Plugins": {
    "Plugins": null
  },
  "Provide": {
    "DHT": {}
  },
  "Provider": {},
  "Pubsub": {
    "DisableSigning": false,
    "Router": ""
  },
  "Reprovider": {},
  "Routing": {
    "DelegatedRouters": [],
    "Type": "dht"
  },
  "Swarm": {
    "AddrFilters": null,
    "ConnMgr": {},
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": false,
    "RelayClient": {},
    "RelayService": {},
    "ResourceMgr": {},
    "Transports": {
      "Multiplexers": {},
      "Network": {},
      "Security": {}
    }
  },
  "Version": {}
}

Description

caught panic: runtime error: invalid memory address or nil pointer dereference
goroutine 7469704 [running]:
runtime/debug.Stack()
	runtime/debug/stack.go:26 +0x64
github.com/libp2p/go-libp2p/core/internal/catch.HandlePanic({0x1066ee8c0, 0x1080a6750}, 0x14008189f00, {0x105d9f5ea, 0x18})
	github.com/libp2p/go-libp2p@v0.45.0/core/internal/catch/catch.go:15 +0x40
github.com/libp2p/go-libp2p/core/peer.AddrInfo.MarshalJSON.func1()
	github.com/libp2p/go-libp2p@v0.45.0/core/peer/addrinfo_serde.go:18 +0x40
panic({0x1066ee8c0?, 0x1080a6750?})
	runtime/panic.go:783 +0x120
github.com/multiformats/go-multiaddr.Multiaddr.String({0x2f?, 0x104821bc8?, 0x14002a10e00?})
	github.com/multiformats/go-multiaddr@v0.16.1/multiaddr.go:113 +0x30
github.com/libp2p/go-libp2p/core/peer.AddrInfo.MarshalJSON({{0x14005a94c90, 0x26}, {0x14000ce2d70, 0x6, 0x3}})
	github.com/libp2p/go-libp2p@v0.45.0/core/peer/addrinfo_serde.go:22 +0xc0
encoding/json.marshalerEncoder(0x14008388240, {0x10682bf40?, 0x140080bb7d0?, 0x10497c9ec?}, {0xc0?, 0x1?})
	encoding/json/encode.go:483 +0xc0
encoding/json.arrayEncoder.encode({0x1400818a108?}, 0x14008388240, {0x10654ef00?, 0x14004a872c0?, 0x0?}, {0x4?, 0x0?})
	encoding/json/encode.go:901 +0xc0
encoding/json.sliceEncoder.encode({0x1400818a278?}, 0x14008388240, {0x10654ef00?, 0x14004a872c0?, 0x10658e800?}, {0xe0?, 0xc6?})
	encoding/json/encode.go:874 +0x25c
encoding/json.(*encodeState).reflectValue(0x14008388240, {0x10654ef00?, 0x14004a872c0?, 0x104823d24?}, {0x28?, 0xa3?})
	encoding/json/encode.go:367 +0x70
encoding/json.interfaceEncoder(0x14008388240, {0x106662b00?, 0x14004c5c780?, 0x4?}, {0x4?, 0xad?})
	encoding/json/encode.go:711 +0xcc
encoding/json.mapEncoder.encode({0x1400818a568?}, 0x14008388240, {0x106761500?, 0x140052b4870?, 0x1400818a5b8?}, {0x9c?, 0xa4?})
	encoding/json/encode.go:815 +0x3d0
encoding/json.(*encodeState).reflectValue(0x14008388240, {0x106761500?, 0x140052b4870?, 0x1518b3220?}, {0x18?, 0xa6?})
	encoding/json/encode.go:367 +0x70
encoding/json.(*encodeState).marshal(0x2e?, {0x106761500?, 0x140052b4870?}, {0x0?, 0x8f?})
	encoding/json/encode.go:343 +0xa8
encoding/json.Marshal({0x106761500, 0x140052b4870})
	encoding/json/encode.go:209 +0x84
github.com/libp2p/go-libp2p/core/routing.(*QueryEvent).MarshalJSON(0x140018f37c0)
	github.com/libp2p/go-libp2p@v0.45.0/core/routing/query_serde.go:10 +0x1dc
encoding/json.marshalerEncoder(0x140062e6180, {0x106717c00?, 0x140018f37c0?, 0x10450a47c?}, {0x78?, 0xa8?})
	encoding/json/encode.go:483 +0xc0
encoding/json.(*encodeState).reflectValue(0x140062e6180, {0x106717c00?, 0x140018f37c0?, 0x1044a72cc?}, {0xf8?, 0xa8?})
	encoding/json/encode.go:367 +0x70
encoding/json.(*encodeState).marshal(0x1400818a918?, {0x106717c00?, 0x140018f37c0?}, {0x60?, 0xe9?})
	encoding/json/encode.go:343 +0xa8
encoding/json.(*Encoder).Encode(0x14008a1ea20, {0x106717c00, 0x140018f37c0})
	encoding/json/stream.go:212 +0x9c
github.com/ipfs/go-ipfs-cmds/http.(*responseEmitter).Emit(0x14005d856c0, {0x106717c00, 0x140018f37c0})
	github.com/ipfs/go-ipfs-cmds@v0.15.0/http/responseemitter.go:135 +0x2c4
github.com/ipfs/kubo/core/commands.init.func140(0x14005d85650, {0x14f3c47f0, 0x14005d856c0}, {0x10691a800?, 0x140009e7180?})
	github.com/ipfs/kubo/core/commands/routing.go:100 +0x208
github.com/ipfs/go-ipfs-cmds.(*Command).call(0x18?, 0x14005d85650, {0x14f3c47f0, 0x14005d856c0}, {0x10691a800, 0x140009e7180})
	github.com/ipfs/go-ipfs-cmds@v0.15.0/command.go:189 +0x124
github.com/ipfs/go-ipfs-cmds.(*Command).Call(0x140009e7180?, 0x14005d85650?, {0x14f3c47f0, 0x14005d856c0}, {0x10691a800?, 0x140009e7180?})
	github.com/ipfs/go-ipfs-cmds@v0.15.0/command.go:159 +0x30
github.com/ipfs/go-ipfs-cmds/http.(*handler).ServeHTTP(0x14001ac8600, {0x106b592d0, 0x14008a1e9c0}, 0x140015a8280)
	github.com/ipfs/go-ipfs-cmds@v0.15.0/http/handler.go:202 +0x840
github.com/ipfs/go-ipfs-cmds/http.prefixHandler.ServeHTTP({{0x105d763f1?, 0x105bd609c?}, {0x106b42460?, 0x14001ac8600?}}, {0x106b592d0, 0x14008a1e9c0}, 0x140015a8280)
	github.com/ipfs/go-ipfs-cmds@v0.15.0/http/apiprefix.go:24 +0x100
github.com/ipfs/go-ipfs-cmds/http.NewHandler.(*Cors).Handler.func1({0x106b592d0, 0x14008a1e9c0}, 0x140015a8280)
	github.com/rs/cors@v1.11.1/cors.go:289 +0x15c
net/http.HandlerFunc.ServeHTTP(0x106b5f090?, {0x106b592d0?, 0x14008a1e9c0?}, 0x1063a6f78?)
	net/http/server.go:2322 +0x38
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP(0x14001ada0d0, {0x14f3c46c0, 0x14006776cd0}, 0x140015a8140, {0x106b40240, 0x14001ac8640})
	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.63.0/handler.go:180 +0xee0
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1({0x14f3c46c0?, 0x14006776cd0?}, 0x14008295418?)
	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.63.0/handler.go:67 +0x3c
net/http.HandlerFunc.ServeHTTP(0x14001391740?, {0x14f3c46c0?, 0x14006776cd0?}, 0x105c7d858?)
	net/http/server.go:2322 +0x38
net/http.(*ServeMux).ServeHTTP(0x140072fa360?, {0x14f3c46c0, 0x14006776cd0}, 0x140015a8140)
	net/http/server.go:2861 +0x190
github.com/ipfs/kubo/cmd/ipfs/kubo.serveHTTPApi.CheckVersionOption.func8.1({0x14f3c46c0, 0x14006776cd0}, 0x140015a8140)
	github.com/ipfs/kubo/core/corehttp/commands.go:228 +0x1f8
net/http.HandlerFunc.ServeHTTP(0x140013912c0?, {0x14f3c46c0?, 0x14006776cd0?}, 0xf?)
	net/http/server.go:2322 +0x38
net/http.(*ServeMux).ServeHTTP(0x14f3c46c0?, {0x14f3c46c0, 0x14006776cd0}, 0x140015a8140)
	net/http/server.go:2861 +0x190
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1({0x14f3c46c0?, 0x14006776c80?}, 0x140015a8140)
	github.com/prometheus/client_golang@v1.23.2/prometheus/promhttp/instrument_server.go:296 +0xa8
net/http.HandlerFunc.ServeHTTP(0x14007472d41?, {0x14f3c46c0?, 0x14006776c80?}, 0x19?)
	net/http/server.go:2322 +0x38
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerRequestSize.func2({0x14f3c46c0?, 0x14006776c80?}, 0x140015a8140)
	github.com/prometheus/client_golang@v1.23.2/prometheus/promhttp/instrument_server.go:255 +0x6c
net/http.HandlerFunc.ServeHTTP(0x10867e6c8?, {0x14f3c46c0?, 0x14006776c80?}, 0x14006776c80?)
	net/http/server.go:2322 +0x38
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2({0x14f3c46c0, 0x14006776c80}, 0x140015a8140)
	github.com/prometheus/client_golang@v1.23.2/prometheus/promhttp/instrument_server.go:109 +0x8c
net/http.HandlerFunc.ServeHTTP(0x106b561b0?, {0x14f3c46c0?, 0x14006776c80?}, 0x9?)
	net/http/server.go:2322 +0x38
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0x106b561b0?, 0x140020a01e0?}, 0x140015a8140)
	github.com/prometheus/client_golang@v1.23.2/prometheus/promhttp/instrument_server.go:147 +0xa0
net/http.HandlerFunc.ServeHTTP(0x14001391200?, {0x106b561b0?, 0x140020a01e0?}, 0x14001642070?)
	net/http/server.go:2322 +0x38
net/http.(*ServeMux).ServeHTTP(0x14003687af8?, {0x106b561b0, 0x140020a01e0}, 0x140015a8140)
	net/http/server.go:2861 +0x190
github.com/ipfs/kubo/core/corehttp.MakeHandler.func1({0x106b561b0?, 0x140020a01e0?}, 0x140011ba1a8?)
	github.com/ipfs/kubo/core/corehttp/corehttp.go:52 +0x90
net/http.HandlerFunc.ServeHTTP(0x0?, {0x106b561b0?, 0x140020a01e0?}, 0x14003687b50?)
	net/http/server.go:2322 +0x38
net/http.serverHandler.ServeHTTP({0x14002fcb480?}, {0x106b561b0?, 0x140020a01e0?}, 0x1?)
	net/http/server.go:3340 +0xb0
net/http.(*conn).serve(0x14007ef21b0, {0x106b5f090, 0x14001ae3350})
	net/http/server.go:2109 +0x528
created by net/http.(*Server).Serve in goroutine 1818
	net/http/server.go:3493 +0x384

[lidel]

@dennis-tra are you by any chance calling (ipfs routing findprovs)?

im 80% sure (late Friday, so ymmv) what is happening is along these lines:

• the {0x14000ce2d70, 0x6, 0x3} represents the go-libp2p's Addrs slice header: data pointer, length=6, capacity=3. This is impossible for a valid Go slice (MUST have len <= cap), so the slice header itself is corrupted, almost certainly due to a data race?
• looking at kubo's findProvidersRoutingCmd, when np := p copies the peer.AddrInfo struct, it copies the slice header (pointer + len + cap), but both p.Addrs and np.Addrs now point to the same underlying array
• if the DHT or peerstore later modifies that array while JSON marshaling is happening on another goroutine, we could get into the data race business 🙈
• this is likely a systemic problem: buggy code can crash go-libp2p under load, and there is no defensive code for it atm 🙈
• good news, i think, this does not occur often, or needs to be triggered by things like manual findprovs, we would catch it already in RC stages otherwise

i'm less certain that the root fix in go-libp2p-kad-dht is enough. doing slice clone there would fix the problem, but the fact we had no nil protection in go-libp2p makes me anxious and feel we should do belt & suspenders fixes for the systemic risk, so the problem does not come back in few months via different thing that mutated peer-related slices.

I've opened bunch of PRs, would appreciate feedback if defensive checks are too much, or ok:

• fix: clone Addrs slice before sending on provider channel libp2p/go-libp2p-kad-dht#1208 (aka "the likely fix")

Defensive checks:

• fix(peer): skip nil addresses in AddrInfo.MarshalJSON libp2p/go-libp2p#3442
• fix(routing): defensive nil checks for multiaddr handling boxo#1081
• fix(routing): defensive clone of AddrInfo from provider channel #11120

cc @aschmahmann @gammazero @guillaumemichel for visibility if there is more to it or if this was discussed elsewhere already

[aschmahmann]

@lidel I took a brief look at this on Friday (haven't seen your PRs yet), but clarifying that go-libp2p does have defenses here and that this issue results in a caught panic. Yes, a bunch of things here (particularly around multiaddr handling) should ideally be better though.

In this case the go-libp2p defense catch is here https://github.com/libp2p/go-libp2p/blob/20ba3c9449809e7348b735f559ad688c11be5338/core/peer/addrinfo_serde.go#L18