Kubo drops freshly published IPNS records when using custom HTTP routers (no bootstrap)

[Rinse12]

Checklist

• This is a bug report, not a question. Ask questions on discuss.ipfs.tech.
• I have searched on the issue tracker for my bug.
• I am running the latest kubo version or have an issue updating.

Installation method

dist.ipfs.tech or ipfs-update

Version

Kubo version: 0.38.0
Repo version: 18
System version: amd64/linux
Golang version: go1.25.1

Config

{
  "API": {
    "HTTPHeaders": {}
  },
  "Addresses": {
    "API": "/ip4/127.0.0.1/tcp/5001",
    "Announce": [],
    "AppendAnnounce": [],
    "Gateway": "/ip4/127.0.0.1/tcp/8080",
    "NoAnnounce": [],
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/::/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/webrtc-direct",
      "/ip4/0.0.0.0/udp/4001/quic-v1",
      "/ip4/0.0.0.0/udp/4001/quic-v1/webtransport",
      "/ip6/::/udp/4001/webrtc-direct",
      "/ip6/::/udp/4001/quic-v1",
      "/ip6/::/udp/4001/quic-v1/webtransport"
    ]
  },
  "AutoNAT": {},
  "AutoTLS": {},
  "Bootstrap": [
    "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/ip4/104.131.131.82/udp/4001/quic-v1/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt"
  ],
  "DNS": {
    "Resolvers": {}
  },
  "Datastore": {
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
            "sync": true,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "10GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": true
    }
  },
  "Experimental": {
    "FilestoreEnabled": false,
    "Libp2pStreamMounting": false,
    "OptimisticProvide": false,
    "OptimisticProvideJobsPoolSize": 0,
    "P2pHttpProxy": false,
    "StrategicProviding": false,
    "UrlstoreEnabled": false
  },
  "Gateway": {
    "DeserializedResponses": null,
    "DisableHTMLErrors": null,
    "ExposeRoutingAPI": null,
    "HTTPHeaders": {},
    "NoDNSLink": false,
    "NoFetch": false,
    "PublicGateways": null,
    "RootRedirect": ""
  },
  "Identity": {
    "PeerID": "12D3KooWBJNtmfaDN1G1enLgEuWVXAHGR98fjYDH2d3wNsKqWr4s"
  },
  "Import": {
    "CidVersion": null,
    "HashFunction": null,
    "UnixFSChunker": null,
    "UnixFSRawLeaves": null
  },
  "Internal": {},
  "Ipns": {
    "RecordLifetime": "",
    "RepublishPeriod": "24h",
    "ResolveCacheSize": 128
  },
  "Migration": {
    "DownloadSources": [],
    "Keep": ""
  },
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns"
  },
  "Peering": {
    "Peers": null
  },
  "Pinning": {
    "RemoteServices": {}
  },
  "Plugins": {
    "Plugins": null
  },
  "Provider": {
    "Strategy": ""
  },
  "Pubsub": {
    "DisableSigning": false,
    "Router": ""
  },
  "Reprovider": {},
  "Routing": {
    "Methods": {
      "find-peers": {
        "RouterName": "HttpRouterNotSupported"
      },
      "find-providers": {
        "RouterName": "HttpRoutersParallel"
      },
      "get-ipns": {
        "RouterName": "HttpRouterNotSupported"
      },
      "provide": {
        "RouterName": "HttpRoutersParallel"
      },
      "put-ipns": {
        "RouterName": "HttpRouterNotSupported"
      }
    },
    "Routers": {
      "HttpRouter1": {
        "Parameters": {
          "Endpoint": "http://127.0.0.1:19575"
        },
        "Type": "http"
      },
      "HttpRouter2": {
        "Parameters": {
          "Endpoint": "http://127.0.0.1:19576"
        },
        "Type": "http"
      },
      "HttpRouter3": {
        "Parameters": {
          "Endpoint": "http://127.0.0.1:19577"
        },
        "Type": "http"
      },
      "HttpRouter4": {
        "Parameters": {
          "Endpoint": "http://127.0.0.1:19578"
        },
        "Type": "http"
      },
      "HttpRouterNotSupported": {
        "Parameters": {
          "Endpoint": "http://kubohttprouternotsupported"
        },
        "Type": "http"
      },
      "HttpRoutersParallel": {
        "Parameters": {
          "Routers": [
            {
              "IgnoreErrors": true,
              "RouterName": "HttpRouter1",
              "Timeout": "10s"
            },
            {
              "IgnoreErrors": true,
              "RouterName": "HttpRouter2",
              "Timeout": "10s"
            },
            {
              "IgnoreErrors": true,
              "RouterName": "HttpRouter3",
              "Timeout": "10s"
            },
            {
              "IgnoreErrors": true,
              "RouterName": "HttpRouter4",
              "Timeout": "10s"
            }
          ]
        },
        "Type": "parallel"
      }
    },
    "Type": "custom"
  },
  "Swarm": {
    "AddrFilters": null,
    "ConnMgr": {},
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": false,
    "RelayClient": {},
    "RelayService": {},
    "ResourceMgr": {},
    "Transports": {
      "Multiplexers": {},
      "Network": {},
      "Security": {}
    }
  },
  "Version": {}
}

Description

Draft GitHub Issue: Kubo 0.38 drops freshly published IPNS records with HTTP router config

Description

While preparing a regression test for the Plebbit project I noticed that Kubo 0.38 consistently loses freshly published IPNS records whenever the node relies on custom HTTP routers and has no bootstrap peers configured. The daemon starts, the ipfs name publish call succeeds, but any subsequent ipfs name resolve hangs until the stream closes without returning the record. After restarting the daemon the behaviour is identical, so the record never reaches the local datastore.

There are no explicit errors in the daemon logs. The only symptom is that the resolve call times out (even with a 60s client-side timeout) and never yields the /ipfs/<new-cid> path that was just published. My best guess is that the HTTP router configuration—where put-ipns/get-ipns are pointed at an unsupported endpoint—causes the daemon to skip persisting the record locally, even though the publish endpoint still reports success.

Environment

• Kubo 0.38.0 installed via npm install kubo@0.38.0, binary executed from node_modules/kubo/bin/ipfs
• Node.js 20.19.1
• Fresh repo initialised with:
  • Bootstrap = []
  • Discovery.MDNS.Enabled = false
  • Custom routing block identical to Plebbit’s configuration (two local HTTP routers for find-providers/provide, and get-ipns/put-ipns pointed at a dummy host)
  • Random high ports for API/Gateway/Swarm to avoid conflicts

What I Was Doing

1. Initialised a new IPFS repo with the routing config above.
2. Started the daemon (ipfs daemon) with no bootstrap peers.
3. Added a text payload via the HTTP API (ipfs add).
4. Published the resulting CID with ipfs name publish --allow-offline --resolve=false --lifetime 24h.
5. Attempted to resolve the name immediately using ipfs name resolve (HTTP API, 60s timeout).
6. Stopped the daemon cleanly, restarted it, and tried resolving again.

Expected vs Actual

• Expected: The resolve calls should return /ipfs/<new-cid> immediately and after restart.
• Actual: Both resolves time out without yielding any value. Even after waiting the full 60s, the stream closes with no entries returned.

Steps To Reproduce

All of the above is automated in this repository:

git clone https://github.com/Rinse12/reproduce_kubo_not_storing_ipns_record
npm install
npm run reproduce

Sample run (trimmed for brevity):

------------------------------
-- Using packaged Kubo 0.38.0 --
------------------------------
Binary path: .../node_modules/kubo/bin/ipfs
Selected ports: { api: 41047, gateway: 40758, swarmTcp: 42351, swarmQuic: 56172 }
...
Added CID: QmfNCoN51J2fLkbSTTFTBbYFX23GEqAhLF5DLjDSjXNf5h
Published IPNS: {
  name: 'k51qzi5uqu5dlws3n28q0wq2xf5vzbadu84buuqz5h5kma6mpmnwgqc38uangp',
  value: '/ipfs/QmfNCoN51J2fLkbSTTFTBbYFX23GEqAhLF5DLjDSjXNf5h'
}
Resolving IPNS name ... with options { timeout: 60000 }
Immediate resolve failed: Error: No IPNS result returned before the stream closed
...
Post-restart resolve failed as expected: No IPNS result returned before the stream closed

Every run so far ends the same way: the publish reports success, but neither the immediate nor post-restart resolve ever returns the new record.

[guillaumemichel]

The current resolution process needs to query the content router (DHT or HTTP endpoint) SearchValue(), and has a 60s default timeout (see --dht-timeout in ipfs name resolve --help, which is applied to all requests, not just DHT). The IPNSPublisher has a local cache in which it writes the published names, but the IPNSResolver doesn't have access to that cache, hence it cannot read from it.

The reason for the remote query is that the local cache may not have the most up-to-date version of the record. If the node is online, it should always try to fetch the last version of the record (and get consensus if DHT).

A few questions:

1. Is this a regression with v0.38.0 only (and it was working as expected before)
2. Is your goal to read the record from the local cache only (and not from remote HTTP endpoint)? If yes, why?
3. Is it expected that the GetIPNS request to the HTTP endpoint hangs until the 60s timeout?

Potential fixes

The IPNSResolver should have access to the cache (datastore) to which the IPNSPublisher writes the published names. The name in the cache (if any) would be considered as one of the potential best records, but can be replaced by a newer remote version.

It isn't enough, since the node will try to get the latest version from the name from the network (or timeout). In you only want to read from local cache, you can either:

1. Set a very low timeout, so that the Resolver immediately aborts the GetIPNS call, resulting in returning the result from the cache (if any)
2. We can introduce a new --offline or --cache-only flag, which will only consider the results from the cache, and won't try to fetch a newer name from the network.

Side note

It seems that the --nocache option (ipfs name resolve) is useless, since the IPNSResolver doesn't have a cache.

[Rinse12]

Our usecase is as follows:

• Each community of plebbit-js is stored in a single SQLite database and is represented as an IPNS record in the network
• That database stores everything related to the community, including its raw IPNS record (most importantly its sequence)
• Consumers of plebbit-js can now move and migrate communities by only copying the sqlite db, without needing to migrate the IPFS repo
• Everytime a community publishes a new IPNS record, plebbit-js updates the IPNS record in the sqlite database with the latest local record from IPFS repo
  • There's no need to fetch the IPNS record from network, since plebbit-js is the signer of the IPNS record and just published the latest record

Answers for your questions:

1. Seems to have the same problem on 0.37, 0.36 as well
2. Yes, the goal is to read from the local cache only, explained by our usecase above
3. We expected since it's a local node, and the same node is the signer and publisher of the IPNS record, the process of retrieving the IPNS record should be instant.

We can introduce a new --offline or --cache-only flag, which will only consider the results from the cache, and won't try to fetch a newer name from the network.

I think --cache-only is ideal, or maybe --cache-if-key-exists if the local node has the key of the IPNS record then it should always fetch it from local store.

[guillaumemichel]

So IIUC there is a single authority for each IPNS record. Others can "move"/"migrate" the community (IPNS record) by fetching it from the single authority, but then this new version won't receive new updates made to the old endpoint. Is that correct?

[guillaumemichel]

Also IPNS specs seem to indicate that by design the IPNS resolver shouldn't access the local cache directly.

Furthermore, IPNS records have size limits, which may be hit if the community grows too large?

[Rinse12]

So IIUC there is a single authority for each IPNS record

Correct.

Others can "move"/"migrate" the community (IPNS record) by fetching it from the single authority, but then this new version won't receive new updates made to the old endpoint. Is that correct?

You "migrate" a community by copying its sqlite db (which includes its raw IPNS record) into a new machine. The whole sqlite db is needed for schemas unrelated to ipfs like comments, votes, edits, etc.

You can't "copy" a community by only copying its IPNS record, you would have a "read-only" version of the IPNS record, but you can't publish a new IPNS record even if you had its keys because you don't have the other schemas.

Also IPNS specs seem to indicate that by design the IPNS resolver shouldn't access the local cache directly.

I suppose we should respect that, for us with plebbit-js our usecase would suffice with something like ipfs routing get --offline, a command specifically to fetch from local store only.

Furthermore, IPNS records have size limits, which may be hit if the community grows too large?

I don't think we have to worry about it in our case, the community's data is in the cid included in its IPNS record, not in the record itself.

[guillaumemichel]

Triage notes:

• We should add an --offline flag to ipfs name resolve
  • This command would return the latest version that is the the cache (or nothing)
  • Doesn't even try to get the latest version from the network
• We should give access to the cache to the IPNSResolver
  • The cache is the datastore used by the IPNSPublisher