Register grid-cbor codec

[stevegt]

Registers grid-cbor as draft IPLD code 0x1027. The code identifies PromiseGrid envelopes. PromiseGrid is a decentralized protocol, computing, and storage infrastructure based in part on Promise Theory.

PromiseGrid uses deterministic CBOR envelopes for messages, files, and CAR blocks, each wrapped in the CBOR grid tag (1735551332). The first slot in a grid envelope is a protocol CID (pCID) which refers to a document that in turn describes the rest of the slots. In order to support graph traversal, codecs need to preserve the 'grid' tag, arbitrary tags, and arbitrary map-key types, as well as tag-42 CID links.

Multicodec specification: https://github.com/promisegrid/promisegrid/blob/main/docs/grid-cbor-multicodec-spec.md
Grid tag specification: https://github.com/promisegrid/promisegrid/blob/main/docs/grid-cbor-tag-spec.md

[vmx]

You've chosen ipld as the tag. Do you plan to make it interoperable with IPLD?

I'm asking as grid-cbor contains things that cannot easily mapped to the IPLD data-model. For example arbitrary tags or map keys.

[stevegt]

You've chosen ipld as the tag. Do you plan to make it interoperable with IPLD?

Yes.

I'm asking as grid-cbor contains things that cannot easily mapped to the IPLD data-model. For example arbitrary tags or map keys.

The plan is to translate non-compliant data types in ways that are consistent enough to round-trip back to the original while retaining deterministic order -- convert them into string-keyed maps with fields such as "type", "key", "value" on read, reverse on write.

[vmx]

The plan is to translate non-compliant data types in ways that are consistent enough to round-trip back to the original

For arbitrary map keys, e.g. integers, you would convert them to strings. How would you make sure that when converting back, only integers that were originally integers are converted? This was a bit cryptic, so an example might help:

Having a map like this (in CBOR diagnostic notation) {32: true} would then e.g. become in the IPLD Data Model: Map([(String(32), Bool(true))]). But {"32": true} (note the quotes) would result in the same data within the IPLD Data Model.

The same problem occurs for aribitrary tags. You woudn't be able to distinguish a tag from someone storing an object that genuinely has that shape.

[stevegt]

Hi Volker,

For arbitrary map keys, e.g. integers, you would convert them to strings.

In my earlier message, I said that we would convert them into string-keyed maps with fields such as "type", "key", "value".

Using that rule and your example, {32: true} becomes:

{
  "type": "boolean",
  "key": 32,
  "value": true
}

I didn't expect this to be controversial. I'm curious -- is there a written standard anywhere for what does and does not constitute a valid entry in table.csv? The overall multiformats project is great, but I see stale PRs here in multicodec that seem to be held up based on subjective measures. Making the multicodec registration process less subjective would probably reduce your workload and help more folks cleanly interoperate with the IPLD ecosystem.

I've followed the instructions at https://github.com/multiformats/multicodec/tree/master#adding-new-multicodecs-to-the-table. Those instructions and "first come first assign" don't mention a design review.

Before submitting this PR, I put in a lot of due diligence in ensuring that, for instance, cbor (0x51) isn't suitable, or we would just use it and I wouldn't need to go through this process. I could recount the entire decision tree that led to this PR, but that's time I don't have -- without a written standard, I can't simply give you a spec that meets the standard, which leaves us with unknown number of days of conversation here instead, and an unknown-length critical path pushing out the delivery date for a project timeline that I'm responsible for.

This situation is similar to the blocking situation that Jeromy ran into 10 years ago at #16 (comment), which is why Juan suggested "first come first assign" in the first place, and why it is still the written procedure we do have.

If this isn't merged, then I have a choice I need to make within the next few days: Use 0x51, or use 0x1027 anyway and hope it never collides. I believe in standards (which is why I filed this PR) so am not likely to do the latter. I also don't want to use 0x51 now, and then convert to 0x1027 later -- that would bake 0x51 into persistent production data on a lot of decentralized systems, and developers would then have to support both codes forever.

In the interest of being a good neighbor, I picked a two-byte code rather than looking for an available one-byte. I'm willing to go to three-byte or even four if that helps. But if the two-byte namespace has special allocation rules, that should be included in a written standard as well.

So, can we follow the written procedure we do have, rather than essentially re-hashing the same conversation from 10 years ago? If the procedure needs to be changed, can that be done in parallel rather than blocking new registrations?

[vmx]

Using that rule and your example, {32: true} becomes:

{
  "type": "boolean",
  "key": 32,
  "value": true
}

Thanks for clarifying that, I misunderstood your original message. That makes sense and clearly maps. I guess for the arbitrary tags you've found a similar solution.

Sorry for the frustration. Sadly there isn't a real process (we never found the time) and the folks looking into PRs are mostly ones that somehow find time to do it. I already saw that you are within a two byte range, which indicated that you really looked into it.

From the outside it's often hard to tell how much someone has thought through things. Hence we try to review things and give advice as sometimes better solutions can be found. Multicodecs/IPLD sounds simple at a first look, but gets more depth the deeper you dive. But it looks like I don't need to tell you that.

One more advice (feel free to ignore :). At DAG-CBOR we decided to encode floats always as 64-bit. The reason is that implementing the "does it fit without loss into 32-bit" is a bit tedious compared to the size savings. Unless you expect that your input data often uses 32-bit floats, then it could make sense.

[stevegt]

Thanks Volker.

Sorry for the frustration. Sadly there isn't a real process (we never found the time) and the folks looking into PRs are mostly ones that somehow find time to do it.

I understand far too well -- you'll see I just opened #410 to, if nothing else, get a gut check from everyone as to whether it's worth a better process or just punting and adopting the CBOR tag registration process.

I already saw that you are within a two byte range, which indicated that you really looked into it.

I agonized. ;-)

From the outside it's often hard to tell how much someone has thought through things.

Yep -- unless they provide a detailed-enough spec doc, it's really hard to tell if someone is registering on a whim or has put any work into it. Mirroring RFC-8949 would mean that a spec doc is required for multicodec registration. That's why I went through that exercise myself for this PR, trying to give you as much as I could ahead of time.

Multicodecs/IPLD sounds simple at a first look, but gets more depth the deeper you dive.

I noticed. ;-)

One more advice (feel free to ignore :). At DAG-CBOR we decided to encode floats always as 64-bit. The reason is that implementing the "does it fit without loss into 32-bit" is a bit tedious compared to the size savings. Unless you expect that your input data often uses 32-bit floats, then it could make sense.

I default to 64 bit these days unless I have a good reason; my Go code is littered with types much wider than they really should be. ;-)

Thanks again,

Steve

[stevegt]

Volker, I've also enshrined your {32: true} example and some float-handling recommendations in the spec doc now.

This sort of spec-doc-centered process is what I think might work. I'll continue that thought over in #410 comments.