[Feature Request] Add `to_bytes` and `from_bytes` to shape classes

[sebastian-goeldi]

Hi Matthias,

Not sure how feasible it would be, but here goes:

I would like to be able to serialize the shapes to bytes, similarly to to_s and from_s on a lot of them. This would be helpful because then I can store some keys for hashing etc in files. I would like to avoid the strings as that seems inefficient, but also I assume for floats that's not necessarily truly accurate ;).

For context: What I want to build is something that allows me to cache results in partial gds files related to function calls. These function calls are cached and if they use e.g. a box, I need to be able to serialize/deserialize them (in python I am doing that with pickle and copyreg and am currently using to_s() and from_s()).

Sebastian

[klayoutmatthias]

I extracted some numbers on pure C++ level (no Python involved). Maybe that is helpful for the decision, if a binary representation will be helpful:

• DBox: full serialization/deserialization via string is about 5µs. Deserialization accounts for 1µs. String memory is about 30 bytes on average, with 32 bytes for a binary representation.
• Box: full serialization/deserialization via string is about 1.7µs. Deserialization accounts for 0.3µs. String memory is about 30 bytes on average, with 16 bytes for a binary representation.

to_string is not well optimized. There is room for improvement.

Matthias

[sebastian-goeldi]

One of the big reasons (for me), would be floating point accuracy. I almost assume you will lose some accuracy for some unfortunate numbers. This is somewhat relevant for me (or us), as we have some transformations that will use off grid coordinates for some positions (basically connection points between optical connections in case they are not on 0/90/180/270 connections).

The speed / data density is only of secondary interest.

Sebastian

[klayoutmatthias]

Hi Sebastian,

The problem I see is that even binary representation may not be 100% accurate because of the limitations of the 64bit floating point-format. String representation may be even able to fix that. If you have a value of "0.001" which is represented once as "0.000999999999" once, and another time as "0.001000000001" (for example), both will be represented as "0.001" in string. On deserialization, both will come out the same way.

Are you referring to the 45 degree rotation issue? I thought that is a more general issue and to solve that one needs to have algorithms that are tolerant against snap problems - like closing gaps during polygon operations or a fuzzy definition of "touching". I don't think that the original data can ever be exact in a sense that floats will have exactly the same value for two points that are computed through different data paths.

Matthias

[sebastian-goeldi]

In optical components for a general problem, any orientation could be desirable. I have seen klayout's matching of equalness being stable enough to survive serialization / deserialization to gds and oasis. I didn't have to do any tests myself yet, although I expected to need to do that at some point :P

I was thinking being able to dump numbers directly as their binary representation would be the most accurate representation, but yeah, maybe you are right and string is good enough.

The closing gap was funnily enough entirely solved by introducing a virtual cell, i.e. a crude python class mimicing a Cell, but deferring any snapping / conversion to dbu until the final transformation is known and a unique cell with the diff to the next accurate DCplxTrans being inserted into the Cell.

But my main concern here are ports positions as they need to stay "ideal". Anything but 0/45/90/135/180/225/270 rotations do not have an accurate center on grid (45 causes also only under further restrictions)

I know other photonic EDA tools essentially don't solve this (I think maybe with the exception of Cadence, but I am not familiar and only saw small snippets/demos). Often they take the " just apply closest complex transformation to the normally rotated instance. Which of leaves tons of snapping errors, or is flat out not allowed by the fab)

[klayoutmatthias]

If you snap flat layouts, the snapping is deterministic, so same exact coordinates will end on the same location. Maybe that explains the positive effect of virtual cells.

I'll build the to_bytes feature, but frankly, I don't think this will really solve your problem.

I have something different in mind, which we could discuss separately or better face to face :)
My idea is based on partial contours and a tolerant stitching algorithm that I would borrow from the DXF reader.

Matthias

[klayoutmatthias]

I created a PR that adds to_bytes/from_bytes to (D)Box, (D)Edge, (D)Point, (D)Vector, (D)Trans, (D)CplxTrans (and variants), (D)EdgePair, (D)Polygon, (D)SimplePolygon, (D)Text and (D)Path and the corresponding "...WithProperties" objects.

The binary format is made for cross-compatibility and future extensibility, so it should be fit for persisting information over a longer period and across operating systems. As of now, I did not check Intel/ARM interoperability yet.

I also added string serialization ("from_s") for the "...WithProperties" objects, which was lacking.

While adding that last feature, I noticed that string serialization of properties (for the "...WithProperties") objects suffered from a syntax problem - the properties cannot be restored properly from these strings. I had to change the format of the properties part in the "to_s" result of "...WithProperties". I hope that does not cause any inconvenience. If it does, I can revert the last change and leave "to_s" broken for these objects.

Matthias