Musings on an architecture that doesn't need a custom linker

[madsmtm]

I just read https://github.com/wasm-bindgen/js-bindgen/blob/main/doc/FAQ.md, very nice overview!

As I understand it, to create web-sys, you'd basically do:

• Pre-generate #[js_sys::js_sys(...)] macro invocations.
• Inside js_sys::js_sys, generate an assembly string placed in a custom js_bindgen.assembly link section that contains a "thunk" function that calls the symbol that you want to call with the ABI that Rust expects.
• Link to and call that thunk with #[link_name = "..."].
• When linking with the custom linker, grab the data from those custom js_bindgen.* sections, extract and assemble them, and turn them into an object file that the actual linker can work with.

I get that the string can't be constructed at macro time if you want to use Rust types to specify how it behaves, but that doesn't sound like it'd be needed for e.g. web-sys, if you're generating from IDL you're gonna have enough information available to also create the required assembly for the thunk?

That is, it isn't clear to me why you don't instead do the following?

• Pre-generate thunk invocations using #[link_name = "..."].
• Pre-generate a bunch of separate *.wat files that contains all the thunks.
• Assemble the thunks in build.rs.

This has the downside that it would duplicate the data (though could be remediated by placing e.g. gen_AbortController.rs right next to gen_AbortController.wat), and that there is no longer a "single source of truth", but with the benefit that you don't need a custom linker?

[madsmtm]

Is the motivation that there's a large need for users to write #[js_bindgen::*] themselves? Would it perhaps make sense to consider that use-case separately?

[madsmtm]

Perhaps there are other considerations that make the tradeoff worth it? For example, I see js_bindgen::embed_js!, that ofc. wouldn't be possible without some sort of wrapper on top.

[daxpedda]

I get that the string can't be constructed at macro time if you want to use Rust types to specify how it behaves, but that doesn't sound like it'd be needed for e.g. web-sys, if you're generating from IDL you're gonna have enough information available to also create the required assembly for the thunk?

Yeah, I have that in a TODO:

js-bindgen/TODO.md

Lines 51 to 52 in 3924fec

	- Add a `disable_hygiene` attribute to `#[js_sys]` to reduce the compile-time of the output to an
	absolute minimum. This can avoid all `interpolate`s.

That is, it isn't clear to me why you don't instead do the following?

• Pre-generate thunk invocations using #[link_name = "..."].
• Pre-generate a bunch of separate *.wat files that contains all the thunks.
• Assemble the thunks in build.rs.

The plan is actually to go a step further: we can just pre-compile everything and just ship object files. No build.rs required.

(Wasm object files are just regular Wasm modules, they are not platform- or compiler-specific)

Is the motivation that there's a large need for users to write #[js_bindgen::*] themselves?

Yes. All these design constrains are because we need users to be able to easily generate bindings themselves. Otherwise we wouldn't be here at all! We could just compile a bunch of object files their Rust bindings and call it a day.

Would it perhaps make sense to consider that use-case separately?

Not exactly. We could of course implement them both separately, but we would be going through similar motions either way. Just to put things in perspective: there's really no use-case without letting users define bindings. I've seen no real-life project that doesn't eventually hit a wall with the provided bindings.

---

You have given me a lot to think about. I never really considered separating these two use-cases, because it has become so normal at this point. I'm not entirely sure yet how I could make use of a design like that, but I will keep it in mind when thinking about the architecture from now on.

[madsmtm]

Re use-cases:

If I were maintaining a library, if I could write an IDL for the things I need access to, run some commandline utility to generate a .rs and .wasm and add a small build.rs script to link that .wasm file, and then only have to add a few lightweight macro dependencies, I would absolutely prefer that! Even more so if my project could work with whatever top-level runner mechanism the user decided to go with (but that might require WIT IIUC?)

But as an end user that wants access to some API, I'd prefer just writing #[js_bindgen::xyz] on some Rust, and have that magically work when I run cargo js-bindgen serve.

Just to put some examples on that "separate use-cases".

---

Actually, a probably noobie question: How does it work when I want to access globalThis.console.log? There needs to be some top-level thing that exposes that to the wasm blob, right? Who does that? cargo wasm-bindgen/js-bindgen, or some lower-level tool?

Relatedly, I'd encourage you to check out manganis, the Dioxus team has put a lot of work into solving the problem of "how do I bundle necessary assets with my Rust library" (in their case mostly images and CSS, in your case probably .js files). (I haven't actually used it myself, I only remember talking to people about it at RustWeek last year).

[daxpedda]

Re use-cases:

If I were maintaining a library, if I could write an IDL for the things I need access to, run some commandline utility to generate a .rs and .wasm and add a small build.rs script to link that .wasm file, and then only have to add a few lightweight macro dependencies, I would absolutely prefer that! Even more so if my project could work with whatever top-level runner mechanism the user decided to go with (but that might require WIT IIUC?)

But as an end user that wants access to some, I'd prefer just writing #[js_bindgen::xyz] on some Rust, and have that magically work when I run cargo js-bindgen serve.

Just to put some examples on that "separate use-cases".

Yes, we are definitely planning that whatever web-sys will do users can have access to as well. I haven't figured out how exactly yet, but access to the web-sys WebIDL generator was requested before, so that could just be it.

The last time I've thought about dropping the macro completely from the core implementation I decided against it because while web-sys can generate everything via WebIDL, js-sys can't. But after gathering some implementation experience and hearing your perspective I think this was a mistake.

I'm trying to formulate a plan on how I can pre-generate all the necessary code for js-sys. Maybe around 10% is handwritten, but the rest has to go through the macro.

• Move js-sys-macro into js-bindgen (behind a crate feature).
• Rename #[js_sys] to #[js_bindgen].
• Move the proc-macro implementation into a lib crate (versus a proc-macro crate).
• Work on a js-sys-generator crate that uses the proc-macro lib to produce pre-compiled files.
• Now that the proc-macro is out of core, switch to syn?
• The WebIDL generator makes use of the proc-macro lib and is the pre-compiler targeted towards users.

Or maybe starting with the proc-macro was a complete mistake to begin with. It might have been better to develop a code generation library that the proc-macro and WebIDL generator could hook into. That way I don't have to frame everything in terms of the proc-macro.

Hah, this would be a complete rewrite! I'm gonna think about this a bit more, but I think this idea is good.

Thank you so much @madsmtm! You have destroyed my work so far 😆

Actually, a probably noobie question: How does it work when I want to access globalThis.console.log? There needs to be some top-level thing that exposes that to the wasm blob, right? Who does that? cargo wasm-bindgen/js-bindgen, or some lower-level tool?

The JS shim. This is how it currently looks like. Our custom linker generates it and puts it right next to the output file.

Relatedly, I'd encourage you to check out manganis, the Dioxus team has put a lot of work into solving the problem of "how do I bundle necessary assets with my Rust library" (in their case mostly images and CSS, in your case probably .js files). (I haven't actually used it myself, I only remember talking to people about it at RustWeek last year).

Oh god, that looks great and really well thought through! I was planning to eventually add a CLI tool like wasm-pack that would take care of bundling, maybe we could simply use the Manganis integration to take care of that!

I will add it to the TODO list!

[daxpedda]

I just took a closer look at your title and the question about the custom linker 😅

Even if we pre-generated everything and the proc-macro would become purely a bonus, we still need it to generate the JS shim. While we could go with something like the wasm-bindgen-cli, the idea was that integrating the post-processor into a custom linker is much easier for users to set up compared to a post-processor CLI that users have to run manually (or rely on yet another tool to do it for them). I'm still thinking of ways to make linking fail without the custom linker to ensure the setup was done correctly.

In any case, the custom linker is absolutely required for the proc-macro. Unlike wasm-bindgen our code is compiled and fed into LLD and is not fixed/injected in post-processing.

[madsmtm]

I'm trying to formulate a plan on how I can pre-generate all the necessary code for js-sys. Maybe around 10% is handwritten, but the rest has to go through the macro.

To be clear, I don't actually mean we need to make that possible to emit that from IDL too. As a library author, I'd be fine with certain types being special-cased / "having" to depend on js-sys.

(At least this would mirror objc2 - we expose things to allow writing your own bindings to e.g. NSData, but the core bindings to the Objective-C runtime and NSObject is special-cased).

You have destroyed my work so far 😆

Haha

JS shim

Yeah, that makes sense. And I agree that integrating it into the linker is the right choice, it is the only place where you actually know which symbols are gonna be used and such.

Curious, do you know how other languages handle declaring imports and exports? If I wanted to write a function in C or Go or whatever that used globalThis.console.log, how would I do so? And how would js-bindgen interface with it?

[madsmtm]

Also, new favorite operator: >>>=

[daxpedda]

I'm trying to formulate a plan on how I can pre-generate all the necessary code for js-sys. Maybe around 10% is handwritten, but the rest has to go through the macro.

To be clear, I don't actually mean we need to make that possible to emit that from IDL too. As a library author, I'd be fine with certain types being special-cased / "having" to depend on js-sys.

Yes indeed! The js-sys-generator would be for internal use only.
The reason why js-sys has to be largely generated is because it would otherwise be extremely repetitive.
I was thinking about the fact that I can't generate js-sys via the WebIDL generator because I didn't know how else to avoid the macro.

Curious, do you know how other languages handle declaring imports and exports? If I wanted to write a function in C or Go or whatever that used globalThis.console.log, how would I do so?

You can declare imports and exports in Wasm and other languages expose ways to access those "instructions". E.g.:

void __attribute__((import_module("bar"))) foo(void);

These imports must be provided when instantiating the Wasm module in JS and there is no way to tell JS to automatically provide imports based on its name or anything like that. So a JS shim is always necessary.

The ES Module Integration Proposal supports importing from JS files and let the JS engine automatically import that JS file and the symbols you want to import from it. But the JS file still has to exist. When that proposal is more widely available we could write a JS shim that supports that.

There is currently no active proposal trying to provide a JS-free way to access globalThis, not even the component model, which currently defines no interactions with JS at all.

And how would js-bindgen interface with it?

Well, because there is no standardized way to generate a JS shim, we have to implement compatibility with toolchains explicitely. I don't know how Emscripten does its JS shim thing, but its definitely worth having compatibility with them. But whatever the story is here, its going to be the same amount of work as compatibility with wasm-bindgen.

[madsmtm]

Yes indeed! The js-sys-generator would be for internal use only.

👍

You can declare imports and exports in Wasm and other languages expose ways to access those "instructions". E.g.:

void attribute((import_module("bar"))) foo(void);

These imports must be provided when instantiating the Wasm module in JS and there is no way to tell JS to automatically provide imports based on its name or anything like that. So a JS shim is always necessary.

The ES Module Integration Proposal supports importing from JS files and let the JS engine automatically import that JS file and the symbols you want to import from it. But the JS file still has to exist. When that proposal is more widely available we could write a JS shim that supports that.

There is currently no active proposal trying to provide a JS-free way to access globalThis, not even the component model, which currently defines no interactions with JS at all.

Makes sense, thanks for the explanation! I think it makes sense too, we'll want to keep WASM "pure".

there is no standardized way to generate a JS shim

Unfortunate, but I kinda suspected that.

In any case, it seems like only having to support js-bindgen.import sections would be simpler for other tooling too (than having to support js-bindgen.assembly as well), and if a standard / common way of doing this across tooling does appear, it'd be possible to swap out pretty transparently.

[daxpedda]

There is currently no active proposal trying to provide a JS-free way to access globalThis, not even the component model, which currently defines no interactions with JS at all.

Makes sense, thanks for the explanation! I think it makes sense too, we'll want to keep WASM "pure".

It has already been corrupted with "builtins", see the JS String Builtins Proposal. These have to still be provided manually during instantiations, but the ES Module Integration Proposal would actually provide these on its own when it detects them. So that would be the first instance of automatic JS imports.

The set of imported functions is however extremely limited, we basically have no use for them.

See my Wishlist for many more builtins I would like to have!

In any case, it seems like only having to support js-bindgen.import sections would be simpler for other tooling too (than having to support js-bindgen.assembly as well), and if a standard / common way of doing this across tooling does appear, it'd be possible to swap out pretty transparently.

That would be really nice actually. Though imports can be really complicated. js-bindgen.import combines with js-bindgen.embed to make it actually work properly.

js-bindgen.assembly will go away as soon as we have inline assembly in Rust.

[daxpedda]

So I'm already working on an experimental implementation of the generator. I can already say its extremely freeing to just go to quote + syn instead of the mess we have in place right now.

I'm still trying to figure out how I can keep all of this transparent. E.g. right now the code in js-sys is very readable, including the assembly. But now I generate the Rust code and assembly from another crate. The Rust code is fine, however its sad that now developers have to work in the generator to implement things instead of working with the actual code directly.

But I don't know what to do with the assembly. I'm basically placing object files in the source folder with no way to inspect them. I still have to generate an .rs file to import the object file. In there I could dump the assembly as a comment. Again, still harder to maintain but at least more transparent.

I also don't know how people feel about these pre-compiled blobs in their libraries. I understand of course that nobody wants actual binary blobs in their library, but these are Wasm binaries and can't actually run on the host machine.

In the very first iteration of js-bindgen I introduced a cfg flag to let users choose between pre-compiled Wasm and build-time compiled Wasm. In the end that experiment completely failed for various reasons, but one definite reason for this experiment right now is that hiding dependencies behind cfg flags doesn't work during cross-compilation.

After writing up all of this, I believe that for now I will table pre-compilation and will just be satisfied with generating pre-expanded proc-macros, including the assembly.

I will say that the biggest downside of this all is that the maintenance experience is much worse. Working on a code generator is much harder then working on the source code directly.

I don't have anything useful to look at right now, but I will be pushing some code tomorrow. Would love to hear your thoughts on my ramblings though 😅

[madsmtm]

But I don't know what to do with the assembly. I'm basically placing object files in the source folder with no way to inspect them. I still have to generate an .rs file to import the object file. In there I could dump the assembly as a comment. Again, still harder to maintain but at least more transparent.

Maybe you could generate both .rs, .wasm and .wat files?

I also don't know how people feel about these pre-compiled blobs in their libraries. I understand of course that nobody wants actual binary blobs in their library, but these are Wasm binaries and can't actually run on the host machine.

Skimming serde-rs/serde#2538 again, I think people are mostly against pre-compiled binaries if they:

1. Cannot be reproduced.
2. Aren't portable.
3. Security concerns.

None of these problems really apply here.

I will say that the biggest downside of this all is that the maintenance experience is much worse. Working on a code generator is much harder then working on the source code directly.

I know ;)

[Spxg]

I understand of course that nobody wants actual binary blobs in their library, but these are Wasm binaries and can't actually run on the host machine.

Yeah, compiling from source code is the best option, or we can use .wat, just like: https://github.com/unicode-org/icu4x/blob/main/components/collections/codepointtrie_builder/cpp/ucptrie_wrap.wat

If binary is absolutely necessary, we need to ensure our binary is verifiable. This can be done by building and signing it using GitHub Actions: https://github.com/actions/attest-build-provenance