Support for asynchronous compilation

[titzer]

The initial design issue had several suggestions how we might support asynchronous compilation, including a number of suggestions. I think these ideas and alternatives can be summarized as:

• 1: A separate instruction func.new_async that explicitly requests asynchronous compilation
  • 1A: it writes the result into a location (e.g. table index or global) when completed
  • 1B: it returns a kind of promise
  • 1C: it is integrated with stack-switching (e.g. it accepts a continuation)
• 2: The implementation of func.new may return a function that blocks until host-compilation is ready
• 3: There is a polling mechanism for asynchronous functions such as func.ready which queries whether a function is ready for use (i.e. won't block for host compilation)

Let's move further discussion/design of asynchrony to this issue.

@cfallin @kg @yuri91

[kg]

Thank you for writing up the ideas/alternatives here!

Re each:

For 1a to be truly useful you need to first fill the table slot with a 'safe to call' placeholder function, which can be costly. But that's what we do right now in one case. It's probably possible to do better.

1b seems most compatible with the existing web model, it's just not suitable for existing synchronous code, so it would only be usable in new from-scratch software where the callers are all async and can yield on a promise. Unless maybe it would be paired with JSPI? JSPI + things like mutexes feels scary.

I don't understand 1c well enough to evaluate it.

2 seems like a good solution - for scenarios where you end up JITting code long enough before it's used there is no pause, but it gives you support for synchronous execution when you need it without the hazard associated with 1a or the code changes implied by 1b.

3 seems like it's functionally equivalent to 1b in terms of what it does to your application code, just with the advantage of not (potentially) needing an event loop turn or a Promise object. Not needing the Promise objects could add up for applications jitting tens of thousands of functions, I suppose? It does have an advantage over 2 which is that if you have a fallback way to execute without the jitted code, you can use that while the compiler is working.

To briefly restate the motivation for wanting something like 1a or 3, during application startup you can end up with time consuming compute tasks like parsing a big json file or a timezone database. Our tiering model approaches this by running everything in a slow baseline interpreter, then tiering hot code up with on-stack replacement to optimized interpreter IR, then finally JITing compiled WASM versions of that optimized interpreter IR that we can execute as a replacement for blocks of opcodes.
So if we had to wait for an event loop turn, we wouldn't be able to optimize the rest of that big JSON parsing operation or tzdb parsing operation, etc, only future ones. This scenario is worse since it is typically going to manifest as slow application startup which is bad for metrics.

[cfallin]

Another idea that I had recently that I feel might address some of the usability weirdness of the true async (option 1a / option 3) variants, at the cost of a little flexibility: func.replace. The idea would be that one passes all the args to func.new, plus the "default" callee as another funcref with a compatible signature. The semantics are nondeterministic: calling the returned funcref can invoke either the default callee, or the new function. This neatly wraps up the question of how the async completion of compilation is fed back into the funcref (it's done "inside the opcode", with the mechanics of the conditional-on-call or whatever hidden from the Wasm), and also nicely handles the JIT-not-available question: one can always return the default callee as the result directly, and do nothing else. (In other words, a trivial polyfill is the identity function.) What do you think?

(Feel free to bikeshed the name: func.new_or_else, func.new_or_default, ...)