Changing Wasmtime's versioning for unreleased branches

[alexcrichton]

I'd like to propose a change to how we manage versions in the Wasmtime repository motivated by downstream usage in Spin. Specifically I'd like to propose improving signalling of in-development versions of Wasmtime and assist in making it as easy as possible to test release branches before we publish them. The goal here would be to retain the "feel" of our release process today (just hit merge on PRs) while changing the underlying implementation.

Specifically what I'd like to propose is:

• The main branch perpetually has a version number X.0.0-dev. The "-dev" part here signifies that it'll never be published anywhere and it's entirely in development. While not strictly necessary this'll help differentiate development on a branch from release branches I believe.
• Upon making the release-X.0.0 branch, currently automation sends a PR to main to bump to (X+1).0.0. It'd still do that, but it'd also create a PR against the release-X.0.0 branch which changes the version number to X.0.0-rc.1.
• Upon bumping to X.0.0-rc.1, this PR would additionally have "[automatically-tag-and-release-this]" which means that X.0.0-rc.1 would be published to crates.io and additionally be published to this repository's github releases.
• I'd expect that the github releases for X.0.0-rc.1 would permanently be listed as a "pre-release" in the GitHub sense. We'd create corresponding tags and such for this exactly as-is today.
• If necessary we could bump the rc on the release branch manually (triggered via a specific input to github actions) and make multiple release candidate publishes before a branch is actually released.
• As is today the actual publication of the release branch would happen via automation where we'd automatically change X.0.0-rc.N to X.0.0 and then tag/publish that.

This would enable publishing pre-release artifacts of Wasmtime to crates.io to enable as-easy-as-possible downstream consumption to help report issues and test out what's about to be published. This would enable Spin, for example, to update and test out easily and see if anything needs updating. While this is all possible in theory to do with git dependencies it's generally much easier to work with crates.io dependencies and I think it's wortwhile to lower the bar as much as possible to make it easy to test out pre-release artifacts.

I wanted to open an issue here and see if anyone had any thoughts about this though and/or opposed this. My intention is that the work here is largely just changing scripts/publish.rs w.r.t. version bumps, adjusting the precise schedule of what makes PRs where, and then updating documentation of our release process.

[tschneidereit]

For some context, we've had situations in the past where downstream embedders of wasmtime discovered that something unexpectedly had regressed in ways that made updating to the current version of wasmtime not viable without further changes. The hope is that by making it as easy as possible to test upcoming versions, we'll get more downstream projects testing them and reporting regressions while they can still be fixed, make that scenario less likely.

[cfallin]

This seems fine to me, as long as we're careful in our documentation and policies on the release candidates.

In particular it would be good to think ahead of time: if we discover a security issue that is only in main and a release branch, but we haven't released yet, do we:

• Cut a new release candidate?
• Yank the existing release candidate crates from crates.io?
• Have to release a security advisory for the affected release candidate?

My answers would be "yes; yes; no" -- we shouldn't leave known-vulnerable code on crates.io, but it is also expected that RCs do not have release-grade support or advisory cover.

And more on the social-policy and -expectation side, the main danger I'd want to watch out for is that over time this becomes an "unofficial release" with higher and higher expectations. The whole reason we have the 2-week "bake in" delay is because we were burned by bad PR merges just before cutting a release; if we start to get nervous merging a PR on the 3rd or 4th of the month because it will go into an RC, we should reconsider this policy IMHO.

[alexcrichton]

Those are good points yeah, and I'd personally want to be clear in documentation that release candidates have virtually no support and should never be a permanent dependency in a system. I'd actually answer your questions as "maybe, maybe, no" insofar as we'd subjectively decide whether a new release candidate is necessary, and not yanking something is a reflection of "this is not an actively supported piece of code".

Put another way, I'd say that if we feel required to go out of our way and actively maintain release candidates (e.g. via listing out affected versions in advisories/etc) then we shouldn't do this. The goal here is to minimally adjust our release process to make it easier to have downstream testing, and if we have to do more than minimally adjust things that might point more towards "use a git dependency". I'd definitely consider anxiety-to-merge-on-main a failure of process if that's what happens as a result of this.

[tschneidereit]

Agreed: good points! I wonder if we could go as far as automatically yanking pre-releases once we do the final one? Though that's probably against the spirit of what yanking a release means, so perhaps not. In any case, I very much agree that we should hold a hard line that means that anyone using these in production is entirely on their own, exactly as though they used a git dependency on some commit from the release branch.

I also think that that's a line we can hold, and wouldn't expect anyone to push back against in in ways we'd want to take seriously.