Replace Valgrind check with sanitizers

[simolus3]

The core extension is a fairly low-level library with plenty of unsafe blocks around the FFI layer for SQLite, so it makes sense to test for memory issues despite using Rust.

However, I think Valgrind is not the best fit for this: We only use it for Rust unit tests, and those don't link SQLite. Since they only test safe Rust, chances of finding memory issues there are slim. miri would be a much more powerful tool here, but IMO it's still not that helpful without linking SQLite. Running some tests of the native SDK through miri might be interesting.

So to actually test for memory issues, we need to run our Dart testsuite loading the core extension and SQLite. Dart doesn't directly work with Valgrind, but it has builtin support for common sanitizers like:

• AddressSanitizer, which tracks valid heap allocations and aborts if memory is used after being freed.
• MemorySanitizer, which tracks reads to uninitialized memory.

By compiling SQLite and the core extension with those sanitizers enabled, we can catch issues affecting any component. For example, we'd get a report if a user-defined SQL function in Rust returned uninitialized memory that is then passed to SQLite and finally read by package:sqlite3 in Dart.

I found a real memory leak with this, which I've also fixed in this PR: When installing update hooks with core extension methods, these own a Rc<DatabaseState> and need to be uninstalled explicitly to free data. I've adopted our existing pre_close_vtab for that. Then this found an additional issue in package:sqlite3 which calls sqlite3_{update|commit|rollback}_hook(null) when closing databases, even if no hooks were installed from Dart. So I'd say that's a pretty nice win here, two issues that would have been very difficult to track without sanitizers.

[rkistner]

I wasn't familiar with the sanitizers before, but these look really useful.

What is the impact of the issues on users? The memory leak is self-explanatory, but what's the effect of the other one? E.g. potential crashes?

[simolus3]

but what's the effect of the other one? E.g. potential crashes?

The package:sqlite3 issue is only a memory leak too (but it's definitely a near-miss). sqlite3_..._hook(db, callback, context) returns the previous context pointer with the idea that the caller would then be able to free it. package:sqlite3 doesn't do that and always passes a null pointer as context, so it doesn't try to free something it doesn't own (even if it did, the practical impact would be freeing a Rust struct without dropping it, which IIRC is also just a leak and not unsound).

Either way, this is all theoretical because we use sqlite3_connection_pool to manage and close connections which is not affected by this bug. So it pretty much only affects our tests.