add codspeed

[pedrocarlo]

Description

Add CodSpeed to also monitor benchmarks and have access to some other nice features: https://codspeed.io/docs/features/profiling

[codspeed-hq]

Congrats! CodSpeed is installed 🎉

🆕 353 new benchmarks were detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

Detected benchmarks

• limbo_execute_select_1: 7.8 µs
• limbo_wal_concurrent_writes: 756.5 ms
• sqlite_concurrent_writes: 188.6 ms
• limbo_mvcc_concurrent_writes: 2.5 s
• limbo_execute_select_count: 115.9 µs
• sqlite_execute_select_1: 4.2 µs
• sqlite_execute_select_count: 20.1 µs
• limbo_execute_select_rows[10]: 44.7 µs
• limbo_execute_select_rows[1]: 26.1 µs
• limbo_execute_select_rows[50]: 122.9 µs
• sqlite_execute_select_rows[50]: 53.5 µs
• limbo_execute_select_rows[100]: 219 µs
• sqlite_execute_select_rows[100]: 90.9 µs
• fast_path_datetime_hhmm: 9.6 µs
• begin_tx + rollback_tx: 12.9 µs
• sqlite_execute_select_rows[10]: 23 µs
• begin_tx-update-commit_tx: 16.1 µs
• begin_tx-read-commit_tx: 15 µs
• sqlite_execute_select_rows[1]: 16.2 µs
• limbo_insert_100_randomblob: 141 ms
• ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

---

[turso-bot]

Please review @sivukhin

[art049]

Hey @pedrocarlo, just checking how the integration is going 😄

Normally, you don't need to import directly from codspeed-criterion-compat

You can rename the package during the install so it doesn't require code changes in the benchmarks:

criterion = { package = "codspeed-criterion-compat", version = "*" }

Or did doing it create some compatibility issues?

[pedrocarlo]

Hi @art049,



Thanks for checking in on us! 

We currently use the pprof profiler for Criterion, and pprof hasn’t upgraded their Criterion dependency to 0.8.1. I don’t know exactly what version Codspeed is using for the criterion-compat, but it is incompatible with the profiler. So the easiest thing for me to do was to just gate the compat crate under a feature flag and disable the profiler for codspeed. 



While you are here, do you know why codspeed emitted this notice for the tpc-h-benchmark? 

NOTICE: codspeed is enabled, but no performance measurement will be made since it's running in an unknown environment.



Did I do the setup wrong for this benchmark?
https://github.com/tursodatabase/turso/actions/runs/20736437443/job/59534571622?pr=4461

[art049]

While you are here, do you know why codspeed emitted this notice for the tpc-h-benchmark? 

NOTICE: codspeed is enabled, but no performance measurement will be made since it's running in an unknown environment.



Thanks for bringing this up, this is a bug that was introduced in the latest release. Will be fixed with CodSpeedHQ/codspeed-rust#158

Make sense for pprof. I didn't know this was the reason.

[pedrocarlo]

Awesome. Thank you @art049 .

Please ping me when there is a new release of the cargo codspeed so that I can upgrade.

Also, I just noticed that some tests are generating traces and some are not.

Does not Generate Trace
https://codspeed.io/tursodatabase/turso/runs/695c7cc0ac86ac4f82c9775a?uri=core%2Fbenches%2Fbenchmark.rs%3A%3Abenches%3A%3Abench_execute_select_1%3A%3AExecute%2520%2560SELECT%25201%2560%3A%3Alimbo_execute_select_1&runnerMode=Instrumentation&sectionId=benchmark-comparison-section-single-run-benchmarks

Generates Trace
https://codspeed.io/tursodatabase/turso/runs/695c7cc0ac86ac4f82c9775a?uri=core%2Fbenches%2Fjson_benchmark.rs%3A%3Abenches%3A%3Abench%3A%3AJSONB%2520Size%2520-%2520Real%2520world%2520json%25205%3A%3ALimbo&runnerMode=Instrumentation&sectionId=benchmark-comparison-section-single-run-benchmarks

[GuillaumeLagrange]

Hey @pedrocarlo

We have released v4.2.1 of our rust integrations, which fixes the erroneous warning.

I'm also starting to look into why the trace generation was not successful, I'll keep you up when I find anything.

[GuillaumeLagrange]

Hello @pedrocarlo

Looking into it, I found that the profiling data generation actually succeeded for the example that you pointed out, but we failed to save it because one of the benchmarks has an URI that is too long.

That's definitely on us, we're working on making sure things do not break if the URI is too long, are at least to make sure that things break with an actionable feedback for the user. We'll fix this asap and let you know when the fix has been deployed!

[GuillaumeLagrange]

FYI, the offender is the bench with the name
limbo_parse_query[SELECT first_name, last_name, state, city, age + 10, LENGTH(email), UPPER(first_name), LOWER(last_name), SUBSTR(phone_number, 1, 3), zipcode || '-' || state, AVG(age) + 5, MAX(age) - MIN(age), ROUND(AVG(age), 1), SUM(age) / COUNT(*), COUNT(*), COUNT(email), SUM(age), AVG(age), MIN(age), MAX(age), SUM(CASE WHEN age >= 18 THEN 1 ELSE 0 END), SUM(CASE WHEN age < 18 THEN 1 ELSE 0 END), AVG(CASE WHEN age >= 18 THEN age ELSE NULL END), MAX(CASE WHEN age >= 18 THEN age ELSE NULL END) FROM users GROUP BY state, city]

If you change it so a shorter ID, it should work immediately.

What we intend to change is to have an explicit error message when this happens, so this is a change that you will have to do anyway.

[jussisaurio]

thread 'tokio-runtime-worker' panicked at stress/main.rs:894:33:
thread#1 integrity check failed: Text("NULL value in hard_fish_311.slow_moon_468")

Hmm... this is highly likely to be the New And Improved ™️ Integrity check reporting a false positive. Should still check though

[pedrocarlo]

Uhm I don't remember if I rebased with your changes that you just merged.

[pedrocarlo]

Hey @GuillaumeLagrange,

Awesome thanks for looking into it. I truncated the test names and now I can see a bunch traces for the benchmarks!

Can I bother you hopefully one last time? The TPC-H benchmark for some reason is timing out. However, in this other CI run we have it does not: https://github.com/tursodatabase/turso/actions/runs/20831566769/job/59846777522?pr=4461

Do you know what might be the reason? Is there something I should be doing from my side?

[GuillaumeLagrange]

@pedrocarlo are these benchmark heavily multithreaded ?

We've had cases where we observed deadlocks in simulation mode due to how valgrind flattens the thread executions. For these cases we generally recommend trying out our walltime instrument.

I'll let you know if I find anything.

[pedrocarlo]

Hi @GuillaumeLagrange,

Thanks! They are all single threaded!