What is the max time a benchmark should take?

[moorepants]

@bjodah What do you think is a reasonable max time for a single benchmark?

[bjodah]

Good question, we need not test all commits to find big regressions:
http://asv.readthedocs.org/en/latest/using.html#finding-a-commit-that-produces-a-large-regression

so that would mean that it's not too bad having long running benchmarks. Ideally I think benchmarks should be on the order of a few seconds, but maybe a few benchmarks being on the order of minute(s) is alright if motivated? (e.g. perfomance regressions wouldn't show up in smaller benchmarks)

[moorepants]

I thinking shorter benchmarks that test the core will be very important. But we may need some longer ones of solve, simplification, etc. And like you say, if a longer one is introduced we should be sure it is required to demonstrate the regressions.

[bjodah]

An other option is to have a separate benchmark folder for very long running benchmarks and have a special asv.conf.json file pointing to this folder (or add some magic to ./run_benchmarks.sh).

[moorepants]

Here are the latest timings on my benchmark machine:

[  0.01%] ··· Running dsolve.TimeDsolve01.time_dsolve                                                                                                                               1.29s
[  0.02%] ··· Running integrate.TimeIntegration01.time_doit                                                                                                                      402.57ms
[  0.03%] ··· Running integrate.TimeIntegration01.time_doit_meijerg                                                                                                              111.50ms
[  0.04%] ··· Running large_exprs.TimeLargeExpressionOperations.peakmem_jacobian_wrt_functions                                                                                        37M
[  0.05%] ··· Running large_exprs.TimeLargeExpressionOperations.peakmem_jacobian_wrt_symbols                                                                                          37M
[  0.05%] ··· Running large_exprs.TimeLargeExpressionOperations.peakmem_subs                                                                                                          37M
[  0.06%] ··· Running large_exprs.TimeLargeExpressionOperations.time_count_ops                                                                                                    63.59ms
[  0.07%] ··· Running large_exprs.TimeLargeExpressionOperations.time_cse                                                                                                          76.11ms
[  0.08%] ··· Running large_exprs.TimeLargeExpressionOperations.time_free_symbols                                                                                                 13.40ms
[  0.09%] ··· Running large_exprs.TimeLargeExpressionOperations.time_jacobian_wrt_functions                                                                                      150.91ms
[  0.10%] ··· Running large_exprs.TimeLargeExpressionOperations.time_jacobian_wrt_symbols                                                                                         69.40ms
[  0.11%] ··· Running large_exprs.TimeLargeExpressionOperations.time_manual_jacobian_wrt_functions                                                                               149.35ms
[  0.12%] ··· Running large_exprs.TimeLargeExpressionOperations.time_subs                                                                                                          7.04ms
[  0.13%] ··· Running logic.LogicSuite.time_dpll                                                                                                                                    4.87s
[  0.14%] ··· Running logic.LogicSuite.time_dpll2                                                                                                                                164.79ms
[  0.15%] ··· Running logic.LogicSuite.time_load_file                                                                                                                             12.71ms
[  0.15%] ··· Running physics.mechanics.kane.KanesMethodMassSpringDamper.time_kanesmethod_mass_spring_damper                                                                       6.13ms
[  0.16%] ··· Running physics.mechanics.lagrange.LagrangesMethodMassSpringDamper.time_lagrangesmethod_mass_spring_damper                                                           3.74ms
[  0.17%] ··· Running refine.TimeRefine01.time_refine                                                                                                                              12.91s
[  0.18%] ··· Running solve.TimeSolve01.time_solve                                                                                                                                  5.14s
[  0.19%] ··· Running solve.TimeSolve01.time_solve_nocheck                                                                                                                          5.15s
[  0.20%] ··· Running sum.TimeSum.time_doit                                                                                                                                       29.23ms

The are a few above 4 seconds, which is long. It'd be nice to keep all benchmarks under a second at least.