Skip to content

Add a progressr-based progress bar for sampling.#1138

Open
josswright wants to merge 23 commits intostan-dev:masterfrom
josswright:progressr
Open

Add a progressr-based progress bar for sampling.#1138
josswright wants to merge 23 commits intostan-dev:masterfrom
josswright:progressr

Conversation

@josswright
Copy link

@josswright josswright commented Feb 13, 2026

Submission Checklist

  • Run unit tests
  • Declare copyright holder and agree to license (see below)

Summary

As discussed in #18 , this pull request adds a progress bar to CmdStanR's sample() function via the progressr framework.

This can be enabled via two new options to model$sample(...):

  • show_progress_bar - boolean - default FALSE
    Enables the progress bar.
  • suppress_iteration_messages - boolean - default FALSE if show_progress_bar is FALSE, and TRUE if show_progress_bar is TRUE.
    Causes any output lines reporting an iteration (such as Chain 1 Iteration 1 / 2000 (Warmup)) to be suppressed.

For the progress bar to be displayed, a progress bar handler must be created and registered by the end user. (See: https://progressr.futureverse.org/) For ease of use, a register_default_progress_handler() function has been added. Calling register_default_progress_handler() before a model$sample(...) call will spawn a default progress bar via the cli library.

Copyright and Licensing

Please list the copyright holder for the work you are submitting
(this will be you or your assignee, such as a university or company):

Joss Wright, University of Oxford

By submitting this pull request, the copyright holder is agreeing to
license the submitted work under the following licenses:

josswright and others added 10 commits December 21, 2025 22:55
This uses the
[`progressr`](https://progressr.futureverse.org/index.html) framework to
enable a progress bar for sampling operations. By default, this replaces
standard iteration messages, but not other informative messages produced
during sampling.

From a user point of view, this adds two arguments to the `$sample()`
method:
- `show_progress_bar`: Default = FALSE

  If TRUE, registers a progress bar via `progressr` and signals an
  update for every output line that matches "Iteration:". The user is
  responsible for registering those progress updates with an appropriate
  handler.

- `suppress_iteration_messages`: Defaults to the value of
  show_progress_bar, but can also be set directly.

  If TRUE, disables display of output lines matching "Iteration:", while
  still causing the progress bar to update. This keeps all of Stan's
  other informative output, but just removes the superfluous iteration
  messages when using a progress bar.

I've tried to keep all additions to the code as non-intrusive as
possible.

One minor decision I made was to pass the value of `refresh` from the
`$sample()` method through to the `CmdStanProcs` class so that the
progress bar can report and update the number of steps in the bar to the
same 'scale' as the number of iterations. (Calling `$sample()` with
`iter_sampling=8000` and `refresh=100` will result in 80 'ticks' of the
progress bar, each increasing the progress by 100.)

By default, `progressr` doesn't register a handler to display the
progress bar, as this is the responsibility of the user. Multiple
packages can be used to display the progress bar. A default progress bar
can be registered using the [`cli`](https://cli.r-lib.org/) library in
this way:

```r
library(progressr)
library(cli)

handlers( global=TRUE )
handlers("cli")

options( cli.spinner = "moon",
         cli.progress_show_after = 0,
         cli.progress_clear = FALSE )

handlers( handler_cli(
            format = "{cli::pb_spin} Progress: |{cli::pb_bar}| {cli::pb_current}/{cli::pb_total} | {cli::pb_percent} | ETA: {cli::pb_eta}",
            clear = FALSE ))
```

A variety of alternative progress bar handlers are available, including
audible and notification-based handlers, and ones that interact with
RStudio's jobs pane:
<https://progressr.futureverse.org/articles/progressr-11-handlers.html>
In addition to updating the progress bar on stdout lines that match
"Iteration:", the CmdStanProcs object(s) now pass the current stdout
line to the progress bar as a message for _every_ line, even if the
number of completed iterations hasn't increased.
Added a `register_default_progress_handler()` function to `utils.R` that
creates a default progress bar for sampling operations and registers it
as global handler for progressr. Requires `cli` and `progressr`.
Was accidentally passing `refresh` to each informational progress bar
update, rather than the calculated `progress_amount`. This caused each
message line to update the progress bar, even if it didn't match an
iteration message.
Moved code to close the progress bar to after the `check_finished()`
function.
The nature of CmdStan's output makes pulling appropriate refresh and
update values for the progress bar slightly tricky. In the original
verison of the code, this led to the bar occasionally updating past the
point where it was full, or terminating early due to an incorrectly
calculated number of steps.

This commit reworks the calculations of updates and refresh values for
the progress bar, as well as adding some extra conditions to avoid the
bar potentially crashing in odd scenarios.
New arguments for `iter_warmup` and `iter_sampling` were not being
assigned a default value if not specified in sampling statement.
@VisruthSK VisruthSK linked an issue Feb 15, 2026 that may be closed by this pull request
@VisruthSK
Copy link
Member

Looks like some tests are failing because the lower bound for iter_sampling should be 0 instead of 1.

@josswright
Copy link
Author

Looks like some tests are failing because the lower bound for iter_sampling should be 0 instead of 1.

Great. Thanks for this. (First pull request for me on a project like this, so I was a little unfamiliar with the process and running the unit tests.)

I've fixed the initial errors, and a couple of other warnings. Now running the tests and documentation generation commands locally before I push an update.

josswright and others added 2 commits February 15, 2026 15:46
- `iter_{warmup,sampling}` arguments to MCMCProcs are allowed to be 0.
- Documentation for `register_default_progress_handler` added to
  repository.
- Ensured that calls to `progressr` functions are appropriately
  prefixed.
@josswright josswright marked this pull request as draft February 15, 2026 18:37
@codecov-commenter
Copy link

codecov-commenter commented Feb 15, 2026

Codecov Report

❌ Patch coverage is 37.50000% with 45 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.46%. Comparing base (6b7f04a) to head (fa4ad10).

Files with missing lines Patch % Lines
R/utils.R 0.00% 20 Missing ⚠️
R/run.R 50.00% 19 Missing ⚠️
R/model.R 57.14% 6 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1138      +/-   ##
==========================================
- Coverage   87.97%   87.46%   -0.52%     
==========================================
  Files          14       14              
  Lines        5937     6030      +93     
==========================================
+ Hits         5223     5274      +51     
- Misses        714      756      +42     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

josswright and others added 4 commits February 15, 2026 19:13
Added documentation for `show_progress_bar` and
`suppress_iteration_messages` in `sample()` function.

Changed `require()` calls for `progressr` and `cli` to suggested
`requireNamespace()` alternatives.
@josswright josswright marked this pull request as ready for review February 16, 2026 17:44
@josswright
Copy link
Author

Thanks for the help with the unit tests.

In the spirit of peer review, some notes and thoughts:

  • I'm not entirely happy with CmdStan's default iter_warmup and iter_sampling values being hard-coded into MCMCProcs default call rather than derived somehow. I imagine that the default values are fairly fixed, but I would have liked not to rely on that. Every route I could see to getting them was clunky and brought in unnecessary code.

  • As mentioned above, I toyed with the idea of setting refresh to 1 by default when showing the progress bar with iteration messages suppressed. In the end that seemed to introduce yet another case of a parameter depending on another parameter, as well as a potential source of fragility. I'll note, though, that refresh can safely be set to 1 for a nice, smooth progress bar without any noticeable overhead or slowdown that I saw in testing.

  • The code to derive the number of steps for the progress bar is more complicated than I originally imagined because of how CmdStan reports iterations.

    At one stage, when the number of steps was incorrectly calculated as too low, the progress bar was crashing, which also killed the sampling run. I've now ensured that the progress bar automatically closes when it reaches its finished number of steps, which should avoid crashes and just print a warning in the case of some output oddity causing more update steps to be signalled.

    I've tested as extensively as I can, but I live in fear of this killing someone's multi-day sampling run.

@jgabry
Copy link
Member

jgabry commented Feb 18, 2026

Awesome, thank you for the PR! I think I have the flu right now, but I will take a look when I'm feeling better and caught up on work.

@josswright
Copy link
Author

Awesome, thank you for the PR! I think I have the flu right now, but I will take a look when I'm feeling better and caught up on work.

No rush on my side! Hope you feel better soon.

@jgabry
Copy link
Member

jgabry commented Feb 26, 2026

I haven't forgotten about this, just dealing with a backlog of work. In the meantime, I posted on the Stan forum to hopefully get some more people to try it out and let us know how it goes:

https://discourse.mc-stan.org/t/please-help-test-a-new-progress-bar-for-sampling-in-cmdstanr/40946

I've tested as extensively as I can, but I live in fear of this killing someone's multi-day sampling run.

Should we maybe add something to the documentation about this? At least while the progress bar is still new we'll be gathering info about whether any users suffer this fate, so we could warn users that it's an experimental feature and to be cautious using it. Then eventually we can remove the experimental caveat from the doc. What do you think @josswright @VisruthSK?

@jgabry
Copy link
Member

jgabry commented Feb 27, 2026

The changes to model.R and utils.R look OK. I just pushed a few very minor changes to make not having the suggested packages an error instead of a warning (that's how we've been handling similar situations) and use the internal %||% function instead of ifelse to handle the NULL arguments (also chains does have a default of 4 in cmdstanr so we don't need to worry about that being NULL, just the iters).

I haven't yet looked at the procs code. I assume most of my review comments will be related to the procs code in run.R and about how we can add tests for all of this.

@josswright
Copy link
Author

I haven't forgotten about this, just dealing with a backlog of work. In the meantime, I posted on the Stan forum to hopefully get some more people to try it out and let us know how it goes:

https://discourse.mc-stan.org/t/please-help-test-a-new-progress-bar-for-sampling-in-cmdstanr/40946

I've tested as extensively as I can, but I live in fear of this killing someone's multi-day sampling run.

Should we maybe add something to the documentation about this? At least while the progress bar is still new we'll be gathering info about whether any users suffer this fate, so we could warn users that it's an experimental feature and to be cautious using it. Then eventually we can remove the experimental caveat from the doc. What do you think @josswright @VisruthSK?

I'd be happy with that. I do think I'm probably just being overly cautious, based on having prodded it quite hard to try and break it, but I don't see any harm in marking it as experimental until more people have had a chance to use it in anger.

(Apologies for slow replies--hectic part of term right now.)

Copy link
Member

@jgabry jgabry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks again for working on this! I just added a first round of review comments. And here are also some responses to some of your comments above:

In the spirit of peer review, some notes and thoughts:

  • I'm not entirely happy with CmdStan's default iter_warmup and iter_sampling values being hard-coded into MCMCProcs default call rather than derived somehow. I imagine that the default values are fairly fixed, but I would have liked not to rely on that. Every route I could see to getting them was clunky and brought in unnecessary code.

Yeah not ideal, but this is tricky. I'll think about it but we may be stuck with this.

  • As mentioned above, I toyed with the idea of setting refresh to 1 by default when showing the progress bar with iteration messages suppressed. In the end that seemed to introduce yet another case of a parameter depending on another parameter, as well as a potential source of fragility. I'll note, though, that refresh can safely be set to 1 for a nice, smooth progress bar without any noticeable overhead or slowdown that I saw in testing.

When I test refresh = 1 it's nice and smooth but it seems to add a lot of overhead. Here's what I tried along with system.time output:

system.time({
  cmdstanr_example(show_progress_bar = TRUE)
})
user  system elapsed 
3.235   0.982   4.106 

system.time({
  fit <- cmdstanr_example(show_progress_bar = TRUE, refresh = 1)
})
   user  system elapsed 
182.875  13.757 199.954 

Drastic difference! Are you not seeing that on your end? Maybe I'm doing something wrong. This is also all with the default progress bar. Maybe other ones have less overhead?

  • The code to derive the number of steps for the progress bar is more complicated than I originally imagined because of how CmdStan reports iterations.

I put a few questions/comments about this in the review comments.

I've tested as extensively as I can, but I live in fear of this killing someone's multi-day sampling run.

I haven't yet gotten around to thinking about what tests we should add to the unit tests. But we should definitely add some that will run automatically whenever our unit tests are run.


# Update progress bar if the iteration is a multiple of the
# refresh rate, or is the final sampling iteration.
if (((iter_current %% private$refresh_) == 0) ||
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If refresh=0 then I think iter_current %% 0 will return NaN.

More generally, what is the expected behavior of the progress bar if refresh=0? refresh = 0 is allowed and pretty common, but we could consider not allowing it when using the progress bar.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It feels like refresh=0 should be mutually exclusive with the progress bar. I'd probably want to throw an error for that case.

private$num_procs_
},
iter_warmup = function() {
privatea$iter_warmup_
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
privatea$iter_warmup_
private$iter_warmup_

Is this used anywhere? I guess not since that typo should have caused an error?

Comment on lines +1141 to +1144
iter_current <- as.numeric(gsub(".*Iteration:\\s*([0-9]+) \\/.*", "\\1", line, perl = TRUE))
if (iter_current > private$iter_warmup_) {
iter_current <- iter_current - private$iter_warmup_
}
Copy link
Member

@jgabry jgabry Feb 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question about the logic here (and below). Is my understanding here correct?

  1. We parse Iteration: X / ... from CmdStan output.
  2. If X > iter_warmup we subtract warmup, so iter_current becomes sampling-only (e.g., 1..iter_sampling)
  3. But then we check the final step (this happens in the code below, not these highlighted lines) with iter_current == iter_warmup + iter_sampling, so then we're now back on the total iterations scale.

So during sampling, can that final step condition ever actually be true? I'm not entirely confident in my analysis here, so correct me if I'm wrong, but here's a simple example (not very realistic, but just to demonstrate). Suppose:

  • iter_warmup = 100
  • iter_sampling = 70
  • Total iterations per chain = 170
  • refresh = 50

Then at the final line (Iteration: 170 / 170):

  • We would subtract warmup: iter_current = 170 - 100 = 70
  • Then check 70 == 170 (false)
  • 70 %% 50 != 0, so no progress update happens for the final chunk?

Comment on lines +1150 to +1153
progress_amount <- private$refresh_
}
}
private$progress_bar_(amount = progress_amount, message = line)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If progress_amount is refresh then updates always add refresh, but the final chunk could be smaller than refresh (a remainder), right? Could this cause any issues?

}
# Ensure at this point that any created progress bar is closed.
if (!is.null(private$progress_bar_)) {
private$progress_bar_(type = "finish")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be procs$progress_bar()(type = "finish")? Does private$progress_bar exists in the CmdStanRun object or just in CmdStanProcs?

#' display sampling progress via the `progressr` framework. The user is
#' responsible for registering a handler to display the progress bar. A
#' default handler, using the `cli` package, can be registered via
#' [register_default_progress_handler()]. The default is `FALSE`.
Copy link
Member

@jgabry jgabry Feb 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think in the doc we should:

  • advise that this is an experimental feature (for now)
  • mention that the progress bar combines all chains due to limitations of progressr (I'm guessing this will be a common request/question otherwise)

Copy link
Author

@josswright josswright Mar 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a common request in progressr as well!

One of the main blockers seems to be that RStudio's terminal only supports the necessary ANSI sequences to allow single-line progress bars, making it not worth the time to add multi-line bars as a feature.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One other thing for the doc: an example of configuring a non-default progress bar

Comment on lines +1073 to +1088
progressr::handlers(global = TRUE)
progressr::handlers("cli")

options(
cli.spinner = "moon",
cli.progress_show_after = 0,
cli.progress_clear = FALSE
)

# Default informative progress output for sampling
progressr::handlers(
progressr::handler_cli(
format = "{cli::pb_spin} Progress: |{cli::pb_bar}| {cli::pb_current}/{cli::pb_total} | {cli::pb_percent} | ETA: {cli::pb_eta}",
clear = FALSE
)
)
Copy link
Member

@jgabry jgabry Feb 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It just occurred to me that these settings will affect things people do outside of CmdStanR too, right? I would think it would be very rare that this would cause a problem for someone, but if we change global settings we should document it.

Typically I would approach something like this by restoring the options to their previous state before exiting the function, but that doesn't make sense here. The user would then need to call this function before every sampling run instead of just once. So documenting this seems sufficient. Maybe something like this (if what I wrote is actually correct):

 `register_default_progress_handler()` modifies global progress settings for the current R session. It installs a global progressr handler and sets cli progress options, which may change progress display behavior in other packages. 

Another option (in addition to the doc) would be to add a function the user can use to restore the settings.

@VisruthSK any thoughts on this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a succinct suppressable message + documentation would suffice; I do agree that doing something is necessary. I think a function to reset would be nice to have but not neccessary, unlike the doc/message.

save_metric = NULL,
save_cmdstan_config = NULL) {
save_cmdstan_config = NULL,
show_progress_bar = FALSE,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about a global option that could be used to turn this on for a whole R session? For example,

show_progress_bar = getOption("cmdstanr_progress_bar", FALSE) 

Copy link
Member

@VisruthSK VisruthSK Feb 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for a global option

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems sensible to me.

@avehtari
Copy link
Member

  1. Can you add this also for other methods? At least laplace and pathfinder take often long time and provide iteration output.
  2. Could it be possible to have global option, e.g. cmdstanr_progressbar with default value FALSE? Then there would not be need to change it for every call separately. This would be especially handy when working with notebooks, so that you like to see the progressbar when running code interactively, but not when rendering the notebook. For example, projpred has global option projpred.use_progressr
  3. Seems to work also with brms when using options(brms.backend = "cmdstanr")
  4. If register_default_progress_handler() is called when quarto rendering, there is an error. The function could use if (interactive()) {... } to register the handler only in interactive session.
�[31mError in `globalCallingHandlers()`:
! should not be called with handlers on the stack
Backtrace:
    ▆
 1. └─cmdstanr::register_default_progress_handler()
 2.   └─progressr::handlers(global = TRUE) at cmdstanr/R/utils.R:1073:3
 3.     └─progressr:::register_global_progression_handler(action = action)
 4.       └─base::globalCallingHandlers(condition = global_progression_handler)
  1. The output in the end looks sometimes silly with progressbar in the middle
Start sampling
Running MCMC with 4 chains, at most 2 in parallel...

Chain 1 finished in 0.1 seconds.
Chain 2 finished in 0.1 seconds.
Chain 3 finished in 0.1 seconds.
🌔  Progress: |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ | 8000/8000 | 100% | ETA:  0s
Chain 4 finished in 0.1 seconds.

@avehtari
Copy link
Member

Forgot to say that this is awesome!

@jgabry
Copy link
Member

jgabry commented Mar 2, 2026

  1. Can you add this also for other methods? At least laplace and pathfinder take often long time and provide iteration output.

I think this would be great, but we don't need to wait for that for this PR. It could be done separately and even potentially by someone else (using the same approach), depending on how much time @josswright has available to dedicate to this.

@josswright
Copy link
Author

  1. Can you add this also for other methods? At least laplace and pathfinder take often long time and provide iteration output.

I think this would be great, but we don't need to wait for that for this PR. It could be done separately and even potentially by someone else (using the same approach), depending on how much time @josswright has available to dedicate to this.

I'd be happy to have a look, although I wouldn't want to give any guarantees about how much time I'm likely to have in the near future.

In that case, though, it would almost certainly be worth abstracting some of the iteration extraction and calculation code, though.

@josswright
Copy link
Author

When I test refresh = 1 it's nice and smooth but it seems to add a lot of overhead. Here's what I tried along with system.time output:

system.time({
  cmdstanr_example(show_progress_bar = TRUE)
})
user  system elapsed 
3.235   0.982   4.106 

system.time({
  fit <- cmdstanr_example(show_progress_bar = TRUE, refresh = 1)
})
   user  system elapsed 
182.875  13.757 199.954 

Drastic difference! Are you not seeing that on your end? Maybe I'm doing something wrong. This is also all with the default progress bar. Maybe other ones have less overhead?

Ah, this was my fault in how I tested--I was running against a GP model, for which each iteration was relatively chunky in itself. I can confirm that using cmdstanr_example( show_progress_bar = TRUE ) does cause a hefty slowdown.

progressr does warn against sending updates too frequently here: https://cran.r-project.org/web/packages/progressr/vignettes/progressr-91-appendix.html

Given this, I wonder if it might be worth throwing a warning if refresh is set too low?

@jgabry
Copy link
Member

jgabry commented Mar 2, 2026

Given this, I wonder if it might be worth throwing a warning if refresh is set too low?

Since it could make sense to set refresh=1 for slower models (like your GP model), maybe just adding some documentation about this is enough instead of throwing a warning? We can just say that the overhead of setting refresh to a low number will be noticeable for fast models but refresh=1 can give you "a nice, smooth progress bar" for slower models, like you mentioned in some comments above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Progress bar

5 participants