Add Support for Migration Statistics API Call #43

phip1611 · 2025-11-25T17:07:56Z

TL;DR

This extends the internal API with a vm_progress function and adds a vm.migration-progress HTTP endpoint including support in ch-remote via ch-remote migration-process to query the latest migration progress.

Motivation

Monitoring a live-migration with live updated information is very important for debugging, development, and monitoring. This is something that verbose logging can't achieve as there is a clear desire to have that structured information somewhere on the outside - and also to prevent spammy logs.

The most interesting part is the pre-copy phase where we get information on each new memory iteration. The first version is rather coarse-grained with one update per memory iteration. More to follow.

The ch driver in libvirt will use these information to populate its
virsh domjobinfo information.

Further, the endpoint will be interesting to query information about a
previously failed or canceled live migration.

Prerequisites

The two major pre-requisites were:

asynchronize the internal API vmm: live migration asynchronization (dedicated thread) #24
asynchronize the HTTP API Make HTTP Server multi threaded #52

Steps Before Merge

test it locally using ch-remote
add libvirt-tests testcase and verify everything works
deploy it on a node in SAP land and see if it works
merge this when we know the libvirt part is also fine (I however think that this is 99% already)

vm-migration/src/progress.rs

src/bin/ch-remote.rs

vmm/src/api/mod.rs

vmm/src/lib.rs

phip1611 · 2026-01-13T07:40:53Z

vm-migration/src/progress.rs

+/// [live-migration protocol]: super::protocol
+#[derive(Clone, Debug, serde::Serialize, serde::Deserialize)]
+pub struct MigrationProgressAndStatus {
+    /// UNIX timestamp of the start of the live-migration process.


please note that this structure will be public API for ever and once we reploy it, it will be hard to change

vm-migration/src/progress.rs

olivereanderson

Thanks for your work on this!

I wish the PR description, or some other form of documentation, explained the intended use cases for this feature.

Is the main use case debugging slow/hanging live migrations, or is this intended to be used to collect metrics for every live migration which will then be used to provide statistical insights? If it is the latter, it would be good to document (somewhere) the compatibility story with standard metrics services like Prometheus. I think @snue kowns a thing or two about such topics 🙂

The metrics crate seems to be the metrics analogue of log/tracing and it would be good to know why that isn't a good fit for the task(s) addressed by this PR. I am assuming that there is some context I am missing and that it is quite possible that I have some misconceptions about the problems you are trying to address.

Otherwise I would just like to say that the code looks good and it was easy to follow the implementation for the most part.

olivereanderson · 2026-01-14T07:13:02Z

vm-migration/src/progress.rs

+    pub memory_bytes_remaining_iteration: u64,
+    /// The amount of transmitted 4k pages.
+    pub memory_pages_4k_transmitted: u64,
+    /// The amount of remaining 4k pages for this iteration.
+    pub memory_pages_4k_remaining_iteration: u64,


Question: What about larger pages?

VMM + KVM transparently splits huge-pages for a migration. I'm however not sure if this also applies to read-only pages. We should discuss this again with Thomas

From offline discussion. KVM splits all pages to 4k for the scope of the migration

vm-migration/src/progress.rs

olivereanderson · 2026-01-14T07:44:52Z

vmm/src/api/mod.rs

The commit message contains:

an vm_migration API call

but it should be "a vm_migration API call".

next time, please write the title of the commit message to the comment. I have no chance to find out which commit message this comment belongs to other than brute force :)

olivereanderson · 2026-01-14T07:47:48Z

vmm/src/api/mod.rs

+    /// The progress of a possibly ongoing live migration.
+    VmMigrationProgress(Box<Option<MigrationProgressAndStatus>>),


I guess the None variant will only ever be present if the API is called before a live migration has started?

I think about refactoring the whole thing to Result<MigrationProgress> and drop the Option

Hm I think Option makes the most sense as is the most consistent API without suddenly changing semantics

olivereanderson · 2026-01-14T07:58:00Z

vmm/src/api/http/http_endpoint.rs

+            {
+                Ok(info) => {
+                    let mut response = Response::new(Version::Http11, StatusCode::OK);
+                    let info_serialized = serde_json::to_string(&info).unwrap();


Nit: It would be good to either replace unwrap with expect or to add a comment explaining why it is OK to unwrap in this case.

I'm not happy with it but it is done like this everywhere in ch-remote. I'd much rather change this in upstream CHV

vmm/src/lib.rs

vm-migration/src/progress.rs

vmm/src/lib.rs

phip1611 · 2026-01-22T12:36:00Z

Thanks for your work on this!

I wish the PR description, or some other form of documentation, explained the intended use cases for this feature.

I've updated the code.

The metrics crate seems to be the metrics analogue of log/tracing and it would be good to know why that isn't a good fit for the task(s) addressed by this PR. I am assuming that there is some context I am missing and that it is quite possible that I have some misconceptions about the problems you are trying to address.

I don't see how this would help. These statistics are not "nice helpers that we need sometimes to debug". They are a fundamental and important functionality we want to export via API.

I hope the new PR description makes the intent here clearer.

amphi

Looks good, just a few small comments and nits.

vm-migration/src/progress.rs

amphi · 2026-01-22T14:27:13Z

vmm/src/lib.rs

+        // Give management software a chance to fetch the migration state.
+        // The VMM already executes on the other side and keeping Cloud Hypervisor running for a
+        // couple of more seconds is fine.
+        info!("Sleeping five seconds before shutting off.");


I think this waiting, and maybe also the mark_as_finished() a few lines above, should happen in check_migration_result. Otherwise the management software would see the mark_as_failed very late.

Agree about the sleep. Not sure about the other. Other thoughts?

Update. all of this need to happen in the end of the migration thread, not in check_migration_result. Otherwise, the API thread is blocked and API users can never query the latest state.

On-behalf-of: SAP philipp.schuster@sap.com Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>

This is the first commit in a series of commits to introduce a new API endpoint in Cloud Hypervisor to report progress and live-insights about an ongoing live migration. Having live and frequently refreshing statistics/metrics about an ongoing live migration is especially interesting for debugging and monitoring. For the first time, we will be able to see how live-migrations behave and create benchmarking infrastructure around it. The ch driver in libvirt will use these information to populate its `virsh domjobinfo` information. We will add a new API endpoint to query information. Further, the endpoint will be interesting to query information about a previously failed or canceled live migration. Specifically interesting about this API endpoint is that it will be the first endpoint that needs the "asynchronization" of the API: more than one API request in parallel. This needs support at least in the HTTP API and the internal API. The "SendMigration" call is long-running and active even if someone is querying the new endpoint. On-behalf-of: SAP philipp.schuster@sap.com Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>

This is part of the commit series to enable live updates about an ongoing live migration. See the first commit for an introduction. On-behalf-of: SAP philipp.schuster@sap.com Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>

This is part of the commit series to enable live updates about an ongoing live migration. See the first commit for an introduction. In this commit, we add the HTTP endpoint to export ongoing VM live-migration progress. This work was made possible because of the following fundamental prerequisites: - internal API was made async - http thread was made async This way, one can send requests to fetch the latest state without blocking in any code path of the API. On-behalf-of: SAP philipp.schuster@sap.com Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>

This is part of the commit series to enable live updates about an ongoing live migration. See the first commit for an introduction. This commit prepares the avoidance of naming clashes in the following. On-behalf-of: SAP philipp.schuster@sap.com Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>

This is part of the commit series to enable live updates about an ongoing live migration. See the first commit for an introduction. This commit actually brings all the functionality together. The first version has the limitation that we populate the latest snapshot once per memory iteration, although this is the most interesting part by far. In a follow-up, we can make this more fine-grained. On-behalf-of: SAP philipp.schuster@sap.com Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>

This is part of the commit series to enable live updates about an ongoing live-migration. See the first commit for an introduction. There isn't really an error that can happen when we query this endpoint. A previous snapshot may either be there or not. It also doesn't make sense here to check if the current VM is running, as users should always be able to query information about the past (failed or canceled) live migration. On-behalf-of: SAP philipp.schuster@sap.com Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>

This is part of the commit series to enable live updates about an ongoing live migration. See the first commit for an introduction. On-behalf-of: SAP philipp.schuster@sap.com Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>

The logging is not very spammy nor costly (iterations take seconds to dozens of minutes) and is clearly a win for us to debug things. On-behalf-of: SAP philipp.schuster@sap.com Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>

phip1611 self-assigned this Nov 25, 2025

phip1611 force-pushed the poc-migration-statistics branch from e29356d to 4101b11 Compare November 28, 2025 08:46

phip1611 commented Nov 28, 2025

View reviewed changes

vm-migration/src/progress.rs Show resolved Hide resolved

tpressure reviewed Nov 28, 2025

View reviewed changes

vm-migration/src/progress.rs Outdated Show resolved Hide resolved

phip1611 force-pushed the poc-migration-statistics branch 5 times, most recently from f49e577 to fdf5858 Compare December 9, 2025 11:42

phip1611 commented Dec 9, 2025

View reviewed changes

src/bin/ch-remote.rs Outdated Show resolved Hide resolved

phip1611 commented Dec 9, 2025

View reviewed changes

vmm/src/api/mod.rs Outdated Show resolved Hide resolved

phip1611 force-pushed the poc-migration-statistics branch from f612cf6 to e9a3321 Compare December 15, 2025 14:24

phip1611 force-pushed the poc-migration-statistics branch from e9a3321 to ecb5f45 Compare January 8, 2026 14:31

phip1611 changed the base branch from gardenlinux-v48 to gardenlinux January 8, 2026 14:32

phip1611 force-pushed the poc-migration-statistics branch 3 times, most recently from e6c80dd to fe8cd0d Compare January 12, 2026 16:30

phip1611 marked this pull request as ready for review January 12, 2026 16:30

phip1611 changed the title ~~WIP XXX Migration Statistics~~ Add Support for Migration Statistics API Call Jan 12, 2026

phip1611 requested review from amphi and olivereanderson January 12, 2026 16:33

phip1611 commented Jan 12, 2026

View reviewed changes

vmm/src/lib.rs Show resolved Hide resolved

phip1611 commented Jan 12, 2026

View reviewed changes

vmm/src/lib.rs Show resolved Hide resolved

phip1611 marked this pull request as draft January 12, 2026 16:35

phip1611 force-pushed the poc-migration-statistics branch from fe8cd0d to a87fe95 Compare January 12, 2026 16:41

phip1611 requested a review from tpressure January 12, 2026 16:42

phip1611 force-pushed the poc-migration-statistics branch from a87fe95 to 6d68c0c Compare January 12, 2026 16:43

phip1611 commented Jan 13, 2026

View reviewed changes

vm-migration/src/progress.rs Outdated Show resolved Hide resolved

phip1611 force-pushed the poc-migration-statistics branch from 6d68c0c to c934e0e Compare January 13, 2026 10:36

phip1611 force-pushed the poc-migration-statistics branch from c934e0e to 78699eb Compare January 13, 2026 10:42

olivereanderson reviewed Jan 14, 2026

View reviewed changes

vmm/src/lib.rs Outdated Show resolved Hide resolved

phip1611 force-pushed the poc-migration-statistics branch from a1b5313 to 708695d Compare January 22, 2026 12:27

phip1611 marked this pull request as ready for review January 22, 2026 12:27

phip1611 force-pushed the poc-migration-statistics branch from 708695d to 40e58e3 Compare January 22, 2026 12:31

phip1611 requested a review from olivereanderson January 22, 2026 12:31

phip1611 force-pushed the poc-migration-statistics branch 3 times, most recently from ceb915e to 5b63ea5 Compare January 22, 2026 14:27

amphi reviewed Jan 22, 2026

View reviewed changes

phip1611 force-pushed the poc-migration-statistics branch 3 times, most recently from a83f554 to 7ee8334 Compare January 27, 2026 14:50

phip1611 mentioned this pull request Jan 28, 2026

vmm: migration: streamline and cleanup memory_copy_iterations() #72

Merged

1 task

phip1611 force-pushed the poc-migration-statistics branch from 7ee8334 to f937d87 Compare January 29, 2026 12:10

phip1611 requested review from amphi and scholzp January 29, 2026 12:10

phip1611 force-pushed the poc-migration-statistics branch from f937d87 to 679c6f1 Compare January 29, 2026 14:08

phip1611 added 9 commits January 30, 2026 14:26

vmm: silence some messages in http_endpoint

d727284

On-behalf-of: SAP philipp.schuster@sap.com Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>

phip1611 force-pushed the poc-migration-statistics branch from 679c6f1 to 307475e Compare February 2, 2026 13:03

		/// The progress of a possibly ongoing live migration.
		VmMigrationProgress(Box<Option<MigrationProgressAndStatus>>),

Add Support for Migration Statistics API Call #43

Are you sure you want to change the base?

Add Support for Migration Statistics API Call #43

Uh oh!

Conversation

phip1611 commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TL;DR

Motivation

Prerequisites

Steps Before Merge

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

olivereanderson left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

phip1611 Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

phip1611 Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

olivereanderson Jan 14, 2026 • edited by phip1611 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

phip1611 commented Jan 22, 2026

Uh oh!

amphi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

phip1611 commented Nov 25, 2025 •

edited

Loading

olivereanderson left a comment •

edited

Loading

phip1611 Jan 14, 2026 •

edited

Loading

phip1611 Jan 29, 2026 •

edited

Loading

olivereanderson Jan 14, 2026 •

edited by phip1611

Loading