[PATCH QEMU HCI-8.0 v5 00/13] usb-host: support CPR migration

[PATCH QEMU HCI-8.0 v5 00/13] usb-host: support CPR migration

[Andrey Drobyshev]

v4 -> v5:

  * Patch 5: deliver URBs that already completed in the kernel instead
    of dropping their data, and error out any a drain can't reap in
    time;

  * Patch 6 (NEW): drain each usb-host before its host controller is
    migrated, so a shared-context completion can't advance xHCI after
    it was saved;

  * Patch 7: remember the packets to replay from the reap callback, so
    only the genuinely in-flight ones are re-issued.

v4: http://lore.virtuozzo.com/qemu/20260924-usb-host_v3-v4-0-1cc48bf630e7@virtuozzo.com

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
Andrey Drobyshev (13):
      usb-host: don't leak hostdev FD on open failure #VSTOR-137800
      usb-host: add migration blocker for CPR modes #VSTOR-137800
      usb-host: preserve hostdev FD during CPR migration #VSTOR-137800
      usb-host: factor out usb_host_reap_xfers() #VSTOR-137800
      usb-host: drain in-flight URBs across CPR #VSTOR-137800
      usb-host: drain before the host controller is migrated #VSTOR-137800
      usb-host: re-issue drained URBs if CPR is aborted #VSTOR-137800
      usb-host: skip product-string read on CPR incoming #VSTOR-137800
      usb-host: hand off the device across cpr-transfer #VSTOR-137800
      usb: migrate the interface altsetting #VSTOR-137800
      usb-host: reconfigure endpoints on CPR incoming #VSTOR-137800
      hcd-xhci: resync isochronous endpoints after CPR #VSTOR-137800
      usb-host: make CPR migration blocker conditional #VSTOR-137800

 hw/usb/bus.c                |  36 ++++
 hw/usb/hcd-xhci.c           |  20 ++
 hw/usb/host-libusb.c        | 512 ++++++++++++++++++++++++++++++++++++++++----
 include/migration/vmstate.h |   1 +
 4 files changed, 530 insertions(+), 39 deletions(-)
---
base-commit: f2f4b3de1a1be3360a77ca6ab79762466d238804
change-id: 20260914-usb-host_v3-bbfea86d1dad

Best regards,
--  
Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>

[PATCH QEMU HCI-8.0 v5 01/13] usb-host: don't leak hostdev FD on open failure #VSTOR-137800

[Andrey Drobyshev]

The FD passed into usb_host_open() for a wrapped device is only stored in
s->hostfd after libusb_wrap_sys_device() succeeds, and the failure path
doesn't close it.  if wrapping fails, or a later step of the open sequence
fails (e.g. usb_device_attach), the FD is leaked, as usb_host_close() only
runs for fully opened devices, and the realize error path doesn't clean it
up either.

Store the FD in s->hostfd before wrapping and close it on the failure
path.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/usb/host-libusb.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/hw/usb/host-libusb.c b/hw/usb/host-libusb.c
index b74670ae256..83645fbe0ce 100644
--- a/hw/usb/host-libusb.c
+++ b/hw/usb/host-libusb.c
@@ -954,6 +954,11 @@ static int usb_host_open(USBHostDevice *s, libusb_device *dev, int hostfd)
     int rc;
     Error *local_err = NULL;
 
+    /* Make sure failure path frees caller-owned FD */
+    if (!dev) {
+        s->hostfd = hostfd;
+    }
+
     if (s->bh_postld_pending) {
         return -1;
     }
@@ -978,7 +983,6 @@ static int usb_host_open(USBHostDevice *s, libusb_device *dev, int hostfd)
         if (rc != 0) {
             goto fail;
         }
-        s->hostfd  = hostfd;
         dev = libusb_get_device(s->dh);
         bus_num = libusb_get_bus_number(dev);
         addr = libusb_get_device_address(dev);
@@ -1066,6 +1070,10 @@ fail:
         s->dh = NULL;
         s->dev = NULL;
     }
+    if (s->hostfd != -1) {
+        close(s->hostfd);
+        s->hostfd = -1;
+    }
     return -1;
 }
 

-- 
2.47.1

[PATCH QEMU HCI-8.0 v5 02/13] usb-host: add migration blocker for CPR modes #VSTOR-137800

[Andrey Drobyshev]

We're about to add support for CPR migration to usb-host in the upcoming
commits.  Right now CPR is crashing with usb-host devices.  Let's add
migration blocker upfront, for both cpr-exec and cpr-transfer.  The
blocker is added unconditionally, so that we resuse CPR migration instead
of crashing on it.  That is for the sake of bisectability of the upcoming
commits.  It is going to be lifted (conditioned) in a following patch
once CPR support is added.  Also, blocker is added at the end of
usb_host_realize(), so we have to cleanup all the side effects done
earlier in .realize() in case migrate_add_blocker_modes() fails.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/usb/host-libusb.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/hw/usb/host-libusb.c b/hw/usb/host-libusb.c
index 83645fbe0ce..5c3da00d8eb 100644
--- a/hw/usb/host-libusb.c
+++ b/hw/usb/host-libusb.c
@@ -46,6 +46,7 @@
 #endif
 
 #include "qapi/error.h"
+#include "migration/blocker.h"
 #include "migration/vmstate.h"
 #include "monitor/monitor.h"
 #include "qemu/error-report.h"
@@ -103,6 +104,7 @@ struct USBHostDevice {
     char                             port[16];
 
     int                              hostfd;
+    Error                            *cpr_blocker;
     libusb_device                    *dev;
     libusb_device_handle             *dh;
     struct libusb_device_descriptor  ddesc;
@@ -1258,6 +1260,18 @@ static void usb_host_realize(USBDevice *udev, Error **errp)
 
     s->exit.notify = usb_host_exit_notifier;
     qemu_add_exit_notifier(&s->exit);
+
+    error_setg(&s->cpr_blocker, "usb-host device %s does not support CPR",
+               DEVICE(s)->id ?: "(anonymous)");
+    if (migrate_add_blocker_modes(&s->cpr_blocker, errp,
+                                  MIG_MODE_CPR_TRANSFER,
+                                  MIG_MODE_CPR_EXEC, -1) < 0) {
+        qemu_remove_exit_notifier(&s->exit);
+        if (s->needs_autoscan) {
+            QTAILQ_REMOVE(&hostdevs, s, next);
+        }
+        usb_host_close(s);
+    }
 }
 
 static void usb_host_instance_init(Object *obj)
@@ -1279,6 +1293,7 @@ static void usb_host_unrealize(USBDevice *udev)
         QTAILQ_REMOVE(&hostdevs, s, next);
     }
     usb_host_close(s);
+    migrate_del_blocker(&s->cpr_blocker);
 }
 
 static void usb_host_cancel_packet(USBDevice *udev, USBPacket *p)

-- 
2.47.1

[PATCH QEMU HCI-8.0 v5 03/13] usb-host: preserve hostdev FD during CPR migration #VSTOR-137800

[Andrey Drobyshev]

In .realize(), save freshly opened host device FD into the CPR registry,
and then reuse it on CPR target.  Also, add deletion of that FD from
registry to usb_host_open() / usb_host_close() cleanup paths.

For this to work, we also need to skip the .post-load() code which
closes, detaches the device, and then rescans the host bus to reopen
matching hostdevs.  For the CPR-case migration we don't want any of that
as the hostdev FD stays preserved, and the guest shouldn't notice the
switchover.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/usb/host-libusb.c | 33 ++++++++++++++++++++++++++++++++-
 1 file changed, 32 insertions(+), 1 deletion(-)

diff --git a/hw/usb/host-libusb.c b/hw/usb/host-libusb.c
index 5c3da00d8eb..b946b0ef350 100644
--- a/hw/usb/host-libusb.c
+++ b/hw/usb/host-libusb.c
@@ -47,6 +47,7 @@
 
 #include "qapi/error.h"
 #include "migration/blocker.h"
+#include "migration/cpr.h"
 #include "migration/vmstate.h"
 #include "monitor/monitor.h"
 #include "qemu/error-report.h"
@@ -947,6 +948,21 @@ static void usb_host_ep_update(USBHostDevice *s)
     libusb_free_config_descriptor(conf);
 }
 
+static char *usb_host_cpr_fd_name(USBHostDevice *s)
+{
+    /* Prefix the key so it can't clash with another CPR label */
+    return DEVICE(s)->id ? g_strdup_printf("usb-host/%s", DEVICE(s)->id)
+                         : NULL;
+}
+
+static void usb_host_cpr_delete_fd(USBHostDevice *s)
+{
+    g_autofree char *name = usb_host_cpr_fd_name(s);
+    if (name) {
+        cpr_delete_fd(name, 0);
+    }
+}
+
 static int usb_host_open(USBHostDevice *s, libusb_device *dev, int hostfd)
 {
     USBDevice *udev = USB_DEVICE(s);
@@ -1073,6 +1089,7 @@ fail:
         s->dev = NULL;
     }
     if (s->hostfd != -1) {
+        usb_host_cpr_delete_fd(s);
         close(s->hostfd);
         s->hostfd = -1;
     }
@@ -1134,6 +1151,7 @@ static int usb_host_close(USBHostDevice *s)
     s->dev = NULL;
 
     if (s->hostfd != -1) {
+        usb_host_cpr_delete_fd(s);
         close(s->hostfd);
         s->hostfd = -1;
     }
@@ -1220,9 +1238,12 @@ static void usb_host_realize(USBDevice *udev, Error **errp)
 
 #if LIBUSB_API_VERSION >= 0x01000107 && !defined(CONFIG_WIN32)
     if (s->hostdevice) {
+        g_autofree char *name = usb_host_cpr_fd_name(s);
         int fd;
+
         s->needs_autoscan = false;
-        fd = qemu_open(s->hostdevice, O_RDWR, errp);
+        fd = name ? cpr_open_fd(s->hostdevice, O_RDWR, name, 0, errp)
+                  : qemu_open(s->hostdevice, O_RDWR, errp);
         if (fd < 0) {
             return;
         }
@@ -1761,6 +1782,16 @@ static int usb_host_post_load(void *opaque, int version_id)
 {
     USBHostDevice *dev = opaque;
 
+    /*
+     * For CPR migration, device wasn't released/reset, and the guest
+     * is unaware of the switchover.  The detach/rescan performed in
+     * usb_host_post_load_bh() only exists for cross-host migration.
+     * Skip it for CPR.
+     */
+    if (cpr_is_incoming()) {
+        return 0;
+    }
+
     if (!dev->bh_postld) {
         dev->bh_postld = qemu_bh_new_guarded(usb_host_post_load_bh, dev,
                                              &DEVICE(dev)->mem_reentrancy_guard);

-- 
2.47.1

[PATCH QEMU HCI-8.0 v5 04/13] usb-host: factor out usb_host_reap_xfers() #VSTOR-137800

[Andrey Drobyshev]

usb_host_abort_xfers() cancels every pending request and then pumps
libusb events until all of them are reaped, with a bounded wait.  Split
that reap-wait loop out into usb_host_reap_xfers().  No functional
change.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/usb/host-libusb.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/hw/usb/host-libusb.c b/hw/usb/host-libusb.c
index b946b0ef350..9b70d6deef9 100644
--- a/hw/usb/host-libusb.c
+++ b/hw/usb/host-libusb.c
@@ -1096,15 +1096,11 @@ fail:
     return -1;
 }
 
-static void usb_host_abort_xfers(USBHostDevice *s)
+static void usb_host_reap_xfers(USBHostDevice *s)
 {
     USBHostRequest *r, *rtmp;
     int limit = 100;
 
-    QTAILQ_FOREACH_SAFE(r, &s->requests, next, rtmp) {
-        usb_host_req_abort(r);
-    }
-
     while (QTAILQ_FIRST(&s->requests) != NULL) {
         struct timeval tv;
         memset(&tv, 0, sizeof(tv));
@@ -1126,6 +1122,16 @@ static void usb_host_abort_xfers(USBHostDevice *s)
     }
 }
 
+static void usb_host_abort_xfers(USBHostDevice *s)
+{
+    USBHostRequest *r, *rtmp;
+
+    QTAILQ_FOREACH_SAFE(r, &s->requests, next, rtmp) {
+        usb_host_req_abort(r);
+    }
+    usb_host_reap_xfers(s);
+}
+
 static int usb_host_close(USBHostDevice *s)
 {
     USBDevice *udev = USB_DEVICE(s);

-- 
2.47.1

[PATCH QEMU HCI-8.0 v5 05/13] usb-host: drain in-flight URBs across CPR #VSTOR-137800

[Andrey Drobyshev]

URBs submitted by the old QEMU must not outlive it across the
switchover: the kernel would complete them into buffers of an address
space which is gone after exec, and the new libusb context would reap
URB pointers it never submitted.

So, in .pre_save() for the CPR modes, cancel and reap every pending
transfer.  One that already completed in the kernel is delivered
normally - its data is real and the guest must see it.  One still in
flight is suppressed: its packet stays ASYNC and its TD stays on the
transfer ring, so the target re-executes it from the start of the TD
when usb_xhci_post_load() kicks the endpoints.  Completing a transfer
can make xHCI queue the next TD, so keep cancelling until the request
list stays empty.

If the device won't release its URBs within the bounded wait, error
the still in-flight packets out -> the resumed guest retries them --
leak the requests, since libusb still owns their transfers, and abort
the migration.

Isochronous URBs are not on the request list but on their own rings,
and freeing them while in-flight would leave them queued on the
preserved FD.  Cancel them so they are unlinked and reaped: an iso URB
in flight when the guest stops may never complete on its own.  The
target reconstructs the iso stream from the guest's transfer ring.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/usb/host-libusb.c | 158 +++++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 135 insertions(+), 23 deletions(-)

diff --git a/hw/usb/host-libusb.c b/hw/usb/host-libusb.c
index 9b70d6deef9..ebad04ec305 100644
--- a/hw/usb/host-libusb.c
+++ b/hw/usb/host-libusb.c
@@ -48,6 +48,7 @@
 #include "qapi/error.h"
 #include "migration/blocker.h"
 #include "migration/cpr.h"
+#include "migration/misc.h"
 #include "migration/vmstate.h"
 #include "monitor/monitor.h"
 #include "qemu/error-report.h"
@@ -403,7 +404,10 @@ static void usb_host_req_free(USBHostRequest *r)
 #ifdef CONFIG_WIN32
     request_count--;
 #endif
-    QTAILQ_REMOVE(&r->host->requests, r, next);
+    /* Skip if a timed-out reap already unlinked it (avoid a double remove) */
+    if (QTAILQ_IN_USE(r, next)) {
+        QTAILQ_REMOVE(&r->host->requests, r, next);
+    }
     libusb_free_transfer(r->xfer);
     g_free(r->buffer);
     g_free(r);
@@ -430,6 +434,9 @@ static void LIBUSB_CALL usb_host_req_complete_ctrl(struct libusb_transfer *xfer)
     if (r->p == NULL) {
         goto out; /* request was canceled */
     }
+    if (xfer->status == LIBUSB_TRANSFER_CANCELLED) {
+        goto out; /* CPR drain: the target will replay this packet */
+    }
 
     r->p->status = status_map[xfer->status];
     r->p->actual_length = xfer->actual_length;
@@ -480,6 +487,9 @@ static void LIBUSB_CALL usb_host_req_complete_data(struct libusb_transfer *xfer)
     if (r->p == NULL) {
         goto out; /* request was canceled */
     }
+    if (xfer->status == LIBUSB_TRANSFER_CANCELLED) {
+        goto out; /* CPR drain: the target will replay this packet */
+    }
 
     r->p->status = status_map[xfer->status];
     if (r->in && xfer->actual_length) {
@@ -500,22 +510,29 @@ out:
     }
 }
 
-static void usb_host_req_abort(USBHostRequest *r)
+/* Fail r's in-flight packet with an error status and detach it from r. */
+static void usb_host_req_complete_err(USBHostRequest *r, int status)
 {
-    USBHostDevice  *s = r->host;
-    bool inflight = (r->p && r->p->state == USB_PACKET_ASYNC);
+    USBHostDevice *s = r->host;
 
-    if (inflight) {
-        r->p->status = USB_RET_NODEV;
-        trace_usb_host_req_complete(s->bus_num, s->addr, r->p,
-                                    r->p->status, r->p->actual_length);
-        if (r->p->ep->nr == 0) {
-            usb_generic_async_ctrl_complete(USB_DEVICE(s), r->p);
-        } else {
-            usb_packet_complete(USB_DEVICE(s), r->p);
-        }
-        r->p = NULL;
+    if (!r->p || r->p->state != USB_PACKET_ASYNC) {
+        return;
+    }
+    r->p->status = status;
+    trace_usb_host_req_complete(s->bus_num, s->addr, r->p,
+                                r->p->status, r->p->actual_length);
+    if (r->p->ep->nr == 0) {
+        usb_generic_async_ctrl_complete(USB_DEVICE(s), r->p);
+    } else {
+        usb_packet_complete(USB_DEVICE(s), r->p);
+    }
+    r->p = NULL;
+}
 
+static void usb_host_req_abort(USBHostRequest *r)
+{
+    if (r->p && r->p->state == USB_PACKET_ASYNC) {
+        usb_host_req_complete_err(r, USB_RET_NODEV);
         libusb_cancel_transfer(r->xfer);
     }
 }
@@ -1096,9 +1113,18 @@ fail:
     return -1;
 }
 
-static void usb_host_reap_xfers(USBHostDevice *s)
+/* libusb still owns cancelled-but-unreaped transfers: unlink, don't free. */
+static void usb_host_drop_requests(USBHostDevice *s)
 {
     USBHostRequest *r, *rtmp;
+
+    QTAILQ_FOREACH_SAFE(r, &s->requests, next, rtmp) {
+        QTAILQ_REMOVE(&s->requests, r, next);
+    }
+}
+
+static void usb_host_reap_xfers(USBHostDevice *s)
+{
     int limit = 100;
 
     while (QTAILQ_FIRST(&s->requests) != NULL) {
@@ -1108,15 +1134,10 @@ static void usb_host_reap_xfers(USBHostDevice *s)
         libusb_handle_events_timeout(ctx, &tv);
         if (--limit == 0) {
             /*
-             * Don't wait forever for libusb calling the complete
-             * callback (which will unlink and free the request).
-             *
-             * Leaking memory here, to make sure libusb will not
-             * access memory which we have released already.
+             * Don't wait forever for libusb to call the complete callback
+             * (which unlinks and frees the request).
              */
-            QTAILQ_FOREACH_SAFE(r, &s->requests, next, rtmp) {
-                QTAILQ_REMOVE(&s->requests, r, next);
-            }
+            usb_host_drop_requests(s);
             return;
         }
     }
@@ -1132,6 +1153,81 @@ static void usb_host_abort_xfers(USBHostDevice *s)
     usb_host_reap_xfers(s);
 }
 
+static bool usb_host_iso_inflight(USBHostDevice *s)
+{
+    USBHostIsoRing *ring;
+
+    QTAILQ_FOREACH(ring, &s->isorings, next) {
+        if (!QTAILQ_EMPTY(&ring->inflight)) {
+            return true;
+        }
+    }
+    return false;
+}
+
+/*
+ * Quiesce the device for a CPR switchover.  Cancel and reap all URBs,
+ * so that no URB submitted by this process can outlive it.
+ */
+static int usb_host_cpr_drain_xfers(USBHostDevice *s)
+{
+    USBHostRequest *r, *rtmp;
+    USBHostIsoRing *ring;
+    USBHostIsoXfer *xfer;
+    int limit = 100;
+
+    /*
+     * Cancel and reap every URB.  Completing a reaped URB can make xHCI queue
+     * the next TD, so re-cancel whatever reappears until the queue stays empty.
+     */
+    while (!QTAILQ_EMPTY(&s->requests)) {
+        struct timeval tv = { .tv_usec = 2500 };
+
+        QTAILQ_FOREACH_SAFE(r, &s->requests, next, rtmp) {
+            libusb_cancel_transfer(r->xfer);
+        }
+        libusb_handle_events_timeout(ctx, &tv);
+        if (--limit == 0) {
+            /*
+             * The device won't release these URBs.  Error the packets out so
+             * the resumed guest retries them, leak the requests (libusb still
+             * owns their transfers) and abort the migration.
+             */
+            QTAILQ_FOREACH_SAFE(r, &s->requests, next, rtmp) {
+                usb_host_req_complete_err(r, USB_RET_IOERROR);
+            }
+            usb_host_drop_requests(s);
+            return -1;
+        }
+    }
+
+    /*
+     * Iso URBs are on the rings, not s->requests.  Cancel them: one in
+     * flight when the guest stops may never complete on its own, and none
+     * may be left on the preserved fd.  The target replays from the ring.
+     */
+    QTAILQ_FOREACH(ring, &s->isorings, next) {
+        QTAILQ_FOREACH(xfer, &ring->inflight, next) {
+            libusb_cancel_transfer(xfer->xfer);
+        }
+    }
+
+    /*
+     * Cap the reap at 2x a full ring:
+     * iso_urb_count URBs x iso_urb_frames packets each
+     */
+    limit = 2 * s->iso_urb_count * s->iso_urb_frames;
+    while (usb_host_iso_inflight(s)) {
+        struct timeval tv = { .tv_usec = 1000 };
+        libusb_handle_events_timeout(ctx, &tv);
+        if (limit-- == 0) {
+            return -1;
+        }
+    }
+    usb_host_iso_free_all(s);
+    return 0;
+}
+
 static int usb_host_close(USBHostDevice *s)
 {
     USBDevice *udev = USB_DEVICE(s);
@@ -1807,10 +1903,26 @@ static int usb_host_post_load(void *opaque, int version_id)
     return 0;
 }
 
+static int usb_host_pre_save(void *opaque)
+{
+    USBHostDevice *s = opaque;
+    MigMode mode = migrate_mode();
+
+    /* URBs submitted by this process must not outlive CPR migration */
+    if ((mode == MIG_MODE_CPR_EXEC || mode == MIG_MODE_CPR_TRANSFER) &&
+        s->dh) {
+        if (usb_host_cpr_drain_xfers(s) < 0) {
+            return -1;
+        }
+    }
+    return 0;
+}
+
 static const VMStateDescription vmstate_usb_host = {
     .name = "usb-host",
     .version_id = 1,
     .minimum_version_id = 1,
+    .pre_save = usb_host_pre_save,
     .post_load = usb_host_post_load,
     .fields = (const VMStateField[]) {
         VMSTATE_USB_DEVICE(parent_obj, USBHostDevice),

-- 
2.47.1

[PATCH QEMU HCI-8.0 v5 06/13] usb-host: drain before the host controller is migrated #VSTOR-137800

[Andrey Drobyshev]

.pre_save() drains the device's URBs, but at the default section
priority the xHCI controller is saved before any usb-host device -> its
state is already on the wire when the drain runs.

All usb-host devices share one libusb context, so draining one reaps
another's completed transfers too.  Each completion pokes xHCI (writes
an event, queues the next TD) after xHCI has been saved, and the target
then resumes from a stale controller: OUT data sent twice, IN data lost.

Raise usb-host above MIG_PRI_DEFAULT so every device drains before its
host controller is saved.  Sections are matched by name, so the
reordered stream stays compatible.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/usb/host-libusb.c        | 1 +
 include/migration/vmstate.h | 1 +
 2 files changed, 2 insertions(+)

diff --git a/hw/usb/host-libusb.c b/hw/usb/host-libusb.c
index ebad04ec305..9e9ee7d305f 100644
--- a/hw/usb/host-libusb.c
+++ b/hw/usb/host-libusb.c
@@ -1922,6 +1922,7 @@ static const VMStateDescription vmstate_usb_host = {
     .name = "usb-host",
     .version_id = 1,
     .minimum_version_id = 1,
+    .priority = MIG_PRI_USB_HOST,
     .pre_save = usb_host_pre_save,
     .post_load = usb_host_post_load,
     .fields = (const VMStateField[]) {
diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
index 6839117be87..3358d7fedc8 100644
--- a/include/migration/vmstate.h
+++ b/include/migration/vmstate.h
@@ -160,6 +160,7 @@ typedef enum {
 
     MIG_PRI_LOW,                /* Must happen after default */
     MIG_PRI_DEFAULT,
+    MIG_PRI_USB_HOST,           /* Must be saved before the host controller */
     MIG_PRI_IOMMU,              /* Must happen before PCI devices */
     MIG_PRI_PCI_BUS,            /* Must happen before IOMMU */
     MIG_PRI_VIRTIO_MEM,         /* Must happen before IOMMU */

-- 
2.47.1

[PATCH QEMU HCI-8.0 v5 07/13] usb-host: re-issue drained URBs if CPR is aborted #VSTOR-137800

[Andrey Drobyshev]

The previous commit drains the in-flight URBs in .pre_save() and leaves
their packets ASYNC for the target to replay.  If migration fails, the
source resumes instead, with those packets still owned by the
controller and nothing left to complete them.

Remember the drained packets in a list, and re-issue them on the
preserved FD from a MIG_EVENT_PRECOPY_FAILED notifier.  Control
transfers can't be reconstructed from the packet alone, so complete
those as errors and let the guest driver retry.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/usb/host-libusb.c | 96 +++++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 92 insertions(+), 4 deletions(-)

diff --git a/hw/usb/host-libusb.c b/hw/usb/host-libusb.c
index 9e9ee7d305f..a14a3748838 100644
--- a/hw/usb/host-libusb.c
+++ b/hw/usb/host-libusb.c
@@ -107,6 +107,9 @@ struct USBHostDevice {
 
     int                              hostfd;
     Error                            *cpr_blocker;
+    NotifierWithReturn               cpr_notifier;
+    GSList                           *cpr_inflight;
+    bool                             cpr_drained;
     libusb_device                    *dev;
     libusb_device_handle             *dh;
     struct libusb_device_descriptor  ddesc;
@@ -425,6 +428,23 @@ static USBHostRequest *usb_host_req_find(USBHostDevice *s, USBPacket *p)
     return NULL;
 }
 
+/*
+ * A CANCELLED reap with r->p still set -> the CPR drain cancelled it (every
+ * other path clears r->p first).  Don't complete it, let the target replay it.
+ * Returns true if the caller should just free the request.
+ */
+static bool usb_host_req_cpr_replay(USBHostRequest *r,
+                                    struct libusb_transfer *xfer)
+{
+    if (xfer->status != LIBUSB_TRANSFER_CANCELLED) {
+        return false;
+    }
+    if (r->p->state == USB_PACKET_ASYNC) {
+        r->host->cpr_inflight = g_slist_append(r->host->cpr_inflight, r->p);
+    }
+    return true;
+}
+
 static void LIBUSB_CALL usb_host_req_complete_ctrl(struct libusb_transfer *xfer)
 {
     USBHostRequest *r = xfer->user_data;
@@ -434,7 +454,7 @@ static void LIBUSB_CALL usb_host_req_complete_ctrl(struct libusb_transfer *xfer)
     if (r->p == NULL) {
         goto out; /* request was canceled */
     }
-    if (xfer->status == LIBUSB_TRANSFER_CANCELLED) {
+    if (usb_host_req_cpr_replay(r, xfer)) {
         goto out; /* CPR drain: the target will replay this packet */
     }
 
@@ -487,7 +507,7 @@ static void LIBUSB_CALL usb_host_req_complete_data(struct libusb_transfer *xfer)
     if (r->p == NULL) {
         goto out; /* request was canceled */
     }
-    if (xfer->status == LIBUSB_TRANSFER_CANCELLED) {
+    if (usb_host_req_cpr_replay(r, xfer)) {
         goto out; /* CPR drain: the target will replay this packet */
     }
 
@@ -1166,8 +1186,13 @@ static bool usb_host_iso_inflight(USBHostDevice *s)
 }
 
 /*
- * Quiesce the device for a CPR switchover.  Cancel and reap all URBs,
- * so that no URB submitted by this process can outlive it.
+ * Quiesce the device for a CPR switchover.  Cancel and reap all URBs
+ * without completing their packets, so that no URB submitted by this
+ * process can outlive it.  CPR target re-executes them in
+ * usb_xhci_post_load() kicking the running endpoints.
+ *
+ * Keep the list of packets so that they can be re-issued on this side
+ * instead if MIG_EVENT_PRECOPY_FAILED fires.
  */
 static int usb_host_cpr_drain_xfers(USBHostDevice *s)
 {
@@ -1176,6 +1201,8 @@ static int usb_host_cpr_drain_xfers(USBHostDevice *s)
     USBHostIsoXfer *xfer;
     int limit = 100;
 
+    s->cpr_drained = true;
+
     /*
      * Cancel and reap every URB.  Completing a reaped URB can make xHCI queue
      * the next TD, so re-cancel whatever reappears until the queue stays empty.
@@ -1308,6 +1335,53 @@ static libusb_device *usb_host_find_ref(int bus, int addr)
     return ret;
 }
 
+static void usb_host_handle_data(USBDevice *udev, USBPacket *p);
+
+/*
+ * A failed CPR migration resumes the source VM with the drained
+ * packets still owned by the host controller as in-flight.  Re-issue
+ * them on the preserved fd.  Control transfers cannot be reconstructed
+ * from the packet alone; complete them as errors and let the guest
+ * driver retry.
+ */
+static int usb_host_cpr_notifier(NotifierWithReturn *notifier,
+                                 MigrationEvent *e, Error **errp)
+{
+    USBHostDevice *s = container_of(notifier, USBHostDevice, cpr_notifier);
+    USBDevice *udev = USB_DEVICE(s);
+    GSList *it;
+    USBPacket *p;
+
+    if (e->type != MIG_EVENT_PRECOPY_FAILED || !s->cpr_drained) {
+        return 0;
+    }
+
+    for (it = s->cpr_inflight; it; it = it->next) {
+        p = it->data;
+        /* Replay from the start of the TD */
+        p->actual_length = 0;
+        if (p->ep->nr == 0) {
+            /*
+             * p->ep[0] is control endpoint.  Control transfers can't
+             * be reconstructed reliably, so complete them as errors
+             * and let the guest retry.
+             */
+            p->status = USB_RET_IOERROR;
+            usb_generic_async_ctrl_complete(udev, p);
+        } else {
+            usb_host_handle_data(udev, p);
+            if (p->status != USB_RET_ASYNC) {
+                /* Completed synchronously with error */
+                usb_packet_complete(udev, p);
+            }
+        }
+    }
+    g_slist_free(s->cpr_inflight);
+    s->cpr_inflight = NULL;
+    s->cpr_drained = false;
+    return 0;
+}
+
 static void usb_host_realize(USBDevice *udev, Error **errp)
 {
     USBHostDevice *s = USB_HOST_DEVICE(udev);
@@ -1384,6 +1458,14 @@ static void usb_host_realize(USBDevice *udev, Error **errp)
     s->exit.notify = usb_host_exit_notifier;
     qemu_add_exit_notifier(&s->exit);
 
+#if LIBUSB_API_VERSION >= 0x01000107 && !defined(CONFIG_WIN32)
+    if (s->hostdevice && DEVICE(s)->id) {
+        migration_add_notifier_modes(&s->cpr_notifier, usb_host_cpr_notifier,
+                                     MIG_MODE_CPR_TRANSFER,
+                                     MIG_MODE_CPR_EXEC, -1);
+    }
+#endif
+
     error_setg(&s->cpr_blocker, "usb-host device %s does not support CPR",
                DEVICE(s)->id ?: "(anonymous)");
     if (migrate_add_blocker_modes(&s->cpr_blocker, errp,
@@ -1417,6 +1499,9 @@ static void usb_host_unrealize(USBDevice *udev)
     }
     usb_host_close(s);
     migrate_del_blocker(&s->cpr_blocker);
+    migration_remove_notifier(&s->cpr_notifier);
+    g_slist_free(s->cpr_inflight);
+    s->cpr_inflight = NULL;
 }
 
 static void usb_host_cancel_packet(USBDevice *udev, USBPacket *p)
@@ -1424,6 +1509,9 @@ static void usb_host_cancel_packet(USBDevice *udev, USBPacket *p)
     USBHostDevice *s = USB_HOST_DEVICE(udev);
     USBHostRequest *r;
 
+    /* Packet is being cancelled -> it mustn't be re-issued on failed CPR */
+    s->cpr_inflight = g_slist_remove(s->cpr_inflight, p);
+
     if (p->combined) {
         usb_combined_packet_cancel(udev, p);
         return;

-- 
2.47.1

[PATCH QEMU HCI-8.0 v5 08/13] usb-host: skip product-string read on CPR incoming #VSTOR-137800

[Andrey Drobyshev]

usb_host_open() reads the product string descriptor from the device
with a synchronous control transfer.  On the CPR target this runs while
the device is being handed over, and the transfer can block there,
hanging the incoming migration.

The string only fills the cosmetic product_desc, so skip the read when
we're the CPR target and fall back to the synthetic "host:bus.addr"
name.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/usb/host-libusb.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/hw/usb/host-libusb.c b/hw/usb/host-libusb.c
index a14a3748838..6b8c0cf0a82 100644
--- a/hw/usb/host-libusb.c
+++ b/hw/usb/host-libusb.c
@@ -1097,7 +1097,7 @@ static int usb_host_open(USBHostDevice *s, libusb_device *dev, int hostfd)
     udev->speed = speed_map[libusb_speed];
     usb_host_speed_compat(s);
 
-    if (s->ddesc.iProduct) {
+    if (s->ddesc.iProduct && !cpr_is_incoming()) {
         libusb_get_string_descriptor_ascii(s->dh, s->ddesc.iProduct,
                                            (unsigned char *)udev->product_desc,
                                            sizeof(udev->product_desc));

-- 
2.47.1

[PATCH QEMU HCI-8.0 v5 09/13] usb-host: hand off the device across cpr-transfer #VSTOR-137800

[Andrey Drobyshev]

Unlike cpr-exec, cpr-transfer keeps the source QEMU alive: it hands the
usbfs FD to the new QEMU over the migration channel and only exits once
the target is up.  While the handoff is in flight both processes hold the
same FD, and only one may reap URB events on it - if both do, one reaps
the other's in-flight completions and wedges an isochronous transfer that
cannot be retried.  So reaping is handed off along with the FD:

  * The target doesn't reap until it takes over: it stops events right
    after opening the FD in .realize() and restarts in .post_load().
  * The source stops reaping in .pre_save() and, since it stays alive,
    restarts from the MIG_EVENT_PRECOPY_FAILED notifier on failure.
  * On success the source's .exit() leaves the device untouched - no
    reset, interface release or host-driver rebind - and just closes its
    own copy of the FD.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/usb/host-libusb.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 83 insertions(+), 11 deletions(-)

diff --git a/hw/usb/host-libusb.c b/hw/usb/host-libusb.c
index 6b8c0cf0a82..742844ce426 100644
--- a/hw/usb/host-libusb.c
+++ b/hw/usb/host-libusb.c
@@ -110,6 +110,7 @@ struct USBHostDevice {
     NotifierWithReturn               cpr_notifier;
     GSList                           *cpr_inflight;
     bool                             cpr_drained;
+    bool                             cpr_handed_off;
     libusb_device                    *dev;
     libusb_device_handle             *dh;
     struct libusb_device_descriptor  ddesc;
@@ -1265,16 +1266,19 @@ static int usb_host_close(USBHostDevice *s)
 
     trace_usb_host_close(s->bus_num, s->addr);
 
-    usb_host_abort_xfers(s);
-    usb_host_iso_free_all(s);
+    if (!s->cpr_handed_off) {
+        usb_host_abort_xfers(s);
+        usb_host_iso_free_all(s);
 
-    if (udev->attached) {
-        usb_device_detach(udev);
+        if (udev->attached) {
+            usb_device_detach(udev);
+        }
+
+        usb_host_release_interfaces(s);
+        libusb_reset_device(s->dh);
+        usb_host_attach_kernel(s);
     }
 
-    usb_host_release_interfaces(s);
-    libusb_reset_device(s->dh);
-    usb_host_attach_kernel(s);
     libusb_close(s->dh);
     s->dh = NULL;
     s->dev = NULL;
@@ -1309,10 +1313,17 @@ static void usb_host_exit_notifier(struct Notifier *n, void *data)
     USBHostDevice *s = container_of(n, USBHostDevice, exit);
 
     if (s->dh) {
-        usb_host_abort_xfers(s);
-        usb_host_release_interfaces(s);
-        libusb_reset_device(s->dh);
-        usb_host_attach_kernel(s);
+        /*
+         * Handed to new QEMU across cpr-transfer: do not reset, release
+         * interfaces or rebind the host driver underneath it; just close
+         * our handle.
+         */
+        if (!s->cpr_handed_off) {
+            usb_host_abort_xfers(s);
+            usb_host_release_interfaces(s);
+            libusb_reset_device(s->dh);
+            usb_host_attach_kernel(s);
+        }
         libusb_close(s->dh);
     }
 }
@@ -1336,6 +1347,8 @@ static libusb_device *usb_host_find_ref(int bus, int addr)
 }
 
 static void usb_host_handle_data(USBDevice *udev, USBPacket *p);
+static void usb_host_cpr_stop_events(void);
+static void usb_host_cpr_restart_events(void);
 
 /*
  * A failed CPR migration resumes the source VM with the drained
@@ -1356,6 +1369,11 @@ static int usb_host_cpr_notifier(NotifierWithReturn *notifier,
         return 0;
     }
 
+    if (s->cpr_handed_off) {
+        usb_host_cpr_restart_events();
+        s->cpr_handed_off = false;
+    }
+
     for (it = s->cpr_inflight; it; it = it->next) {
         p = it->data;
         /* Replay from the start of the TD */
@@ -1428,6 +1446,14 @@ static void usb_host_realize(USBDevice *udev, Error **errp)
             error_setg(errp, "failed to open host usb device %s", s->hostdevice);
             return;
         }
+        if (cpr_is_incoming()) {
+            /*
+             * The source still owns the shared fd until it hands off.  Don't
+             * reap URBs on it yet, or we would steal the source's in-flight
+             * completions; .post_load() restarts events once we take over.
+             */
+            usb_host_cpr_stop_events();
+        }
     } else
 #endif
     if (s->match.addr && s->match.bus_num &&
@@ -1979,6 +2005,7 @@ static int usb_host_post_load(void *opaque, int version_id)
      * Skip it for CPR.
      */
     if (cpr_is_incoming()) {
+        usb_host_cpr_restart_events();
         return 0;
     }
 
@@ -1991,6 +2018,47 @@ static int usb_host_post_load(void *opaque, int version_id)
     return 0;
 }
 
+#ifndef CONFIG_WIN32
+
+/*
+ * cpr-transfer: the usbfs fd is shared with the new QEMU via SCM_RIGHTS.
+ * Stop this (source) process from reaping URBs on it, so the new QEMU can
+ * drive the device without both processes racing on the same fd.  Restart
+ * on migration failure, when the source resumes.
+ */
+static void usb_host_cpr_stop_events(void)
+{
+    const struct libusb_pollfd **poll = libusb_get_pollfds(ctx);
+
+    libusb_set_pollfd_notifiers(ctx, NULL, NULL, NULL);
+    if (poll) {
+        for (int i = 0; poll[i] != NULL; i++) {
+            usb_host_del_fd(poll[i]->fd, ctx);
+        }
+        free(poll);
+    }
+}
+
+static void usb_host_cpr_restart_events(void)
+{
+    const struct libusb_pollfd **poll = libusb_get_pollfds(ctx);
+
+    libusb_set_pollfd_notifiers(ctx, usb_host_add_fd, usb_host_del_fd, ctx);
+    if (poll) {
+        for (int i = 0; poll[i] != NULL; i++) {
+            usb_host_add_fd(poll[i]->fd, poll[i]->events, ctx);
+        }
+        free(poll);
+    }
+}
+
+#else
+
+static void usb_host_cpr_stop_events(void) {}
+static void usb_host_cpr_restart_events(void) {}
+
+#endif
+
 static int usb_host_pre_save(void *opaque)
 {
     USBHostDevice *s = opaque;
@@ -2002,6 +2070,10 @@ static int usb_host_pre_save(void *opaque)
         if (usb_host_cpr_drain_xfers(s) < 0) {
             return -1;
         }
+        if (mode == MIG_MODE_CPR_TRANSFER) {
+            usb_host_cpr_stop_events();
+            s->cpr_handed_off = true;
+        }
     }
     return 0;
 }

-- 
2.47.1

[PATCH QEMU HCI-8.0 v5 10/13] usb: migrate the interface altsetting #VSTOR-137800

[Andrey Drobyshev]

The active altsetting of each interface is host controller state that
the guest selected with SET_INTERFACE, but it isn't part of the
migration stream.  For most devices the target happens to reopen on the
same altsettings, so this went unnoticed.

It matters for CPR of a device whose endpoints only exist on a non-zero
altsetting, such as an isochronous streaming interface.  Migrate the
altsetting array in a subsection, gated to the CPR modes: only the CPR
target consumes it to rebuild its endpoints, while a regular migration
reopens the device and reconstructs the altsetting itself.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/usb/bus.c | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/hw/usb/bus.c b/hw/usb/bus.c
index 8dd2ce415eb..ee13acea658 100644
--- a/hw/usb/bus.c
+++ b/hw/usb/bus.c
@@ -8,6 +8,7 @@
 #include "qemu/module.h"
 #include "system/system.h"
 #include "migration/vmstate.h"
+#include "migration/cpr.h"
 #include "monitor/monitor.h"
 #include "trace.h"
 #include "qemu/cutils.h"
@@ -63,6 +64,37 @@ static int usb_device_post_load(void *opaque, int version_id)
     return 0;
 }
 
+static bool usb_altsetting_needed(void *opaque)
+{
+    USBDevice *dev = opaque;
+    int i;
+
+    /*
+     * Only the CPR target consumes this, to rebuild its endpoints; other
+     * migrations reopen the device and reconstruct the altsetting.
+     */
+    if (!cpr_incoming_needed(NULL)) {
+        return false;
+    }
+    for (i = 0; i < USB_MAX_INTERFACES; i++) {
+        if (dev->altsetting[i]) {
+            return true;
+        }
+    }
+    return false;
+}
+
+static const VMStateDescription vmstate_usb_device_altsetting = {
+    .name = "USBDevice/altsetting",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .needed = usb_altsetting_needed,
+    .fields = (const VMStateField[]) {
+        VMSTATE_INT32_ARRAY(altsetting, USBDevice, USB_MAX_INTERFACES),
+        VMSTATE_END_OF_LIST(),
+    }
+};
+
 const VMStateDescription vmstate_usb_device = {
     .name = "USBDevice",
     .version_id = 1,
@@ -77,6 +109,10 @@ const VMStateDescription vmstate_usb_device = {
         VMSTATE_INT32(setup_index, USBDevice),
         VMSTATE_UINT8_ARRAY(setup_buf, USBDevice, 8),
         VMSTATE_END_OF_LIST(),
+    },
+    .subsections = (const VMStateDescription * const []) {
+        &vmstate_usb_device_altsetting,
+        NULL
     }
 };
 

-- 
2.47.1

[PATCH QEMU HCI-8.0 v5 11/13] usb-host: reconfigure endpoints on CPR incoming #VSTOR-137800

[Andrey Drobyshev]

On CPR incoming we skip the usb_host_open() rescan, so libusb only sets
up the endpoints from the device's default configuration.  If the guest
had enabled an endpoint by selecting a non-zero altsetting (e.g. an
isochronous streaming endpoint), that endpoint is now missing, and the
guest's transfers to it simply stall.

The altsetting itself is restored by vmload now, so let's call
usb_host_ep_update() on the CPR target to rebuild the endpoints out of
it.

The kernel claims are still there on the preserved FD, but libusb's own
handle knows nothing about them, so a later SET_INTERFACE would fail with
LIBUSB_ERROR_NOT_FOUND.  Let's re-claim the interfaces on the target as
well to bring libusb's view back in sync - on the preserved FD this is a
no-op at the usbfs level anyway.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/usb/host-libusb.c | 83 +++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 82 insertions(+), 1 deletion(-)

diff --git a/hw/usb/host-libusb.c b/hw/usb/host-libusb.c
index 742844ce426..8078de01260 100644
--- a/hw/usb/host-libusb.c
+++ b/hw/usb/host-libusb.c
@@ -937,7 +937,7 @@ static void usb_host_ep_update(USBHostDevice *s)
         alt = udev->altsetting[intf->bInterfaceNumber];
 
         if (alt != 0) {
-            assert(alt < conf->interface[i].num_altsetting);
+            assert(alt >= 0 && alt < conf->interface[i].num_altsetting);
             intf = &conf->interface[i].altsetting[alt];
         }
 
@@ -1994,6 +1994,75 @@ static void usb_host_post_load_bh(void *opaque)
     usb_host_auto_check(NULL);
 }
 
+/*
+ * On the CPR target libusb knows nothing about the interface claims the
+ * preserved fd still holds, so a later SET_INTERFACE would fail with
+ * NOT_FOUND.  Re-claim them to bring libusb's view back in sync; on the
+ * preserved fd it's a no-op at the usbfs level anyway.
+ */
+static int usb_host_cpr_reclaim_interfaces(USBHostDevice *s)
+{
+    USBDevice *udev = USB_DEVICE(s);
+    struct libusb_config_descriptor *conf;
+    int n, rc;
+
+    rc = libusb_get_active_config_descriptor(s->dev, &conf);
+    if (rc != 0) {
+        usb_host_libusb_error("libusb_get_active_config_descriptor", rc);
+        return -1;
+    }
+
+    n = MIN(conf->bNumInterfaces, USB_MAX_INTERFACES);
+    for (int i = 0; i < n; i++) {
+        rc = libusb_claim_interface(s->dh, i);
+        if (rc != 0) {
+            usb_host_libusb_error("libusb_claim_interface", rc);
+            libusb_free_config_descriptor(conf);
+            return -1;
+        }
+        s->ifs[i].claimed = true;
+    }
+
+    udev->ninterfaces = conf->bNumInterfaces;
+    udev->configuration = conf->bConfigurationValue;
+    libusb_free_config_descriptor(conf);
+
+    return 0;
+}
+
+/*
+ * During CPR the altsetting array is included in the migration stream.
+ * Reject a value this device can't honour, so that usb_host_ep_update()
+ * on the target never sees an out of range altsetting.  In this case
+ * we fail the incoming migration instead of aborting on it.
+ */
+static int usb_host_cpr_check_altsettings(USBHostDevice *s)
+{
+    USBDevice *udev = USB_DEVICE(s);
+    struct libusb_config_descriptor *conf;
+
+    if (libusb_get_active_config_descriptor(s->dev, &conf) != 0) {
+        error_report("usb-host: cannot read config descriptor on "
+                     "CPR incoming");
+        return -1;
+    }
+
+    for (int i = 0; i < conf->bNumInterfaces; i++) {
+        int ifnum = conf->interface[i].altsetting[0].bInterfaceNumber;
+        int alt = udev->altsetting[ifnum];
+
+        if (alt < 0 || alt >= conf->interface[i].num_altsetting) {
+            error_report("usb-host: migrated altsetting %d out of range for "
+                         "interface %d", alt, ifnum);
+            libusb_free_config_descriptor(conf);
+            return -1;
+        }
+    }
+
+    libusb_free_config_descriptor(conf);
+    return 0;
+}
+
 static int usb_host_post_load(void *opaque, int version_id)
 {
     USBHostDevice *dev = opaque;
@@ -2005,6 +2074,18 @@ static int usb_host_post_load(void *opaque, int version_id)
      * Skip it for CPR.
      */
     if (cpr_is_incoming()) {
+        /*
+         * We kept the device open across CPR, so libusb still needs its
+         * interface claims and endpoints rebuilt here.  And we only start
+         * reaping now, once the source has handed the fd off to us.
+         */
+        if (usb_host_cpr_check_altsettings(dev) < 0) {
+            return -1;
+        }
+        if (usb_host_cpr_reclaim_interfaces(dev) < 0) {
+            return -1;
+        }
+        usb_host_ep_update(dev);
         usb_host_cpr_restart_events();
         return 0;
     }

-- 
2.47.1

[PATCH QEMU HCI-8.0 v5 12/13] hcd-xhci: resync isochronous endpoints after CPR #VSTOR-137800

[Andrey Drobyshev]

An isochronous endpoint has no retransmission, so the transfer events
the controller reports have to line up with the TDs the guest driver is
sitting on.  Across CPR we rebuild the controller's transient transfer
and scheduling state out of guest memory, and it no longer matches what
the guest has already consumed - so the very first event after resume
lands on the wrong TD and the guest wedges with "not part of TD".

Let's report CC_MISSED_SERVICE_ERROR on the first isochronous transfer
after a CPR resume, just like a real xHCI does when it misses a service
interval.  That arms the guest driver's skip logic, and it resyncs to
the live stream instead of getting stuck.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/usb/hcd-xhci.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/hw/usb/hcd-xhci.c b/hw/usb/hcd-xhci.c
index 292c378bfc9..0370c784627 100644
--- a/hw/usb/hcd-xhci.c
+++ b/hw/usb/hcd-xhci.c
@@ -25,6 +25,7 @@
 #include "qemu/module.h"
 #include "qemu/queue.h"
 #include "migration/vmstate.h"
+#include "migration/cpr.h"
 #include "hw/qdev-properties.h"
 #include "trace.h"
 #include "qapi/error.h"
@@ -282,6 +283,7 @@ struct XHCIEPContext {
     unsigned int max_psize;
     uint32_t state;
     uint32_t kick_active;
+    bool cpr_iso_resync; /* resync iso stream on first kick after CPR resume */
 
     /* streams */
     unsigned int max_pstreams;
@@ -1811,6 +1813,19 @@ static int xhci_submit(XHCIState *xhci, XHCITransfer *xfer, XHCIEPContext *epctx
         xfer->pkts = 1;
         xfer->iso_xfer = true;
         xfer->timed_xfer = true;
+        if (epctx->cpr_iso_resync) {
+            /*
+             * First iso TD after a CPR resume: our transfer state was
+             * rebuilt from guest memory and no longer lines up with what
+             * the guest consumed.  Report a missed service, like real HW,
+             * to kick the driver's skip logic into resyncing.
+             */
+            epctx->cpr_iso_resync = false;
+            xfer->status = CC_MISSED_SERVICE_ERROR;
+            xhci_xfer_report(xfer);
+            xfer->complete = 1;
+            return 0;
+        }
         mfindex = xhci_mfindex_get(xhci);
         xhci_calc_iso_kick(xhci, xfer, epctx, mfindex);
         xhci_check_intr_iso_kick(xhci, xfer, epctx, mfindex);
@@ -3510,6 +3525,11 @@ static int usb_xhci_post_load(void *opaque, int version_id)
             xhci_init_epctx(epctx, pctx, ep_ctx);
             epctx->state = state;
             if (state == EP_RUNNING) {
+                if (cpr_is_incoming() &&
+                    (epctx->type == ET_ISO_IN || epctx->type == ET_ISO_OUT)) {
+                    /* only iso streams need resyncing; see xhci_submit() */
+                    epctx->cpr_iso_resync = true;
+                }
                 /* kick endpoint after vmload is finished */
                 timer_mod(epctx->kick_timer, qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL));
             }

-- 
2.47.1

[PATCH QEMU HCI-8.0 v5 13/13] usb-host: make CPR migration blocker conditional #VSTOR-137800

[Andrey Drobyshev]

A previous commit added an unconditional blocker for CPR migrations.
That is to refuse CPR gracefully instead of crashing and for the previous
commits to remain bisectable.

Now the restore path is complete: the FD is preserved, in-flight URBs
are drained and replayed, the device is handed off across cpr-transfer,
and the endpoints and isochronous streams are reconstructed on the
target.  So let's lift the CPR blocker for the devices which satisfy the
preconditions.

Still, the remaining preconditions for CPR to work are:

  * We must open host device FD ourselves, so that libusb owns the FD.
    This way we're able to preserve the FD during CPR;
  * Device ID must be provided as a unique stable key for CPR FD registry.

Use them to guard the blocker addition.  Apart from that, prohibit CPR on
WIN32 platform and on hosts with older libusb versions.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/usb/host-libusb.c | 50 +++++++++++++++++++++++++++++++++++---------------
 1 file changed, 35 insertions(+), 15 deletions(-)

diff --git a/hw/usb/host-libusb.c b/hw/usb/host-libusb.c
index 8078de01260..0623d6a7183 100644
--- a/hw/usb/host-libusb.c
+++ b/hw/usb/host-libusb.c
@@ -1484,24 +1484,44 @@ static void usb_host_realize(USBDevice *udev, Error **errp)
     s->exit.notify = usb_host_exit_notifier;
     qemu_add_exit_notifier(&s->exit);
 
-#if LIBUSB_API_VERSION >= 0x01000107 && !defined(CONFIG_WIN32)
-    if (s->hostdevice && DEVICE(s)->id) {
-        migration_add_notifier_modes(&s->cpr_notifier, usb_host_cpr_notifier,
-                                     MIG_MODE_CPR_TRANSFER,
-                                     MIG_MODE_CPR_EXEC, -1);
+#ifdef CONFIG_WIN32
+    error_setg(&s->cpr_blocker,
+               "usb-host device %s does not support CPR: "
+               "WIN32 platform is not supported",
+               DEVICE(s)->id ?: "(anonymous)");
+#elif LIBUSB_API_VERSION < 0x01000107
+    error_setg(&s->cpr_blocker,
+               "usb-host device %s does not support CPR: "
+               "libusb API version 0x%08x is too old",
+               DEVICE(s)->id ?: "(anonymous)", (unsigned) LIBUSB_API_VERSION);
+#else
+    if (!(s->hostdevice && DEVICE(s)->id)) {
+        /*
+         * Host FD must be owned by libusb to support CPR migration.  That's
+         * equivalent to hostdevice= property being present.   Also device ID
+         * is required as stable key for CPR FD registry.
+         */
+        error_setg(&s->cpr_blocker,
+                   "usb-host device %s does not support CPR: "
+                   "hostdevice= and a device id are required",
+                   DEVICE(s)->id ?: "(anonymous)");
     }
 #endif
 
-    error_setg(&s->cpr_blocker, "usb-host device %s does not support CPR",
-               DEVICE(s)->id ?: "(anonymous)");
-    if (migrate_add_blocker_modes(&s->cpr_blocker, errp,
-                                  MIG_MODE_CPR_TRANSFER,
-                                  MIG_MODE_CPR_EXEC, -1) < 0) {
-        qemu_remove_exit_notifier(&s->exit);
-        if (s->needs_autoscan) {
-            QTAILQ_REMOVE(&hostdevs, s, next);
-        }
-        usb_host_close(s);
+    if (s->cpr_blocker) {
+        if (migrate_add_blocker_modes(&s->cpr_blocker, errp,
+                                      MIG_MODE_CPR_TRANSFER,
+                                      MIG_MODE_CPR_EXEC, -1) < 0) {
+            qemu_remove_exit_notifier(&s->exit);
+            if (s->needs_autoscan) {
+                QTAILQ_REMOVE(&hostdevs, s, next);
+            }
+            usb_host_close(s);
+        }
+    } else {
+        migration_add_notifier_modes(&s->cpr_notifier, usb_host_cpr_notifier,
+                                     MIG_MODE_CPR_TRANSFER,
+                                     MIG_MODE_CPR_EXEC, -1);
     }
 }
 

-- 
2.47.1

Re: [PATCH QEMU HCI-8.0 v5 00/13] usb-host: support CPR migration

[Denis V. Lunev]

On 9/25/26 16:11, Andrey Drobyshev wrote:
> v4 -> v5:
>
>   * Patch 5: deliver URBs that already completed in the kernel instead
>     of dropping their data, and error out any a drain can't reap in
>     time;
>
>   * Patch 6 (NEW): drain each usb-host before its host controller is
>     migrated, so a shared-context completion can't advance xHCI after
>     it was saved;
>
>   * Patch 7: remember the packets to replay from the reap callback, so
>     only the genuinely in-flight ones are re-issued.
>
> v4: http://lore.virtuozzo.com/qemu/20260924-usb-host_v3-v4-0-1cc48bf630e7@virtuozzo.com
>
> Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
> ---
> Andrey Drobyshev (13):
>       usb-host: don't leak hostdev FD on open failure #VSTOR-137800
>       usb-host: add migration blocker for CPR modes #VSTOR-137800
>       usb-host: preserve hostdev FD during CPR migration #VSTOR-137800
>       usb-host: factor out usb_host_reap_xfers() #VSTOR-137800
>       usb-host: drain in-flight URBs across CPR #VSTOR-137800
>       usb-host: drain before the host controller is migrated #VSTOR-137800
>       usb-host: re-issue drained URBs if CPR is aborted #VSTOR-137800
>       usb-host: skip product-string read on CPR incoming #VSTOR-137800
>       usb-host: hand off the device across cpr-transfer #VSTOR-137800
>       usb: migrate the interface altsetting #VSTOR-137800
>       usb-host: reconfigure endpoints on CPR incoming #VSTOR-137800
>       hcd-xhci: resync isochronous endpoints after CPR #VSTOR-137800
>       usb-host: make CPR migration blocker conditional #VSTOR-137800
>
>  hw/usb/bus.c                |  36 ++++
>  hw/usb/hcd-xhci.c           |  20 ++
>  hw/usb/host-libusb.c        | 512 ++++++++++++++++++++++++++++++++++++++++----
>  include/migration/vmstate.h |   1 +
>  4 files changed, 530 insertions(+), 39 deletions(-)
> ---
> base-commit: f2f4b3de1a1be3360a77ca6ab79762466d238804
> change-id: 20260914-usb-host_v3-bbfea86d1dad
>
> Best regards,
> --  
> Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
  What v5 fixes from v4:
  - Draining now happens before the xHCI controller's state is saved
(new patch 6).
  - Transfers that already finished in the kernel are delivered to the
guest instead of dropped.
  - If migration fails, only the transfers that were really cancelled
are sent again, so none go to the device
    twice.

  Blocking:
  1. The guest can lose an interrupt and USB stops working on the
target. The CPU state, including the KVM
     interrupt controller, is saved (migration/savevm.c:1668) before
usb_host_pre_save() drains. Any transfer
     completed during the drain makes xHCI send an MSI
(message-signalled interrupt) after that snapshot, so the
     new process never sees it. The saved xHCI state still has ERDP_EHB
set, so xhci_intr_raise() sends no more
     interrupts until the guest writes ERDP, which it only does from the
handler that never runs. That kills the
     whole interrupter, not just delays one event. v5 makes this more
likely than v4, because it now delivers
     completed transfers during the drain.
  2. A transfer cancelled halfway is re-run from the start. For a bulk
IN transfer, the bytes already read from
     the device are lost. For bulk OUT, bytes the device already
accepted are sent again. This is the normal case
     whenever the VM stops under I/O. The fix is to deliver a cancelled
transfer that has actual_length != 0 as a
     short transfer, and replay only the ones that moved no data.

  Proposed direction: drain from a VM state change handler when the VM
enters FINISH_MIGRATE in a CPR mode, and
  re-issue from the handler that runs when the VM starts again. That
runs before the CPU state is saved, which
  fixes item 1 and makes patch 6 unnecessary. It also closes the window
from the v2 review where vm_start() runs
  before the failure notifier. That window now has a second problem:
with pipelined bulk endpoints, newer OUT data
  can reach the device before the re-issued older data. I left one
question open for Andrey: a VM already paused
  before migration starts never gets that handler call, so it still
needs some fallback.