[PATCH QEMU HCI-8.0 v3 00/12] usb-host: support CPR migration

[PATCH QEMU HCI-8.0 v3 00/12] usb-host: support CPR migration

[Andrey Drobyshev]

v3 fixes the issues reported by Den.

v2 -> v3:

  * v2 patch 1 ("migration/cpr: fix use-after-free in cpr_delete_fd()"):
    dropped, already applied.

  * Patch 1: record the caller's FD before the early exits so every
    failure path frees it.

  * Patch 2: make the blocker message a complete sentence;

  * Patch 3: use cpr_open_fd();

  * Patch 5:
    - fail .pre_save() when a URB can't be reaped;
    - fix a block-comment style warning;
    - reword the TD-replay note.

  * Patch 6: drop the packet from the re-issue list in
    usb_host_cancel_packet();

  * Patch 8:
    - honour cpr_handed_off in usb_host_close();
    - move the forward declarations above the notifier's doc comment.

  * Patch 10:
    - validate the migrated altsetting and fail the incoming migration
      instead of asserting on it;
    - check libusb_claim_interface() failures instead of silently
      continuing;
    - reword commit message.

  * Patch 11: gate the resync to cpr_is_incoming().

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
Andrey Drobyshev (12):
      usb-host: don't leak hostdev FD on open failure #VSTOR-137800
      usb-host: add migration blocker for CPR modes #VSTOR-137800
      usb-host: preserve hostdev FD during CPR migration #VSTOR-137800
      usb-host: factor out usb_host_reap_xfers() #VSTOR-137800
      usb-host: drain in-flight URBs across CPR #VSTOR-137800
      usb-host: re-issue drained URBs if CPR is aborted #VSTOR-137800
      usb-host: skip product-string read on CPR incoming #VSTOR-137800
      usb-host: hand off the device across cpr-transfer #VSTOR-137800
      usb: migrate the interface altsetting #VSTOR-137800
      usb-host: reconfigure endpoints on CPR incoming #VSTOR-137800
      hcd-xhci: resync isochronous endpoints after CPR #VSTOR-137800
      usb-host: make CPR migration blocker conditional #VSTOR-137800

 hw/usb/bus.c         |  36 +++++
 hw/usb/hcd-xhci.c    |  20 +++
 hw/usb/host-libusb.c | 435 ++++++++++++++++++++++++++++++++++++++++++++++++---
 3 files changed, 471 insertions(+), 20 deletions(-)
---
base-commit: 87f111d087cff956908ccb22daedb596b3734877
change-id: 20260914-usb-host_v3-bbfea86d1dad

Best regards,
--  
Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>

[PATCH QEMU HCI-8.0 v3 01/12] usb-host: don't leak hostdev FD on open failure #VSTOR-137800

[Andrey Drobyshev]

The FD passed into usb_host_open() for a wrapped device is only stored in
s->hostfd after libusb_wrap_sys_device() succeeds, and the failure path
doesn't close it.  if wrapping fails, or a later step of the open sequence
fails (e.g. usb_device_attach), the FD is leaked, as usb_host_close() only
runs for fully opened devices, and the realize error path doesn't clean it
up either.

Store the FD in s->hostfd before wrapping and close it on the failure
path.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/usb/host-libusb.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/hw/usb/host-libusb.c b/hw/usb/host-libusb.c
index b74670ae256..83645fbe0ce 100644
--- a/hw/usb/host-libusb.c
+++ b/hw/usb/host-libusb.c
@@ -954,6 +954,11 @@ static int usb_host_open(USBHostDevice *s, libusb_device *dev, int hostfd)
     int rc;
     Error *local_err = NULL;
 
+    /* Make sure failure path frees caller-owned FD */
+    if (!dev) {
+        s->hostfd = hostfd;
+    }
+
     if (s->bh_postld_pending) {
         return -1;
     }
@@ -978,7 +983,6 @@ static int usb_host_open(USBHostDevice *s, libusb_device *dev, int hostfd)
         if (rc != 0) {
             goto fail;
         }
-        s->hostfd  = hostfd;
         dev = libusb_get_device(s->dh);
         bus_num = libusb_get_bus_number(dev);
         addr = libusb_get_device_address(dev);
@@ -1066,6 +1070,10 @@ fail:
         s->dh = NULL;
         s->dev = NULL;
     }
+    if (s->hostfd != -1) {
+        close(s->hostfd);
+        s->hostfd = -1;
+    }
     return -1;
 }
 

-- 
2.47.1

[PATCH QEMU HCI-8.0 v3 02/12] usb-host: add migration blocker for CPR modes #VSTOR-137800

[Andrey Drobyshev]

We're about to add support for CPR migration to usb-host in the upcoming
commits.  Right now CPR is crashing with usb-host devices.  Let's add
migration blocker upfront, for both cpr-exec and cpr-transfer.  The
blocker is added unconditionally, so that we resuse CPR migration instead
of crashing on it.  That is for the sake of bisectability of the upcoming
commits.  It is going to be lifted (conditioned) in a following patch
once CPR support is added.  Also, blocker is added at the end of
usb_host_realize(), so we have to cleanup all the side effects done
earlier in .realize() in case migrate_add_blocker_modes() fails.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/usb/host-libusb.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/hw/usb/host-libusb.c b/hw/usb/host-libusb.c
index 83645fbe0ce..5c3da00d8eb 100644
--- a/hw/usb/host-libusb.c
+++ b/hw/usb/host-libusb.c
@@ -46,6 +46,7 @@
 #endif
 
 #include "qapi/error.h"
+#include "migration/blocker.h"
 #include "migration/vmstate.h"
 #include "monitor/monitor.h"
 #include "qemu/error-report.h"
@@ -103,6 +104,7 @@ struct USBHostDevice {
     char                             port[16];
 
     int                              hostfd;
+    Error                            *cpr_blocker;
     libusb_device                    *dev;
     libusb_device_handle             *dh;
     struct libusb_device_descriptor  ddesc;
@@ -1258,6 +1260,18 @@ static void usb_host_realize(USBDevice *udev, Error **errp)
 
     s->exit.notify = usb_host_exit_notifier;
     qemu_add_exit_notifier(&s->exit);
+
+    error_setg(&s->cpr_blocker, "usb-host device %s does not support CPR",
+               DEVICE(s)->id ?: "(anonymous)");
+    if (migrate_add_blocker_modes(&s->cpr_blocker, errp,
+                                  MIG_MODE_CPR_TRANSFER,
+                                  MIG_MODE_CPR_EXEC, -1) < 0) {
+        qemu_remove_exit_notifier(&s->exit);
+        if (s->needs_autoscan) {
+            QTAILQ_REMOVE(&hostdevs, s, next);
+        }
+        usb_host_close(s);
+    }
 }
 
 static void usb_host_instance_init(Object *obj)
@@ -1279,6 +1293,7 @@ static void usb_host_unrealize(USBDevice *udev)
         QTAILQ_REMOVE(&hostdevs, s, next);
     }
     usb_host_close(s);
+    migrate_del_blocker(&s->cpr_blocker);
 }
 
 static void usb_host_cancel_packet(USBDevice *udev, USBPacket *p)

-- 
2.47.1

[PATCH QEMU HCI-8.0 v3 03/12] usb-host: preserve hostdev FD during CPR migration #VSTOR-137800

[Andrey Drobyshev]

In .realize(), save freshly opened host device FD into the CPR registry,
and then reuse it on CPR target.  Also, add deletion of that FD from
registry to usb_host_open() / usb_host_close() cleanup paths.

For this to work, we also need to skip the .post-load() code which
closes, detaches the device, and then rescans the host bus to reopen
matching hostdevs.  For the CPR-case migration we don't want any of that
as the hostdev FD stays preserved, and the guest shouldn't notice the
switchover.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/usb/host-libusb.c | 33 ++++++++++++++++++++++++++++++++-
 1 file changed, 32 insertions(+), 1 deletion(-)

diff --git a/hw/usb/host-libusb.c b/hw/usb/host-libusb.c
index 5c3da00d8eb..b946b0ef350 100644
--- a/hw/usb/host-libusb.c
+++ b/hw/usb/host-libusb.c
@@ -47,6 +47,7 @@
 
 #include "qapi/error.h"
 #include "migration/blocker.h"
+#include "migration/cpr.h"
 #include "migration/vmstate.h"
 #include "monitor/monitor.h"
 #include "qemu/error-report.h"
@@ -947,6 +948,21 @@ static void usb_host_ep_update(USBHostDevice *s)
     libusb_free_config_descriptor(conf);
 }
 
+static char *usb_host_cpr_fd_name(USBHostDevice *s)
+{
+    /* Prefix the key so it can't clash with another CPR label */
+    return DEVICE(s)->id ? g_strdup_printf("usb-host/%s", DEVICE(s)->id)
+                         : NULL;
+}
+
+static void usb_host_cpr_delete_fd(USBHostDevice *s)
+{
+    g_autofree char *name = usb_host_cpr_fd_name(s);
+    if (name) {
+        cpr_delete_fd(name, 0);
+    }
+}
+
 static int usb_host_open(USBHostDevice *s, libusb_device *dev, int hostfd)
 {
     USBDevice *udev = USB_DEVICE(s);
@@ -1073,6 +1089,7 @@ fail:
         s->dev = NULL;
     }
     if (s->hostfd != -1) {
+        usb_host_cpr_delete_fd(s);
         close(s->hostfd);
         s->hostfd = -1;
     }
@@ -1134,6 +1151,7 @@ static int usb_host_close(USBHostDevice *s)
     s->dev = NULL;
 
     if (s->hostfd != -1) {
+        usb_host_cpr_delete_fd(s);
         close(s->hostfd);
         s->hostfd = -1;
     }
@@ -1220,9 +1238,12 @@ static void usb_host_realize(USBDevice *udev, Error **errp)
 
 #if LIBUSB_API_VERSION >= 0x01000107 && !defined(CONFIG_WIN32)
     if (s->hostdevice) {
+        g_autofree char *name = usb_host_cpr_fd_name(s);
         int fd;
+
         s->needs_autoscan = false;
-        fd = qemu_open(s->hostdevice, O_RDWR, errp);
+        fd = name ? cpr_open_fd(s->hostdevice, O_RDWR, name, 0, errp)
+                  : qemu_open(s->hostdevice, O_RDWR, errp);
         if (fd < 0) {
             return;
         }
@@ -1761,6 +1782,16 @@ static int usb_host_post_load(void *opaque, int version_id)
 {
     USBHostDevice *dev = opaque;
 
+    /*
+     * For CPR migration, device wasn't released/reset, and the guest
+     * is unaware of the switchover.  The detach/rescan performed in
+     * usb_host_post_load_bh() only exists for cross-host migration.
+     * Skip it for CPR.
+     */
+    if (cpr_is_incoming()) {
+        return 0;
+    }
+
     if (!dev->bh_postld) {
         dev->bh_postld = qemu_bh_new_guarded(usb_host_post_load_bh, dev,
                                              &DEVICE(dev)->mem_reentrancy_guard);

-- 
2.47.1

[PATCH QEMU HCI-8.0 v3 04/12] usb-host: factor out usb_host_reap_xfers() #VSTOR-137800

[Andrey Drobyshev]

usb_host_abort_xfers() cancels every pending request and then pumps
libusb events until all of them are reaped, with a bounded wait.
Split the reap-wait loop into usb_host_reap_xfers(), so that the
following commit can reuse it to reap canceled transfers without going
through the abort path.  No functional change.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/usb/host-libusb.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/hw/usb/host-libusb.c b/hw/usb/host-libusb.c
index b946b0ef350..9b70d6deef9 100644
--- a/hw/usb/host-libusb.c
+++ b/hw/usb/host-libusb.c
@@ -1096,15 +1096,11 @@ fail:
     return -1;
 }
 
-static void usb_host_abort_xfers(USBHostDevice *s)
+static void usb_host_reap_xfers(USBHostDevice *s)
 {
     USBHostRequest *r, *rtmp;
     int limit = 100;
 
-    QTAILQ_FOREACH_SAFE(r, &s->requests, next, rtmp) {
-        usb_host_req_abort(r);
-    }
-
     while (QTAILQ_FIRST(&s->requests) != NULL) {
         struct timeval tv;
         memset(&tv, 0, sizeof(tv));
@@ -1126,6 +1122,16 @@ static void usb_host_abort_xfers(USBHostDevice *s)
     }
 }
 
+static void usb_host_abort_xfers(USBHostDevice *s)
+{
+    USBHostRequest *r, *rtmp;
+
+    QTAILQ_FOREACH_SAFE(r, &s->requests, next, rtmp) {
+        usb_host_req_abort(r);
+    }
+    usb_host_reap_xfers(s);
+}
+
 static int usb_host_close(USBHostDevice *s)
 {
     USBDevice *udev = USB_DEVICE(s);

-- 
2.47.1

[PATCH QEMU HCI-8.0 v3 05/12] usb-host: drain in-flight URBs across CPR #VSTOR-137800

[Andrey Drobyshev]

URBs submitted by the old QEMU must not outlive it across the
switchover: the kernel would complete them into buffers of an address
space which is gone after exec, and the new libusb context would reap
URB pointers it never submitted.

So, in .pre_save() for the CPR modes, cancel and reap every pending
transfer, without completing its packet.  The packets stay ASYNC and
their TDs remain on the transfer rings, so the target re-executes them
when usb_xhci_post_load() kicks the endpoints - the same in-flight
replay as on regular live migration.  A partially executed transfer is
simply re-run from the start of the TD.

Isochronous URBs are not on the request list but on their own rings,
and freeing them while in-flight would leave them queued on the
preserved FD.  Cancel them so they are unlinked and reaped: an iso URB
in flight when the guest stops may never complete on its own.  The
target reconstructs the iso stream from the guest's transfer ring.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/usb/host-libusb.c | 92 ++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 90 insertions(+), 2 deletions(-)

diff --git a/hw/usb/host-libusb.c b/hw/usb/host-libusb.c
index 9b70d6deef9..1331eeefee4 100644
--- a/hw/usb/host-libusb.c
+++ b/hw/usb/host-libusb.c
@@ -48,6 +48,7 @@
 #include "qapi/error.h"
 #include "migration/blocker.h"
 #include "migration/cpr.h"
+#include "migration/misc.h"
 #include "migration/vmstate.h"
 #include "monitor/monitor.h"
 #include "qemu/error-report.h"
@@ -1096,7 +1097,7 @@ fail:
     return -1;
 }
 
-static void usb_host_reap_xfers(USBHostDevice *s)
+static bool usb_host_reap_xfers(USBHostDevice *s)
 {
     USBHostRequest *r, *rtmp;
     int limit = 100;
@@ -1117,9 +1118,10 @@ static void usb_host_reap_xfers(USBHostDevice *s)
             QTAILQ_FOREACH_SAFE(r, &s->requests, next, rtmp) {
                 QTAILQ_REMOVE(&s->requests, r, next);
             }
-            return;
+            return false;
         }
     }
+    return true;
 }
 
 static void usb_host_abort_xfers(USBHostDevice *s)
@@ -1132,6 +1134,76 @@ static void usb_host_abort_xfers(USBHostDevice *s)
     usb_host_reap_xfers(s);
 }
 
+static bool usb_host_iso_inflight(USBHostDevice *s)
+{
+    USBHostIsoRing *ring;
+
+    QTAILQ_FOREACH(ring, &s->isorings, next) {
+        if (!QTAILQ_EMPTY(&ring->inflight)) {
+            return true;
+        }
+    }
+    return false;
+}
+
+/*
+ * Quiesce the device for a CPR switchover.  Cancel and reap all URBs,
+ * so that no URB submitted by this process can outlive it.
+ */
+static int usb_host_cpr_drain_xfers(USBHostDevice *s)
+{
+    USBHostRequest *r, *rtmp;
+    USBHostIsoRing *ring;
+    USBHostIsoXfer *xfer;
+    int limit;
+
+    QTAILQ_FOREACH_SAFE(r, &s->requests, next, rtmp) {
+        if (r->p) {
+            /*
+             * Clear r->p so the reap callback early-outs on it: the
+             * callback must still run (it frees the request), but it
+             * must not complete the packet.
+             */
+            r->p = NULL;
+        }
+        libusb_cancel_transfer(r->xfer);
+    }
+
+    if (!usb_host_reap_xfers(s)) {
+        /*
+         * No URB can be left live on the FD across CPR migration.  If
+         * we can't reap them -> abort the migration.
+         */
+        return -1;
+    }
+
+    /*
+     * Iso URBs are on the rings, not s->requests.  Cancel them: one in
+     * flight when the guest stops may never complete on its own, and none
+     * may be left on the preserved fd.  The target replays from the ring.
+     */
+    QTAILQ_FOREACH(ring, &s->isorings, next) {
+        QTAILQ_FOREACH(xfer, &ring->inflight, next) {
+            libusb_cancel_transfer(xfer->xfer);
+        }
+    }
+
+    /*
+     * Cap the reap at 2x a full ring:
+     * iso_urb_count URBs x iso_urb_frames packets each
+     */
+    limit = 2 * s->iso_urb_count * s->iso_urb_frames;
+    while (usb_host_iso_inflight(s)) {
+        struct timeval tv = { .tv_usec = 1000 };
+        libusb_handle_events_timeout(ctx, &tv);
+        if (limit-- == 0) {
+            return -1;
+        }
+    }
+    usb_host_iso_free_all(s);
+    return 0;
+}
+
 static int usb_host_close(USBHostDevice *s)
 {
     USBDevice *udev = USB_DEVICE(s);
@@ -1807,10 +1879,26 @@ static int usb_host_post_load(void *opaque, int version_id)
     return 0;
 }
 
+static int usb_host_pre_save(void *opaque)
+{
+    USBHostDevice *s = opaque;
+    MigMode mode = migrate_mode();
+
+    /* URBs submitted by this process must not outlive CPR migration */
+    if ((mode == MIG_MODE_CPR_EXEC || mode == MIG_MODE_CPR_TRANSFER) &&
+        s->dh) {
+        if (usb_host_cpr_drain_xfers(s) < 0) {
+            return -1;
+        }
+    }
+    return 0;
+}
+
 static const VMStateDescription vmstate_usb_host = {
     .name = "usb-host",
     .version_id = 1,
     .minimum_version_id = 1,
+    .pre_save = usb_host_pre_save,
     .post_load = usb_host_post_load,
     .fields = (const VMStateField[]) {
         VMSTATE_USB_DEVICE(parent_obj, USBHostDevice),

-- 
2.47.1

[PATCH QEMU HCI-8.0 v3 06/12] usb-host: re-issue drained URBs if CPR is aborted #VSTOR-137800

[Andrey Drobyshev]

The previous commit drains the in-flight URBs in .pre_save() and leaves
their packets ASYNC for the target to replay.  If migration fails, the
source resumes instead, with those packets still owned by the
controller and nothing left to complete them.

Remember the drained packets in a list, and re-issue them on the
preserved FD from a MIG_EVENT_PRECOPY_FAILED notifier.  Control
transfers can't be reconstructed from the packet alone, so complete
those as errors and let the guest driver retry.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/usb/host-libusb.c | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 76 insertions(+), 2 deletions(-)

diff --git a/hw/usb/host-libusb.c b/hw/usb/host-libusb.c
index 1331eeefee4..77ee3ca1dd7 100644
--- a/hw/usb/host-libusb.c
+++ b/hw/usb/host-libusb.c
@@ -107,6 +107,9 @@ struct USBHostDevice {
 
     int                              hostfd;
     Error                            *cpr_blocker;
+    NotifierWithReturn               cpr_notifier;
+    GSList                           *cpr_inflight;
+    bool                             cpr_drained;
     libusb_device                    *dev;
     libusb_device_handle             *dh;
     struct libusb_device_descriptor  ddesc;
@@ -1147,8 +1150,13 @@ static bool usb_host_iso_inflight(USBHostDevice *s)
 }
 
 /*
- * Quiesce the device for a CPR switchover.  Cancel and reap all URBs,
- * so that no URB submitted by this process can outlive it.
+ * Quiesce the device for a CPR switchover.  Cancel and reap all URBs
+ * without completing their packets, so that no URB submitted by this
+ * process can outlive it.  CPR target re-executes them in
+ * usb_xhci_post_load() kicking the running endpoints.
+ *
+ * Keep the list of packets so that they can be re-issued on this side
+ * instead if MIG_EVENT_PRECOPY_FAILED fires.
  */
 static int usb_host_cpr_drain_xfers(USBHostDevice *s)
 {
@@ -1159,6 +1167,10 @@ static int usb_host_cpr_drain_xfers(USBHostDevice *s)
 
     QTAILQ_FOREACH_SAFE(r, &s->requests, next, rtmp) {
         if (r->p) {
+            if (r->p->state == USB_PACKET_ASYNC) {
+                /* In-flight req, in submission order, for the failure path */
+                s->cpr_inflight = g_slist_append(s->cpr_inflight, r->p);
+            }
             /*
              * Clear r->p so the reap callback early-outs on it: the
              * callback must still run (it frees the request), but it
@@ -1168,6 +1180,7 @@ static int usb_host_cpr_drain_xfers(USBHostDevice *s)
         }
         libusb_cancel_transfer(r->xfer);
     }
+    s->cpr_drained = true;
 
     if (!usb_host_reap_xfers(s)) {
         /*
@@ -1284,6 +1297,53 @@ static libusb_device *usb_host_find_ref(int bus, int addr)
     return ret;
 }
 
+static void usb_host_handle_data(USBDevice *udev, USBPacket *p);
+
+/*
+ * A failed CPR migration resumes the source VM with the drained
+ * packets still owned by the host controller as in-flight.  Re-issue
+ * them on the preserved fd.  Control transfers cannot be reconstructed
+ * from the packet alone; complete them as errors and let the guest
+ * driver retry.
+ */
+static int usb_host_cpr_notifier(NotifierWithReturn *notifier,
+                                 MigrationEvent *e, Error **errp)
+{
+    USBHostDevice *s = container_of(notifier, USBHostDevice, cpr_notifier);
+    USBDevice *udev = USB_DEVICE(s);
+    GSList *it;
+    USBPacket *p;
+
+    if (e->type != MIG_EVENT_PRECOPY_FAILED || !s->cpr_drained) {
+        return 0;
+    }
+
+    for (it = s->cpr_inflight; it; it = it->next) {
+        p = it->data;
+        /* Replay from the start of the TD */
+        p->actual_length = 0;
+        if (p->ep->nr == 0) {
+            /*
+             * p->ep[0] is control endpoint.  Control transfers can't
+             * be reconstructed reliably, so complete them as errors
+             * and let the guest retry.
+             */
+            p->status = USB_RET_IOERROR;
+            usb_generic_async_ctrl_complete(udev, p);
+        } else {
+            usb_host_handle_data(udev, p);
+            if (p->status != USB_RET_ASYNC) {
+                /* Completed synchronously with error */
+                usb_packet_complete(udev, p);
+            }
+        }
+    }
+    g_slist_free(s->cpr_inflight);
+    s->cpr_inflight = NULL;
+    s->cpr_drained = false;
+    return 0;
+}
+
 static void usb_host_realize(USBDevice *udev, Error **errp)
 {
     USBHostDevice *s = USB_HOST_DEVICE(udev);
@@ -1360,6 +1420,14 @@ static void usb_host_realize(USBDevice *udev, Error **errp)
     s->exit.notify = usb_host_exit_notifier;
     qemu_add_exit_notifier(&s->exit);
 
+#if LIBUSB_API_VERSION >= 0x01000107 && !defined(CONFIG_WIN32)
+    if (s->hostdevice && DEVICE(s)->id) {
+        migration_add_notifier_modes(&s->cpr_notifier, usb_host_cpr_notifier,
+                                     MIG_MODE_CPR_TRANSFER,
+                                     MIG_MODE_CPR_EXEC, -1);
+    }
+#endif
+
     error_setg(&s->cpr_blocker, "usb-host device %s does not support CPR",
                DEVICE(s)->id ?: "(anonymous)");
     if (migrate_add_blocker_modes(&s->cpr_blocker, errp,
@@ -1393,6 +1461,9 @@ static void usb_host_unrealize(USBDevice *udev)
     }
     usb_host_close(s);
     migrate_del_blocker(&s->cpr_blocker);
+    migration_remove_notifier(&s->cpr_notifier);
+    g_slist_free(s->cpr_inflight);
+    s->cpr_inflight = NULL;
 }
 
 static void usb_host_cancel_packet(USBDevice *udev, USBPacket *p)
@@ -1400,6 +1471,9 @@ static void usb_host_cancel_packet(USBDevice *udev, USBPacket *p)
     USBHostDevice *s = USB_HOST_DEVICE(udev);
     USBHostRequest *r;
 
+    /* Packet is being cancelled -> it mustn't be re-issued on failed CPR */
+    s->cpr_inflight = g_slist_remove(s->cpr_inflight, p);
+
     if (p->combined) {
         usb_combined_packet_cancel(udev, p);
         return;

-- 
2.47.1

[PATCH QEMU HCI-8.0 v3 07/12] usb-host: skip product-string read on CPR incoming #VSTOR-137800

[Andrey Drobyshev]

usb_host_open() reads the product string descriptor from the device
with a synchronous control transfer.  On the CPR target this runs while
the device is being handed over, and the transfer can block there,
hanging the incoming migration.

The string only fills the cosmetic product_desc, so skip the read when
we're the CPR target and fall back to the synthetic "host:bus.addr"
name.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/usb/host-libusb.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/hw/usb/host-libusb.c b/hw/usb/host-libusb.c
index 77ee3ca1dd7..922c1a5585e 100644
--- a/hw/usb/host-libusb.c
+++ b/hw/usb/host-libusb.c
@@ -1064,7 +1064,7 @@ static int usb_host_open(USBHostDevice *s, libusb_device *dev, int hostfd)
     udev->speed = speed_map[libusb_speed];
     usb_host_speed_compat(s);
 
-    if (s->ddesc.iProduct) {
+    if (s->ddesc.iProduct && !cpr_is_incoming()) {
         libusb_get_string_descriptor_ascii(s->dh, s->ddesc.iProduct,
                                            (unsigned char *)udev->product_desc,
                                            sizeof(udev->product_desc));

-- 
2.47.1

[PATCH QEMU HCI-8.0 v3 08/12] usb-host: hand off the device across cpr-transfer #VSTOR-137800

[Andrey Drobyshev]

Unlike cpr-exec, cpr-transfer keeps the source QEMU alive: it hands the
usbfs FD to the new QEMU over the migration channel and only exits once
the target is up.  While the handoff is in flight both processes hold the
same FD, and only one may reap URB events on it - if both do, one reaps
the other's in-flight completions and wedges an isochronous transfer that
cannot be retried.  So reaping is handed off along with the FD:

  * The target doesn't reap until it takes over: it stops events right
    after opening the FD in .realize() and restarts in .post_load().
  * The source stops reaping in .pre_save() and, since it stays alive,
    restarts from the MIG_EVENT_PRECOPY_FAILED notifier on failure.
  * On success the source's .exit() leaves the device untouched - no
    reset, interface release or host-driver rebind - and just closes its
    own copy of the FD.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/usb/host-libusb.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 83 insertions(+), 11 deletions(-)

diff --git a/hw/usb/host-libusb.c b/hw/usb/host-libusb.c
index 922c1a5585e..e29e6210fdd 100644
--- a/hw/usb/host-libusb.c
+++ b/hw/usb/host-libusb.c
@@ -110,6 +110,7 @@ struct USBHostDevice {
     NotifierWithReturn               cpr_notifier;
     GSList                           *cpr_inflight;
     bool                             cpr_drained;
+    bool                             cpr_handed_off;
     libusb_device                    *dev;
     libusb_device_handle             *dh;
     struct libusb_device_descriptor  ddesc;
@@ -1227,16 +1228,19 @@ static int usb_host_close(USBHostDevice *s)
 
     trace_usb_host_close(s->bus_num, s->addr);
 
-    usb_host_abort_xfers(s);
-    usb_host_iso_free_all(s);
+    if (!s->cpr_handed_off) {
+        usb_host_abort_xfers(s);
+        usb_host_iso_free_all(s);
 
-    if (udev->attached) {
-        usb_device_detach(udev);
+        if (udev->attached) {
+            usb_device_detach(udev);
+        }
+
+        usb_host_release_interfaces(s);
+        libusb_reset_device(s->dh);
+        usb_host_attach_kernel(s);
     }
 
-    usb_host_release_interfaces(s);
-    libusb_reset_device(s->dh);
-    usb_host_attach_kernel(s);
     libusb_close(s->dh);
     s->dh = NULL;
     s->dev = NULL;
@@ -1271,10 +1275,17 @@ static void usb_host_exit_notifier(struct Notifier *n, void *data)
     USBHostDevice *s = container_of(n, USBHostDevice, exit);
 
     if (s->dh) {
-        usb_host_abort_xfers(s);
-        usb_host_release_interfaces(s);
-        libusb_reset_device(s->dh);
-        usb_host_attach_kernel(s);
+        /*
+         * Handed to new QEMU across cpr-transfer: do not reset, release
+         * interfaces or rebind the host driver underneath it; just close
+         * our handle.
+         */
+        if (!s->cpr_handed_off) {
+            usb_host_abort_xfers(s);
+            usb_host_release_interfaces(s);
+            libusb_reset_device(s->dh);
+            usb_host_attach_kernel(s);
+        }
         libusb_close(s->dh);
     }
 }
@@ -1298,6 +1309,8 @@ static libusb_device *usb_host_find_ref(int bus, int addr)
 }
 
 static void usb_host_handle_data(USBDevice *udev, USBPacket *p);
+static void usb_host_cpr_stop_events(void);
+static void usb_host_cpr_restart_events(void);
 
 /*
  * A failed CPR migration resumes the source VM with the drained
@@ -1318,6 +1331,11 @@ static int usb_host_cpr_notifier(NotifierWithReturn *notifier,
         return 0;
     }
 
+    if (s->cpr_handed_off) {
+        usb_host_cpr_restart_events();
+        s->cpr_handed_off = false;
+    }
+
     for (it = s->cpr_inflight; it; it = it->next) {
         p = it->data;
         /* Replay from the start of the TD */
@@ -1390,6 +1408,14 @@ static void usb_host_realize(USBDevice *udev, Error **errp)
             error_setg(errp, "failed to open host usb device %s", s->hostdevice);
             return;
         }
+        if (cpr_is_incoming()) {
+            /*
+             * The source still owns the shared fd until it hands off.  Don't
+             * reap URBs on it yet, or we would steal the source's in-flight
+             * completions; .post_load() restarts events once we take over.
+             */
+            usb_host_cpr_stop_events();
+        }
     } else
 #endif
     if (s->match.addr && s->match.bus_num &&
@@ -1941,6 +1967,7 @@ static int usb_host_post_load(void *opaque, int version_id)
      * Skip it for CPR.
      */
     if (cpr_is_incoming()) {
+        usb_host_cpr_restart_events();
         return 0;
     }
 
@@ -1953,6 +1980,47 @@ static int usb_host_post_load(void *opaque, int version_id)
     return 0;
 }
 
+#ifndef CONFIG_WIN32
+
+/*
+ * cpr-transfer: the usbfs fd is shared with the new QEMU via SCM_RIGHTS.
+ * Stop this (source) process from reaping URBs on it, so the new QEMU can
+ * drive the device without both processes racing on the same fd.  Restart
+ * on migration failure, when the source resumes.
+ */
+static void usb_host_cpr_stop_events(void)
+{
+    const struct libusb_pollfd **poll = libusb_get_pollfds(ctx);
+
+    libusb_set_pollfd_notifiers(ctx, NULL, NULL, NULL);
+    if (poll) {
+        for (int i = 0; poll[i] != NULL; i++) {
+            usb_host_del_fd(poll[i]->fd, ctx);
+        }
+        free(poll);
+    }
+}
+
+static void usb_host_cpr_restart_events(void)
+{
+    const struct libusb_pollfd **poll = libusb_get_pollfds(ctx);
+
+    libusb_set_pollfd_notifiers(ctx, usb_host_add_fd, usb_host_del_fd, ctx);
+    if (poll) {
+        for (int i = 0; poll[i] != NULL; i++) {
+            usb_host_add_fd(poll[i]->fd, poll[i]->events, ctx);
+        }
+        free(poll);
+    }
+}
+
+#else
+
+static void usb_host_cpr_stop_events(void) {}
+static void usb_host_cpr_restart_events(void) {}
+
+#endif
+
 static int usb_host_pre_save(void *opaque)
 {
     USBHostDevice *s = opaque;
@@ -1964,6 +2032,10 @@ static int usb_host_pre_save(void *opaque)
         if (usb_host_cpr_drain_xfers(s) < 0) {
             return -1;
         }
+        if (mode == MIG_MODE_CPR_TRANSFER) {
+            usb_host_cpr_stop_events();
+            s->cpr_handed_off = true;
+        }
     }
     return 0;
 }

-- 
2.47.1

[PATCH QEMU HCI-8.0 v3 09/12] usb: migrate the interface altsetting #VSTOR-137800

[Andrey Drobyshev]

The active altsetting of each interface is host controller state that
the guest selected with SET_INTERFACE, but it isn't part of the
migration stream.  For most devices the target happens to reopen on the
same altsettings, so this went unnoticed.

It matters for CPR of a device whose endpoints only exist on a non-zero
altsetting, such as an isochronous streaming interface.  Migrate the
altsetting array in a subsection, gated to the CPR modes: only the CPR
target consumes it to rebuild its endpoints, while a regular migration
reopens the device and reconstructs the altsetting itself.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/usb/bus.c | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/hw/usb/bus.c b/hw/usb/bus.c
index 8dd2ce415eb..ee13acea658 100644
--- a/hw/usb/bus.c
+++ b/hw/usb/bus.c
@@ -8,6 +8,7 @@
 #include "qemu/module.h"
 #include "system/system.h"
 #include "migration/vmstate.h"
+#include "migration/cpr.h"
 #include "monitor/monitor.h"
 #include "trace.h"
 #include "qemu/cutils.h"
@@ -63,6 +64,37 @@ static int usb_device_post_load(void *opaque, int version_id)
     return 0;
 }
 
+static bool usb_altsetting_needed(void *opaque)
+{
+    USBDevice *dev = opaque;
+    int i;
+
+    /*
+     * Only the CPR target consumes this, to rebuild its endpoints; other
+     * migrations reopen the device and reconstruct the altsetting.
+     */
+    if (!cpr_incoming_needed(NULL)) {
+        return false;
+    }
+    for (i = 0; i < USB_MAX_INTERFACES; i++) {
+        if (dev->altsetting[i]) {
+            return true;
+        }
+    }
+    return false;
+}
+
+static const VMStateDescription vmstate_usb_device_altsetting = {
+    .name = "USBDevice/altsetting",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .needed = usb_altsetting_needed,
+    .fields = (const VMStateField[]) {
+        VMSTATE_INT32_ARRAY(altsetting, USBDevice, USB_MAX_INTERFACES),
+        VMSTATE_END_OF_LIST(),
+    }
+};
+
 const VMStateDescription vmstate_usb_device = {
     .name = "USBDevice",
     .version_id = 1,
@@ -77,6 +109,10 @@ const VMStateDescription vmstate_usb_device = {
         VMSTATE_INT32(setup_index, USBDevice),
         VMSTATE_UINT8_ARRAY(setup_buf, USBDevice, 8),
         VMSTATE_END_OF_LIST(),
+    },
+    .subsections = (const VMStateDescription * const []) {
+        &vmstate_usb_device_altsetting,
+        NULL
     }
 };
 

-- 
2.47.1

[PATCH QEMU HCI-8.0 v3 10/12] usb-host: reconfigure endpoints on CPR incoming #VSTOR-137800

[Andrey Drobyshev]

On CPR incoming we skip the usb_host_open() rescan, so libusb only sets
up the endpoints from the device's default configuration.  If the guest
had enabled an endpoint by selecting a non-zero altsetting (e.g. an
isochronous streaming endpoint), that endpoint is now missing, and the
guest's transfers to it simply stall.

The altsetting itself is restored by vmload now, so let's call
usb_host_ep_update() on the CPR target to rebuild the endpoints out of
it.

The kernel claims are still there on the preserved FD, but libusb's own
handle knows nothing about them, so a later SET_INTERFACE would fail with
LIBUSB_ERROR_NOT_FOUND.  Let's re-claim the interfaces on the target as
well to bring libusb's view back in sync - on the preserved FD this is a
no-op at the usbfs level anyway.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/usb/host-libusb.c | 81 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 81 insertions(+)

diff --git a/hw/usb/host-libusb.c b/hw/usb/host-libusb.c
index e29e6210fdd..a3fdde16c1a 100644
--- a/hw/usb/host-libusb.c
+++ b/hw/usb/host-libusb.c
@@ -1956,6 +1956,75 @@ static void usb_host_post_load_bh(void *opaque)
     usb_host_auto_check(NULL);
 }
 
+/*
+ * On the CPR target libusb knows nothing about the interface claims the
+ * preserved fd still holds, so a later SET_INTERFACE would fail with
+ * NOT_FOUND.  Re-claim them to bring libusb's view back in sync; on the
+ * preserved fd it's a no-op at the usbfs level anyway.
+ */
+static int usb_host_cpr_reclaim_interfaces(USBHostDevice *s)
+{
+    USBDevice *udev = USB_DEVICE(s);
+    struct libusb_config_descriptor *conf;
+    int n, rc;
+
+    rc = libusb_get_active_config_descriptor(s->dev, &conf);
+    if (rc != 0) {
+        usb_host_libusb_error("libusb_get_active_config_descriptor", rc);
+        return -1;
+    }
+
+    n = MIN(conf->bNumInterfaces, USB_MAX_INTERFACES);
+    for (int i = 0; i < n; i++) {
+        rc = libusb_claim_interface(s->dh, i);
+        if (rc != 0) {
+            usb_host_libusb_error("libusb_claim_interface", rc);
+            libusb_free_config_descriptor(conf);
+            return -1;
+        }
+        s->ifs[i].claimed = true;
+    }
+
+    udev->ninterfaces = conf->bNumInterfaces;
+    udev->configuration = conf->bConfigurationValue;
+    libusb_free_config_descriptor(conf);
+
+    return 0;
+}
+
+/*
+ * During CPR the altsetting array is included in the migration stream.
+ * Reject a value this device can't honour, so that usb_host_ep_update()
+ * on the target never sees an out of range altsetting.  In this case
+ * we fail the incoming migration instead of aborting on it.
+ */
+static int usb_host_cpr_check_altsettings(USBHostDevice *s)
+{
+    USBDevice *udev = USB_DEVICE(s);
+    struct libusb_config_descriptor *conf;
+
+    if (libusb_get_active_config_descriptor(s->dev, &conf) != 0) {
+        error_report("usb-host: cannot read config descriptor on "
+                     "CPR incoming");
+        return -1;
+    }
+
+    for (int i = 0; i < conf->bNumInterfaces; i++) {
+        int ifnum = conf->interface[i].altsetting[0].bInterfaceNumber;
+        int alt = udev->altsetting[ifnum];
+
+        if (alt != 0 && alt >= conf->interface[i].num_altsetting) {
+            error_report("usb-host: migrated altsetting %d out of range for "
+                         "interface %d", alt, ifnum);
+            libusb_free_config_descriptor(conf);
+            return -1;
+        }
+    }
+
+    libusb_free_config_descriptor(conf);
+    return 0;
+}
+
 static int usb_host_post_load(void *opaque, int version_id)
 {
     USBHostDevice *dev = opaque;
@@ -1967,6 +2036,18 @@ static int usb_host_post_load(void *opaque, int version_id)
      * Skip it for CPR.
      */
     if (cpr_is_incoming()) {
+        /*
+         * We kept the device open across CPR, so libusb still needs its
+         * interface claims and endpoints rebuilt here.  And we only start
+         * reaping now, once the source has handed the fd off to us.
+         */
+        if (usb_host_cpr_check_altsettings(dev) < 0) {
+            return -1;
+        }
+        if (usb_host_cpr_reclaim_interfaces(dev) < 0) {
+            return -1;
+        }
+        usb_host_ep_update(dev);
         usb_host_cpr_restart_events();
         return 0;
     }

-- 
2.47.1

[PATCH QEMU HCI-8.0 v3 11/12] hcd-xhci: resync isochronous endpoints after CPR #VSTOR-137800

[Andrey Drobyshev]

An isochronous endpoint has no retransmission, so the transfer events
the controller reports have to line up with the TDs the guest driver is
sitting on.  Across CPR we rebuild the controller's transient transfer
and scheduling state out of guest memory, and it no longer matches what
the guest has already consumed - so the very first event after resume
lands on the wrong TD and the guest wedges with "not part of TD".

Let's report CC_MISSED_SERVICE_ERROR on the first isochronous transfer
after a CPR resume, just like a real xHCI does when it misses a service
interval.  That arms the guest driver's skip logic, and it resyncs to
the live stream instead of getting stuck.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/usb/hcd-xhci.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/hw/usb/hcd-xhci.c b/hw/usb/hcd-xhci.c
index 292c378bfc9..0370c784627 100644
--- a/hw/usb/hcd-xhci.c
+++ b/hw/usb/hcd-xhci.c
@@ -25,6 +25,7 @@
 #include "qemu/module.h"
 #include "qemu/queue.h"
 #include "migration/vmstate.h"
+#include "migration/cpr.h"
 #include "hw/qdev-properties.h"
 #include "trace.h"
 #include "qapi/error.h"
@@ -282,6 +283,7 @@ struct XHCIEPContext {
     unsigned int max_psize;
     uint32_t state;
     uint32_t kick_active;
+    bool cpr_iso_resync; /* resync iso stream on first kick after CPR resume */
 
     /* streams */
     unsigned int max_pstreams;
@@ -1811,6 +1813,19 @@ static int xhci_submit(XHCIState *xhci, XHCITransfer *xfer, XHCIEPContext *epctx
         xfer->pkts = 1;
         xfer->iso_xfer = true;
         xfer->timed_xfer = true;
+        if (epctx->cpr_iso_resync) {
+            /*
+             * First iso TD after a CPR resume: our transfer state was
+             * rebuilt from guest memory and no longer lines up with what
+             * the guest consumed.  Report a missed service, like real HW,
+             * to kick the driver's skip logic into resyncing.
+             */
+            epctx->cpr_iso_resync = false;
+            xfer->status = CC_MISSED_SERVICE_ERROR;
+            xhci_xfer_report(xfer);
+            xfer->complete = 1;
+            return 0;
+        }
         mfindex = xhci_mfindex_get(xhci);
         xhci_calc_iso_kick(xhci, xfer, epctx, mfindex);
         xhci_check_intr_iso_kick(xhci, xfer, epctx, mfindex);
@@ -3510,6 +3525,11 @@ static int usb_xhci_post_load(void *opaque, int version_id)
             xhci_init_epctx(epctx, pctx, ep_ctx);
             epctx->state = state;
             if (state == EP_RUNNING) {
+                if (cpr_is_incoming() &&
+                    (epctx->type == ET_ISO_IN || epctx->type == ET_ISO_OUT)) {
+                    /* only iso streams need resyncing; see xhci_submit() */
+                    epctx->cpr_iso_resync = true;
+                }
                 /* kick endpoint after vmload is finished */
                 timer_mod(epctx->kick_timer, qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL));
             }

-- 
2.47.1

[PATCH QEMU HCI-8.0 v3 12/12] usb-host: make CPR migration blocker conditional #VSTOR-137800

[Andrey Drobyshev]

A previous commit added an unconditional blocker for CPR migrations.
That is to refuse CPR gracefully instead of crashing and for the previous
commits to remain bisectable.

Now the restore path is complete: the FD is preserved, in-flight URBs
are drained and replayed, the device is handed off across cpr-transfer,
and the endpoints and isochronous streams are reconstructed on the
target.  So let's lift the CPR blocker for the devices which satisfy the
preconditions.

Still, the remaining preconditions for CPR to work are:

  * We must open host device FD ourselves, so that libusb owns the FD.
    This way we're able to preserve the FD during CPR;
  * Device ID must be provided as a unique stable key for CPR FD registry.

Use them to guard the blocker addition.  Apart from that, prohibit CPR on
WIN32 platform and on hosts with older libusb versions.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/usb/host-libusb.c | 50 +++++++++++++++++++++++++++++++++++---------------
 1 file changed, 35 insertions(+), 15 deletions(-)

diff --git a/hw/usb/host-libusb.c b/hw/usb/host-libusb.c
index a3fdde16c1a..578262518de 100644
--- a/hw/usb/host-libusb.c
+++ b/hw/usb/host-libusb.c
@@ -1446,24 +1446,44 @@ static void usb_host_realize(USBDevice *udev, Error **errp)
     s->exit.notify = usb_host_exit_notifier;
     qemu_add_exit_notifier(&s->exit);
 
-#if LIBUSB_API_VERSION >= 0x01000107 && !defined(CONFIG_WIN32)
-    if (s->hostdevice && DEVICE(s)->id) {
-        migration_add_notifier_modes(&s->cpr_notifier, usb_host_cpr_notifier,
-                                     MIG_MODE_CPR_TRANSFER,
-                                     MIG_MODE_CPR_EXEC, -1);
+#ifdef CONFIG_WIN32
+    error_setg(&s->cpr_blocker,
+               "usb-host device %s does not support CPR: "
+               "WIN32 platform is not supported",
+               DEVICE(s)->id ?: "(anonymous)");
+#elif LIBUSB_API_VERSION < 0x01000107
+    error_setg(&s->cpr_blocker,
+               "usb-host device %s does not support CPR: "
+               "libusb API version 0x%08x is too old",
+               DEVICE(s)->id ?: "(anonymous)", (unsigned) LIBUSB_API_VERSION);
+#else
+    if (!(s->hostdevice && DEVICE(s)->id)) {
+        /*
+         * Host FD must be owned by libusb to support CPR migration.  That's
+         * equivalent to hostdevice= property being present.   Also device ID
+         * is required as stable key for CPR FD registry.
+         */
+        error_setg(&s->cpr_blocker,
+                   "usb-host device %s does not support CPR: "
+                   "hostdevice= and a device id are required",
+                   DEVICE(s)->id ?: "(anonymous)");
     }
 #endif
 
-    error_setg(&s->cpr_blocker, "usb-host device %s does not support CPR",
-               DEVICE(s)->id ?: "(anonymous)");
-    if (migrate_add_blocker_modes(&s->cpr_blocker, errp,
-                                  MIG_MODE_CPR_TRANSFER,
-                                  MIG_MODE_CPR_EXEC, -1) < 0) {
-        qemu_remove_exit_notifier(&s->exit);
-        if (s->needs_autoscan) {
-            QTAILQ_REMOVE(&hostdevs, s, next);
-        }
-        usb_host_close(s);
+    if (s->cpr_blocker) {
+        if (migrate_add_blocker_modes(&s->cpr_blocker, errp,
+                                      MIG_MODE_CPR_TRANSFER,
+                                      MIG_MODE_CPR_EXEC, -1) < 0) {
+            qemu_remove_exit_notifier(&s->exit);
+            if (s->needs_autoscan) {
+                QTAILQ_REMOVE(&hostdevs, s, next);
+            }
+            usb_host_close(s);
+        }
+    } else {
+        migration_add_notifier_modes(&s->cpr_notifier, usb_host_cpr_notifier,
+                                     MIG_MODE_CPR_TRANSFER,
+                                     MIG_MODE_CPR_EXEC, -1);
     }
 }
 

-- 
2.47.1