From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from DUZPR83CU001.outbound.protection.outlook.com (mail-northeuropeazon11022130.outbound.protection.outlook.com [52.101.66.130]) by lore.virtuozzo.com (Postfix) with ESMTPS id C835A81377 for ; Mon, 9 Nov 2026 12:27:02 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=tpk3PWfEXxwT+E61ilsutNpKgUboB1yxs/dVLfOmNJ7QmQGY5i2tygrE6vnCVye1Nr8gWZVEtWUVp/PKDlnWkzY91ZKNqgGrWjisF4jpSnx54xAgIfdAODEGURexqm4gOvDgfFeqNxyOqwatd6ocdFzNzTfM49RBGVDOfOpy+u1T2wNS8X2yXmGPRumfxO26eO9v3t0IMV+TMjlCNboRe0VYHmmWJzfJdqierMIWdYHlsPV92oCZB9Dr8/3FTUxual7uYlovnZ5Sim6q/0whbKw9lPobrQEvBFSTz4Nq4+tvNk2uJfRqXME4eSHlLI7V/rB1ZMTZn8VqdCfUbMnR1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=LYIc7I9Wp1YGqVflL+YLbIgRUqy1XWt3AIuKUbZuQi0=; b=enjmi/UilV6NgQtvEHbXAbjwWhqbz58knq1cyf7yBWNK9KnzAgq46rGMVssNDWyEXSl/sZfaf04svw+Snq3AhRWKbp2KEx8/3df6bKOBw0BI2TByxQfH5h9MLNT2HYVgWkd2ZvCvNAUsNOK66mzTm1vcouGaAbsodPWZk8P6rceORZI6MXgMKkkAjrB1amKaf4P7MElfluRYMRMEs04bq4coU97CbgwHCtQVSf3v3B2RL3e0Nd7W4yDvEwvJV3YEfaL91osz6i3KwFD1GrvkoQFUFV5KzajqYYBE+yJwBYQHNGCyXcD61V8NeEyYUFNlDDgUBj4uaQ+OPuj7UnXVww== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LYIc7I9Wp1YGqVflL+YLbIgRUqy1XWt3AIuKUbZuQi0=; b=OwIVhWAo1p86iS1oZgE8QfLEn6f5wMFIAen1BEjerS6xpMi+zgLwTgB6udoKR0KX/hkXNzejU1kA+dlKR1CyBJOABybyiSk62SwG791KbzA3LTJjmBVWWZT+oPSCfE+Au5ZCGOokC1RtO/BWLNANTinTkZE0/K18gsjzWWW8nVzeFpLQdx6acz7xRnGPwx60ewDaL4pSTfJBzVNNnMEetapX6vIE1rywHVkun4ZSJE0+yRlVG+UHh4Ovqt42ZktX2i7LiCV7PXGEiLKq2GxkJPVaUpHb8ipKYUwXZtCLP5OGuEo0OLpADL39q8OAMIwCCnNdqaj7ueUuhNlgeiUiUw== Authentication-Results: mx.microsoft.com 1; dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=virtuozzo.com; Received: from AM9PR08MB5892.eurprd08.prod.outlook.com (2603:10a6:20b:2dd::16) by DU0PR08MB9131.eurprd08.prod.outlook.com (2603:10a6:10:470::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.592.12; Mon, 9 Nov 2026 12:27:00 +0000 Received: from AM9PR08MB5892.eurprd08.prod.outlook.com ([fe80::94bb:633f:1f55:4bbd]) by AM9PR08MB5892.eurprd08.prod.outlook.com ([fe80::94bb:633f:1f55:4bbd%5]) with mapi id 15.21.0592.010; Mon, 9 Nov 2026 12:26:58 +0000 Message-ID: Date: Mon, 9 Nov 2026 13:26:57 +0100 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH hci-vev v2 1/1] vfio/migration: don't touch the device on guest shutdown #VSTOR-153976 To: Andrey Drobyshev , svt-core@virtuozzo.com Cc: den@openvz.org References: <20261108205050.37277-1-andrey.drobyshev@virtuozzo.com> <20261108205050.37277-2-andrey.drobyshev@virtuozzo.com> Content-Language: en-US From: "Denis V. Lunev" In-Reply-To: <20261108205050.37277-2-andrey.drobyshev@virtuozzo.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-ClientProxiedBy: VIUP296CA0003.AUTP296.PROD.OUTLOOK.COM (2603:10a6:800:34f::6) To AM9PR08MB5892.eurprd08.prod.outlook.com (2603:10a6:20b:2dd::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AM9PR08MB5892:EE_|DU0PR08MB9131:EE_ X-MS-Office365-Filtering-Correlation-Id: d2f72683-de21-4838-12ff-08df3e5cbf8d X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr List-Id: svt-core@virtuozzo.com X-MS-Exchange-Transport-Rules-Loop: 1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|23010399003|376014|1800799024|10070799003|18002099003|22082099003|5113699003|10067099003|4143699003|5023799004|261028144727099003|261102123227099003|261022105027099003|56012099006|261021121427099003|261026121527099003|261026140227099003|261009223027099003; X-Microsoft-Antispam-Message-Info: UYXDdqMKYbPN4pIkCzzLAz37KDE+n1K8OapJvcXjnN9Ut0DqzYGoduK7LdcIPoH2KdS/2GG8k3gOL5qYMFR6vz5PtgW1FBvQu1umXwURBZPfMWaD3bOTiuVuuXOfZFa3+MsNIQK0h0Ta3ng6w+brzC3DCUCu8mYJC7c7Qgf0fB38SMbNSF+IOTLDBKVYcTc3UlQlUKJGacAyFh7tGMiTlh7Udynt8wNWaeCn2I5c5fYRvYBZEQrBWEBTSWbuG/I1udGO7BNyMZb48+QpkTnP8Nml6N+ndUg0xLkF4zp/IlgtCcUamPLU0LkzoUeG6anIpO4aNbQP1zE1GssTdEKtKR1nNqrlyySERO6RIodELOliFlu1YThRjnaI2d0utMCgtGht3icUnhbhJlai/lCiwppIUQBgsbYwEt7oRZqeNxpda6zgFTqcoz09ILZhSjK9is9GfURW4W0U2CWfEj3+4O4xaALpctE60Lvf2tBCJl78KAIL4nFEEJ4OO/Xx+7MN70gmYjZGADzqfnPW9T5/yoSc6rFJoX1An3XU4Q86OKR2hjXHc25jI8Ul9b62FIg6I/lUCPExUyQ1w6ymFFBldpTGGQMbGuovbP0ze5xpex8LMDZd0aiAj1RkGjMbeHq/iTFN5HAgsgrI5FNTpEPI8uKSQ4b/xxrIcqhzTUeqQ5c= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AM9PR08MB5892.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(23010399003)(376014)(1800799024)(10070799003)(18002099003)(22082099003)(5113699003)(10067099003)(4143699003)(5023799004)(261028144727099003)(261102123227099003)(261022105027099003)(56012099006)(261021121427099003)(261026121527099003)(261026140227099003)(261009223027099003);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 2 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?K2dYV0RSR0VaK3pOa09oNFdSZ3Z4OU53MjRBY1JqR2p2SjcwalVySlZ4a1Nm?= =?utf-8?B?WUJ4eXdodjR5ZWhwcFNrbFM0VUV3YjF6M0dsWjVteUZrZVlHbklCbHFLbXVv?= =?utf-8?B?S2pTRVpmR3BKbUxpdkk4S0cvUnV0MnlpUU1KcktsMUl3VFFRYnhCckpieCtr?= =?utf-8?B?NHRpaFFiUEw1eUFsWXNvNy9QVEVVUGYyVzlEa2hlMU16aEtVYnBmSnlxTER1?= =?utf-8?B?K3l2Z3Z6bjJzSlJuaVNxQjVNcnpRVUFpRzdNVmV6MnMrVGtlZ3hCZTMxeUY2?= =?utf-8?B?czhVUENnVVNlTmx0Z2lTWWdpVVlLSThmWjBYN2hxL0oveFQrRS9KT3R0NUJ1?= =?utf-8?B?ZUdsYTI5WlZnNWdlM21HZEtjK0h6TmdVbGUramZuclYxODJsL0RMd2xkdDZL?= =?utf-8?B?R2NCMG1ja1VXLytRT3k3NVQvSzhNUDdoVzE4bDRvWUZZR0hRM21lRWdMUmlw?= =?utf-8?B?UEI4WVZRMU56cXNDZmZ6aXc0bFlIeVJ6WThtMDhRbXl3WkxXNzdpcGlzSEJN?= =?utf-8?B?eHBUcE1oVXJqT3Nod3lIeGRDVU9YK255WWkzNjQzV0dBS0tIcG9qNFRXZjlp?= =?utf-8?B?c0dtSGFmUnpPamFHUVM5SDdWZHJndUtYQnlua3dPcnFqQ0VLZm4yWSt6dmpD?= =?utf-8?B?MGFTUW5UM1kxQ2U4UlA0VUJubmQyOXJBanZJeVJXc3d6UVlGelVjcmwvdHEy?= =?utf-8?B?UVRHN0VOeEUvTU9QZmFIbjN2RERMS3JEZHRPb2NXR0hhWkNkeTFSOG5CMXRD?= =?utf-8?B?REtpRXVpaFVDN0pPV2lkbGhMa2xFdWNJMjB2Mk5RMHlEcWEwbnhLdHUzNGQ5?= =?utf-8?B?QlhSYUR5dklBR1Erb3FqbXplWldNQjZON1kwRmE3RHU2SFlOYWxhWWx5bkt6?= =?utf-8?B?ZFU4T3h0V1JZWmdMUHRkd0hldDRMUjhuRFBJbE4yNGd6NVJZMW91aFV0UW56?= =?utf-8?B?b0JRUzZ6SmRyM21jVFRRY09lWVk4V2Nra3hPbmdnWlNXbWk0b0pzb3Rpc0hQ?= =?utf-8?B?OVJSa0o5bE9ta1oyK1lKdzZSekNnQ3dUeHJUbEE4SW8yRDRHVHpTMDFsNUxq?= =?utf-8?B?RTBneWZYaGg3djl2TXJQMkNKcFQwMHNKUlpWYUdtTzl1UTNoVHBObkU4S1Fy?= =?utf-8?B?UXh0Y0FiSkFYc0hnaFlObjRFby9aQmJSd1QwemRFcGpna056ejcvZFAveXNU?= =?utf-8?B?Z1Y5NU5xWGRlRjNtZTBybHRQSXh2RnlqcGQzVU5PUkpkelVMQUlWajk4U21w?= =?utf-8?B?a1d0cTRaczNEU2pHSzRTLzVmTGZFNkwvR3Q1azR1NHNReVFKL2dNbWdMU2xa?= =?utf-8?B?ZzBMekFxNEp1WlNRZ1duK3lHUHNhZVJwemM2SzNCMVRsWTdoNHpyV3pyMXNC?= =?utf-8?B?NHlSU2JtUjBKZFh2eHg0SUp0V0U2ek9aSmszS3RHWkFtN3FwOWNRUkdQS210?= =?utf-8?B?YzErS2JEcW02UlZldS9RSXJ3emVESm5IWGVmUG94RTZhaXEvUVNzWTVNUGpz?= =?utf-8?B?R1NBNFg0SlNVMlR3VjRXU1dtOEV2UGYrR3ZLaWZ6d1hVMU9ZTkVVWDJJMm9p?= =?utf-8?B?L3VBa09QNTEvQTJIWE1jRlZTSVNTRE54dzVDNWU1RDE0MlpCdFNjMHpyelB4?= =?utf-8?B?QlczclVIZmhEYkhtQWgvOFNxN0hoWElTb0Q2SWZta0MzbTc3cjJFMDdIOElK?= =?utf-8?B?MFhHay9nWXBwTlYyMG9lMHpwVGRseWpSV3IwdzlzL0pkbVJVTjRXUDlxRDJR?= =?utf-8?B?YUNyV2RlRnVIdzFOamx3d2dHNDUrUmRSZnc0SU9PVHhkVDk0SHByOUlCeHF5?= =?utf-8?B?djc4MDh1MnN3UzlOcjBiem41elZuc0IxZS9DWDlTY200d0xWbnVGSWk4Ukh2?= =?utf-8?B?dlZ4akQwaHhUYzkycmczQ1pjUk05Z0dsaHRxSVEwRGc0bUhKUktBWld5N2lv?= =?utf-8?B?ckFEOFozSDIvdDZJL0tRY1B4dkg2WVNTQ2dNcFdBOEtRNy9vNTFld2VZaWJR?= =?utf-8?B?L1R3QnNkQURxeXIzbGhOb1p0WXJVZzJncmlwRlVHUDdDVGNOWGFmT1RQSTA3?= =?utf-8?B?NkwwYmpTdFN3bmdJUFdJZmJsZU0wbG5Bek5KOWlwV0ZLY1lxbG1UVVplUnZO?= =?utf-8?B?cnZ6Wi8zTEhNckVNamRSRUxYNWd0YS9uRXI3NFdIbUVGRmlaZFE1R1RIbStH?= =?utf-8?B?M1cxSTVMRk9QTktwSk16RnlDUDhZN2QzU1dqVzVITkRIMTQ4WWZLZVJJZ0hr?= =?utf-8?B?a2QxMGhjaEFpQ0V3azlLN1JTTXJUVENVR2N5b0NZNjhXNW9ONHBTYjMrakJJ?= =?utf-8?B?eFhnVnZMOUU3N01IS3hyL2wxaGdVU0ZndmlJS295OVQ5ckZ5cVdxVkJITmpW?= =?utf-8?Q?snteUGzEv9mOSJ6RFGxKhvcPBsXMFY0OD5qaLHM2hskFO?= X-MS-Exchange-AntiSpam-MessageData-1: ESgbktZEFleG4g== X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: d2f72683-de21-4838-12ff-08df3e5cbf8d X-MS-Exchange-CrossTenant-AuthSource: AM9PR08MB5892.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Nov 2026 12:26:58.2994 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: lwgDbZC47a5oqTdw2kamDOC6iRIMU6SHDXTT4Okb/XFz2nIg/JBnvlJm+vwSmhHD5TkP6z3y5ftGMUmsNrbKaQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR08MB9131 On 11/8/26 21:50, Andrey Drobyshev wrote: > When the guest shuts down and QEMU runs with -no-shutdown, the VM stays > stopped and can only be reset or quit. A libvirt soft reboot is exactly > that: guest shutdown, system_reset, cont. Today vfio_vmstate_change() > treats this stop like any other and moves the device to STOP, then cont > brings it back to RUNNING. > > NVIDIA's vGPU manager (595.71.03) takes that STOP for the start of a > migration, finds no guest driver, breaks its dirty page tracking and > never repairs it when the driver loads again. Every live migration of > the VM after a soft reboot then fails: "Error saving page in pipelined > mode". > > Do not touch the device state in this case. There is nothing to > quiesce, and the reset that must follow resets the device anyway. The > case is told apart by the shutdown action: only "pause" (-no-shutdown) > keeps the VM alive. quit, a signal and a plain guest shutdown use > "poweroff" and still stop the device on the way out, as before. If a > migration or savevm is started on the stopped VM, no VM state change > comes at switchover, so vfio_save_setup() stops the device itself. > > With this patch applied, soft reboot + subsequent live migration pass > successfully. > > Signed-off-by: Andrey Drobyshev > --- > hw/vfio/migration.c | 30 ++++++++++++++++++++++++++++++ > include/system/runstate.h | 1 + > system/runstate.c | 5 +++++ > 3 files changed, 36 insertions(+) > > diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c > index 7e7fb496bbe..2ff3e49d5de 100644 > --- a/hw/vfio/migration.c > +++ b/hw/vfio/migration.c > @@ -473,6 +473,19 @@ static int vfio_save_setup(QEMUFile *f, void *opaque, Error **errp) > migration->event_save_iterate_started = false; > migration->event_precopy_empty_hit = false; > > + /* > + * A VM that is already stopped gets no VM state change at switchover. > + * If a guest shutdown left the device running, stop it now. > + */ > + if (!runstate_is_running() && > + migration->device_state == VFIO_DEVICE_STATE_RUNNING) { > + ret = vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_STOP, > + VFIO_DEVICE_STATE_RUNNING, errp); > + if (ret) { > + return ret; > + } > + } > + > if (vfio_precopy_supported(vbasedev)) { > switch (migration->device_state) { > case VFIO_DEVICE_STATE_RUNNING: > @@ -899,6 +912,12 @@ static void vfio_vmstate_change_prepare(void *opaque, bool running, > return; > } > > + /* The guest has shut down and QEMU keeps the VM stopped: leave it */ > + if (!running && state == RUN_STATE_SHUTDOWN && > + qemu_shutdown_action_is_pause()) { > + return; > + } > + > new_state = migration->device_state == VFIO_DEVICE_STATE_PRE_COPY ? > VFIO_DEVICE_STATE_PRE_COPY_P2P : > VFIO_DEVICE_STATE_RUNNING_P2P; > @@ -936,6 +955,17 @@ static void vfio_vmstate_change(void *opaque, bool running, RunState state) > return; > } > > + /* > + * The guest has shut down and QEMU keeps the VM stopped. It can only > + * be reset or quit now, so leave the device state as it is: the reset > + * resets the device too. The exit path uses the same run state with > + * the poweroff action and stops the device as before. > + */ > + if (!running && state == RUN_STATE_SHUTDOWN && > + qemu_shutdown_action_is_pause()) { > + return; > + } > + > if (running) { > new_state = VFIO_DEVICE_STATE_RUNNING; > } else { > diff --git a/include/system/runstate.h b/include/system/runstate.h > index 929379adae4..29a919348da 100644 > --- a/include/system/runstate.h > +++ b/include/system/runstate.h > @@ -148,6 +148,7 @@ void qemu_system_vmstop_request_prepare(void); > bool qemu_vmstop_requested(RunState *r); > ShutdownCause qemu_shutdown_requested_get(void); > bool qemu_force_shutdown_requested(void); > +bool qemu_shutdown_action_is_pause(void); > ShutdownCause qemu_reset_requested_get(void); > void qemu_system_killed(int signal, pid_t pid); > void qemu_system_reset(ShutdownCause reason); > diff --git a/system/runstate.c b/system/runstate.c > index 6178b0091a3..1cdecde0fcf 100644 > --- a/system/runstate.c > +++ b/system/runstate.c > @@ -433,6 +433,11 @@ bool qemu_force_shutdown_requested(void) > return force_shutdown; > } > > +bool qemu_shutdown_action_is_pause(void) > +{ > + return shutdown_action == SHUTDOWN_ACTION_PAUSE; > +} > + > ShutdownCause qemu_reset_requested_get(void) > { > return reset_requested; Reviewed-by: Denis V. Lunev