From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from GVXPR05CU001.outbound.protection.outlook.com (mail-swedencentralazon11023112.outbound.protection.outlook.com [52.101.83.112]) by lore.virtuozzo.com (Postfix) with ESMTPS id 2FF1481377 for ; Mon, 9 Nov 2026 15:45:08 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ATKEEIjgsVmqnc025NWW7pIR0H0wf/RSQGVvQZCwIIPjYEAzY2rDqlE0wx5HVrwF6Ppxt31+Ab0AaVBDBUihyFg0ysc+dIVR+pmrof49taygbekOL1z4cxPqLFGVypk/aep3tYRv3dxyFrkG8HzoPJn69WdbCuX53WfKaJ5KsuscY1uYJ2G3EMhsVIiJWqVXsOT8zhrra+T0kxj0pLSC1nhGBlmihDEenjkVa2J9WZphe/5c0e85K/CnaKvZxcWW9uGn2ZEXqt6sxMozyi8A7KW3ToIi1FRvWohwOWaQadILcd9r27oDeC5/UV9FxGBn6+7eqCLBAXFIUq9VPSVCDQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=EKvgb0L48jIzc1JLucJAyQM/evB/jPFzzr4NCMYcido=; b=oZ3fyxgHmFtMgsBj0zYLuO1eA7p3xL2Ct+7PEkL/0PfESaHnvfE1UZbHT44fIW51mSokjpstnK3JoF2t6qZ186LetZzBDHY99xMO6IwWYkz9NRotjpBqaLvZIs15nHAG0j1h4LXh7LClqCF85SVMu5Zac/yG8sb5ykXEaSoufdkQATjMTxm6btlsuN5q9mf5tjrKP3SbEabcFJ9ubTOmbBZRlPSxdjvLtpB9Al+w5dSQeFXJTQK1OXnuf1KfObd5sfmaKcvz0TDa7YBoh+AQc/mZ5n3WiNUBekexIW4pXALGiBsCEjvvtkrBD8OiBHxTe8SH3qp7MoKhwN6agAM1jA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=EKvgb0L48jIzc1JLucJAyQM/evB/jPFzzr4NCMYcido=; b=yOZ6YJaiYpbgJRFiWgAT9ot759buDY8ibUXDeuJC1dHW03PnVppDuNSvNIcL56tjloXWYGxpn19XU7mpDcLCubXXs5oKLQxMX5TgbHXt6YbVxuZjSJZKLhgcM+SyONjrrB/b8cRdwEDeZJzHBBsJxidYlxJdkQ/3z9GFVq+FaIjC+hITVa2o7S7CZCFAVY+TFbhUnAI5dpBvlQaMzwc2EjY+lgrLabFvBJYlMDgm9+JM2fRcKEWQ6cN3q99GCX7CQQ6DXj8MYdGQ4JL0xIAmuweQ7PNJxeaAq9YaAxgjvBgTAPjvPfUU5Vm83s/mL47ya9CaMEf2kvoHcmrOeM/vSg== Authentication-Results: mx.microsoft.com 1; dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=virtuozzo.com; Received: from VI0PR08MB10656.eurprd08.prod.outlook.com (2603:10a6:800:20a::12) by PA6PR08MB10444.eurprd08.prod.outlook.com (2603:10a6:102:3c7::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.592.12; Mon, 9 Nov 2026 15:45:04 +0000 Received: from VI0PR08MB10656.eurprd08.prod.outlook.com ([fe80::4e37:b189:ddcd:3dd8]) by VI0PR08MB10656.eurprd08.prod.outlook.com ([fe80::4e37:b189:ddcd:3dd8%4]) with mapi id 15.21.0592.010; Mon, 9 Nov 2026 15:45:04 +0000 Message-ID: Date: Mon, 9 Nov 2026 17:45:03 +0200 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH hci-vev v2 1/1] vfio/migration: don't touch the device on guest shutdown #VSTOR-153976 To: "Denis V. Lunev" , svt-core@virtuozzo.com Cc: den@openvz.org References: <20261108205050.37277-1-andrey.drobyshev@virtuozzo.com> <20261108205050.37277-2-andrey.drobyshev@virtuozzo.com> Content-Language: en-US From: Andrey Drobyshev In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-ClientProxiedBy: ZR1PEPF0000779F.CHEP278.PROD.OUTLOOK.COM (2603:10a6:918::431) To VI0PR08MB10656.eurprd08.prod.outlook.com (2603:10a6:800:20a::12) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: VI0PR08MB10656:EE_|PA6PR08MB10444:EE_ X-MS-Office365-Filtering-Correlation-Id: 09e3f5e4-585c-42d2-578d-08df3e786c80 X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr List-Id: svt-core@virtuozzo.com X-MS-Exchange-Transport-Rules-Loop: 1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|1800799024|23010399003|376014|5023799004|4143699003|261021121427099003|56012099006|261022105027099003|261009223027099003|261026140227099003|261028144727099003|261102123227099003|261026121527099003|10067099003|18002099003|22082099003|5113699003; X-Microsoft-Antispam-Message-Info: 2tcr8OIAwc6JCFuMgpD/k06rlymuh4rLX8vTqsz6GR0+ZvHMEIO6Rdv1PFxTj0Bhmqku7U335pJ2tI0ZswLVT4cTVFyL/be7MVYPzWhBjiayZrFR1SICKKpX+OphACRoneceVLzRvI3FvfHnrlVGW2CJmhLymGxEv21PWfPYgH9+je4QRdmdztpfpxATsN2+3npzwZwUMsJ0T4SdyOwmr4uD3O7gLUK3NRCsjSuW6SDk2s9zQrN22XOkkHVEcVOycq5+MTenwWuijAOErE0PxVqIdT6gjND37DGkoTEoleLv6H0BwjQJcagHVayFbglBgmuUhSXNuNuvQdoMWt1SjP3BhkBA6AAxbO8oeMk8aw48svxHCXGLx91UWZZ5WzYjLDNnTJpja703ONoxiGn5wniBsC6V4PDnlL49FBtbJn1sHBg0Y0qxzIVPyD3RJthuFUVrcZ7svb5BNfsaOMNSOHRg9g/4X3qVuM5BpYlEtAJr8B18z9lMl3x29wyx7++LqmUYEc4QkSQDInEpURFBqOjmuXR4ZpH3WvnniJFIa5m4uJrujcbjR6YRb/AhSmmoBC6Bq1H9Ks4Epfl1z7q9pdHJRKlJi/CMvsSz7O9G/pMGUFTN5CyQgZM8kDbm7G8hexLKlb8dg1zFLofwHq4OorLcBKVyCH3r4/MH2jBsJow= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI0PR08MB10656.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(1800799024)(23010399003)(376014)(5023799004)(4143699003)(261021121427099003)(56012099006)(261022105027099003)(261009223027099003)(261026140227099003)(261028144727099003)(261102123227099003)(261026121527099003)(10067099003)(18002099003)(22082099003)(5113699003);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?N0VZU1Zja1h4T2lGR1R0a09uWkV3SHQ1T1d3VGI5RGkzMjFjNm56NWRRckVi?= =?utf-8?B?MHhJZ3hncUhybDJxSkNSb2VrUklPRUJHU202UUtraEd5ZjBHR29QT0VvWVBR?= =?utf-8?B?azdmZlhuQ0Zsa0s0YVZlYXVHTHZ6QkhOd2I0ZC82RUFFdGV3ZndVODd4cUlS?= =?utf-8?B?ME5HcWp3b0Jsell5Y1VWaVlhQ1BpVzZpeGU1N2NBcWZncjlVaVRFWHN4enA4?= =?utf-8?B?Z3duMGdrK0dqNDVZZXJNMVorSzRHa1J0aTRPRzBORGJBMHliakJFaHp1bWgr?= =?utf-8?B?NHQxaExseEJsMEVPanZBb1FIMmYxd2xlWHYrczlEeEVxZGg2M0NnblVPQ01u?= =?utf-8?B?a21FcWVvNGQ1S256WFhlR0txZ0ZKNG1KWnJGZ2NqL2xmV3lweno4ME81bXpD?= =?utf-8?B?cUtVYjhSeG9tbXRDbDJNU09nZ0dXeTZrYy9PdFFmN3ZCTm0zQUxJekplRnVq?= =?utf-8?B?eUh3a2JsYktpekJPdjlON2RBSkNYMGMzSWY5b3JKUWx1b2cvQ1dOa2dXRTJ4?= =?utf-8?B?UTVRdFNzUVEweW5KRXIxWUFXYmIvMkdCakk1NlBTNWRpb1Jvd2hHM0ZMbnNz?= =?utf-8?B?QktXTFBUNUprcTk0V2l1L1p4N2w0V2VZUVlLVDR1dTB4dmpEZ0hCQ3VjdEYx?= =?utf-8?B?Slg5dnNQVFBLczE2UUJtdTZpWFpHU1RxSC9JRkQvUzdWYjFBSG42YmNRSThW?= =?utf-8?B?cWtFb1dHdXVjYTl2QjF4enVGUlVNejE3M05HRkhTTUhhdjNHUWEzQk1VMnRq?= =?utf-8?B?cXc1ckxnTEp2OWtJUzVGeUpFazBkQXRkNVJTZXB4Tyt5elFMUW03NjYvNUMy?= =?utf-8?B?UlFjQ3FOWitTakwvOUE1c0J0UHUwMTFDaVJhQWVVQlhHTXAremtMZUpSeFM0?= =?utf-8?B?ejdJSEl6Sm1sY3NuaGMwSnZ5ZTU1QmJmY2xGeDdZR08vY090RDB4ZHRuV0dF?= =?utf-8?B?UElHeE4ySlAxTlFwZ1JyQVZWMFJleHVZbG1RZWM2S2FPZkVySGpSL21yNFg4?= =?utf-8?B?OCsvRkFBZDVxdnhWSXNmTUl1Ty9aajJhM21BamwrTHR3UTc5RzNjY1JzVmk5?= =?utf-8?B?d0hjZjNNUUZaTWhxcDNmR25zWGJXK0pVd0kwbXVJdmIrc3dWZjY2eThGZVNP?= =?utf-8?B?dmZzUEJFK3l1WldTVTkrUHZ5dnZZaWRKSVJ5b3MyNjlFZVZTQ1h4em9ONTQw?= =?utf-8?B?VWdGOU5mSXhQRU9zY0lOWnRkM3VBZVFWMm9BOWhDQlpzczI0K0hpZnhqRis5?= =?utf-8?B?QmtGc2pPRldGQnZaRmRNaFhRVGtlVG10OTBBQjNxMElxbWp0U0I4ZTZLNmV3?= =?utf-8?B?Ky9vWjYrQ3FIZDJuclQ2WG8zUGkrd0ZFcS82YUF3TCsvd01YV2txbVdiYys4?= =?utf-8?B?WmpBOGhuVXpjekxUMUIwdnFTSDdvRkpXL0VSekcvcWkyV0xHdkx1U2R5c09O?= =?utf-8?B?YTlucnlGN3IrVzBqL1czeUZQVXd6dTFPeEFOOU1PZ3FRdGtHKzFkbXp2Y0Zx?= =?utf-8?B?bnZUZmtXT0VlWjZkaFlKNjV6cUlGSnlWSXFBV1JsUy9lMEJBQXF2MkwrZDBp?= =?utf-8?B?dWxRM0doL0Mzc0RFRlAveU0xb0d5eFBQTjY5ZEplZis2Y04weUsvRnZuTVQ3?= =?utf-8?B?ZWRQcWJ3aHQvMzRDanVWR0FOWVlwZGNzdXJFbmhUVlkwd0pMZlBxR2pOallm?= =?utf-8?B?K3AxREw2aGNVY0dVeE1RZ0F4ZndmYi9PbDIwUFl4T2ZwbnBHRVMwMWp0T0t6?= =?utf-8?B?VGh2S0w5WDd4Qzh0bi9rdm8xYTk0SFlpRnkxYlZmblU5aHdVL2RCQWlmSXJx?= =?utf-8?B?SnB2RGYyOGV0TzhSdzVwMW9qd3RadEtoVUU1emtJRXNBTnRSWThSL0NpWmNS?= =?utf-8?B?bGoxVXh4Zm84cC9DSWo3cERoZVMxUHBpNWNmTnhoK2FHZlhvYVltVEZ5eXdv?= =?utf-8?B?cExkWWlrbFdJdG9CU3B2WG9EU1hTZlFBcUZaaFlST0dRbXp2WmNEaWhvQTBY?= =?utf-8?B?K3Q3TWNCRHhaZHgxY0YxUXRVeDdSZktLdlhFdTcxcW5iSWlBcW8yWHE2ekZl?= =?utf-8?B?ZU5aZUVDQm0wdGlFZ280OHpWa3BYaU14MEFJa3hZQXZ5NEorN3AyM1pFNEM4?= =?utf-8?B?TlQxQTg0aW5HUUxnc2ZOQnBadER2NU54cHFhS1UwN2F4R0trMmt5NEdjQ0ht?= =?utf-8?B?WS8vOFM2Q0RxL1ZhQW5mam9Ea1kyK2xlYUwzREZod3IwZ0tuaHlhbkZVTmtC?= =?utf-8?B?NEZqbDJIYXFCMDhvak81WFFrMmlJWm56SVc5R0JuMHNVZnFUZHNCVHlyeGlt?= =?utf-8?B?amZXM0FuOU83OFAvNVJnL1ZKUnpQSzF5R1h6SHN4T1RXR1dZamtEaGlkS0do?= =?utf-8?Q?PnWaz4Ts98qH/54A=3D?= X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: 09e3f5e4-585c-42d2-578d-08df3e786c80 X-MS-Exchange-CrossTenant-AuthSource: VI0PR08MB10656.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Nov 2026 15:45:04.8271 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Bi8JCa1v3cPNXQjhPdEYW6HkRPg2cU+fmsXDWIsq8QlSfGCoMgqNhy1DGXs1kK7BcV1DNYWZ5lfSZMcSiEW6k9P5/rbNND5R0dXtaIw3eB8= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PA6PR08MB10444 On 11/9/26 2:26 PM, Denis V. Lunev wrote: > On 11/8/26 21:50, Andrey Drobyshev wrote: >> When the guest shuts down and QEMU runs with -no-shutdown, the VM stays >> stopped and can only be reset or quit. A libvirt soft reboot is exactly >> that: guest shutdown, system_reset, cont. Today vfio_vmstate_change() >> treats this stop like any other and moves the device to STOP, then cont >> brings it back to RUNNING. >> >> NVIDIA's vGPU manager (595.71.03) takes that STOP for the start of a >> migration, finds no guest driver, breaks its dirty page tracking and >> never repairs it when the driver loads again. Every live migration of >> the VM after a soft reboot then fails: "Error saving page in pipelined >> mode". >> >> Do not touch the device state in this case. There is nothing to >> quiesce, and the reset that must follow resets the device anyway. The >> case is told apart by the shutdown action: only "pause" (-no-shutdown) >> keeps the VM alive. quit, a signal and a plain guest shutdown use >> "poweroff" and still stop the device on the way out, as before. If a >> migration or savevm is started on the stopped VM, no VM state change >> comes at switchover, so vfio_save_setup() stops the device itself. >> >> With this patch applied, soft reboot + subsequent live migration pass >> successfully. >> >> Signed-off-by: Andrey Drobyshev >> --- >> hw/vfio/migration.c | 30 ++++++++++++++++++++++++++++++ >> include/system/runstate.h | 1 + >> system/runstate.c | 5 +++++ >> 3 files changed, 36 insertions(+) >> >> diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c >> index 7e7fb496bbe..2ff3e49d5de 100644 >> --- a/hw/vfio/migration.c >> +++ b/hw/vfio/migration.c >> @@ -473,6 +473,19 @@ static int vfio_save_setup(QEMUFile *f, void *opaque, Error **errp) >> migration->event_save_iterate_started = false; >> migration->event_precopy_empty_hit = false; >> >> + /* >> + * A VM that is already stopped gets no VM state change at switchover. >> + * If a guest shutdown left the device running, stop it now. >> + */ >> + if (!runstate_is_running() && >> + migration->device_state == VFIO_DEVICE_STATE_RUNNING) { >> + ret = vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_STOP, >> + VFIO_DEVICE_STATE_RUNNING, errp); >> + if (ret) { >> + return ret; >> + } >> + } >> + >> if (vfio_precopy_supported(vbasedev)) { >> switch (migration->device_state) { >> case VFIO_DEVICE_STATE_RUNNING: >> @@ -899,6 +912,12 @@ static void vfio_vmstate_change_prepare(void *opaque, bool running, >> return; >> } >> >> + /* The guest has shut down and QEMU keeps the VM stopped: leave it */ >> + if (!running && state == RUN_STATE_SHUTDOWN && >> + qemu_shutdown_action_is_pause()) { >> + return; >> + } >> + >> new_state = migration->device_state == VFIO_DEVICE_STATE_PRE_COPY ? >> VFIO_DEVICE_STATE_PRE_COPY_P2P : >> VFIO_DEVICE_STATE_RUNNING_P2P; >> @@ -936,6 +955,17 @@ static void vfio_vmstate_change(void *opaque, bool running, RunState state) >> return; >> } >> >> + /* >> + * The guest has shut down and QEMU keeps the VM stopped. It can only >> + * be reset or quit now, so leave the device state as it is: the reset >> + * resets the device too. The exit path uses the same run state with >> + * the poweroff action and stops the device as before. >> + */ >> + if (!running && state == RUN_STATE_SHUTDOWN && >> + qemu_shutdown_action_is_pause()) { >> + return; >> + } >> + >> if (running) { >> new_state = VFIO_DEVICE_STATE_RUNNING; >> } else { >> diff --git a/include/system/runstate.h b/include/system/runstate.h >> index 929379adae4..29a919348da 100644 >> --- a/include/system/runstate.h >> +++ b/include/system/runstate.h >> @@ -148,6 +148,7 @@ void qemu_system_vmstop_request_prepare(void); >> bool qemu_vmstop_requested(RunState *r); >> ShutdownCause qemu_shutdown_requested_get(void); >> bool qemu_force_shutdown_requested(void); >> +bool qemu_shutdown_action_is_pause(void); >> ShutdownCause qemu_reset_requested_get(void); >> void qemu_system_killed(int signal, pid_t pid); >> void qemu_system_reset(ShutdownCause reason); >> diff --git a/system/runstate.c b/system/runstate.c >> index 6178b0091a3..1cdecde0fcf 100644 >> --- a/system/runstate.c >> +++ b/system/runstate.c >> @@ -433,6 +433,11 @@ bool qemu_force_shutdown_requested(void) >> return force_shutdown; >> } >> >> +bool qemu_shutdown_action_is_pause(void) >> +{ >> + return shutdown_action == SHUTDOWN_ACTION_PAUSE; >> +} >> + >> ShutdownCause qemu_reset_requested_get(void) >> { >> return reset_requested; > Reviewed-by: Denis V. Lunev This version didn't work either. I've just sent another. Andrey