From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.openvz.org (mail.openvz.org [69.168.225.77]) by lore.virtuozzo.com (Postfix) with ESMTPS id 1F22F81303 for ; Wed, 4 Nov 2026 20:10:06 +0000 (UTC) Received: from mail.openvz.org (localhost [127.0.0.1]) by mail.openvz.org (8.14.4/8.14.4) with ESMTP id 6A4K8rmp026624; Wed, 4 Nov 2026 23:08:54 +0300 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.openvz.org 6A4K8rmp026624 Authentication-Results: mail.openvz.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=virtuozzo.com header.i=@virtuozzo.com header.b="b4N018ev" Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) by mail.openvz.org (8.14.4/8.14.4) with ESMTP id 6A4K8iuK026599 (version=TLSv1/SSLv3 cipher=AES128-GCM-SHA256 bits=128 verify=FAIL) for ; Wed, 4 Nov 2026 23:08:44 +0300 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.openvz.org 6A4K8iuK026599 Received: by mail-wm1-f70.google.com with SMTP id 5b1f17b1804b1-49febef74e3so16244175e9.0 for ; Wed, 04 Nov 2026 12:08:44 -0800 (PST) X-Gm-Message-State: APuU4Z8rLqazRzeriAgnD/Pvj1utGxguyKobGl45q+h6DxhZ8FY7LR1U rzEhwwEX16p+TV2Cq1GdxYO2bKMdFgQvqF/A4Mt+k1BoqPKbS2+DlvLqs4TBJjoSF9r9v6dEMQp 2+lTBAcd/KTZ67XoCnXbXLgy7FywfgdDUkdgAvdU6Anqj2VeRiVS9EA== X-Gm-Gg: AYBFou3XaGA98ECMUj2dvtCk1uENZf4IwFX6rd82zGwk4Y8ADOapiHW1k87oz/ePr6V HJYhhsm1ewvuiP32La9UV14zHnjRpiWJxQnYFPFZq25l/lUALrUX9OgK0qsCYk2Gc6DtZxv/BBN WfpH8Ji4QWcDdQYGLWNqZLPAeBjrtpvmbE67MGhLyiaZKTkEL8ham4bvuv356NTR62j4iOXuIxM pI01dNw//h3e5Ds5RkMkat6j99hDts3CCufnZFajiYtur3vTyr25htyHFIjxp/OAKsNhv+HfL9B 8Q4vae4d9LrS2uxuOfIMZ2FTwrlahZtUjYsKkSnVkTwKkYB629mX+b3/rp4XqCZZxfCkoDjPo5f 7HxI4v7QXSZObvhJ//Q== X-Received: by 2002:a05:600d:82e2:b0:4a1:c27d:e1a8 with SMTP id 5b1f17b1804b1-4b55d7d7a2amr8696785e9.10.1793822923870; Wed, 04 Nov 2026 12:08:43 -0800 (PST) X-Received: by 2002:a05:600d:82e2:b0:4a1:c27d:e1a8 with SMTP id 5b1f17b1804b1-4b55d7d7a2amr8696515e9.10.1793822923438; Wed, 04 Nov 2026 12:08:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1793822923; cv=none; d=google.com; s=arc-20260327; b=b51WIw1XNgbAQ4O9BYnlqEunrBy+g6yimM3Mnl8ZKAVpf46PZLn/zTuytcbaiexh3+ Nc1GnIVQqnrXYeK/GMf1PttNFOS6Rg0oiGYAbDYeJWrtmt7MyOIfqBOZ2fx7heROaZBX H5ieEtH+Vj0Sh3G2twriJpu64U8gFJfYAJZYaLaaafF6/gDDxWx3tduOd9U01QrWLf7q T1wxFaE1QxZQegLYe58pnaASXb4sRMpHkGVZeB9H/bDXNLa1Frkva+0n9o8EfJz+/ZUe zhVwVUbAoVSCFZquYmJww2MRBqzScgxq5X9QbCcYArUK37NJe3dweFhIgrxvIkA0PkQU 5wQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=message-id:date:references:in-reply-to:cc:to:from:subject :content-transfer-encoding:mime-version:dkim-signature; bh=eBA05qAaSwAxovvaHvvJK0jFt9SOKNsoL4bx6neR8kg=; fh=5AmFLEo6nHxL6WDW39LXC01IwU+CKjcYMpOuqqVTSIM=; b=RBzlehBBjg3xMQZfnngkK88i55ZYIq2WG0/2luQyCHcfg5j/ZDEjpSKvPcRLwLBox0 bOgZa/NDB4QWQHIMutnpMP9la9XapcD7hvGH1xuP9mK7c7a1bShcBBxvemxegpNjaTU/ /kK1HvKlQEuMaGk7uJoqxu2/9brH8EJKOHOSwvAEP4mKKQyLYMuRn1maqYWh2B32/9iR XEWaoh4Uy6I6hTk57ifQ/RMl8eXJIUz3sW/rKRWsOsc7ALblX2HjYVPnOy4XrwUVRtQ7 A0c2NSZhUjYjjZQozO2TN9FX2txEgB9MBXmyfm+L23A7oYc1OB0Co9+8ea5TQIdEg3GI UoLQ==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@virtuozzo.com header.s=relay header.b=b4N018ev; spf=pass (google.com: domain of khorenko@virtuozzo.com designates 130.117.225.111 as permitted sender) smtp.mailfrom=khorenko@virtuozzo.com; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=virtuozzo.com Received: from relay.virtuozzo.com (relay.virtuozzo.com. [130.117.225.111]) by mx.google.com with ESMTPS id ffacd0b85a97d-48fed271737si5868903f8f.50.2026.11.04.12.08.43 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Nov 2026 12:08:43 -0800 (PST) Received-SPF: pass (google.com: domain of khorenko@virtuozzo.com designates 130.117.225.111 as permitted sender) client-ip=130.117.225.111; Authentication-Results: mx.google.com; dkim=pass header.i=@virtuozzo.com header.s=relay header.b=b4N018ev; spf=pass (google.com: domain of khorenko@virtuozzo.com designates 130.117.225.111 as permitted sender) smtp.mailfrom=khorenko@virtuozzo.com; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=virtuozzo.com DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=virtuozzo.com; s=relay; h=Message-Id:Date:From:Subject:Content-Type: MIME-Version; bh=eBA05qAaSwAxovvaHvvJK0jFt9SOKNsoL4bx6neR8kg=; b=b4N018evysSn oEf5Spjcqm83MOTeZk13/cl9uug2jDlsu7M1YR9TrmAPdXxKHdfScU8c0iszisK5E8eTPl6vP09AQ PRZwCdBRtFkMyeLfVXavJaXqwEwJy68kFqYhmPTWlIOWl9m0VypJ6KVzOIRzVeIcaDfPCP4bSq6rz 72OlksP1uZNI7Cr16hez8ElgBkg9hX7EC9MRL7C2g/XHMUpJsxFTPUEhqZwMlIRPgvEejTuuEr5+P oYuK66msN1CaE3xeFHy0Q8EC7XsSYNGMJCwqrFwVuJK7emHAinHGzbX3tqOALaDw5Iu+L/HZW8hUr +U3sYw6C9FiKbqvRJV0B7w==; Received: from ch-demo-asa.virtuozzo.com ([130.117.225.8] helo=f0.sw.ru) by relay.virtuozzo.com with esmtp (Exim 4.96) (envelope-from ) id 1xOhDp-000aSY-2r; Wed, 04 Nov 2026 21:08:42 +0100 MIME-Version: 1.0 From: Konstantin Khorenko To: Andrey Drobyshev In-Reply-To: <20261030144933.944360-2-andrey.drobyshev@virtuozzo.com> References: <20261030144933.944360-1-andrey.drobyshev@virtuozzo.com> <20261030144933.944360-2-andrey.drobyshev@virtuozzo.com> Date: Wed, 04 Nov 2026 21:08:34 +0100 Message-Id: <179382291494.2764784.3910448479668484261.b4-review@b4> X-Mailer: b4 0.16.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3273; i=khorenko@virtuozzo.com; h=from:subject:message-id; bh=loNfeQzpX+nRj0/UQGaHphuTdr6m52fBuLG5yF0DyNI=; b=owEBbQKS/ZANAwAKAVGWCkf5YVwqAcsmYgBq65DKJSDbhjHNI91toRR6/LC3NobwKGNy7zv30 urCKEjUte2JAjMEAAEKAB0WIQRWD23IPcXT0GO9CCxRlgpH+WFcKgUCauuQygAKCRBRlgpH+WFc KtvVD/sF9tnkPhqBaPfdVnkRxuV09n23UhmYaaLcVnvsXZFLz+RWUNoQG5Ig3cZetS1tuG8+aQ5 ETpbFPD17m+zNyM61jt+TOPcKYfHwHd722lBjNHOBXYDabMh2PHTEuAd+4GrelHy9PbAHDZipk5 8jrW9vXoE9dAyOwdj1QZ9M2GOHZjFrNh+6OjcfTFais9KsKUBCGEgjYdnfGhmYvlAOdWlUkYFNc N0FUgyiCxlv+Tt396bYETz5zOQ8wdec2raiyjNQSuUg4mR+Zk7gq4IOyUHqBadKwIGiPtIEvBme URdUrJTQ8YE/2TqHM/ycK6XVUx6E7v6rHmNGXFXoYZP8VTUqCDFcbwQPTJoMGb/ooeoLUUR/Pdg i8uuUzOwvdR198rAeOU/2WdEqhKNT5qulkorpJ0kxcrxvG3XB0hKLW5taN4+nW3fi3eOSMwS1Ye eAhsKiZfbaugQPJbqLcBjxTqUQMYzDTQq3Cuob4e2g52Wct/xvOrQqFDGr2PzfDi7PVI3ZD/RLA NxpQ4IqIcTgBgiBrqeT02gns1QGLTFv8CZYh8qYbB6/5w0LJE9v8Sg9W0AJXU0sbSzfv3I/T4U8 OXanjgVAkigk7L0z5GJUo+XzlpTn4b/tthseEZcL4Vz8cvvlk/evierCXeK6EKXHOB32n5zMJP9 wtPHBDMlXElVtPw== X-Developer-Key: i=khorenko@virtuozzo.com; a=openpgp; fpr=560F6DC83DC5D3D063BD082C51960A47F9615C2A X-OZ-Fwd: true Cc: devel@openvz.org Subject: Re: [Devel] [PATCH hci-8.1 1/2] vfio: move an open device's kvm reference when its group gets a new kvm X-BeenThere: devel@openvz.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: OpenVZ development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: devel-bounces@openvz.org Errors-To: devel-bounces@openvz.org I won't apply this version, please send v2 with the changes below. On Fri, 30 Oct 2026 16:49:32 +0200, Andrey Drobyshev wrote: > + mutex_lock(&group->device_lock); > + list_for_each_entry(device, &group->device_list, group_next) { > + mutex_lock(&device->dev_set->lock); > + if (device->kvm && device->kvm != kvm) { > + vfio_device_put_kvm(device); > + vfio_device_get_kvm_safe(device, kvm); > + } Item 1 of the cover letter reply: do not change device->kvm under a driver that still uses it after open_device(). Check the NVIDIA vGPU driver and say in the message which driver this was tested with. Why: until now device->kvm did not change between open_device() and close_device(), and drivers rely on that. KVMGT registers its page-track notifier on device->kvm at open, and that takes a reference on the kvm. At close it unregisters from device->kvm, which drops a reference. After the move the close drops a reference on the new VM that it never took, so the new VM is freed while QEMU still uses it. KVMGT is not built in vz10, but the vGPU driver from the ticket is out of tree and may do the same. > + /* > + * Open devices hold a reference to the kvm they were opened with. > + * Move it to the new kvm, or the old one stays pinned until the > + * device is closed. Nothing to move to on NULL. > + */ > + if (!kvm) > + return; Item 2 of the cover letter reply: say in this comment why NULL is skipped: the lock order. Update the kernel-doc of vfio_file_set_kvm() too, it says the device gets the kvm when it is first opened. Why: the NULL call from kvm_device_release() comes with kvm->lock held. vfio_ap takes kvm->lock in its open_device(), which runs under dev_set->lock. Taking dev_set->lock here for NULL would close an ABBA deadlock of the kind 2b48f52f2bff fixed. "Nothing to move to" is not the reason. > Move the reference to the new kvm. Leave NULL alone: that is the old > kvm-vfio device going away under kvm->lock, and the new kvm, if any, > arrives on its own. Item 2 of the cover letter reply: drop "and the new kvm, if any, arrives on its own" and give the lock order as the reason instead. Why: with cpr-transfer the new QEMU adds the group to its kvm-vfio first, and the old QEMU's NULL comes last. It leaves group->kvm NULL. The open devices keep the new kvm, so the code is fine, the sentence is not. A NULL from KVM_DEV_VFIO_FILE_DEL also holds only kv->lock, not kvm->lock. > https://virtuozzo.atlassian.net/browse/VSTOR-150597 Item 3 of the cover letter reply: add a Fixes tag for 2b48f52f2bff ("vfio: fix deadlock between group lock and kvm lock"). Why: that commit made the device hold a reference to the kvm until its last close. Item 6 of the cover letter reply: single space after a period, here and in the code comment, and the Feature: line. This needs to go to mainline too, I created a task for that: https://virtuozzo.atlassian.net/browse/VSTOR-153437 Mainline vfio_group_set_kvm() also swaps only group->kvm, and since 5c3ea6d3e836d the open device even holds the old VM's file. The change of the device->kvm contract is something the vfio maintainers have to agree to. -- Konstantin Khorenko, Virtuozzo Linux Kernel Team _______________________________________________ Devel mailing list Devel@openvz.org https://lists.openvz.org/mailman/listinfo/devel