From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.openvz.org (unknown [69.168.225.77]) by lore.virtuozzo.com (Postfix) with ESMTPS id 52EE4802E4 for ; Mon, 31 Aug 2026 15:18:03 +0000 (UTC) Received: from mail.openvz.org (localhost [127.0.0.1]) by mail.openvz.org (8.14.4/8.14.4) with ESMTP id 67VFGhUn017025; Mon, 31 Aug 2026 18:16:44 +0300 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.openvz.org 67VFGhUn017025 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=openvz.org; s=default; t=1788189408; bh=5IE8vXtO/PSZ+lmxkw7UzDHHU4DrzAljV3f6mocN4wY=; h=From:To:Date:Subject:List-Id:List-Unsubscribe:List-Archive: List-Post:List-Help:List-Subscribe; b=cW3n2zmSm0qtIJ/iiC7S6ddWyZTzN11aNTftST3SzYgO+lIYeaeeL4eTAt26k2b7l 7RIVl/HQac9uAHdZAwZlHSZG9rpE6KpMXMxWjJrO7rSox7awIE+1mnzzK5O/3VnJBk 5P1Znf3jcnbn+R3UlWRuSdW4/hXVdrEn5xm3NWjYfc39PLlfgCLBf6lbvif9mqX9Gn AUPjCrn5NA5ikBx8mzxZLdGubuM0OBvsEfCko0n4A4tkRrw2NOOZ8fWR1+MTEoqcG5 RO+X40cofb1EURUZP8GAjXCoQGNuogaveOLpVm+cXqP60QGGk6CUr+Wh7fyKbtoLe/ ERfCLB60tOblQ== Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by mail.openvz.org (8.14.4/8.14.4) with ESMTP id 67VFGgu3017021 (version=TLSv1/SSLv3 cipher=AES128-GCM-SHA256 bits=128 verify=FAIL) for ; Mon, 31 Aug 2026 18:16:42 +0300 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.openvz.org 67VFGgu3017021 Authentication-Results: mail.openvz.org; dkim=pass (2048-bit key) header.d=openvz.org header.i=@openvz.org header.b="IEJPiDpd" Received: by mail-wm1-f69.google.com with SMTP id 5b1f17b1804b1-49cced8309bso16887825e9.3 for ; Mon, 31 Aug 2026 08:16:42 -0700 (PDT) X-Gm-Message-State: AFuF++mKb3GN2Mf83UKfk8OvunguYg0LV9SJ7zmEyhqj3LraoM4r02fE 8ghU9qjdMN/GpME0QzMEVtXohapLDimLE2Ol9qlo8qpcx+H6oTTOXosjwLn8Vv9ur4cFq6JUX7m VvqMRzQZ7AE+ladCRO+2bp3Ew1siUO+zp4KXsT1JiOYv2YEZtQcSAIA== X-Gm-Gg: AR+sD12PN9WFC0we2tF4MCLqW8W3bC71Zei0HztP3pT0Tp8uWz0XWLdKxXEG5OFGm+6 D3lbXg9XHHX+/5tBGCb1iwDz88rzy+kmS/4kXYx9hdRnAJZxc11WRWewQFv4LW2sfcFTU7M0iV5 FhR3ogJJ++wwTF1WKi+eiRgMr0yZs4dywtud6ov8PxCDY7HBhBByXyuVV0EpNJUizvZi9FjXjTN I3HX9RyW7ZI8a648Gt+oT6JiE0HNlBlH0r9cMVduaZYBsOjx2QW94L2r8qFlk6UG4IGYbq2raQ7 gH/ld3dHeA== X-Received: by 2002:a05:600c:354b:b0:495:4d88:e630 with SMTP id 5b1f17b1804b1-49b91c4fb4amr394265385e9.10.1788189401692; Mon, 31 Aug 2026 08:16:41 -0700 (PDT) X-Received: by 2002:a05:600c:354b:b0:495:4d88:e630 with SMTP id 5b1f17b1804b1-49b91c4fb4amr394264305e9.10.1788189401127; Mon, 31 Aug 2026 08:16:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1788189401; cv=none; d=google.com; s=arc-20260327; b=HKvhxlh/XRt92LkOgwOKONZPQu9/aVJCVrmwGOVoPUq7/dk6OhJa/nT5SCxUdSUm8k Re9PJ8Hn67cEvSeQYkRtIlklH5pn58S17/czfY8BVNliCsGKG9nyyWzJZ9uIQLvOaiJ+ nzs6orhezDCP28IIRhQUIfYijlgZu0eUF3T7TqBP6uzQf93tKVyv+68sWt9UpyAN80e9 UCXIHykCEiwB0lJMNG7QeSMaZudnqk4GFfolLPcet3ZzpH1ZUn8DWKNkdydJsXsWWe/2 Wr5+PkRop1Kvjz1jX32Umk/BVjrYqmOM5r57naf8y7R+vw3so1uz0crUc4S1fzL9243C 7tyQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:dkim-signature; bh=fF73LcbYfMkhoA6XHxlIU+6bECb+9b6MIHlUX0O8tz0=; fh=yZ3SQEeCA88XmTynhqwyDnY3KAx9/rtIDrjAb3vSECw=; b=dGCudHw4XFSmxzTlrBf7roTu+7yK91071yum83aQyzOW3ldwtmg5MGZCd1GbPVidxg 6TzkSb41eolVr/fsBxKF2hhXbdATPJOXu7wpiDZIb57WXPKtcKp8/dnoiJaDXkLu2ehk q+SozRf+dCoK+bGYugmONpdd/InNZcG6jiE/O+B3ZgDu5wqTC7DnTwd8n7VBAPcIFHYr zFYS4363azcc8u+C2zyAIYE1T6hb9vu2G0E7KIq1+u4pT2q67HCb5RWBxdNdrOiSYJxh YDMLz9Vo8eJI63vBF8L0MoLNVOXeJgqyny+M9EttkcIMbyMjZMZc1PkDUVkyB/b7Y0aV O9hQ==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@openvz.org header.s=google header.b=IEJPiDpd; spf=softfail (google.com: domain of transitioning den@openvz.org does not designate 2a06:5b06:b600:300:3bc1:c53b:30b:203e as permitted sender) smtp.mailfrom=den@openvz.org; dara=neutral header.i=@openvz.org Received: from mail-sor-f41.google.com (mail-sor-f41.google.com. [209.85.220.41]) by mx.google.com with SMTPS id 5b1f17b1804b1-49b95096c4esor16063585e9.16.2026.08.31.08.16.41 for (Google Transport Security); Mon, 31 Aug 2026 08:16:41 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning den@openvz.org does not designate 2a06:5b06:b600:300:3bc1:c53b:30b:203e as permitted sender) client-ip=2a06:5b06:b600:300:3bc1:c53b:30b:203e; Authentication-Results: mx.google.com; dkim=pass header.i=@openvz.org header.s=google header.b=IEJPiDpd; spf=softfail (google.com: domain of transitioning den@openvz.org does not designate 2a06:5b06:b600:300:3bc1:c53b:30b:203e as permitted sender) smtp.mailfrom=den@openvz.org; dara=neutral header.i=@openvz.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=openvz.org; s=google; t=1788189401; x=1788794201; darn=openvz.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to:content-type; bh=fF73LcbYfMkhoA6XHxlIU+6bECb+9b6MIHlUX0O8tz0=; b=IEJPiDpdBMH7pq03QPqhwstQOVIDa9aHSy+PpO3nTc2PtWgIDx20JNCY85d5NvEhY0 +3FPOUWjpEmhRyhH6+IXDN+I9Gw6KAuVpttRew9XF3X4HGB2F98FlaBJen3tatgjsO/d kadgznMkYZOq7Zn7C7XIjw7C+EBVPoL7A2hN3DeeTdzpf3B0PfOHLvFFc+JZr03asfY4 0zSYi4FU770fcNhSmbqkPVYNiJQF970zwIJzHgpAiiSCMhplhyff6acGUIiqvDYPf8+5 CerihLFDHKjrWMnEezSBhkD5cBXK9R48zOPgh5Zm4SzbMOddU09BjW12WKh56NeXpih5 6O2g== X-Received: by 2002:a05:600c:4e01:b0:49c:cedc:768a with SMTP id 5b1f17b1804b1-49ccedc76a1mr318996215e9.16.1788189400508; Mon, 31 Aug 2026 08:16:40 -0700 (PDT) Received: from athena.sw.ru ([2a06:5b06:b600:300:3bc1:c53b:30b:203e]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-49cd775dae8sm137234415e9.9.2026.08.31.08.16.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 Aug 2026 08:16:40 -0700 (PDT) From: "Denis V. Lunev" To: devel@openvz.org Date: Mon, 31 Aug 2026 17:16:37 +0200 Message-ID: <20260831151637.1335671-1-den@openvz.org> X-Mailer: git-send-email 2.53.0 MIME-Version: 1.0 X-OZ-Fwd: true Subject: [Devel] [PATCH 1/1] ms/virtio_ring: fix infinite loop in virtnet_poll_cleantx when device is broken X-BeenThere: devel@openvz.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: OpenVZ development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: devel-bounces@openvz.org Errors-To: devel-bounces@openvz.org From: Jinqian Yang virtnet_poll_cleantx() contains a do-while loop that cleans up transmitted TX buffers and calls virtqueue_enable_cb_delayed() to check whether more buffers need processing. When the virtio backend stops responding during guest reboot, used->idx is never updated, so virtqueue_enable_cb_delayed() always returns false and the loop never terminates. Then it will block reboot process, and the guest will hang. The problem occurs during guest reboot under network traffic: 1. kernel_restart() -> device_shutdown() traverses the device list 2. virtio_dev_shutdown() calls virtio_break_device() which sets vq->broken = true 3. virtio_dev_shutdown() then calls virtio_synchronize_cbs() to wait for in-flight callbacks to complete 4. A virtio interrupt fires, softirq is deferred to ksoftirqd which calls net_rx_action() -> virtnet_poll() -> virtnet_poll_cleantx() 5. virtnet_poll_cleantx() enters the do-while loop and never exits because the QEMU backend has stopped updating used->idx, despite vq->broken having been set to true in step 2. Since the loop runs inside ksoftirqd (a SCHED_OTHER kthread), it is visible to the scheduler and does not trigger a hard lockup. However, the kthread never leaves the loop, so RCU detects it as a CPU stall and reports it periodically. Meanwhile, the reboot process remains blocked in device_shutdown() because virtio_dev_shutdown() cannot complete its synchronization step, and the guest hangs permanently. This can be reproduced on a guest with a virtio-net device: run iperf3 traffic in the guest, then trigger reboot. The reboot occasionally hangs permanently with RCU stall on ksoftirqd. Observed on ARM64 KVM guest: CPU#1 RCU stall (ksoftirqd/1), repeated periodically: virtqueue_enable_cb_delayed_split <- virtnet_poll <- __napi_poll <- net_rx_action <- handle_softirqs <- run_ksoftirqd <- smpboot_thread_fn <- kthread Fix by adding a vq->broken check in virtqueue_enable_cb_delayed(), so that the loop exits immediately when the device is broken, allowing the device shutdown to proceed. Signed-off-by: Jinqian Yang Reviewed-by: Xuan Zhuo Signed-off-by: Michael S. Tsirkin Message-ID: <20260716115940.394832-1-yangjinqian1@huawei.com> (cherry picked from commit 0d0eff39ceb3dcbf7847a6f4517086c207c60081) Applied by hand: this tree has no data_race() annotation on vq->event_triggered and dispatches the split and packed helpers directly, so only the surrounding context differs. The added check is identical. This tree is hit harder than the kernel the fix was written against. Upstream commit e13b6da7045f ("virtio-net: tweak for better TX performance in NAPI mode") replaced the same do-while loop in start_xmit() with a single check and is not here yet, so the livelock is also reachable from a timer softirq: a delack timer transmitting on a broken queue never returns from start_xmit(), and softlockup_panic turns that into a panic rather than an RCU stall on ksoftirqd. https://virtuozzo.atlassian.net/browse/VSTOR-143525 Feature: fix ms/virtio_ring Signed-off-by: Denis V. Lunev --- drivers/virtio/virtio_ring.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 65674af37918..8097e49456d3 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -2529,6 +2529,14 @@ bool virtqueue_enable_cb_delayed(struct virtqueue *_vq) { struct vring_virtqueue *vq = to_vvq(_vq); + /* + * When the device is broken there is no point in polling used->idx, + * the backend will never update it. Return true to let callers + * exit their cleanup loops instead of spinning forever. + */ + if (unlikely(vq->broken)) + return true; + if (vq->event_triggered) vq->event_triggered = false; -- 2.53.0 _______________________________________________ Devel mailing list Devel@openvz.org https://lists.openvz.org/mailman/listinfo/devel