From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.openvz.org (unknown [69.168.225.77]) by lore.virtuozzo.com (Postfix) with ESMTPS id 396A78015C for ; Tue, 15 Sep 2026 11:50:15 +0000 (UTC) Received: from mail.openvz.org (localhost [127.0.0.1]) by mail.openvz.org (8.14.4/8.14.4) with ESMTP id 68FBmtQp005550; Tue, 15 Sep 2026 14:48:56 +0300 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.openvz.org 68FBmtQp005550 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=openvz.org; s=default; t=1789472939; bh=oHcbLTOQm/JtgjLn/hKDJdeHMWRQnHjkjY8lA0o8HIM=; h=From:To:Date:Subject:List-Id:List-Unsubscribe:List-Archive: List-Post:List-Help:List-Subscribe; b=Lj/7wDHP0n/XYUwgIwoTHtSac6k6bKZXxwa388n6BIDxEmaY57ghGDi2TEuYVxXtV JB5G8qoJE99IboxI/WucDRLZ2rv7eWvaL8W/fU4xxpkQxBkPrCc5+x+qeVq0zASxT3 qLDq36QRYNSDdGcPYBH420ugSNq16naE4t0LIU/TyJhrj893FPIEXYc5ZbNrYCtL2w m0jT3mQKsZdDRThREK+FZ7jL4kZked3YEwQ4V7zWN9WjBlLXalMJoVPUlZ61bxtri1 o/TTeFwYQnapdDDOIrtKWe9WLbCXt53U06qI9+e/KKpc6Z41QxvME5X+vKnOodQnXG B0bVNXA0+SkVQ== Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by mail.openvz.org (8.14.4/8.14.4) with ESMTP id 68FBmrYt005546 (version=TLSv1/SSLv3 cipher=AES128-GCM-SHA256 bits=128 verify=FAIL) for ; Tue, 15 Sep 2026 14:48:54 +0300 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.openvz.org 68FBmrYt005546 Authentication-Results: mail.openvz.org; dkim=pass (2048-bit key) header.d=openvz.org header.i=@openvz.org header.b="ZTEI2MVt" Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-49e823c4868so2441255e9.0 for ; Tue, 15 Sep 2026 04:48:54 -0700 (PDT) X-Gm-Message-State: AFuF++lgAXkg1fVp9Rh9OQrJnt+BusVQjBXs5w0NuPd7WjTgxEPmmQB6 6+9PN9XQK6AkyI1RZW2FdoZH7sKFScbGlhhnNMe5f0miz6iKpDBxHKpXcU3QB2kZ2sKG6KjfJI5 pdOiGSYwYrbOTG5v5ykj+DR4+5X9tO7bBKrRXB5er4ArkaQz/Anmzjw== X-Gm-Gg: AYBFou0d4DMxURnw3lUc6Q8XlxXyFl9wcPkMJIjQce6B2Ccgc5sS7RAo+pzdl8CHSoV b963ly7D34JPy0lBBOV7e3Cg0e8o69ZsQ9SvG3IycUGXlNOSBD9sFMqj7XTnJGmxsqNEXuN06RP EwWfU0cQzfCPsI1XmW+lt7N0hM88P1cQkuUAjStnhPU5u1OGoJ1VCEMN4iTiWl5CBDyzliqrzsY M8sUR6umTTzsE2bmE8/Hkk4fYJ9lRKgnptu2iazfM2TkzcK9QL0QhMoq9n3xdJUwdcyivS/+ebB KKzjnKvilw== X-Received: by 2002:a05:600c:34c1:b0:49c:e3c3:5efd with SMTP id 5b1f17b1804b1-49e7a67e4d5mr176026115e9.9.1789472933610; Tue, 15 Sep 2026 04:48:53 -0700 (PDT) X-Received: by 2002:a05:600c:34c1:b0:49c:e3c3:5efd with SMTP id 5b1f17b1804b1-49e7a67e4d5mr176025045e9.9.1789472932979; Tue, 15 Sep 2026 04:48:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1789472932; cv=none; d=google.com; s=arc-20260327; b=rX1pvHlIDtcyKG4Xq7hiM3mBS52ME46PMspvUff+GDNvUgVi9+YcmaeH4NeW9cCAzo M9gTQX+hAK4lN81EiKUJ5/9LoQuqqwWAbSCgBxNC5P5MG2zzFxBC6ZpdkLs18ZtkpfWc 4vYdpmHrvkZtjenDzs6d922WLK+9FefrXvY08IWw+jntKSrWat24xUI8eDR6eyA99mC2 +kdah8kJ+Lv42OOqZv5pDoB6mFsNv+0J0kIn3zABJpXsu1F9pMsD5oW6nFc6v7vS6mHL 6NhCWZtJGOUuTraiyhHozLjKVJ9EZdnnXnsXyaFSyrJzp805lyKmy4FQYip6Q6y8OuHP vrKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:dkim-signature; bh=OZYqLBfiBzJAQ0PlCm1uTD4BQ6hc4v4S6MIA5PMbPO0=; fh=yZ3SQEeCA88XmTynhqwyDnY3KAx9/rtIDrjAb3vSECw=; b=XqPpiwzySsW0QOoMbjbpTBoI7WKsX4jkueih8augBWRb2CBlXyuqFCsY9FwLo/hk9E Z24fIbUqKX442WQUrYxfEF6rKoUmI+toaT2/mgOEOIRh3qOrrPYxAsIMZkFuSNZSflpU HtaQ2KKuGQ9N5K750rT8Lacw7+EyOUp7CULkkwMGYeeDiGvqQyB8KRq8sFERm5O//jVa tHa7xhTeMFZVuj3VdBpF3MuXIpUEYdZqiUSxlixM7t9XQpJsntgSv9pWQgt1co51MrQV UVN828ivJ1SwiirGwXf/av6F1VBgrw0m9ZrnCmlwFF/anUJjeKUwYocUFbDuS7fpTJV6 EsJQ==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@openvz.org header.s=google header.b=ZTEI2MVt; spf=softfail (google.com: domain of transitioning den@openvz.org does not designate 2a06:5b06:b600:300:33e6:6107:4678:322f as permitted sender) smtp.mailfrom=den@openvz.org; dara=neutral header.i=@openvz.org Received: from mail-sor-f41.google.com (mail-sor-f41.google.com. [209.85.220.41]) by mx.google.com with SMTPS id 5b1f17b1804b1-49e7a7ad7bdsor7202095e9.4.2026.09.15.04.48.52 for (Google Transport Security); Tue, 15 Sep 2026 04:48:52 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning den@openvz.org does not designate 2a06:5b06:b600:300:33e6:6107:4678:322f as permitted sender) client-ip=2a06:5b06:b600:300:33e6:6107:4678:322f; Authentication-Results: mx.google.com; dkim=pass header.i=@openvz.org header.s=google header.b=ZTEI2MVt; spf=softfail (google.com: domain of transitioning den@openvz.org does not designate 2a06:5b06:b600:300:33e6:6107:4678:322f as permitted sender) smtp.mailfrom=den@openvz.org; dara=neutral header.i=@openvz.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=openvz.org; s=google; t=1789472932; x=1790077732; darn=openvz.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to:content-type; bh=OZYqLBfiBzJAQ0PlCm1uTD4BQ6hc4v4S6MIA5PMbPO0=; b=ZTEI2MVt+QRV/dU1bpBSLZTWqPM3WYBk7p3IurFEfDl48rcImGIrB4o42u/FEy4eDH 6H4f/7aVmbDgslfnXmH9U3WCEbHtbkoehlX8q+XNq/wUSXUVpW5PvdvgF7K/X3/gYQMH 2gRrMXcb9kLut46veu0Y1OPKsYOZEwR3S8tkgYZ8/66IWpg97s4o5gV4l9UEXnP7b7Sf uSH5Yn0uCPM6+JQSAy9jtTSch2LHzxa7hOR7asWB0iBAYhW3N+QtsHuaBSu8apuCh4wI v+8AWI3yqjuHevGsOFMrH3vfgU2XDGpFq6cRcqzPaQXGYy7nNvM6rv0pW69HkJP5u4jF Unzw== X-Received: by 2002:a05:600c:4e45:b0:49e:6050:9fbe with SMTP id 5b1f17b1804b1-49e7a68cefdmr186078895e9.15.1789472932559; Tue, 15 Sep 2026 04:48:52 -0700 (PDT) Received: from athena.sw.ru ([2a06:5b06:b600:300:33e6:6107:4678:322f]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-49e7d27c88fsm57519805e9.2.2026.09.15.04.48.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2026 04:48:51 -0700 (PDT) From: "Denis V. Lunev" To: devel@openvz.org Date: Tue, 15 Sep 2026 13:48:49 +0200 Message-ID: <20260915114849.3869609-1-den@openvz.org> X-Mailer: git-send-email 2.53.0 MIME-Version: 1.0 X-OZ-Fwd: true Subject: [Devel] [PATCH vz10-latest 1/1] openvswitch: fix soft lockup in the netlink dump callbacks #VSTOR-144650 X-BeenThere: devel@openvz.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: OpenVZ development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: devel-bounces@openvz.org Errors-To: devel-bounces@openvz.org A production compute node carrying a few thousand datapath flows and ports hit a soft lockup inside a single netlink flow dump and panicked. ovs_flow_cmd_dump() calls ovs_flow_stats_get() for every flow it emits, and that releases stats->lock with spin_unlock_bh() once per CPU that has touched the flow. ovs_vport_cmd_dump() calls peernet2id_alloc() for every vport that lives in a foreign netns, and that releases nsid_lock with spin_unlock_bh(). Every release is a local_bh_enable(), and each one runs the pending softirq backlog in the dumping thread's own context. The skb bounds how many items one callback emits, and a large but sparse table adds only a walk over empty buckets, so no dumper carries a budget of its own. Neither bounds the softirq work the callback absorbs. On a CPU that carries the box's packet load the backlog refills as fast as it drains, so the dumping thread becomes that CPU's softirq engine. It never sleeps and it has no reschedule point, so under voluntary preemption nothing can take the CPU away from it: neither the ksoftirqd the kernel woke to take the work over, nor the stopper thread the softlockup detector dispatches to refresh its timestamp. Hold BH off across the whole callback instead, the way ctnetlink_dump_table() does, so the nested spin_unlock_bh() stop draining softirqs. Both loops already run under rcu_read_lock() and cannot sleep. What they give up is preemption under CONFIG_PREEMPT, since a BH-off region is not preemptible outside PREEMPT_RT. That region is bounded by the skb and the table size, where the softirq backlog it used to absorb is not. The vport half is downstream-only. Mainline made nsid_lock a plain spinlock in commit aed4969f2bdf ("net: net->nsid_lock does not need BH safety"), so the vport dump there never drains softirqs and the upstream submission touches ovs_flow_cmd_dump() alone. This kernel predates that commit and still takes nsid_lock with spin_lock_bh(). Signed-off-by: Denis V. Lunev --- net/openvswitch/datapath.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/net/openvswitch/datapath.c b/net/openvswitch/datapath.c index f2f50324424b..762bd6d8e49f 100644 --- a/net/openvswitch/datapath.c +++ b/net/openvswitch/datapath.c @@ -1478,6 +1478,11 @@ static int ovs_flow_cmd_dump(struct sk_buff *skb, struct netlink_callback *cb) return -ENODEV; } + /* + * Not needed for safety. Stops every spin_unlock_bh() in + * ovs_flow_stats_get() from running the softirq backlog here. + */ + local_bh_disable(); ti = rcu_dereference(dp->table.ti); for (;;) { struct sw_flow *flow; @@ -1498,6 +1503,7 @@ static int ovs_flow_cmd_dump(struct sk_buff *skb, struct netlink_callback *cb) cb->args[0] = bucket; cb->args[1] = obj; } + local_bh_enable(); rcu_read_unlock(); return skb->len; } @@ -2502,6 +2508,11 @@ static int ovs_vport_cmd_dump(struct sk_buff *skb, struct netlink_callback *cb) rcu_read_unlock(); return -ENODEV; } + /* + * Not needed for safety. Stops every spin_unlock_bh() in + * peernet2id_alloc() from running the softirq backlog here. + */ + local_bh_disable(); for (i = bucket; i < DP_VPORT_HASH_BUCKETS; i++) { struct vport *vport; @@ -2522,6 +2533,7 @@ static int ovs_vport_cmd_dump(struct sk_buff *skb, struct netlink_callback *cb) skip = 0; } out: + local_bh_enable(); rcu_read_unlock(); cb->args[0] = i; -- 2.53.0 _______________________________________________ Devel mailing list Devel@openvz.org https://lists.openvz.org/mailman/listinfo/devel