* [Devel] [PATCH hci-8.1 0/3] iommu/amd: make iommu_completion_wait() wait for flushes queued by other CPUs
@ 2026-11-10 19:18 Denis V. Lunev
2026-11-10 19:18 ` [Devel] [PATCH hci-8.1 1/3] ms/iommu/amd: move wait_on_sem() out of spinlock Denis V. Lunev
` (3 more replies)
0 siblings, 4 replies; 8+ messages in thread
From: Denis V. Lunev @ 2026-11-10 19:18 UTC (permalink / raw)
To: khorenko; +Cc: devel
iommu_completion_wait() checks iommu->need_sync without iommu->lock and
returns at once when it is clear. A completion wait queued by another
CPU clears the flag before the IOMMU has executed the commands in front
of it, so the caller can return while its own invalidations are still
pending. After an unmap that lets an IOVA or a page table page be
reused while the IOMMU may still hold the old translation. It also
leaves a hole in "iommu/amd/pgtbl: fix DMA faults on IOVAs mapped while
the table grows": the DTE flush done before the new page table level is
published can return before the IOMMU has dropped the old DTE.
Patch 3 is the mainline fix 1e75a8255f11c. It waits for the last
allocated completion sequence number, which relies on the model
introduced by patches 1 and 2, also from mainline: the wait moves out of
iommu->lock and the sequence number is allocated under it. vz10 still
had the old model, an atomic increment outside the lock and the wait
under it. 6.12.y carries the same three commits. Patch 2 needed a
context fix only: vz10 has no cmd_sem_paddr in struct amd_iommu.
Tested on epyc06 (EPYC 7352, four Optane in one IOMMU group) with the
VSTOR-154276 reproducer: an out-of-tree module exhausts the group's
32-bit IOVA space while fio reads the drives, and a kprobe holds the DTE
update open for 5 ms. The stock 8.0 kernel gives IO_PAGE_FAULTs and fio
verify mismatches with no I/O error. 22.19 with the VSTOR-154276 fix and
this series gives no faults, no mismatches and no completion wait
timeouts.
https://virtuozzo.atlassian.net/browse/VSTOR-154424
Ankit Soni (2):
ms/iommu/amd: move wait_on_sem() out of spinlock
ms/iommu/amd: serialize sequence allocation under concurrent TLB
invalidations
Guanghui Feng (1):
ms/iommu/amd: Wait for completion instead of returning early in
iommu_completion_wait()
drivers/iommu/amd/amd_iommu_types.h | 2 +-
drivers/iommu/amd/init.c | 2 +-
drivers/iommu/amd/iommu.c | 61 ++++++++++++++++++++---------
3 files changed, 45 insertions(+), 20 deletions(-)
--
2.53.0
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* [Devel] [PATCH hci-8.1 1/3] ms/iommu/amd: move wait_on_sem() out of spinlock
2026-11-10 19:18 [Devel] [PATCH hci-8.1 0/3] iommu/amd: make iommu_completion_wait() wait for flushes queued by other CPUs Denis V. Lunev
@ 2026-11-10 19:18 ` Denis V. Lunev
2026-11-10 19:23 ` no-reply
2026-11-10 19:18 ` [Devel] [PATCH hci-8.1 2/3] ms/iommu/amd: serialize sequence allocation under concurrent TLB invalidations Denis V. Lunev
` (2 subsequent siblings)
3 siblings, 1 reply; 8+ messages in thread
From: Denis V. Lunev @ 2026-11-10 19:18 UTC (permalink / raw)
To: khorenko; +Cc: devel
From: Ankit Soni <Ankit.Soni@amd.com>
With iommu.strict=1, the existing completion wait path can cause soft
lockups under stressed environment, as wait_on_sem() busy-waits under the
spinlock with interrupts disabled.
Move the completion wait in iommu_completion_wait() out of the spinlock.
wait_on_sem() only polls the hardware-updated cmd_sem and does not require
iommu->lock, so holding the lock during the busy wait unnecessarily
increases contention and extends the time with interrupts disabled.
Signed-off-by: Ankit Soni <Ankit.Soni@amd.com>
Reviewed-by: Vasant Hegde <vasant.hegde@amd.com>
Signed-off-by: Joerg Roedel <joerg.roedel@amd.com>
(cherry picked from commit d2a0cac10597068567d336e85fa3cbdbe8ca62bf)
https://virtuozzo.atlassian.net/browse/VSTOR-154424
Feature: fix ms/iommu
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
drivers/iommu/amd/iommu.c | 25 +++++++++++++++++--------
1 file changed, 17 insertions(+), 8 deletions(-)
diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index 7c502c7d514e..00608e55e70c 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -1161,7 +1161,12 @@ static int wait_on_sem(struct amd_iommu *iommu, u64 data)
{
int i = 0;
- while (*iommu->cmd_sem != data && i < LOOP_TIMEOUT) {
+ /*
+ * cmd_sem holds a monotonically non-decreasing completion sequence
+ * number.
+ */
+ while ((__s64)(READ_ONCE(*iommu->cmd_sem) - data) < 0 &&
+ i < LOOP_TIMEOUT) {
udelay(1);
i += 1;
}
@@ -1406,14 +1411,13 @@ static int iommu_completion_wait(struct amd_iommu *iommu)
raw_spin_lock_irqsave(&iommu->lock, flags);
ret = __iommu_queue_command_sync(iommu, &cmd, false);
+ raw_spin_unlock_irqrestore(&iommu->lock, flags);
+
if (ret)
- goto out_unlock;
+ return ret;
ret = wait_on_sem(iommu, data);
-out_unlock:
- raw_spin_unlock_irqrestore(&iommu->lock, flags);
-
return ret;
}
@@ -3094,13 +3098,18 @@ static void iommu_flush_irt_and_complete(struct amd_iommu *iommu, u16 devid)
raw_spin_lock_irqsave(&iommu->lock, flags);
ret = __iommu_queue_command_sync(iommu, &cmd, true);
if (ret)
- goto out;
+ goto out_err;
ret = __iommu_queue_command_sync(iommu, &cmd2, false);
if (ret)
- goto out;
+ goto out_err;
+ raw_spin_unlock_irqrestore(&iommu->lock, flags);
+
wait_on_sem(iommu, data);
-out:
+ return;
+
+out_err:
raw_spin_unlock_irqrestore(&iommu->lock, flags);
+ return;
}
static inline u8 iommu_get_int_tablen(struct iommu_dev_data *dev_data)
--
2.53.0
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* [Devel] [PATCH hci-8.1 2/3] ms/iommu/amd: serialize sequence allocation under concurrent TLB invalidations
2026-11-10 19:18 [Devel] [PATCH hci-8.1 0/3] iommu/amd: make iommu_completion_wait() wait for flushes queued by other CPUs Denis V. Lunev
2026-11-10 19:18 ` [Devel] [PATCH hci-8.1 1/3] ms/iommu/amd: move wait_on_sem() out of spinlock Denis V. Lunev
@ 2026-11-10 19:18 ` Denis V. Lunev
2026-11-10 19:22 ` no-reply
2026-11-10 19:18 ` [Devel] [PATCH hci-8.1 3/3] ms/iommu/amd: Wait for completion instead of returning early in iommu_completion_wait() Denis V. Lunev
2026-11-12 12:40 ` [Devel] [PATCH hci-8.1 0/3] iommu/amd: make iommu_completion_wait() wait for flushes queued by other CPUs Konstantin Khorenko
3 siblings, 1 reply; 8+ messages in thread
From: Denis V. Lunev @ 2026-11-10 19:18 UTC (permalink / raw)
To: khorenko; +Cc: devel
From: Ankit Soni <Ankit.Soni@amd.com>
With concurrent TLB invalidations, completion wait randomly gets timed out
because cmd_sem_val was incremented outside the IOMMU spinlock, allowing
CMD_COMPL_WAIT commands to be queued out of sequence and breaking the
ordering assumption in wait_on_sem().
Move the cmd_sem_val increment under iommu->lock so completion sequence
allocation is serialized with command queuing.
And remove the unnecessary return.
Fixes: d2a0cac10597 ("iommu/amd: move wait_on_sem() out of spinlock")
Tested-by: Srikanth Aithal <sraithal@amd.com>
Reported-by: Srikanth Aithal <sraithal@amd.com>
Signed-off-by: Ankit Soni <Ankit.Soni@amd.com>
Reviewed-by: Vasant Hegde <vasant.hegde@amd.com>
Signed-off-by: Joerg Roedel <joerg.roedel@amd.com>
(cherry picked from commit 9e249c48412828e807afddc21527eb734dc9bd3d)
Context adjusted: vz10 does not have cmd_sem_paddr in struct amd_iommu.
https://virtuozzo.atlassian.net/browse/VSTOR-154424
Feature: fix ms/iommu
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
drivers/iommu/amd/amd_iommu_types.h | 2 +-
drivers/iommu/amd/init.c | 2 +-
drivers/iommu/amd/iommu.c | 18 ++++++++++++------
3 files changed, 14 insertions(+), 8 deletions(-)
diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
index 95f63c5f6159..05a54f8f0434 100644
--- a/drivers/iommu/amd/amd_iommu_types.h
+++ b/drivers/iommu/amd/amd_iommu_types.h
@@ -791,7 +791,7 @@ struct amd_iommu {
u32 flags;
volatile u64 *cmd_sem;
- atomic64_t cmd_sem_val;
+ u64 cmd_sem_val;
#ifdef CONFIG_AMD_IOMMU_DEBUGFS
/* DebugFS Info */
diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c
index 07f1159a5e28..36ea9ce775ec 100644
--- a/drivers/iommu/amd/init.c
+++ b/drivers/iommu/amd/init.c
@@ -1749,7 +1749,7 @@ static int __init init_iommu_one(struct amd_iommu *iommu, struct ivhd_header *h,
iommu->pci_seg = pci_seg;
raw_spin_lock_init(&iommu->lock);
- atomic64_set(&iommu->cmd_sem_val, 0);
+ iommu->cmd_sem_val = 0;
/* Add IOMMU to internal data structures */
list_add_tail(&iommu->list, &amd_iommu_list);
diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index 00608e55e70c..373596fa07c0 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -1391,6 +1391,12 @@ static int iommu_queue_command(struct amd_iommu *iommu, struct iommu_cmd *cmd)
return iommu_queue_command_sync(iommu, cmd, true);
}
+static u64 get_cmdsem_val(struct amd_iommu *iommu)
+{
+ lockdep_assert_held(&iommu->lock);
+ return ++iommu->cmd_sem_val;
+}
+
/*
* This function queues a completion wait command into the command
* buffer of an IOMMU
@@ -1405,11 +1411,11 @@ static int iommu_completion_wait(struct amd_iommu *iommu)
if (!iommu->need_sync)
return 0;
- data = atomic64_inc_return(&iommu->cmd_sem_val);
- build_completion_wait(&cmd, iommu, data);
-
raw_spin_lock_irqsave(&iommu->lock, flags);
+ data = get_cmdsem_val(iommu);
+ build_completion_wait(&cmd, iommu, data);
+
ret = __iommu_queue_command_sync(iommu, &cmd, false);
raw_spin_unlock_irqrestore(&iommu->lock, flags);
@@ -3092,10 +3098,11 @@ static void iommu_flush_irt_and_complete(struct amd_iommu *iommu, u16 devid)
return;
build_inv_irt(&cmd, devid);
- data = atomic64_inc_return(&iommu->cmd_sem_val);
- build_completion_wait(&cmd2, iommu, data);
raw_spin_lock_irqsave(&iommu->lock, flags);
+ data = get_cmdsem_val(iommu);
+ build_completion_wait(&cmd2, iommu, data);
+
ret = __iommu_queue_command_sync(iommu, &cmd, true);
if (ret)
goto out_err;
@@ -3109,7 +3116,6 @@ static void iommu_flush_irt_and_complete(struct amd_iommu *iommu, u16 devid)
out_err:
raw_spin_unlock_irqrestore(&iommu->lock, flags);
- return;
}
static inline u8 iommu_get_int_tablen(struct iommu_dev_data *dev_data)
--
2.53.0
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* [Devel] [PATCH hci-8.1 3/3] ms/iommu/amd: Wait for completion instead of returning early in iommu_completion_wait()
2026-11-10 19:18 [Devel] [PATCH hci-8.1 0/3] iommu/amd: make iommu_completion_wait() wait for flushes queued by other CPUs Denis V. Lunev
2026-11-10 19:18 ` [Devel] [PATCH hci-8.1 1/3] ms/iommu/amd: move wait_on_sem() out of spinlock Denis V. Lunev
2026-11-10 19:18 ` [Devel] [PATCH hci-8.1 2/3] ms/iommu/amd: serialize sequence allocation under concurrent TLB invalidations Denis V. Lunev
@ 2026-11-10 19:18 ` Denis V. Lunev
2026-11-10 19:24 ` no-reply
2026-11-12 12:40 ` [Devel] [PATCH hci-8.1 0/3] iommu/amd: make iommu_completion_wait() wait for flushes queued by other CPUs Konstantin Khorenko
3 siblings, 1 reply; 8+ messages in thread
From: Denis V. Lunev @ 2026-11-10 19:18 UTC (permalink / raw)
To: khorenko; +Cc: devel
From: Guanghui Feng <guanghuifeng@linux.alibaba.com>
need_sync is a per-IOMMU flag shared by all domains and devices behind
that IOMMU. It is set whenever a command is queued with sync == true and
cleared when a completion-wait (CWAIT) command is queued. However, a
cleared need_sync only means that a covering CWAIT has been queued, not
that all previously queued commands have actually completed in hardware.
iommu_completion_wait() read need_sync locklessly and returned early
when it was false. This breaks the "block until all previously queued
commands have completed" contract in a multi-CPU scenario:
CPU2: queue inv-B => need_sync = true
CPU1: queue CWAIT(N); need_sync = false; then wait_on_sem(N)
CPU2: read need_sync == false => return 0 (no wait!)
CPU2 returns without waiting for any sequence number even though its
inv-B may not have completed yet (CWAIT(N), queued after inv-B, has not
been signaled). CPU2 then proceeds to, for example, free page-table
pages while the IOMMU can still walk stale translations, opening a
use-after-free window. This is a logical race in the meaning of the
flag, not a memory-visibility issue, so barriers alone do not help.
Fix it without losing the optimization of avoiding redundant CWAIT
commands: take iommu->lock before testing need_sync, and when it is
false do not return early but wait for the last allocated sequence
number (cmd_sem_val). Since need_sync == false implies no sync command
was queued after the last CWAIT, that CWAIT is FIFO-ordered after every
not-yet-completed command, so waiting for its sequence number guarantees
all prior commands (possibly queued by another CPU) have completed. The
common path with pending work is unchanged and no extra hardware command
is issued.
Signed-off-by: Guanghui Feng <guanghuifeng@linux.alibaba.com>
Fixes: 815b33fdc279 ("x86/amd-iommu: Cleanup completion-wait handling")
Reviewed-by: Vasant Hegde <vasant.hegde@amd.com>
Signed-off-by: Will Deacon <will@kernel.org>
(cherry picked from commit 1e75a8255f11c81fb07e81e5029cfd75804350a0)
https://virtuozzo.atlassian.net/browse/VSTOR-154424
Feature: fix ms/iommu
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
drivers/iommu/amd/iommu.c | 22 ++++++++++++++++------
1 file changed, 16 insertions(+), 6 deletions(-)
diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index 373596fa07c0..91cf277f1293 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -1408,11 +1408,23 @@ static int iommu_completion_wait(struct amd_iommu *iommu)
int ret;
u64 data;
- if (!iommu->need_sync)
- return 0;
-
raw_spin_lock_irqsave(&iommu->lock, flags);
+ if (!iommu->need_sync) {
+ /*
+ * No command has been queued since the last completion-wait.
+ * A concurrent CPU may have already queued that CWAIT and
+ * cleared need_sync; need_sync == false only means a covering
+ * CWAIT is queued, not that all prior commands have completed.
+ * Wait for the last allocated sequence number so that any
+ * command queued before this call (possibly on another CPU)
+ * is guaranteed to have completed before returning.
+ */
+ data = iommu->cmd_sem_val;
+ raw_spin_unlock_irqrestore(&iommu->lock, flags);
+ return wait_on_sem(iommu, data);
+ }
+
data = get_cmdsem_val(iommu);
build_completion_wait(&cmd, iommu, data);
@@ -1422,9 +1434,7 @@ static int iommu_completion_wait(struct amd_iommu *iommu)
if (ret)
return ret;
- ret = wait_on_sem(iommu, data);
-
- return ret;
+ return wait_on_sem(iommu, data);
}
static void domain_flush_complete(struct protection_domain *domain)
--
2.53.0
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Devel] [PATCH hci-8.1 2/3] ms/iommu/amd: serialize sequence allocation under concurrent TLB invalidations
2026-11-10 19:18 ` [Devel] [PATCH hci-8.1 2/3] ms/iommu/amd: serialize sequence allocation under concurrent TLB invalidations Denis V. Lunev
@ 2026-11-10 19:22 ` no-reply
0 siblings, 0 replies; 8+ messages in thread
From: no-reply @ 2026-11-10 19:22 UTC (permalink / raw)
To: Denis V. Lunev, khorenko; +Cc: devel
> From: Ankit Soni <Ankit.Soni@amd.com>
>
> With concurrent TLB invalidations, completion wait randomly gets timed out
> because cmd_sem_val was incremented outside the IOMMU spinlock, allowing
> CMD_COMPL_WAIT commands to be queued out of sequence and breaking the
> ordering assumption in wait_on_sem().
> Move the cmd_sem_val increment under iommu->lock so completion sequence
> allocation is serialized with command queuing.
> And remove the unnecessary return.
>
> Fixes: d2a0cac10597 ("iommu/amd: move wait_on_sem() out of spinlock")
>
> Tested-by: Srikanth Aithal <sraithal@amd.com>
> Reported-by: Srikanth Aithal <sraithal@amd.com>
> Signed-off-by: Ankit Soni <Ankit.Soni@amd.com>
> Reviewed-by: Vasant Hegde <vasant.hegde@amd.com>
> Signed-off-by: Joerg Roedel <joerg.roedel@amd.com>
> (cherry picked from commit 9e249c48412828e807afddc21527eb734dc9bd3d)
>
> Context adjusted: vz10 does not have cmd_sem_paddr in struct amd_iommu.
>
> https://virtuozzo.atlassian.net/browse/VSTOR-154424
> Feature: fix ms/iommu
>
> Signed-off-by: Denis V. Lunev <den@openvz.org>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261110191836.3000352-1-den@openvz.org?part=2
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Devel] [PATCH hci-8.1 1/3] ms/iommu/amd: move wait_on_sem() out of spinlock
2026-11-10 19:18 ` [Devel] [PATCH hci-8.1 1/3] ms/iommu/amd: move wait_on_sem() out of spinlock Denis V. Lunev
@ 2026-11-10 19:23 ` no-reply
0 siblings, 0 replies; 8+ messages in thread
From: no-reply @ 2026-11-10 19:23 UTC (permalink / raw)
To: khorenko, Denis V. Lunev; +Cc: devel
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] iommu/amd: cmd_sem seq allocated outside lock, signed wait_on_sem() unsafe
- [Low] iommu/amd: wait_on_sem() semantics change to signed >= is not in commit message
--
commit 1ce3255ad1464c0ff15e279b0d0ff6234294a548
Author: Ankit Soni <Ankit.Soni@amd.com>
ms/iommu/amd: move wait_on_sem() out of spinlock
This commit moves the wait_on_sem() busy-wait in iommu_completion_wait()
and iommu_flush_irt_and_complete() out of the iommu->lock critical section.
It also changes the wait condition in wait_on_sem() from an equality test
to a signed comparison.
> diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
> --- a/drivers/iommu/amd/iommu.c
> +++ b/drivers/iommu/amd/iommu.c
[ ... ]
> @@ -1161,7 +1161,12 @@ static int wait_on_sem(struct amd_iommu *iommu, u64 data)
> {
> int i = 0;
>
> - while (*iommu->cmd_sem != data && i < LOOP_TIMEOUT) {
> + /*
> + * cmd_sem holds a monotonically non-decreasing completion sequence
> + * number.
> + */
> + while ((__s64)(READ_ONCE(*iommu->cmd_sem) - data) < 0 &&
> + i < LOOP_TIMEOUT) {
> udelay(1);
> i += 1;
> }
[Severity: Low]
The commit message says only that the wait is moved out of the spinlock and
that wait_on_sem() "only polls the hardware-updated cmd_sem and does not
require iommu->lock". Is that a complete description of the change?
The patch also changes the wait condition in wait_on_sem() from an exact
match to a signed comparison, adds READ_ONCE(), and restructures the error
path in iommu_flush_irt_and_complete(). None of this is mentioned.
The new comment states "cmd_sem holds a monotonically non-decreasing
completion sequence number". Is that guaranteed by this patch alone? The
sequence number is still allocated with atomic64_inc_return() before
iommu->lock is taken. That invariant only becomes true with the follow-up
patch, which allocates the number under the lock. The later patches in the
series do not correct this commit message.
[ ... ]
> @@ -1406,14 +1411,13 @@ static int iommu_completion_wait(struct amd_iommu *iommu)
> raw_spin_lock_irqsave(&iommu->lock, flags);
>
> ret = __iommu_queue_command_sync(iommu, &cmd, false);
> + raw_spin_unlock_irqrestore(&iommu->lock, flags);
> +
> if (ret)
> - goto out_unlock;
> + return ret;
>
> ret = wait_on_sem(iommu, data);
>
> -out_unlock:
> - raw_spin_unlock_irqrestore(&iommu->lock, flags);
> -
> return ret;
> }
[Severity: Medium]
Can wait_on_sem() return before this CPU's commands have executed?
In iommu_completion_wait(), the sequence number is allocated before the
lock is taken:
data = atomic64_inc_return(&iommu->cmd_sem_val);
build_completion_wait(&cmd, iommu, data);
raw_spin_lock_irqsave(&iommu->lock, flags);
ret = __iommu_queue_command_sync(iommu, &cmd, false);
raw_spin_unlock_irqrestore(&iommu->lock, flags);
Queue order can therefore differ from sequence order. Consider two CPUs
running this concurrently, for example with iommu.strict=1:
CPU A: data = 5, then is delayed before taking iommu->lock
CPU B: data = 6, takes the lock, queues COMPLETION_WAIT(6), unlocks
CPU A: takes the lock, queues its commands and COMPLETION_WAIT(5)
The hardware processes the queue in order, so *cmd_sem becomes 6 and later
5. This contradicts the new comment in wait_on_sem() about monotonicity.
CPU A then sees 6 and computes (6 - 5) >= 0 in the new signed test, so it
stops waiting. Its own invalidations and COMPLETION_WAIT(5) have not
executed yet. Before this patch, the lock held across queue and wait, plus
the equality test, prevented this early return.
CPU B can also poll only after the hardware has written 5, so (5 - 6) < 0.
It spins for LOOP_TIMEOUT and then returns -EIO with the "Completion-Wait
loop timed out" alert. The equality test had the same weakness in this
case, so that part is not new.
The follow-up patch "ms/iommu/amd: serialize sequence allocation under
concurrent TLB invalidations" moves the allocation under iommu->lock, so
this looks like a regression only in this intermediate commit, but it
affects bisection.
[ ... ]
> @@ -3094,13 +3098,18 @@ static void iommu_flush_irt_and_complete(struct amd_iommu *iommu, u16 devid)
> raw_spin_lock_irqsave(&iommu->lock, flags);
> ret = __iommu_queue_command_sync(iommu, &cmd, true);
> if (ret)
> - goto out;
> + goto out_err;
> ret = __iommu_queue_command_sync(iommu, &cmd2, false);
> if (ret)
> - goto out;
> + goto out_err;
> + raw_spin_unlock_irqrestore(&iommu->lock, flags);
> +
> wait_on_sem(iommu, data);
> -out:
> + return;
> +
> +out_err:
> raw_spin_unlock_irqrestore(&iommu->lock, flags);
> + return;
> }
[Severity: Medium]
Does iommu_flush_irt_and_complete() have the same exposure?
Its sequence number is also allocated outside the lock:
data = atomic64_inc_return(&iommu->cmd_sem_val);
build_completion_wait(&cmd2, iommu, data);
raw_spin_lock_irqsave(&iommu->lock, flags);
...
raw_spin_unlock_irqrestore(&iommu->lock, flags);
wait_on_sem(iommu, data);
With the signed comparison in wait_on_sem(), a CPU holding a lower number
can stop waiting once another CPU's higher number has been written, before
its IRTE invalidation has executed. The return value of wait_on_sem() is
also ignored here, so a timeout would not be reported.
--
Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261110191836.3000352-1-den@openvz.org?part=1
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Devel] [PATCH hci-8.1 3/3] ms/iommu/amd: Wait for completion instead of returning early in iommu_completion_wait()
2026-11-10 19:18 ` [Devel] [PATCH hci-8.1 3/3] ms/iommu/amd: Wait for completion instead of returning early in iommu_completion_wait() Denis V. Lunev
@ 2026-11-10 19:24 ` no-reply
0 siblings, 0 replies; 8+ messages in thread
From: no-reply @ 2026-11-10 19:24 UTC (permalink / raw)
To: khorenko, Denis V. Lunev; +Cc: devel
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] iommu/amd: iommu_completion_wait() can wait on a cmd_sem_val with no queued CWAIT
--
commit 17e0a27d76ec180494393cc226f8033b509119b2
Author: Guanghui Feng <guanghuifeng@linux.alibaba.com>
ms/iommu/amd: Wait for completion instead of returning early in iommu_completion_wait()
This is a backport of an upstream fix. iommu_completion_wait() now takes
iommu->lock before testing need_sync. When need_sync is false, it waits
on the last allocated sequence number (cmd_sem_val) instead of returning
early.
> diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
> index 373596fa07c0..91cf277f1293 100644
> --- a/drivers/iommu/amd/iommu.c
> +++ b/drivers/iommu/amd/iommu.c
> @@ -1408,11 +1408,23 @@ static int iommu_completion_wait(struct amd_iommu *iommu)
> int ret;
> u64 data;
>
> - if (!iommu->need_sync)
> - return 0;
> -
> raw_spin_lock_irqsave(&iommu->lock, flags);
>
> + if (!iommu->need_sync) {
> + /*
> + * No command has been queued since the last completion-wait.
> + * A concurrent CPU may have already queued that CWAIT and
> + * cleared need_sync; need_sync == false only means a covering
> + * CWAIT is queued, not that all prior commands have completed.
> + * Wait for the last allocated sequence number so that any
> + * command queued before this call (possibly on another CPU)
> + * is guaranteed to have completed before returning.
> + */
> + data = iommu->cmd_sem_val;
[Severity: Low]
Can cmd_sem_val here be a number for which no CWAIT was ever queued?
get_cmdsem_val() increments cmd_sem_val before anything is queued. In
iommu_flush_irt_and_complete() the sequence is:
data = get_cmdsem_val(iommu);
build_completion_wait(&cmd2, iommu, data);
ret = __iommu_queue_command_sync(iommu, &cmd, true);
if (ret)
goto out_err;
If the first __iommu_queue_command_sync() fails with "Command buffer
timeout", it returns -EIO before copy_cmd_to_buffer() and before
"iommu->need_sync = sync". So need_sync keeps its previous value, which is
false if the last queued command was a CWAIT. No CWAIT carrying number N is
queued.
A later iommu_completion_wait() then sees need_sync == false, takes this
new branch, reads data = cmd_sem_val (N), and calls wait_on_sem(iommu, N).
*cmd_sem can never reach N, so the wait spins for LOOP_TIMEOUT iterations
of udelay(1) and returns -EIO with "Completion-Wait loop timed out".
Before the patch the same state returned 0 immediately. Callers such as
domain_flush_complete() would now stall for the full timeout while holding
domain->lock. The state lasts until another command sets need_sync and a
CWAIT with a higher sequence number completes.
This needs a command-buffer-full timeout, so the IOMMU is already
misbehaving. The failure of the second queue call in
iommu_flush_irt_and_complete() is not affected, because the first call has
already set need_sync = true.
The commit message says "Since need_sync == false implies no sync command
was queued after the last CWAIT, that CWAIT is FIFO-ordered after every
not-yet-completed command, so waiting for its sequence number guarantees
all prior commands ... have completed." Is that accurate, given that
cmd_sem_val can be ahead of the last CWAIT that was actually queued?
> + raw_spin_unlock_irqrestore(&iommu->lock, flags);
> + return wait_on_sem(iommu, data);
> + }
> +
> data = get_cmdsem_val(iommu);
> build_completion_wait(&cmd, iommu, data);
>
--
Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261110191836.3000352-1-den@openvz.org?part=3
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Devel] [PATCH hci-8.1 0/3] iommu/amd: make iommu_completion_wait() wait for flushes queued by other CPUs
2026-11-10 19:18 [Devel] [PATCH hci-8.1 0/3] iommu/amd: make iommu_completion_wait() wait for flushes queued by other CPUs Denis V. Lunev
` (2 preceding siblings ...)
2026-11-10 19:18 ` [Devel] [PATCH hci-8.1 3/3] ms/iommu/amd: Wait for completion instead of returning early in iommu_completion_wait() Denis V. Lunev
@ 2026-11-12 12:40 ` Konstantin Khorenko
3 siblings, 0 replies; 8+ messages in thread
From: Konstantin Khorenko @ 2026-11-12 12:40 UTC (permalink / raw)
To: Denis V. Lunev; +Cc: devel
On Tue, 10 Nov 2026 20:18:33 +0100, Denis V. Lunev wrote:
> iommu/amd: make iommu_completion_wait() wait for flushes queued by other CPUs
>
> iommu_completion_wait() checks iommu->need_sync without iommu->lock and
> returns at once when it is clear. A completion wait queued by another
> CPU clears the flag before the IOMMU has executed the commands in front
> of it, so the caller can return while its own invalidations are still
> pending. After an unmap that lets an IOVA or a page table page be
> reused while the IOMMU may still hold the old translation. It also
> leaves a hole in "iommu/amd/pgtbl: fix DMA faults on IOVAs mapped while
> the table grows": the DTE flush done before the new page table level is
> published can return before the IOMMU has dropped the old DTE.
>
> [...]
Applied to branch-rh10-6.12.0-211.52.1.22.x.vz10-ovz, will be released in rh10-6.12.0-211.52.1.22.21.vz10, thanks!
[1/3] ms/iommu/amd: move wait_on_sem() out of spinlock
https://bitbucket.org/virtuozzocore/vzkernel.vzs/commits/3e6384ffedcb
[2/3] ms/iommu/amd: serialize sequence allocation under concurrent TLB invalidations
https://bitbucket.org/virtuozzocore/vzkernel.vzs/commits/5a6596791b87
[3/3] ms/iommu/amd: Wait for completion instead of returning early in iommu_completion_wait()
https://bitbucket.org/virtuozzocore/vzkernel.vzs/commits/516fe340b0d5
--
Konstantin Khorenko,
Virtuozzo Linux Kernel Team
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2026-11-12 12:42 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-10 19:18 [Devel] [PATCH hci-8.1 0/3] iommu/amd: make iommu_completion_wait() wait for flushes queued by other CPUs Denis V. Lunev
2026-11-10 19:18 ` [Devel] [PATCH hci-8.1 1/3] ms/iommu/amd: move wait_on_sem() out of spinlock Denis V. Lunev
2026-11-10 19:23 ` no-reply
2026-11-10 19:18 ` [Devel] [PATCH hci-8.1 2/3] ms/iommu/amd: serialize sequence allocation under concurrent TLB invalidations Denis V. Lunev
2026-11-10 19:22 ` no-reply
2026-11-10 19:18 ` [Devel] [PATCH hci-8.1 3/3] ms/iommu/amd: Wait for completion instead of returning early in iommu_completion_wait() Denis V. Lunev
2026-11-10 19:24 ` no-reply
2026-11-12 12:40 ` [Devel] [PATCH hci-8.1 0/3] iommu/amd: make iommu_completion_wait() wait for flushes queued by other CPUs Konstantin Khorenko
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox