* [Devel] [PATCH hci-8.1 0/3] iommu/amd: make iommu_completion_wait() wait for flushes queued by other CPUs
@ 2026-11-10 19:18 Denis V. Lunev
2026-11-10 19:18 ` [Devel] [PATCH hci-8.1 1/3] ms/iommu/amd: move wait_on_sem() out of spinlock Denis V. Lunev
` (3 more replies)
0 siblings, 4 replies; 8+ messages in thread
From: Denis V. Lunev @ 2026-11-10 19:18 UTC (permalink / raw)
To: khorenko; +Cc: devel
iommu_completion_wait() checks iommu->need_sync without iommu->lock and
returns at once when it is clear. A completion wait queued by another
CPU clears the flag before the IOMMU has executed the commands in front
of it, so the caller can return while its own invalidations are still
pending. After an unmap that lets an IOVA or a page table page be
reused while the IOMMU may still hold the old translation. It also
leaves a hole in "iommu/amd/pgtbl: fix DMA faults on IOVAs mapped while
the table grows": the DTE flush done before the new page table level is
published can return before the IOMMU has dropped the old DTE.
Patch 3 is the mainline fix 1e75a8255f11c. It waits for the last
allocated completion sequence number, which relies on the model
introduced by patches 1 and 2, also from mainline: the wait moves out of
iommu->lock and the sequence number is allocated under it. vz10 still
had the old model, an atomic increment outside the lock and the wait
under it. 6.12.y carries the same three commits. Patch 2 needed a
context fix only: vz10 has no cmd_sem_paddr in struct amd_iommu.
Tested on epyc06 (EPYC 7352, four Optane in one IOMMU group) with the
VSTOR-154276 reproducer: an out-of-tree module exhausts the group's
32-bit IOVA space while fio reads the drives, and a kprobe holds the DTE
update open for 5 ms. The stock 8.0 kernel gives IO_PAGE_FAULTs and fio
verify mismatches with no I/O error. 22.19 with the VSTOR-154276 fix and
this series gives no faults, no mismatches and no completion wait
timeouts.
https://virtuozzo.atlassian.net/browse/VSTOR-154424
Ankit Soni (2):
ms/iommu/amd: move wait_on_sem() out of spinlock
ms/iommu/amd: serialize sequence allocation under concurrent TLB
invalidations
Guanghui Feng (1):
ms/iommu/amd: Wait for completion instead of returning early in
iommu_completion_wait()
drivers/iommu/amd/amd_iommu_types.h | 2 +-
drivers/iommu/amd/init.c | 2 +-
drivers/iommu/amd/iommu.c | 61 ++++++++++++++++++++---------
3 files changed, 45 insertions(+), 20 deletions(-)
--
2.53.0
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 8+ messages in thread* [Devel] [PATCH hci-8.1 1/3] ms/iommu/amd: move wait_on_sem() out of spinlock 2026-11-10 19:18 [Devel] [PATCH hci-8.1 0/3] iommu/amd: make iommu_completion_wait() wait for flushes queued by other CPUs Denis V. Lunev @ 2026-11-10 19:18 ` Denis V. Lunev 2026-11-10 19:23 ` no-reply 2026-11-10 19:18 ` [Devel] [PATCH hci-8.1 2/3] ms/iommu/amd: serialize sequence allocation under concurrent TLB invalidations Denis V. Lunev ` (2 subsequent siblings) 3 siblings, 1 reply; 8+ messages in thread From: Denis V. Lunev @ 2026-11-10 19:18 UTC (permalink / raw) To: khorenko; +Cc: devel From: Ankit Soni <Ankit.Soni@amd.com> With iommu.strict=1, the existing completion wait path can cause soft lockups under stressed environment, as wait_on_sem() busy-waits under the spinlock with interrupts disabled. Move the completion wait in iommu_completion_wait() out of the spinlock. wait_on_sem() only polls the hardware-updated cmd_sem and does not require iommu->lock, so holding the lock during the busy wait unnecessarily increases contention and extends the time with interrupts disabled. Signed-off-by: Ankit Soni <Ankit.Soni@amd.com> Reviewed-by: Vasant Hegde <vasant.hegde@amd.com> Signed-off-by: Joerg Roedel <joerg.roedel@amd.com> (cherry picked from commit d2a0cac10597068567d336e85fa3cbdbe8ca62bf) https://virtuozzo.atlassian.net/browse/VSTOR-154424 Feature: fix ms/iommu Signed-off-by: Denis V. Lunev <den@openvz.org> --- drivers/iommu/amd/iommu.c | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c index 7c502c7d514e..00608e55e70c 100644 --- a/drivers/iommu/amd/iommu.c +++ b/drivers/iommu/amd/iommu.c @@ -1161,7 +1161,12 @@ static int wait_on_sem(struct amd_iommu *iommu, u64 data) { int i = 0; - while (*iommu->cmd_sem != data && i < LOOP_TIMEOUT) { + /* + * cmd_sem holds a monotonically non-decreasing completion sequence + * number. + */ + while ((__s64)(READ_ONCE(*iommu->cmd_sem) - data) < 0 && + i < LOOP_TIMEOUT) { udelay(1); i += 1; } @@ -1406,14 +1411,13 @@ static int iommu_completion_wait(struct amd_iommu *iommu) raw_spin_lock_irqsave(&iommu->lock, flags); ret = __iommu_queue_command_sync(iommu, &cmd, false); + raw_spin_unlock_irqrestore(&iommu->lock, flags); + if (ret) - goto out_unlock; + return ret; ret = wait_on_sem(iommu, data); -out_unlock: - raw_spin_unlock_irqrestore(&iommu->lock, flags); - return ret; } @@ -3094,13 +3098,18 @@ static void iommu_flush_irt_and_complete(struct amd_iommu *iommu, u16 devid) raw_spin_lock_irqsave(&iommu->lock, flags); ret = __iommu_queue_command_sync(iommu, &cmd, true); if (ret) - goto out; + goto out_err; ret = __iommu_queue_command_sync(iommu, &cmd2, false); if (ret) - goto out; + goto out_err; + raw_spin_unlock_irqrestore(&iommu->lock, flags); + wait_on_sem(iommu, data); -out: + return; + +out_err: raw_spin_unlock_irqrestore(&iommu->lock, flags); + return; } static inline u8 iommu_get_int_tablen(struct iommu_dev_data *dev_data) -- 2.53.0 _______________________________________________ Devel mailing list Devel@openvz.org https://lists.openvz.org/mailman/listinfo/devel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Devel] [PATCH hci-8.1 1/3] ms/iommu/amd: move wait_on_sem() out of spinlock 2026-11-10 19:18 ` [Devel] [PATCH hci-8.1 1/3] ms/iommu/amd: move wait_on_sem() out of spinlock Denis V. Lunev @ 2026-11-10 19:23 ` no-reply 0 siblings, 0 replies; 8+ messages in thread From: no-reply @ 2026-11-10 19:23 UTC (permalink / raw) To: khorenko, Denis V. Lunev; +Cc: devel Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] iommu/amd: cmd_sem seq allocated outside lock, signed wait_on_sem() unsafe - [Low] iommu/amd: wait_on_sem() semantics change to signed >= is not in commit message -- commit 1ce3255ad1464c0ff15e279b0d0ff6234294a548 Author: Ankit Soni <Ankit.Soni@amd.com> ms/iommu/amd: move wait_on_sem() out of spinlock This commit moves the wait_on_sem() busy-wait in iommu_completion_wait() and iommu_flush_irt_and_complete() out of the iommu->lock critical section. It also changes the wait condition in wait_on_sem() from an equality test to a signed comparison. > diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c > --- a/drivers/iommu/amd/iommu.c > +++ b/drivers/iommu/amd/iommu.c [ ... ] > @@ -1161,7 +1161,12 @@ static int wait_on_sem(struct amd_iommu *iommu, u64 data) > { > int i = 0; > > - while (*iommu->cmd_sem != data && i < LOOP_TIMEOUT) { > + /* > + * cmd_sem holds a monotonically non-decreasing completion sequence > + * number. > + */ > + while ((__s64)(READ_ONCE(*iommu->cmd_sem) - data) < 0 && > + i < LOOP_TIMEOUT) { > udelay(1); > i += 1; > } [Severity: Low] The commit message says only that the wait is moved out of the spinlock and that wait_on_sem() "only polls the hardware-updated cmd_sem and does not require iommu->lock". Is that a complete description of the change? The patch also changes the wait condition in wait_on_sem() from an exact match to a signed comparison, adds READ_ONCE(), and restructures the error path in iommu_flush_irt_and_complete(). None of this is mentioned. The new comment states "cmd_sem holds a monotonically non-decreasing completion sequence number". Is that guaranteed by this patch alone? The sequence number is still allocated with atomic64_inc_return() before iommu->lock is taken. That invariant only becomes true with the follow-up patch, which allocates the number under the lock. The later patches in the series do not correct this commit message. [ ... ] > @@ -1406,14 +1411,13 @@ static int iommu_completion_wait(struct amd_iommu *iommu) > raw_spin_lock_irqsave(&iommu->lock, flags); > > ret = __iommu_queue_command_sync(iommu, &cmd, false); > + raw_spin_unlock_irqrestore(&iommu->lock, flags); > + > if (ret) > - goto out_unlock; > + return ret; > > ret = wait_on_sem(iommu, data); > > -out_unlock: > - raw_spin_unlock_irqrestore(&iommu->lock, flags); > - > return ret; > } [Severity: Medium] Can wait_on_sem() return before this CPU's commands have executed? In iommu_completion_wait(), the sequence number is allocated before the lock is taken: data = atomic64_inc_return(&iommu->cmd_sem_val); build_completion_wait(&cmd, iommu, data); raw_spin_lock_irqsave(&iommu->lock, flags); ret = __iommu_queue_command_sync(iommu, &cmd, false); raw_spin_unlock_irqrestore(&iommu->lock, flags); Queue order can therefore differ from sequence order. Consider two CPUs running this concurrently, for example with iommu.strict=1: CPU A: data = 5, then is delayed before taking iommu->lock CPU B: data = 6, takes the lock, queues COMPLETION_WAIT(6), unlocks CPU A: takes the lock, queues its commands and COMPLETION_WAIT(5) The hardware processes the queue in order, so *cmd_sem becomes 6 and later 5. This contradicts the new comment in wait_on_sem() about monotonicity. CPU A then sees 6 and computes (6 - 5) >= 0 in the new signed test, so it stops waiting. Its own invalidations and COMPLETION_WAIT(5) have not executed yet. Before this patch, the lock held across queue and wait, plus the equality test, prevented this early return. CPU B can also poll only after the hardware has written 5, so (5 - 6) < 0. It spins for LOOP_TIMEOUT and then returns -EIO with the "Completion-Wait loop timed out" alert. The equality test had the same weakness in this case, so that part is not new. The follow-up patch "ms/iommu/amd: serialize sequence allocation under concurrent TLB invalidations" moves the allocation under iommu->lock, so this looks like a regression only in this intermediate commit, but it affects bisection. [ ... ] > @@ -3094,13 +3098,18 @@ static void iommu_flush_irt_and_complete(struct amd_iommu *iommu, u16 devid) > raw_spin_lock_irqsave(&iommu->lock, flags); > ret = __iommu_queue_command_sync(iommu, &cmd, true); > if (ret) > - goto out; > + goto out_err; > ret = __iommu_queue_command_sync(iommu, &cmd2, false); > if (ret) > - goto out; > + goto out_err; > + raw_spin_unlock_irqrestore(&iommu->lock, flags); > + > wait_on_sem(iommu, data); > -out: > + return; > + > +out_err: > raw_spin_unlock_irqrestore(&iommu->lock, flags); > + return; > } [Severity: Medium] Does iommu_flush_irt_and_complete() have the same exposure? Its sequence number is also allocated outside the lock: data = atomic64_inc_return(&iommu->cmd_sem_val); build_completion_wait(&cmd2, iommu, data); raw_spin_lock_irqsave(&iommu->lock, flags); ... raw_spin_unlock_irqrestore(&iommu->lock, flags); wait_on_sem(iommu, data); With the signed comparison in wait_on_sem(), a CPU holding a lower number can stop waiting once another CPU's higher number has been written, before its IRTE invalidation has executed. The return value of wait_on_sem() is also ignored here, so a timeout would not be reported. -- Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261110191836.3000352-1-den@openvz.org?part=1 _______________________________________________ Devel mailing list Devel@openvz.org https://lists.openvz.org/mailman/listinfo/devel ^ permalink raw reply [flat|nested] 8+ messages in thread
* [Devel] [PATCH hci-8.1 2/3] ms/iommu/amd: serialize sequence allocation under concurrent TLB invalidations 2026-11-10 19:18 [Devel] [PATCH hci-8.1 0/3] iommu/amd: make iommu_completion_wait() wait for flushes queued by other CPUs Denis V. Lunev 2026-11-10 19:18 ` [Devel] [PATCH hci-8.1 1/3] ms/iommu/amd: move wait_on_sem() out of spinlock Denis V. Lunev @ 2026-11-10 19:18 ` Denis V. Lunev 2026-11-10 19:22 ` no-reply 2026-11-10 19:18 ` [Devel] [PATCH hci-8.1 3/3] ms/iommu/amd: Wait for completion instead of returning early in iommu_completion_wait() Denis V. Lunev 2026-11-12 12:40 ` [Devel] [PATCH hci-8.1 0/3] iommu/amd: make iommu_completion_wait() wait for flushes queued by other CPUs Konstantin Khorenko 3 siblings, 1 reply; 8+ messages in thread From: Denis V. Lunev @ 2026-11-10 19:18 UTC (permalink / raw) To: khorenko; +Cc: devel From: Ankit Soni <Ankit.Soni@amd.com> With concurrent TLB invalidations, completion wait randomly gets timed out because cmd_sem_val was incremented outside the IOMMU spinlock, allowing CMD_COMPL_WAIT commands to be queued out of sequence and breaking the ordering assumption in wait_on_sem(). Move the cmd_sem_val increment under iommu->lock so completion sequence allocation is serialized with command queuing. And remove the unnecessary return. Fixes: d2a0cac10597 ("iommu/amd: move wait_on_sem() out of spinlock") Tested-by: Srikanth Aithal <sraithal@amd.com> Reported-by: Srikanth Aithal <sraithal@amd.com> Signed-off-by: Ankit Soni <Ankit.Soni@amd.com> Reviewed-by: Vasant Hegde <vasant.hegde@amd.com> Signed-off-by: Joerg Roedel <joerg.roedel@amd.com> (cherry picked from commit 9e249c48412828e807afddc21527eb734dc9bd3d) Context adjusted: vz10 does not have cmd_sem_paddr in struct amd_iommu. https://virtuozzo.atlassian.net/browse/VSTOR-154424 Feature: fix ms/iommu Signed-off-by: Denis V. Lunev <den@openvz.org> --- drivers/iommu/amd/amd_iommu_types.h | 2 +- drivers/iommu/amd/init.c | 2 +- drivers/iommu/amd/iommu.c | 18 ++++++++++++------ 3 files changed, 14 insertions(+), 8 deletions(-) diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h index 95f63c5f6159..05a54f8f0434 100644 --- a/drivers/iommu/amd/amd_iommu_types.h +++ b/drivers/iommu/amd/amd_iommu_types.h @@ -791,7 +791,7 @@ struct amd_iommu { u32 flags; volatile u64 *cmd_sem; - atomic64_t cmd_sem_val; + u64 cmd_sem_val; #ifdef CONFIG_AMD_IOMMU_DEBUGFS /* DebugFS Info */ diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c index 07f1159a5e28..36ea9ce775ec 100644 --- a/drivers/iommu/amd/init.c +++ b/drivers/iommu/amd/init.c @@ -1749,7 +1749,7 @@ static int __init init_iommu_one(struct amd_iommu *iommu, struct ivhd_header *h, iommu->pci_seg = pci_seg; raw_spin_lock_init(&iommu->lock); - atomic64_set(&iommu->cmd_sem_val, 0); + iommu->cmd_sem_val = 0; /* Add IOMMU to internal data structures */ list_add_tail(&iommu->list, &amd_iommu_list); diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c index 00608e55e70c..373596fa07c0 100644 --- a/drivers/iommu/amd/iommu.c +++ b/drivers/iommu/amd/iommu.c @@ -1391,6 +1391,12 @@ static int iommu_queue_command(struct amd_iommu *iommu, struct iommu_cmd *cmd) return iommu_queue_command_sync(iommu, cmd, true); } +static u64 get_cmdsem_val(struct amd_iommu *iommu) +{ + lockdep_assert_held(&iommu->lock); + return ++iommu->cmd_sem_val; +} + /* * This function queues a completion wait command into the command * buffer of an IOMMU @@ -1405,11 +1411,11 @@ static int iommu_completion_wait(struct amd_iommu *iommu) if (!iommu->need_sync) return 0; - data = atomic64_inc_return(&iommu->cmd_sem_val); - build_completion_wait(&cmd, iommu, data); - raw_spin_lock_irqsave(&iommu->lock, flags); + data = get_cmdsem_val(iommu); + build_completion_wait(&cmd, iommu, data); + ret = __iommu_queue_command_sync(iommu, &cmd, false); raw_spin_unlock_irqrestore(&iommu->lock, flags); @@ -3092,10 +3098,11 @@ static void iommu_flush_irt_and_complete(struct amd_iommu *iommu, u16 devid) return; build_inv_irt(&cmd, devid); - data = atomic64_inc_return(&iommu->cmd_sem_val); - build_completion_wait(&cmd2, iommu, data); raw_spin_lock_irqsave(&iommu->lock, flags); + data = get_cmdsem_val(iommu); + build_completion_wait(&cmd2, iommu, data); + ret = __iommu_queue_command_sync(iommu, &cmd, true); if (ret) goto out_err; @@ -3109,7 +3116,6 @@ static void iommu_flush_irt_and_complete(struct amd_iommu *iommu, u16 devid) out_err: raw_spin_unlock_irqrestore(&iommu->lock, flags); - return; } static inline u8 iommu_get_int_tablen(struct iommu_dev_data *dev_data) -- 2.53.0 _______________________________________________ Devel mailing list Devel@openvz.org https://lists.openvz.org/mailman/listinfo/devel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Devel] [PATCH hci-8.1 2/3] ms/iommu/amd: serialize sequence allocation under concurrent TLB invalidations 2026-11-10 19:18 ` [Devel] [PATCH hci-8.1 2/3] ms/iommu/amd: serialize sequence allocation under concurrent TLB invalidations Denis V. Lunev @ 2026-11-10 19:22 ` no-reply 0 siblings, 0 replies; 8+ messages in thread From: no-reply @ 2026-11-10 19:22 UTC (permalink / raw) To: Denis V. Lunev, khorenko; +Cc: devel > From: Ankit Soni <Ankit.Soni@amd.com> > > With concurrent TLB invalidations, completion wait randomly gets timed out > because cmd_sem_val was incremented outside the IOMMU spinlock, allowing > CMD_COMPL_WAIT commands to be queued out of sequence and breaking the > ordering assumption in wait_on_sem(). > Move the cmd_sem_val increment under iommu->lock so completion sequence > allocation is serialized with command queuing. > And remove the unnecessary return. > > Fixes: d2a0cac10597 ("iommu/amd: move wait_on_sem() out of spinlock") > > Tested-by: Srikanth Aithal <sraithal@amd.com> > Reported-by: Srikanth Aithal <sraithal@amd.com> > Signed-off-by: Ankit Soni <Ankit.Soni@amd.com> > Reviewed-by: Vasant Hegde <vasant.hegde@amd.com> > Signed-off-by: Joerg Roedel <joerg.roedel@amd.com> > (cherry picked from commit 9e249c48412828e807afddc21527eb734dc9bd3d) > > Context adjusted: vz10 does not have cmd_sem_paddr in struct amd_iommu. > > https://virtuozzo.atlassian.net/browse/VSTOR-154424 > Feature: fix ms/iommu > > Signed-off-by: Denis V. Lunev <den@openvz.org> Sashiko has reviewed this patch and found no issues. It looks great! -- Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261110191836.3000352-1-den@openvz.org?part=2 _______________________________________________ Devel mailing list Devel@openvz.org https://lists.openvz.org/mailman/listinfo/devel ^ permalink raw reply [flat|nested] 8+ messages in thread
* [Devel] [PATCH hci-8.1 3/3] ms/iommu/amd: Wait for completion instead of returning early in iommu_completion_wait() 2026-11-10 19:18 [Devel] [PATCH hci-8.1 0/3] iommu/amd: make iommu_completion_wait() wait for flushes queued by other CPUs Denis V. Lunev 2026-11-10 19:18 ` [Devel] [PATCH hci-8.1 1/3] ms/iommu/amd: move wait_on_sem() out of spinlock Denis V. Lunev 2026-11-10 19:18 ` [Devel] [PATCH hci-8.1 2/3] ms/iommu/amd: serialize sequence allocation under concurrent TLB invalidations Denis V. Lunev @ 2026-11-10 19:18 ` Denis V. Lunev 2026-11-10 19:24 ` no-reply 2026-11-12 12:40 ` [Devel] [PATCH hci-8.1 0/3] iommu/amd: make iommu_completion_wait() wait for flushes queued by other CPUs Konstantin Khorenko 3 siblings, 1 reply; 8+ messages in thread From: Denis V. Lunev @ 2026-11-10 19:18 UTC (permalink / raw) To: khorenko; +Cc: devel From: Guanghui Feng <guanghuifeng@linux.alibaba.com> need_sync is a per-IOMMU flag shared by all domains and devices behind that IOMMU. It is set whenever a command is queued with sync == true and cleared when a completion-wait (CWAIT) command is queued. However, a cleared need_sync only means that a covering CWAIT has been queued, not that all previously queued commands have actually completed in hardware. iommu_completion_wait() read need_sync locklessly and returned early when it was false. This breaks the "block until all previously queued commands have completed" contract in a multi-CPU scenario: CPU2: queue inv-B => need_sync = true CPU1: queue CWAIT(N); need_sync = false; then wait_on_sem(N) CPU2: read need_sync == false => return 0 (no wait!) CPU2 returns without waiting for any sequence number even though its inv-B may not have completed yet (CWAIT(N), queued after inv-B, has not been signaled). CPU2 then proceeds to, for example, free page-table pages while the IOMMU can still walk stale translations, opening a use-after-free window. This is a logical race in the meaning of the flag, not a memory-visibility issue, so barriers alone do not help. Fix it without losing the optimization of avoiding redundant CWAIT commands: take iommu->lock before testing need_sync, and when it is false do not return early but wait for the last allocated sequence number (cmd_sem_val). Since need_sync == false implies no sync command was queued after the last CWAIT, that CWAIT is FIFO-ordered after every not-yet-completed command, so waiting for its sequence number guarantees all prior commands (possibly queued by another CPU) have completed. The common path with pending work is unchanged and no extra hardware command is issued. Signed-off-by: Guanghui Feng <guanghuifeng@linux.alibaba.com> Fixes: 815b33fdc279 ("x86/amd-iommu: Cleanup completion-wait handling") Reviewed-by: Vasant Hegde <vasant.hegde@amd.com> Signed-off-by: Will Deacon <will@kernel.org> (cherry picked from commit 1e75a8255f11c81fb07e81e5029cfd75804350a0) https://virtuozzo.atlassian.net/browse/VSTOR-154424 Feature: fix ms/iommu Signed-off-by: Denis V. Lunev <den@openvz.org> --- drivers/iommu/amd/iommu.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c index 373596fa07c0..91cf277f1293 100644 --- a/drivers/iommu/amd/iommu.c +++ b/drivers/iommu/amd/iommu.c @@ -1408,11 +1408,23 @@ static int iommu_completion_wait(struct amd_iommu *iommu) int ret; u64 data; - if (!iommu->need_sync) - return 0; - raw_spin_lock_irqsave(&iommu->lock, flags); + if (!iommu->need_sync) { + /* + * No command has been queued since the last completion-wait. + * A concurrent CPU may have already queued that CWAIT and + * cleared need_sync; need_sync == false only means a covering + * CWAIT is queued, not that all prior commands have completed. + * Wait for the last allocated sequence number so that any + * command queued before this call (possibly on another CPU) + * is guaranteed to have completed before returning. + */ + data = iommu->cmd_sem_val; + raw_spin_unlock_irqrestore(&iommu->lock, flags); + return wait_on_sem(iommu, data); + } + data = get_cmdsem_val(iommu); build_completion_wait(&cmd, iommu, data); @@ -1422,9 +1434,7 @@ static int iommu_completion_wait(struct amd_iommu *iommu) if (ret) return ret; - ret = wait_on_sem(iommu, data); - - return ret; + return wait_on_sem(iommu, data); } static void domain_flush_complete(struct protection_domain *domain) -- 2.53.0 _______________________________________________ Devel mailing list Devel@openvz.org https://lists.openvz.org/mailman/listinfo/devel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Devel] [PATCH hci-8.1 3/3] ms/iommu/amd: Wait for completion instead of returning early in iommu_completion_wait() 2026-11-10 19:18 ` [Devel] [PATCH hci-8.1 3/3] ms/iommu/amd: Wait for completion instead of returning early in iommu_completion_wait() Denis V. Lunev @ 2026-11-10 19:24 ` no-reply 0 siblings, 0 replies; 8+ messages in thread From: no-reply @ 2026-11-10 19:24 UTC (permalink / raw) To: khorenko, Denis V. Lunev; +Cc: devel Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] iommu/amd: iommu_completion_wait() can wait on a cmd_sem_val with no queued CWAIT -- commit 17e0a27d76ec180494393cc226f8033b509119b2 Author: Guanghui Feng <guanghuifeng@linux.alibaba.com> ms/iommu/amd: Wait for completion instead of returning early in iommu_completion_wait() This is a backport of an upstream fix. iommu_completion_wait() now takes iommu->lock before testing need_sync. When need_sync is false, it waits on the last allocated sequence number (cmd_sem_val) instead of returning early. > diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c > index 373596fa07c0..91cf277f1293 100644 > --- a/drivers/iommu/amd/iommu.c > +++ b/drivers/iommu/amd/iommu.c > @@ -1408,11 +1408,23 @@ static int iommu_completion_wait(struct amd_iommu *iommu) > int ret; > u64 data; > > - if (!iommu->need_sync) > - return 0; > - > raw_spin_lock_irqsave(&iommu->lock, flags); > > + if (!iommu->need_sync) { > + /* > + * No command has been queued since the last completion-wait. > + * A concurrent CPU may have already queued that CWAIT and > + * cleared need_sync; need_sync == false only means a covering > + * CWAIT is queued, not that all prior commands have completed. > + * Wait for the last allocated sequence number so that any > + * command queued before this call (possibly on another CPU) > + * is guaranteed to have completed before returning. > + */ > + data = iommu->cmd_sem_val; [Severity: Low] Can cmd_sem_val here be a number for which no CWAIT was ever queued? get_cmdsem_val() increments cmd_sem_val before anything is queued. In iommu_flush_irt_and_complete() the sequence is: data = get_cmdsem_val(iommu); build_completion_wait(&cmd2, iommu, data); ret = __iommu_queue_command_sync(iommu, &cmd, true); if (ret) goto out_err; If the first __iommu_queue_command_sync() fails with "Command buffer timeout", it returns -EIO before copy_cmd_to_buffer() and before "iommu->need_sync = sync". So need_sync keeps its previous value, which is false if the last queued command was a CWAIT. No CWAIT carrying number N is queued. A later iommu_completion_wait() then sees need_sync == false, takes this new branch, reads data = cmd_sem_val (N), and calls wait_on_sem(iommu, N). *cmd_sem can never reach N, so the wait spins for LOOP_TIMEOUT iterations of udelay(1) and returns -EIO with "Completion-Wait loop timed out". Before the patch the same state returned 0 immediately. Callers such as domain_flush_complete() would now stall for the full timeout while holding domain->lock. The state lasts until another command sets need_sync and a CWAIT with a higher sequence number completes. This needs a command-buffer-full timeout, so the IOMMU is already misbehaving. The failure of the second queue call in iommu_flush_irt_and_complete() is not affected, because the first call has already set need_sync = true. The commit message says "Since need_sync == false implies no sync command was queued after the last CWAIT, that CWAIT is FIFO-ordered after every not-yet-completed command, so waiting for its sequence number guarantees all prior commands ... have completed." Is that accurate, given that cmd_sem_val can be ahead of the last CWAIT that was actually queued? > + raw_spin_unlock_irqrestore(&iommu->lock, flags); > + return wait_on_sem(iommu, data); > + } > + > data = get_cmdsem_val(iommu); > build_completion_wait(&cmd, iommu, data); > -- Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261110191836.3000352-1-den@openvz.org?part=3 _______________________________________________ Devel mailing list Devel@openvz.org https://lists.openvz.org/mailman/listinfo/devel ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Devel] [PATCH hci-8.1 0/3] iommu/amd: make iommu_completion_wait() wait for flushes queued by other CPUs 2026-11-10 19:18 [Devel] [PATCH hci-8.1 0/3] iommu/amd: make iommu_completion_wait() wait for flushes queued by other CPUs Denis V. Lunev ` (2 preceding siblings ...) 2026-11-10 19:18 ` [Devel] [PATCH hci-8.1 3/3] ms/iommu/amd: Wait for completion instead of returning early in iommu_completion_wait() Denis V. Lunev @ 2026-11-12 12:40 ` Konstantin Khorenko 3 siblings, 0 replies; 8+ messages in thread From: Konstantin Khorenko @ 2026-11-12 12:40 UTC (permalink / raw) To: Denis V. Lunev; +Cc: devel On Tue, 10 Nov 2026 20:18:33 +0100, Denis V. Lunev wrote: > iommu/amd: make iommu_completion_wait() wait for flushes queued by other CPUs > > iommu_completion_wait() checks iommu->need_sync without iommu->lock and > returns at once when it is clear. A completion wait queued by another > CPU clears the flag before the IOMMU has executed the commands in front > of it, so the caller can return while its own invalidations are still > pending. After an unmap that lets an IOVA or a page table page be > reused while the IOMMU may still hold the old translation. It also > leaves a hole in "iommu/amd/pgtbl: fix DMA faults on IOVAs mapped while > the table grows": the DTE flush done before the new page table level is > published can return before the IOMMU has dropped the old DTE. > > [...] Applied to branch-rh10-6.12.0-211.52.1.22.x.vz10-ovz, will be released in rh10-6.12.0-211.52.1.22.21.vz10, thanks! [1/3] ms/iommu/amd: move wait_on_sem() out of spinlock https://bitbucket.org/virtuozzocore/vzkernel.vzs/commits/3e6384ffedcb [2/3] ms/iommu/amd: serialize sequence allocation under concurrent TLB invalidations https://bitbucket.org/virtuozzocore/vzkernel.vzs/commits/5a6596791b87 [3/3] ms/iommu/amd: Wait for completion instead of returning early in iommu_completion_wait() https://bitbucket.org/virtuozzocore/vzkernel.vzs/commits/516fe340b0d5 -- Konstantin Khorenko, Virtuozzo Linux Kernel Team _______________________________________________ Devel mailing list Devel@openvz.org https://lists.openvz.org/mailman/listinfo/devel ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2026-11-12 12:42 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2026-11-10 19:18 [Devel] [PATCH hci-8.1 0/3] iommu/amd: make iommu_completion_wait() wait for flushes queued by other CPUs Denis V. Lunev 2026-11-10 19:18 ` [Devel] [PATCH hci-8.1 1/3] ms/iommu/amd: move wait_on_sem() out of spinlock Denis V. Lunev 2026-11-10 19:23 ` no-reply 2026-11-10 19:18 ` [Devel] [PATCH hci-8.1 2/3] ms/iommu/amd: serialize sequence allocation under concurrent TLB invalidations Denis V. Lunev 2026-11-10 19:22 ` no-reply 2026-11-10 19:18 ` [Devel] [PATCH hci-8.1 3/3] ms/iommu/amd: Wait for completion instead of returning early in iommu_completion_wait() Denis V. Lunev 2026-11-10 19:24 ` no-reply 2026-11-12 12:40 ` [Devel] [PATCH hci-8.1 0/3] iommu/amd: make iommu_completion_wait() wait for flushes queued by other CPUs Konstantin Khorenko
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox