[PATCH hci-8.0 0/2] pci: batch memory transactions around mapping updates #VSTOR-129348

[PATCH hci-8.0 0/2] pci: batch memory transactions around mapping updates #VSTOR-129348

[Denis V. Lunev]

Restoring a PCI device's config space remaps its BARs, updates the bridge
windows when it is a bridge, and toggles the bus master region. Each of
those ends a memory transaction of its own, and a transaction commit
re-renders the flatviews it touches, at a cost which grows with the number
of memory regions in the machine.

So a machine with a rich PCIe topology pays that render many times over
while an incoming migration stream is read, once or more per device, even
though nothing between the steps looks at an address space.

Both patches wrap an existing sequence in a single transaction. There is
no new API and no behaviour a guest can observe: each transaction opens and
closes inside the function that starts it, so nothing sees an intermediate
state which was not already reachable. pci_bridge_update_mappings() has
worked this way for the bridge windows for a long time; the first patch
just makes pci_update_mappings() consistent with it.

Measured on a Windows Server 2022 guest with 64 vCPUs and 65 pcie-root-ports,
the non-iterable part of an incoming migration drops by about a third. The
saving scales with the number of PCI devices, so it is small for a plain
machine and large for the topologies which hurt today.

Note that pci_update_mappings() is also reached from the runtime config
write path, not only from migration. Several BAR moves in one config write
now share a rebuild instead of taking one each, which is the same win in
miniature and equally invisible to the guest.

CC: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Denis V. Lunev <den@virtuozzo.com>

Denis V. Lunev (2):
  pci: batch BAR remapping into one memory transaction #VSTOR-129348
  pci: load a device's config inside one memory transaction
    #VSTOR-129348

 hw/pci/pci.c | 8 ++++++++
 1 file changed, 8 insertions(+)

-- 
2.53.0

[PATCH hci-8.0 1/2] pci: batch BAR remapping into one memory transaction #VSTOR-129348

[Denis V. Lunev]

From: Denis V. Lunev <den@openvz.org>

pci_update_mappings() deletes and re-adds a subregion for each BAR
whose address changed, and both of those commit their own transaction,
so one BAR move costs two full flatview rebuilds.

Nothing between the delete and the add touches an address space, so
the walk can run inside a single transaction. pci_bridge_update_mappings()
already does this for the bridge windows.

CC: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Denis V. Lunev <den@virtuozzo.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
---
 hw/pci/pci.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/hw/pci/pci.c b/hw/pci/pci.c
index c998ea7f8fc..21eadee2e37 100644
--- a/hw/pci/pci.c
+++ b/hw/pci/pci.c
@@ -1696,6 +1696,8 @@ static void pci_update_mappings(PCIDevice *d)
     int i;
     pcibus_t new_addr;
 
+    memory_region_transaction_begin();
+
     for(i = 0; i < PCI_NUM_REGIONS; i++) {
         r = &d->io_regions[i];
 
@@ -1732,6 +1734,8 @@ static void pci_update_mappings(PCIDevice *d)
     }
 
     pci_update_vga(d);
+
+    memory_region_transaction_commit();
 }
 
 int pci_irq_disabled(PCIDevice *d)
-- 
2.53.0

[PATCH hci-8.0 2/2] pci: load a device's config inside one memory transaction #VSTOR-129348

[Denis V. Lunev]

From: Denis V. Lunev <den@openvz.org>

Restoring config space remaps the BARs, updates the bridge windows when
the device is a bridge, and toggles the bus master region. Each of those
commits a transaction of its own, so one device costs several flatview
rebuilds where one would do, and the cost of a rebuild grows with the
number of regions in the machine.

Nothing between them touches an address space, so the three can share a
transaction. The saving scales with the number of PCI devices.

CC: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Denis V. Lunev <den@virtuozzo.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
---
 hw/pci/pci.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/hw/pci/pci.c b/hw/pci/pci.c
index 21eadee2e37..5d7d17f6529 100644
--- a/hw/pci/pci.c
+++ b/hw/pci/pci.c
@@ -811,6 +811,8 @@ static int get_pci_config_device(QEMUFile *f, void *pv, size_t size,
     }
     memcpy(s->config, config, size);
 
+    memory_region_transaction_begin();
+
     pci_update_mappings(s);
     if (IS_PCI_BRIDGE(s)) {
         pci_bridge_update_mappings(PCI_BRIDGE(s));
@@ -819,6 +821,8 @@ static int get_pci_config_device(QEMUFile *f, void *pv, size_t size,
     pci_set_master(s, pci_get_word(s->config + PCI_COMMAND)
                       & PCI_COMMAND_MASTER);
 
+    memory_region_transaction_commit();
+
     g_free(config);
     return 0;
 }
-- 
2.53.0

Re: [PATCH hci-8.0 0/2] pci: batch memory transactions around mapping updates #VSTOR-129348

[Andrey Drobyshev]

Ack
Applied to hci-8.0

On 9/10/26 11:32 PM, Denis V. Lunev wrote:
> Restoring a PCI device's config space remaps its BARs, updates the bridge
> windows when it is a bridge, and toggles the bus master region. Each of
> those ends a memory transaction of its own, and a transaction commit
> re-renders the flatviews it touches, at a cost which grows with the number
> of memory regions in the machine.
> 
> So a machine with a rich PCIe topology pays that render many times over
> while an incoming migration stream is read, once or more per device, even
> though nothing between the steps looks at an address space.
> 
> Both patches wrap an existing sequence in a single transaction. There is
> no new API and no behaviour a guest can observe: each transaction opens and
> closes inside the function that starts it, so nothing sees an intermediate
> state which was not already reachable. pci_bridge_update_mappings() has
> worked this way for the bridge windows for a long time; the first patch
> just makes pci_update_mappings() consistent with it.
> 
> Measured on a Windows Server 2022 guest with 64 vCPUs and 65 pcie-root-ports,
> the non-iterable part of an incoming migration drops by about a third. The
> saving scales with the number of PCI devices, so it is small for a plain
> machine and large for the topologies which hurt today.
> 
> Note that pci_update_mappings() is also reached from the runtime config
> write path, not only from migration. Several BAR moves in one config write
> now share a rebuild instead of taking one each, which is the same win in
> miniature and equally invisible to the guest.
> 
> CC: Michael S. Tsirkin <mst@redhat.com>
> Signed-off-by: Denis V. Lunev <den@virtuozzo.com>
> 
> Denis V. Lunev (2):
>   pci: batch BAR remapping into one memory transaction #VSTOR-129348
>   pci: load a device's config inside one memory transaction
>     #VSTOR-129348
> 
>  hw/pci/pci.c | 8 ++++++++
>  1 file changed, 8 insertions(+)
>