Bug 1979951

Summary: nodedev-API would face performance issue after define multiple mediated device and mdev of ccw with wrong type
Product: Red Hat Enterprise Linux 8 Reporter: bfu <bfu>
Component: mdevctlAssignee: Virtualization Maintenance <virt-maint>
Status: CLOSED DUPLICATE QA Contact: virt-qe-z
Severity: high Docs Contact:
Priority: high    
Version: 8.5CC: alex.williamson, bfiuczyn, cohuck, dhorak, hannsj_uhl, jinzhao, jjongsma, juzhang, knoel, ngu, pbonzini, qzhang, ribarry, smitterl, thuth, tstaudt, virt-qe-z, yiwei
Target Milestone: betaFlags: pm-rhel: mirror+
Target Release: ---   
Hardware: s390x   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-07-09 14:01:47 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description bfu 2021-07-07 13:31:36 UTC
Description of problem:
after define a ccw mdev with invalid type, message present through nodedev-API would be delay

Version-Release number of selected component (if applicable):
host kernel: 4.18.0-316.el8.s390x
qemu version: qemu-kvm-6.0.0-21.module+el8.5.0+11555+e0ab0d09.s390x
libvirt version: libvirt-7.4.0-1.module+el8.5.0+11218+83343022.s390x
mdevctl version: mdevctl-0.78-1.el8.noarch

How reproducible:
100%

Steps to Reproduce:
1. define mediated devices of ccw and crypto
[root@kernelqe2 ~]# mdevctl define --uuid 566d63bd-8b33-4323-9f13-5d56155cd668 --parent 0.0.26ab --type fbq1
[root@kernelqe2 ~]# mdevctl define --uuid 254193dd-8ffc-4594-a272-28e35d87071g --parent 0.0.26ab --type fbq2
[root@kernelqe2 ~]# mdevctl define --uuid fcbc4814-8e59-4620-a817-92c9a7724a2e --parent matrix --type vfio_ap-passthrough
[root@kernelqe2 ~]# mdevctl modify --uuid fcbc4814-8e59-4620-a817-92c9a7724a2e --addattr=assign_adapter --value=0x02
[root@kernelqe2 ~]# mdevctl modify --uuid fcbc4814-8e59-4620-a817-92c9a7724a2e --addattr=assign_domain --value=0x0011
[root@kernelqe2 ~]# mdevctl modify --uuid fcbc4814-8e59-4620-a817-92c9a7724a2e --addattr=assign_domain --value=0x003a
[root@kernelqe2 ~]# mdevctl modify --uuid fcbc4814-8e59-4620-a817-92c9a7724a2e --addattr=assign_domain --value=0x00ab

2.reboot system and login again
[root@kernelqe2 ~]# reboot
bfu@fedora31 ~> sshpass -p kvmautotest ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root.bos.redhat.com
Warning: Permanently added 'kernelqe2.s390.bos.redhat.com,10.16.106.171' (ECDSA) to the list of known hosts.
Activate the web console with: systemctl enable --now cockpit.socket
This system is not registered to Red Hat Insights. See https://cloud.redhat.com/
To register this system, run: insights-client --register

3. load vfio_ap module
[root@kernelqe2 ~]# modprobe vfio_ap

4. Check if the ap_matrix is loaded
[root@kernelqe2 ~]# virsh nodedev-list --cap ap_matrix


Actual results:
there will be around two minutes delay to load the ap_matrix successfully

Expected results:
the ap_matrix could load immediately

Additional info:
[root@kernelqe2 ~]# journalctl -u libvirtd
-- Logs begin at Wed 2021-07-07 09:12:26 EDT, end at Wed 2021-07-07 09:13:04 EDT. --
Jul 07 09:12:34 kernelqe2.s390.bos.redhat.com systemd[1]: Starting Virtualization daemon...
Jul 07 09:12:34 kernelqe2.s390.bos.redhat.com systemd[1]: Started Virtualization daemon.
Jul 07 09:12:35 kernelqe2.s390.bos.redhat.com dnsmasq[2290]: listening on virbr0(#3): 192.168.122.1
Jul 07 09:12:35 kernelqe2.s390.bos.redhat.com dnsmasq[2303]: started, version 2.79 cachesize 150
Jul 07 09:12:35 kernelqe2.s390.bos.redhat.com dnsmasq[2303]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN2 DHCP DHCPv6 no-Lua>
Jul 07 09:12:35 kernelqe2.s390.bos.redhat.com dnsmasq-dhcp[2303]: DHCP, IP range 192.168.122.2 -- 192.168.122.254, lease time 1h
Jul 07 09:12:35 kernelqe2.s390.bos.redhat.com dnsmasq-dhcp[2303]: DHCP, sockets bound exclusively to interface virbr0
Jul 07 09:12:35 kernelqe2.s390.bos.redhat.com dnsmasq[2303]: reading /etc/resolv.conf
Jul 07 09:12:35 kernelqe2.s390.bos.redhat.com dnsmasq[2303]: using nameserver 10.11.5.19#53
Jul 07 09:12:35 kernelqe2.s390.bos.redhat.com dnsmasq[2303]: read /etc/hosts - 3 addresses
Jul 07 09:12:35 kernelqe2.s390.bos.redhat.com dnsmasq[2303]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Jul 07 09:12:35 kernelqe2.s390.bos.redhat.com dnsmasq-dhcp[2303]: read /var/lib/libvirt/dnsmasq/default.hostsfile
Jul 07 09:12:35 kernelqe2.s390.bos.redhat.com libvirtd[1927]: libvirt version: 7.4.0, package: 1.module+el8.5.0+11218+83343022 (Red Hat>
Jul 07 09:12:35 kernelqe2.s390.bos.redhat.com libvirtd[1927]: hostname: kernelqe2.s390.bos.redhat.com
Jul 07 09:12:35 kernelqe2.s390.bos.redhat.com libvirtd[1927]: internal error: Unexpected format for parent device object
Jul 07 09:12:35 kernelqe2.s390.bos.redhat.com libvirtd[1927]: internal error: failed to query mdevs from mdevctl:

Comment 1 bfu 2021-07-07 13:35:47 UTC
If use "# virsh nodedev-create mdev.xml" to create mdev, it will also have two minutes delay for checking the result with "virsh nodedev-list --cap mdev"
[root@kernelqe2 bfu]# cat mdev.xml 
<device>
<name>mdev_4c1da20a-7f13-4e01-b42a-d0705ba4ffb6</name>
<path>/sys/devices/vfio_ap/matrix/4c1da20a-7f13-4e01-b42a-d0705ba4ffb6</path>
<parent>ap_matrix</parent>
<driver>
<name>vfio_mdev</name>
</driver>
<capability type='mdev'>
<type id='vfio_ap-passthrough'/>
<attr name="assign_adapter" value="0x02"/>
<attr name="assign_domain" value="0x0011"/>
</capability>
</device>

Comment 2 bfu 2021-07-07 14:08:29 UTC
*** Bug 1979949 has been marked as a duplicate of this bug. ***

Comment 3 Cornelia Huck 2021-07-07 14:56:06 UTC
Does mdevctl print anything sane if you try to list the devices? IOW, is it libvirt that chokes on this?

Comment 4 bfu 2021-07-08 05:22:01 UTC
(In reply to Cornelia Huck from comment #3)
> Does mdevctl print anything sane if you try to list the devices? IOW, is it
> libvirt that chokes on this?

As the additional information in comment1, seems yes, I think libvirt chokes on this.
And as for mdevctl print:
[root@kernelqe2 bfu]# mdevctl list -d --dumpjson
[
  {
    "0.0.26ab": [
      {
        "566d63bd-8b33-4323-9f13-5d56155cd668": {
          "mdev_type": "fbq1",
          "start": "manual"
        }
      },
      {
        "254193dd-8ffc-4594-a272-28e35d87071g": {
          "mdev_type": "fbq2",
          "start": "manual"
        }
      },
      {
        "566d63bd-8b33-4323-9f13-5d56155cd669": {
          "mdev_type": "fbq1",
          "start": "manual"
        }
      }
    ],
    "matrix": [
      {
        "fcbc4814-8e59-4620-a817-92c9a7724a2e": {
          "mdev_type": "vfio_ap-passthrough",
          "start": "manual",
          "attrs": [
            {
              "assign_adapter": "0x02"
            },
            {
              "assign_domain": "0x0011"
            },
            {
              "assign_domain": "0x003a"
            },
            {
              "assign_domain": "0x00ab"
            }
          ]
        }
      }
    ]
  }
]

Comment 5 Boris Fiuczynski (IBM) 2021-07-08 08:55:25 UTC
(In reply to Cornelia Huck from comment #3)
> Does mdevctl print anything sane if you try to list the devices? IOW, is it
> libvirt that chokes on this?

Yes, I think that is caused by libvirt.

1. libvirts mdevctl polling runs into parsing errors as long as an mdev definition with an unknown mdev_type type exists.
 I tried it out like this creating an vfio-ccw alike mdev.
# mdevctl list -d --dumpjson
[
  {
    "0.0.0033": [
      {
        "e60cef97-3f6b-485e-ac46-0520f9f66ac2": {
          "mdev_type": "vfio_ccw-io",
          "start": "manual"
        }
      }
    ],
    "0.0.0034": [
      {
        "ffffffff-3f6b-485e-ac46-0520f9f66ac2": {
          "mdev_type": "type1",
          "start": "manual"
        }
      }
    ]
  }
]

Looking into the journal can I confirm this problem.
  Jul 08 09:32:18 t46lp71.lnxne.boe libvirtd[139293]: internal error: Unexpected format for parent device object
  Jul 08 09:32:18 t46lp71.lnxne.boe libvirtd[139293]: internal error: failed to query mdevs from mdevctl:
  Jul 08 09:32:56 t46lp71.lnxne.boe libvirtd[139293]: internal error: Unexpected format for parent device object
  Jul 08 09:32:56 t46lp71.lnxne.boe libvirtd[139293]: internal error: failed to query mdevs from mdevctl:
  Jul 08 09:32:56 t46lp71.lnxne.boe libvirtd[139293]: mdevctl failed to updated mediated devices
  Jul 08 09:32:57 t46lp71.lnxne.boe libvirtd[139293]: internal error: Unexpected format for parent device object
  Jul 08 09:32:57 t46lp71.lnxne.boe libvirtd[139293]: internal error: failed to query mdevs from mdevctl:
  Jul 08 09:32:57 t46lp71.lnxne.boe libvirtd[139293]: mdevctl failed to updated mediated devices

2. loading/unloading the vfio_ap device driver causes a long delay until the ap_matrix gets creating/deleting as nodedev object
 The device driver is actually loaded and available immediately. When trying this out with libvirt v7.5.0+ I had to wait more than 5 minutes until the ap_matrix nodedev object appeared in the list of objects after loading the vfio_ap device driver.
 It looks like the udev event parsing in libvirt gets locked up by the mdevctl polling errors.

Comment 6 Jonathon Jongsma 2021-07-08 18:12:24 UTC
I'm pretty sure this is just the same root cause as bug 1979440 -- failing to properly parse mdevs from multiple parent devices.

Comment 7 Jonathon Jongsma 2021-07-09 14:01:47 UTC

*** This bug has been marked as a duplicate of bug 1979440 ***