Red Hat Bugzilla – 2032795 – [OSP16.2]Unwanted new line/whitespace character in case of vhostuser interface name causes errors when polling for iface statistics
Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
IMO this is not a ceilometer bug but a libvirt bug.
Looking at libvirt implementation, I guess this was fixed by https://github.com/libvirt/libvirt/commit/b29c86ae1042fc44e7c86f3519f048a696e68afe .
However unfortunately this is available in libvirt 7.1.0+ .
We'd need to request backporting this to RHEL8.4 EUS to resolve the issue.
(In reply to Takashi Kajinami from comment #5)
> IMO this is not a ceilometer bug but a libvirt bug.
>
> Looking at libvirt implementation, I guess this was fixed by
> https://github.com/libvirt/libvirt/commit/
> b29c86ae1042fc44e7c86f3519f048a696e68afe .
> However unfortunately this is available in libvirt 7.1.0+ .
>
> We'd need to request backporting this to RHEL8.4 EUS to resolve the issue.
Oh that's a really great find Takashi. I wonder where we'd request the backport for this... any ideas?
@mrunge maybe you might know? If this is part of the base operating system I guess we need to open something against RHEL?
Comment 11yalzhang@redhat.com
2022-04-22 07:48:51 UTC
There is already a libvirt bug 2077328 to track this issue.
And z stream bug will be cloned automatically from bug 2077328 after approved.
I will close this one and please reopen it if any concern. Thank you.
*** This bug has been marked as a duplicate of bug 2077328 ***
##################################################### In case of RHOSP16.2 DPDK enabled compute nodes, XML definition of VM contains new line character in vhostuser interface target dev definition as illustrated bellow: ##################################################### [root@dpdk-compute-1 stdouts]# podman exec nova_libvirt virsh dumpxml 1 | grep -A9 '^.*<interface' <interface type='vhostuser'> <mac address='fa:16:3e:ba:7a:0f'/> <source type='unix' path='/var/lib/vhost_sockets/vhu90ff7386-ef' mode='server'/> <target dev='vhu90ff7386-ef '/> <model type='virtio'/> <driver rx_queue_size='1024' tx_queue_size='1024'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <interface type='vhostuser'> <mac address='fa:16:3e:08:1d:a7'/> <source type='unix' path='/var/lib/vhost_sockets/vhu419de68d-89' mode='server'/> <target dev='vhu419de68d-89 '/> <model type='virtio'/> <driver rx_queue_size='1024' tx_queue_size='1024'/> <alias name='net1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </interface> ##################################################### This does not seem to cause any problems to libvirt itself - VM is up and running, network connectivity isn't affected: ##################################################### [root@dpd-compute-1 stdouts]# podman exec nova_libvirt virsh domiflist 1 Interface Type Source Model MAC --------------------------------------------------------------------------------------------------- vhu90ff7386-ef vhostuser /var/lib/vhost_sockets/vhu90ff7386-ef virtio fa:16:3e:ba:7a:0f vhu419de68d-89 vhostuser /var/lib/vhost_sockets/vhu419de68d-89 virtio fa:16:3e:08:1d:a7 (overcloud) [stack@dir01 ~]$ openstack port list --server 1e6c44e6-e696-4242-8b19-0ddcb92ab6a8 +--------------------------------------+------+-------------------+-------------------------------------------------------------------------------+--------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | +--------------------------------------+------+-------------------+-------------------------------------------------------------------------------+--------+ | 419de68d-893b-4dfa-b46b-d275e7cfa4f5 | | fa:16:3e:08:1d:a7 | ip_address='10.12.151.19', subnet_id='bf4d4d5f-8b2c-4682-b6fe-2a6e21799d18' | ACTIVE | | 90ff7386-ef9c-4a8e-b717-d2e828a74608 | | fa:16:3e:ba:7a:0f | ip_address='10.129.141.210', subnet_id='fca779c5-c2d0-4f35-9522-1304e4bc7577' | ACTIVE | +--------------------------------------+------+-------------------+-------------------------------------------------------------------------------+--------+ (overcloud) [stack@dir01 ~]$ openstack port show 419de68d-893b-4dfa-b46b-d275e7cfa4f5 +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | dpdk-compute-1 | | binding_profile | | | binding_vif_details | port_filter='False', vhostuser_mode='server', vhostuser_ovs_plug='True', vhostuser_socket='/var/lib/vhost_sockets/vhu419de68d-89' | | binding_vif_type | vhostuser | | binding_vnic_type | normal | | created_at | 2021-12-13T10:20:43Z | | data_plane_status | None | | description | | | device_id | 1e6c44e6-e696-4242-8b19-0ddcb92ab6a8 | | device_owner | compute:nova | | dns_assignment | fqdn='host-10-12-151-19.openstacklocal.', hostname='host-10-12-151-19', ip_address='10.12.151.19' | | dns_domain | None | | dns_name | | | extra_dhcp_opts | | | fixed_ips | ip_address='10.12.151.19', subnet_id='bf4d4d5f-8b2c-4682-b6fe-2a6e21799d18' | | id | 419de68d-893b-4dfa-b46b-d275e7cfa4f5 | | location | cloud='', project.domain_id=, project.domain_name='Default', project.id='78e484e5cb074f2ea021bc8212516cee', project.name='admin', region_name='regionOne', zone= | | mac_address | fa:16:3e:08:1d:a7 | | name | | | network_id | 6689e9bb-0558-4e27-a10a-f8290d2cacc2 | | port_security_enabled | False | | project_id | 78e484e5cb074f2ea021bc8212516cee | | propagate_uplink_status | None | | qos_policy_id | None | | resource_request | None | | revision_number | 21 | | security_group_ids | | | status | ACTIVE | | tags | | | trunk_details | None | | updated_at | 2021-12-14T21:05:43Z | +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ (overcloud) [stack@dir01 ~]$ openstack port show 90ff7386-ef9c-4a8e-b717-d2e828a74608 +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | dpdk-compute-1 | | binding_profile | | | binding_vif_details | port_filter='False', vhostuser_mode='server', vhostuser_ovs_plug='True', vhostuser_socket='/var/lib/vhost_sockets/vhu90ff7386-ef' | | binding_vif_type | vhostuser | | binding_vnic_type | normal | | created_at | 2021-12-13T10:20:43Z | | data_plane_status | None | | description | | | device_id | 1e6c44e6-e696-4242-8b19-0ddcb92ab6a8 | | device_owner | compute:nova | | dns_assignment | fqdn='host-10-129-141-210.openstacklocal.', hostname='host-10-129-141-210', ip_address='10.129.141.210' | | dns_domain | None | | dns_name | | | extra_dhcp_opts | | | fixed_ips | ip_address='10.129.141.210', subnet_id='fca779c5-c2d0-4f35-9522-1304e4bc7577' | | id | 90ff7386-ef9c-4a8e-b717-d2e828a74608 | | location | cloud='', project.domain_id=, project.domain_name='Default', project.id='78e484e5cb074f2ea021bc8212516cee', project.name='admin', region_name='regionOne', zone= | | mac_address | fa:16:3e:ba:7a:0f | | name | | | network_id | 2bf94005-96d3-4aab-bdf2-f6db16f1fe51 | | port_security_enabled | False | | project_id | 78e484e5cb074f2ea021bc8212516cee | | propagate_uplink_status | None | | qos_policy_id | None | | resource_request | None | | revision_number | 20 | | security_group_ids | | | status | ACTIVE | | tags | | | trunk_details | None | | updated_at | 2021-12-14T21:05:43Z | +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ##################################################### However once STF was configured on the platform, I can observe errors which are probably caused by this weirdly formated VM definition. For an instance: ##################################################### [root@dpdk-compute-1 stdouts]# tail /var/log/containers/stdouts/ceilometer_agent_compute.log 2021-12-15T08:10:06.135469736+00:00 stderr F libvirt: Domain Config error : invalid argument: 'vhu90ff7386-ef ' is not a known interface 2021-12-15T08:10:06.139186031+00:00 stderr F libvirt: Domain Config error : invalid argument: 'vhu90ff7386-ef ' is not a known interface 2021-12-15T08:10:36.066100169+00:00 stderr F libvirt: Domain Config error : invalid argument: 'vhu90ff7386-ef ' is not a known interface 2021-12-15T08:10:36.103598943+00:00 stderr F libvirt: Domain Config error : invalid argument: 'vhu90ff7386-ef ' is not a known interface 2021-12-15T08:10:36.105059875+00:00 stderr F libvirt: Domain Config error : invalid argument: 'vhu90ff7386-ef ' is not a known interface 2021-12-15T08:10:36.117131111+00:00 stderr F libvirt: Domain Config error : invalid argument: 'vhu90ff7386-ef ' is not a known interface 2021-12-15T08:10:36.120463239+00:00 stderr F libvirt: Domain Config error : invalid argument: 'vhu90ff7386-ef ' is not a known interface 2021-12-15T08:10:36.121913181+00:00 stderr F libvirt: Domain Config error : invalid argument: 'vhu90ff7386-ef ' is not a known interface 2021-12-15T08:10:36.129652791+00:00 stderr F libvirt: Domain Config error : invalid argument: 'vhu90ff7386-ef ' is not a known interface 2021-12-15T08:10:36.135094716+00:00 stderr F libvirt: Domain Config error : invalid argument: 'vhu90ff7386-ef ' is not a known interface [root@dpdk-compute-1 ~]# tail -f /var/log/containers/libvirt/libvirtd.log 2021-12-15 08:33:06.131+0000: 5515: error : virDomainNetFind:30053 : invalid argument: 'vhu90ff7386-ef ' is not a known interface 2021-12-15 08:33:06.132+0000: 5519: error : virDomainNetFind:30053 : invalid argument: 'vhu90ff7386-ef ' is not a known interface 2021-12-15 08:33:06.136+0000: 5521: error : virDomainNetFind:30053 : invalid argument: 'vhu90ff7386-ef ' is not a known interface 2021-12-15 08:33:06.147+0000: 5520: error : virDomainNetFind:30053 : invalid argument: 'vhu90ff7386-ef ' is not a known interface 2021-12-15 08:33:07.149+0000: 5512: error : virDomainNetFind:30053 : invalid argument: 'vhu90ff7386-ef ' is not a known interface 2021-12-15 08:33:07.150+0000: 5516: error : virDomainNetFind:30053 : invalid argument: 'vhu419de68d-89 ' is not a known interface 2021-12-15 08:33:12.191+0000: 5515: error : virDomainNetFind:30053 : invalid argument: 'vhu90ff7386-ef ' is not a known interface 2021-12-15 08:33:12.192+0000: 5514: error : virDomainNetFind:30053 : invalid argument: 'vhu419de68d-89 ' is not a known interface 2021-12-15 08:33:17.171+0000: 5517: error : virDomainNetFind:30053 : invalid argument: 'vhu90ff7386-ef ' is not a known interface 2021-12-15 08:33:17.171+0000: 5512: error : virDomainNetFind:30053 : invalid argument: 'vhu419de68d-89 ' is not a known interface 2021-12-15 08:33:22.162+0000: 5517: error : virDomainNetFind:30053 : invalid argument: 'vhu90ff7386-ef ' is not a known interface 2021-12-15 08:33:22.162+0000: 5515: error : virDomainNetFind:30053 : invalid argument: 'vhu419de68d-89 ' is not a known interface ##################################################### Logs from nova-compute seems to confirm that port name in VM definition is ok, so this new line character has to come from code running on the hypervisor most likely: ##################################################### 2021-12-15 08:29:57.831 6 DEBUG nova.network.base_api [req-36c8d9f7-6db3-4c25-8f76-a4a4ae0543e1 - - - - -] [instance: 1e6c44e6-e696-4242-8b19-0ddcb92ab6a8] Updating instance_info_cache with network_info: [{"id": "90ff7386-ef9c-4a8e-b717-d2e828a74608", "address": "fa:16:3e:ba:7a:0f", "network": {"id": "2bf94005-96d3-4aab-bdf2-f6db16f1fe51", "bridge": "br-int", "label": "test_vms", "subnets": [{"cidr": "10.129.141.192/26", "dns": [{"address": "10.129.141.11", "type": "dns", "version": 4, "meta": {}}, {"address": "10.129.141.12", "type": "dns", "version": 4, "meta": {}}], "gateway": {"address": "10.129.141.193", "type": "gateway", "version": 4, "meta": {}}, "ips": [{"address": "10.129.141.210", "type": "fixed", "version": 4, "meta": {}, "floating_ips": []}], "routes": [], "version": 4, "meta": {}}], "meta": {"injected": false, "tenant_id": "78e484e5cb074f2ea021bc8212516cee", "mtu": 1500, "physical_network": "tenant", "tunneled": false}}, "type": "vhostuser", "details": {"port_filter": false, "vhostuser_mode": "server", "vhostuser_ovs_plug": true, "vhostuser_socket": "/var/lib/vhost_sockets/vhu90ff7386-ef"}, "devname": "tap90ff7386-ef", "ovs_interfaceid": "90ff7386-ef9c-4a8e-b717-d2e828a74608", "qbh_params": null, "qbg_params": null, "active": true, "vnic_type": "normal", "profile": {}, "preserve_on_delete": false, "meta": {}}, {"id": "419de68d-893b-4dfa-b46b-d275e7cfa4f5", "address": "fa:16:3e:08:1d:a7", "network": {"id": "6689e9bb-0558-4e27-a10a-f8290d2cacc2", "bridge": "br-int", "label": "ip_net-x1", "subnets": [{"cidr": "10.12.151.0/24", "dns": [], "gateway": {"address": null, "type": "gateway", "version": null, "meta": {}}, "ips": [{"address": "10.12.151.19", "type": "fixed", "version": 4, "meta": {}, "floating_ips": []}], "routes": [], "version": 4, "meta": {}}], "meta": {"injected": false, "tenant_id": "78e484e5cb074f2ea021bc8212516cee", "mtu": 1500, "physical_network": "tenant", "tunneled": false}}, "type": "vhostuser", "details": {"port_filter": false, "vhostuser_mode": "server", "vhostuser_ovs_plug": true, "vhostuser_socket": "/var/lib/vhost_sockets/vhu419de68d-89"}, "devname": "tap419de68d-89", "ovs_interfaceid": "419de68d-893b-4dfa-b46b-d275e7cfa4f5", "qbh_params": null, "qbg_params": null, "active": true, "vnic_type": "normal", "profile": {}, "preserve_on_delete": false, "meta": {}}] update_instance_cache_with_nw_info /usr/lib/python3.6/site-packages/nova/network/base_api.py:47 ##################################################### Since this weird format of vhostuser interface target dev definition does not affect hypervisor but only Ceilometer related part, I'd consider worth to implement in Ceilometer part some validation/cleaning of unwanted characters in interface name before querying hypervisor for the statistics. ##################################################### Version-Release number of selected component (if applicable): [root@dpdk-compute-1 ~]# podman exec ceilometer_agent_compute rpm -qa | grep -i ceilo python3-ceilometer-13.1.3-2.20210608074808.dad2447.el8ost.1.noarch openstack-ceilometer-common-13.1.3-2.20210608074808.dad2447.el8ost.1.noarch openstack-ceilometer-polling-13.1.3-2.20210608074808.dad2447.el8ost.1.noarch openstack-ceilometer-compute-13.1.3-2.20210608074808.dad2447.el8ost.1.noarch [root@dpdk-compute-1 ~]# podman exec nova_libvirt rpm -qa | grep -i libvirt libvirt-bash-completion-7.0.0-14.5.module+el8.4.0+13026+f38c77ab.x86_64 libvirt-libs-7.0.0-14.5.module+el8.4.0+13026+f38c77ab.x86_64 libvirt-client-7.0.0-14.5.module+el8.4.0+13026+f38c77ab.x86_64 libvirt-daemon-driver-nwfilter-7.0.0-14.5.module+el8.4.0+13026+f38c77ab.x86_64 libvirt-daemon-driver-storage-mpath-7.0.0-14.5.module+el8.4.0+13026+f38c77ab.x86_64 libvirt-daemon-driver-storage-iscsi-direct-7.0.0-14.5.module+el8.4.0+13026+f38c77ab.x86_64 libvirt-daemon-driver-storage-7.0.0-14.5.module+el8.4.0+13026+f38c77ab.x86_64 libvirt-daemon-kvm-7.0.0-14.5.module+el8.4.0+13026+f38c77ab.x86_64 python3-libvirt-7.0.0-1.module+el8.4.0+9469+2eaf72bc.x86_64 libvirt-daemon-driver-storage-core-7.0.0-14.5.module+el8.4.0+13026+f38c77ab.x86_64 libvirt-daemon-driver-qemu-7.0.0-14.5.module+el8.4.0+13026+f38c77ab.x86_64 libvirt-daemon-driver-nodedev-7.0.0-14.5.module+el8.4.0+13026+f38c77ab.x86_64 libvirt-daemon-driver-storage-rbd-7.0.0-14.5.module+el8.4.0+13026+f38c77ab.x86_64 libvirt-daemon-driver-storage-scsi-7.0.0-14.5.module+el8.4.0+13026+f38c77ab.x86_64 libvirt-daemon-driver-storage-disk-7.0.0-14.5.module+el8.4.0+13026+f38c77ab.x86_64 libvirt-daemon-driver-storage-iscsi-7.0.0-14.5.module+el8.4.0+13026+f38c77ab.x86_64 libvirt-daemon-driver-network-7.0.0-14.5.module+el8.4.0+13026+f38c77ab.x86_64 libvirt-daemon-driver-interface-7.0.0-14.5.module+el8.4.0+13026+f38c77ab.x86_64 libvirt-admin-7.0.0-14.5.module+el8.4.0+13026+f38c77ab.x86_64 libvirt-daemon-7.0.0-14.5.module+el8.4.0+13026+f38c77ab.x86_64 libvirt-daemon-driver-secret-7.0.0-14.5.module+el8.4.0+13026+f38c77ab.x86_64 libvirt-daemon-driver-storage-logical-7.0.0-14.5.module+el8.4.0+13026+f38c77ab.x86_64 libvirt-daemon-driver-storage-gluster-7.0.0-14.5.module+el8.4.0+13026+f38c77ab.x86_64 libvirt-daemon-config-nwfilter-7.0.0-14.5.module+el8.4.0+13026+f38c77ab.x86_64 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: