Fedora Account System
Red Hat Associate
Red Hat Customer
In the last two months we have started to see a fair number of virtinterfaced crashes [1] in cockpit-machines tests, on Fedora 38 and 39 (we only just started testing on 40, but likely they happen there as well). As c-machines doesn't talk to virtinterfaced directly, it doesn't actually break the UI, but the tests also watch out for unexpected journal messages like crashes. One of the two common libvirt crashes happens in e.g. [2][3]. All of these have "journal" links to the full systemd journals. (The other common crash is bug 2266014) The stack trace of the relevant thread looks like this, with xdr_remote_interface_lookup_by_mac_string_args() being the key part: #0 0x00007fe1f170a834 __pthread_kill_implementation (libc.so.6 + 0x90834) #1 0x00007fe1f16b88ee raise (libc.so.6 + 0x3e8ee) #2 0x00007fe1f16a08ff abort (libc.so.6 + 0x268ff) #3 0x00007fe1f16a17d0 __libc_message.cold (libc.so.6 + 0x277d0) #4 0x00007fe1f17147a5 malloc_printerr (libc.so.6 + 0x9a7a5) #5 0x00007fe1f1716a9c _int_free (libc.so.6 + 0x9ca9c) #6 0x00007fe1f17193de free (libc.so.6 + 0x9f3de) #7 0x00007fe1f20f99b8 xdr_string (libtirpc.so.3 + 0x199b8) #8 0x00007fe1f1e5e592 xdr_remote_nonnull_string (libvirt.so.0 + 0x25e592) #9 0x00007fe1f1e69ded xdr_remote_interface_lookup_by_mac_string_args (libvirt.so.0 + 0x269ded) #10 0x00007fe1f20f8e6b xdr_free (libtirpc.so.3 + 0x18e6b) #11 0x00007fe1f1e281a7 virNetServerProgramDispatch (libvirt.so.0 + 0x2281a7) #12 0x00007fe1f1e2eb23 virNetServerProcessMsg (libvirt.so.0 + 0x22eb23) #13 0x00007fe1f1e2ec36 virNetServerHandleJob (libvirt.so.0 + 0x22ec36) #14 0x00007fe1f1d66c45 virThreadPoolWorker (libvirt.so.0 + 0x166c45) #15 0x00007fe1f1d65e00 virThreadHelper (libvirt.so.0 + 0x165e00) #16 0x00007fe1f1708897 start_thread (libc.so.6 + 0x8e897) #17 0x00007fe1f178f6fc __clone3 (libc.so.6 + 0x1156fc) For both [2] and [3], the journal just said virtinterfaced[2416]: free(): invalid pointer which isn't super helpful. [1] https://github.com/cockpit-project/cockpit-machines/issues/1391#issuecomment-1963676507 [2] https://cockpit-logs.us-east-1.linodeobjects.com/pull-5903-20240212-230634-e54961ef-fedora-39-firefox-cockpit-project-cockpit-machines/log.html#60 [3] https://cockpit-logs.us-east-1.linodeobjects.com/pull-1280-20240126-150137-d0fc7d9f-fedora-39-firefox/log.html#40 Reproducible: Couldn't Reproduce Steps to Reproduce: Unfortunately there is no direct reproducer. It happens sporadically, and in all tests (i.e. not one or two specific ones), which corresponds to "we don't assert anything in it directly". Like in bug 2266014 I've tried for about half an hour to locally run tests in a loop on my laptop, but as a good Heisenbug it never happened. So is there some extra information which we could collect to make this more useful? Note that it only crashes a few times a week in our CI (during hundreds of test runs).
Yesterday's test run in https://cockpit-logs.us-east-1.linodeobjects.com/pull-6166-a97c224f-20240403-154431-fedora-40-cockpit-project-cockpit-machines/log.html#70 has shown another variant: #0 0x00007fcd502b2144 __pthread_kill_implementation (libc.so.6 + 0x98144) #1 0x00007fcd5025a65e raise (libc.so.6 + 0x4065e) #2 0x00007fcd50242902 abort (libc.so.6 + 0x28902) #3 0x00007fcd4ec5f901 log_assert_failed.cold (libudev.so.1 + 0x6901) #4 0x00007fcd4ec6921f udev_ref (libudev.so.1 + 0x1021f) #5 0x00007fcd4ec958d5 udevInterfaceGetXMLDesc (libvirt_driver_interface.so + 0x48d5) #6 0x00007fcd50942d75 virInterfaceGetXMLDesc (libvirt.so.0 + 0x342d75) #7 0x000055ef3de5a106 remoteDispatchInterfaceGetXMLDescHelper.lto_priv.0 (virtinterfaced + 0x66106) #8 0x00007fcd50806e29 virNetServerProgramDispatch (libvirt.so.0 + 0x206e29) #9 0x00007fcd5080d8a3 virNetServerProcessMsg (libvirt.so.0 + 0x20d8a3) #10 0x00007fcd5080d9b6 virNetServerHandleJob (libvirt.so.0 + 0x20d9b6) #11 0x00007fcd507456f5 virThreadPoolWorker (libvirt.so.0 + 0x1456f5) #12 0x00007fcd507448cd virThreadHelper (libvirt.so.0 + 0x1448cd) #13 0x00007fcd502b01b7 start_thread (libc.so.6 + 0x961b7) #14 0x00007fcd5033239c __clone3 (libc.so.6 + 0x11839c) bug 2266014 is already fixed/closed, and was about udevInterfaceLookupByMACString(). But this looks too close to be a coincidence, so for now I won't file another bug. If you want me to, I can though.
A recent F40 run in https://cockpit-logs.us-east-1.linodeobjects.com/pull-1643-7d20b02f-20240519-191431-fedora-40-devel/log.html#83 has the same remoteDispatchInterfaceGetXMLDescHelper -> virInterfaceGetXMLDesc crash.
> [1] https://github.com/cockpit-project/cockpit-machines/issues/1391#issuecomment-1963676507 That shows: virtinterfaced[1858]: Assertion '*q > 0' failed at src/libudev/libudev.c:91, function udev_ref(). Aborting. which is the missing context I wanted from the stack trace in comment #1 and comment #2. IOW, we have a refcount underflow - it can't have wrapped since there's another assert_se call that should log a message on overflow. I can spot one bug in virInterfaceGetXMLDesc(), where it could fail to release a reference if an invalid flag is passed in, but that should be resulting in clear errors in your app code, and would result in an overflow eventually, not an underflow.
FEDORA-2024-ee96e0c470 (libvirt-10.1.0-2.fc40) has been submitted as an update to Fedora 40. https://bodhi.fedoraproject.org/updates/FEDORA-2024-ee96e0c470
FEDORA-2024-c2e7b82022 (libvirt-9.7.0-4.fc39) has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2024-c2e7b82022
> I can spot one bug in virInterfaceGetXMLDesc(), where it could fail to release a reference if an invalid flag is passed in, but that should be resulting in clear errors in your app code, and would result in an overflow eventually, not an underflow. The update I've issued addresses that flaw. I'm fairly sceptical that it will solve the problem cockpit sees, but its worth a try as thus far I can't explain the problem otherwise.
FEDORA-2024-ee96e0c470 has been pushed to the Fedora 40 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-ee96e0c470` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-ee96e0c470 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2024-c2e7b82022 has been pushed to the Fedora 39 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-c2e7b82022` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-c2e7b82022 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2024-ee96e0c470 (libvirt-10.1.0-2.fc40) has been pushed to the Fedora 40 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2024-c2e7b82022 (libvirt-9.7.0-4.fc39) has been pushed to the Fedora 39 stable repository. If problem still persists, please make note of it in this bug report.
Our CI bots believe the issue has been fixed: https://github.com/cockpit-project/bots/pull/6953 They didn't see any core dumps of virtinterfaced for 25 days. Thanks!
cockpit-machines also stopped using virtinterfaced: https://github.com/cockpit-project/cockpit-machines/pull/1782