Created attachment 841791 [details]
valgrind_libvirtd

Created attachment 841792 [details]
kernel message log

Created attachment 841793 [details]
libvirtd log

I was able to reproduce this on Fedora 20 (which has the same netcf version). The problem is the function aug_get_mac() - it doesn't initialize the local char *path to NULL, but then unconditionally frees it on error; the problem is that one error condition can be encountered prior to path getting set.

I'm changing the component to netcf and will be posting a patch upstream shortly.

Note that repproducing the bug will be quicker if you run this loop concurrent with the loop that is loading/unloading the network card driver (btw, on my system I had an 82576 card / igb driver, so you don't necessarily need an 82559):

  while true; do virsh iface-list; done

A patch has been posted to the upstream netcf mailing list:

https://lists.fedorahosted.org/pipermail/netcf-devel/2014-January/000853.html

I tested this patch by attaching gdb to the libvirtd process (so I would easily see when it crashed) and running these two shell scripts simultaneously in different windows:

---

  while true; do virsh iface-list; done
---

  i=1
  while true; do
    echo $i; i=$(expr $i + 1)
    modprobe -r igb
    usleep 1
    modprobe igb max_vfs=7
    usleep 1
  done

Without the patched netcf, libvirtd would crash within < 30 seconds. Without the patch I ran the test for several minutes with no crashes (the virsh iface-list will periodically fail due to interface config being in an inconsistent state through multiple calls to the virInterface API, but that is an unsolvable (and acceptable) problem.

Pushed this upstream:

commit 8ed36d22fbc792474ca9c3b06c8a326b1fb5af08
Author: Laine Stump <laine>
Date:   Tue Jan 7 20:12:06 2014 +0200

    eliminate use of uninitialized data when getting mac address
    
    If the call to get_augeas() at the top of aug_get_mac() failed, we
    would goto error and FREE(path), which would not have been
    initialized. And if by some magic of fate we happened to get past
    that, we would return garbage for the return code, since r was also
    not initialized. This patch initializes both path and r to fix the
    crash documented in Bug 1046594.
    
    Although it doesn't directly impact the referenced bug, a quick audit
    of other functions in the same file showed that defnode() had the same
    problem with uninitialized "r". Beyond that, I also defensively
    initialized the pointer to mac address to NULL both in aug_get_mac()
    as well as two of its callers, to make future audits of the code
    easier, and to shut up both valgrind and whatever static analyzers
    might be run on the code.

Hanns-Joachim Uhl: Why do you consider this BZ to block an update to the ixbge driver? Did you encounter this crash while testing?

The problem is actually driver-agnostic (so it is just as likely to happen with the old isgbe as the new ixgbe). It can be completely avoided when testing by simply not running virt-manager (which happens to frequently call netcf) during the testing of the new driver (or at least during parts that involve unloading/loading a netdev driver). Depending on your tests, that may be less than ideal, but likely not a large problem.

(That said, a netcf build is likely coming this week, but lack of a build probably shouldn't prevent testing of the ixgbe driver)

To test that the fix is working properly, you will need to restart libvirtd.service after updating the netcf package (there is no method of automatically triggering that other than mandating offline updates). So:


   yum update netcf-*.x86_64.rpm
   systemctl restart libvirtd.service
   (now run the test scripts in Comment 7)

So how is then blocking BZ "Bug 726818 - [Intel 7.0 FEAT] Update ixgbe driver to latest upstream."?  It should not be any more correct?

(In reply to John Ronciak from comment #14)
> So how is then blocking BZ "Bug 726818 - [Intel 7.0 FEAT] Update ixgbe
> driver to latest upstream."?  It should not be any more correct?
.
... correct ...

Since Bug 965845 was also in the "blocks" list, and it is just a duplicate of Bug 726818, I've also removed it from the blocks list.

In latest netcf-0.2.3-6.el7.x86_64, run with those two scripts in Comment 7, there is no crash happened.
So I choose to change the status to VERIFIED.

This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.