Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
augeas was not thread safe, and could leak file descriptors when multiple programs attempted to use the libvirt library simultaneously. This resulted in the failure of the calling program. augeas has been modified to remove the global variable that caused this threading issue.
there were reports from a couple of users of a leak of fd's owned by libvirtd, as illustrated by this example output of "lsof | grep aug":
libvirtd 20159 root 42r REG 253,1 5478 2112374 /usr/share/augeas/lenses/dist/build.aug
libvirtd 20159 root 43r REG 253,1 3557 2112732 /usr/share/augeas/lenses/dist/util.aug
libvirtd 20159 root 44r REG 253,1 3557 2112732 /usr/share/augeas/lenses/dist/util.aug
libvirtd 20159 root 45r REG 253,1 3557 2112732 /usr/share/augeas/lenses/dist/util.aug
libvirtd 20159 root 46r REG 253,1 3557 2112732 /usr/share/augeas/lenses/dist/util.aug
libvirtd 20159 root 47r REG 253,1 3557 2112732 /usr/share/augeas/lenses/dist/util.aug
libvirtd 20159 root 48r REG 253,1 3557 2112732 /usr/share/augeas/lenses/dist/util.aug
libvirtd 20159 root 49r REG 253,1 5478 2112374 /usr/share/augeas/lenses/dist/build.aug
libvirtd 20159 root 50r REG 253,1 3557 2112732 /usr/share/augeas/lenses/dist/util.aug
libvirtd 20159 root 51r REG 253,1 3557 2112732 /usr/share/augeas/lenses/dist/util.aug
libvirtd 20159 root 52r REG 253,1 3557 2112732 /usr/share/augeas/lenses/dist/util.aug
libvirtd 20159 root 53r REG 253,1 5478 2112374 /usr/share/augeas/lenses/dist/build.aug
libvirtd 20159 root 54r REG 253,1 3557 2112732 /usr/share/augeas/lenses/dist/util.aug
This problem can be reproduced by opening two shell windows, and running this command in both:
while true; do virsh iface-list; done
You will occasionally notice an error scroll past in the windows:
error: Failed to list active interfaces
error: internal error failed to get number of interfaces on host (netcf:
unspecified error - errors in loading some config files)
If you run libvirtd with NETCF_DEBUG defined in its environment, you will be able to see more specifics of the errors reported by augeas. They are always similar (but not identical) to this:
=======================================
warning: augeas initialization had errors
please file a bug with the following lines in the bug report:
/augeas/load/Ifcfg/error = "/usr/share/augeas/lenses/dist/util.aug:149:0: Missing *)\n/usr/share/netcf/lenses/sysconfig.aug:7.12-.20:Could not load module Util for Util.eol\n/usr/share/netcf/lenses/sysconfig.aug:7.12-.20:Undefined variable Util.eol\n/usr/share/netcf/lenses/sysconfig.aug:10.11-.23:Could not load module Util for Util.del_str\n/usr/share/netcf/lenses/sysconfig.aug:10.11-.23:Undefined variable Util.del_str\n/usr/share/netcf/lenses/sysconfig.aug:11.16-.28:Could not load module Util for Util.comment\n/usr/share/netcf/lenses/sysconfig.aug:11.16-.28:Undefined variable Util.comment\n/usr/share/netcf/lenses/sysconfig.aug:12.16-.26:Could not load module Util for Util.empty\n/usr/share/netcf/lenses/sysconfig.aug:12.16-.26:Undefined variable Util.empty\n/usr/share/netcf/lenses/sysconfig.aug:13.16-.27:Could not load module Build for Build.xchgs\n/usr/share/netcf/lenses/sysconfig.aug:13.16-.27:Undefined variable Build.xchgs\n/usr/share/netcf/lenses/sysconfig.aug:14.16-.28:Could not load module Util for Util.del_str\n/usr/share/netcf/lenses/sysconfig.aug:14.16-.28:Undefined variable Util.del_str\n/usr/share/netcf/lenses/sysconfig.aug:33.4-.8:Undefined variable dels\n/usr/share/netcf/lenses/sysconfig.aug:35.32-.47:Could not load module Util for Util.del_ws_spc\n/usr/share/netcf/lenses/sysconfig.aug:35.32-.47:Undefined variable Util.del_ws_spc\n/usr/share/netcf/lenses/sysconfig.aug:36.26-.32:Undefined variable export\n/usr/share/netcf/lenses/sysconfig.aug:37.15-.17:Undefined variable kv\n/usr/share/netcf/lenses/sysconfig.aug:40.6-.11:Undefined variable xchgs\n/usr/share/netcf/lenses/sysconfig.aug:42.14-.24:Undefined variable var_action\n/usr/share/netcf/lenses/sysconfig.aug:43.20-.30:Undefined variable var_action\n/usr/share/netcf/lenses/sysconfig.aug:48.6-.21:Could not load module Util for Util.del_ws_spc\n/usr/share/netcf/lenses/sysconfig.aug:48.6-.21:Undefined variable Util.del_ws_spc\n/usr/share/netcf/lenses/sysconfig.aug:51.12-.20:Undefined variable comment"
==========================================
The leaked fd's are caused by improper error recovery from the parsing errors. The parsing errors are caused by the parser using a global variable to keep track of comment nesting level - if multiple threads in libvirtd are calling augeas at the same time, they can stomp on each others' use of comment nesting level, and the parser gets confused.
Both of these problems (the leaked fd's when recovering from the error, and the use of the global variable that causes this particular error) have been fixed in upstream augeas:
commit 39ad6cd46357961dda67a1c9eda65229a25858ae
Author: David Lutterkort <lutter>
Date: Tue Mar 22 17:18:44 2011 -0700
Eliminate global variable 'comment_depth' in lexer
commit 9a5d5834d6a15a0d751233dc7d617b75fd4d7564
Author: David Lutterkort <lutter>
Date: Wed Mar 16 14:06:50 2011 -0700
* src/lexer.l (augl_init_lexer): really close the input file on error
Test runs have verified that these two fixes solve the issue.
There is one more upstream patch needed to fix this:
commit 35c86e740c268ec7fefa5b32a964419cc5e112c0
Author: David Lutterkort <lutter>
Date: Tue Mar 8 13:44:51 2011 -0800
Be more defensive about closing input files for the lexer
verify this bug in RHEL6 with augeas-0.7.2-6.el6
steps :
1.update augeas to 0.7.2-6
2.run the "while true; do virsh iface-list; done" in two shell windows
No more show error message
3.run the libvirtd with NETCF_DEBUG=1
No more show warning message like: "augeas initialization had errors"
4.lsof |grep aug
libvirtd 2812 root mem REG 8,3 301944 1705059 /usr/lib64/libaugeas.so.0.11.0
perl 17548 root DEL REG 8,3 1733213 /usr/lib64/libaugeas.so.0.11.0
Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.
New Contents:
augeas was not thread safe, and could leak file descriptors when multiple programs attempted to use the libvirt library simultaneously. This resulted in the failure of the calling program. augeas has been patched to be more threadsafe, so this issue no longer occurs.
Comment 10David Lutterkort
2011-04-27 17:32:34 UTC
Change the last sentence to 'Augeas has been modified to remove the global variable that caused this threading issue'
(It is by all accounts not just more threadsafe, but just plain threadsafe now)
Technical note updated. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.
Diffed Contents:
@@ -1 +1 @@
-augeas was not thread safe, and could leak file descriptors when multiple programs attempted to use the libvirt library simultaneously. This resulted in the failure of the calling program. augeas has been patched to be more threadsafe, so this issue no longer occurs.+augeas was not thread safe, and could leak file descriptors when multiple programs attempted to use the libvirt library simultaneously. This resulted in the failure of the calling program. augeas has been modified to remove the global variable that caused this threading issue.
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.
http://rhn.redhat.com/errata/RHBA-2011-0650.html
there were reports from a couple of users of a leak of fd's owned by libvirtd, as illustrated by this example output of "lsof | grep aug": libvirtd 20159 root 42r REG 253,1 5478 2112374 /usr/share/augeas/lenses/dist/build.aug libvirtd 20159 root 43r REG 253,1 3557 2112732 /usr/share/augeas/lenses/dist/util.aug libvirtd 20159 root 44r REG 253,1 3557 2112732 /usr/share/augeas/lenses/dist/util.aug libvirtd 20159 root 45r REG 253,1 3557 2112732 /usr/share/augeas/lenses/dist/util.aug libvirtd 20159 root 46r REG 253,1 3557 2112732 /usr/share/augeas/lenses/dist/util.aug libvirtd 20159 root 47r REG 253,1 3557 2112732 /usr/share/augeas/lenses/dist/util.aug libvirtd 20159 root 48r REG 253,1 3557 2112732 /usr/share/augeas/lenses/dist/util.aug libvirtd 20159 root 49r REG 253,1 5478 2112374 /usr/share/augeas/lenses/dist/build.aug libvirtd 20159 root 50r REG 253,1 3557 2112732 /usr/share/augeas/lenses/dist/util.aug libvirtd 20159 root 51r REG 253,1 3557 2112732 /usr/share/augeas/lenses/dist/util.aug libvirtd 20159 root 52r REG 253,1 3557 2112732 /usr/share/augeas/lenses/dist/util.aug libvirtd 20159 root 53r REG 253,1 5478 2112374 /usr/share/augeas/lenses/dist/build.aug libvirtd 20159 root 54r REG 253,1 3557 2112732 /usr/share/augeas/lenses/dist/util.aug This problem can be reproduced by opening two shell windows, and running this command in both: while true; do virsh iface-list; done You will occasionally notice an error scroll past in the windows: error: Failed to list active interfaces error: internal error failed to get number of interfaces on host (netcf: unspecified error - errors in loading some config files) If you run libvirtd with NETCF_DEBUG defined in its environment, you will be able to see more specifics of the errors reported by augeas. They are always similar (but not identical) to this: ======================================= warning: augeas initialization had errors please file a bug with the following lines in the bug report: /augeas/load/Ifcfg/error = "/usr/share/augeas/lenses/dist/util.aug:149:0: Missing *)\n/usr/share/netcf/lenses/sysconfig.aug:7.12-.20:Could not load module Util for Util.eol\n/usr/share/netcf/lenses/sysconfig.aug:7.12-.20:Undefined variable Util.eol\n/usr/share/netcf/lenses/sysconfig.aug:10.11-.23:Could not load module Util for Util.del_str\n/usr/share/netcf/lenses/sysconfig.aug:10.11-.23:Undefined variable Util.del_str\n/usr/share/netcf/lenses/sysconfig.aug:11.16-.28:Could not load module Util for Util.comment\n/usr/share/netcf/lenses/sysconfig.aug:11.16-.28:Undefined variable Util.comment\n/usr/share/netcf/lenses/sysconfig.aug:12.16-.26:Could not load module Util for Util.empty\n/usr/share/netcf/lenses/sysconfig.aug:12.16-.26:Undefined variable Util.empty\n/usr/share/netcf/lenses/sysconfig.aug:13.16-.27:Could not load module Build for Build.xchgs\n/usr/share/netcf/lenses/sysconfig.aug:13.16-.27:Undefined variable Build.xchgs\n/usr/share/netcf/lenses/sysconfig.aug:14.16-.28:Could not load module Util for Util.del_str\n/usr/share/netcf/lenses/sysconfig.aug:14.16-.28:Undefined variable Util.del_str\n/usr/share/netcf/lenses/sysconfig.aug:33.4-.8:Undefined variable dels\n/usr/share/netcf/lenses/sysconfig.aug:35.32-.47:Could not load module Util for Util.del_ws_spc\n/usr/share/netcf/lenses/sysconfig.aug:35.32-.47:Undefined variable Util.del_ws_spc\n/usr/share/netcf/lenses/sysconfig.aug:36.26-.32:Undefined variable export\n/usr/share/netcf/lenses/sysconfig.aug:37.15-.17:Undefined variable kv\n/usr/share/netcf/lenses/sysconfig.aug:40.6-.11:Undefined variable xchgs\n/usr/share/netcf/lenses/sysconfig.aug:42.14-.24:Undefined variable var_action\n/usr/share/netcf/lenses/sysconfig.aug:43.20-.30:Undefined variable var_action\n/usr/share/netcf/lenses/sysconfig.aug:48.6-.21:Could not load module Util for Util.del_ws_spc\n/usr/share/netcf/lenses/sysconfig.aug:48.6-.21:Undefined variable Util.del_ws_spc\n/usr/share/netcf/lenses/sysconfig.aug:51.12-.20:Undefined variable comment" ========================================== The leaked fd's are caused by improper error recovery from the parsing errors. The parsing errors are caused by the parser using a global variable to keep track of comment nesting level - if multiple threads in libvirtd are calling augeas at the same time, they can stomp on each others' use of comment nesting level, and the parser gets confused. Both of these problems (the leaked fd's when recovering from the error, and the use of the global variable that causes this particular error) have been fixed in upstream augeas: commit 39ad6cd46357961dda67a1c9eda65229a25858ae Author: David Lutterkort <lutter> Date: Tue Mar 22 17:18:44 2011 -0700 Eliminate global variable 'comment_depth' in lexer commit 9a5d5834d6a15a0d751233dc7d617b75fd4d7564 Author: David Lutterkort <lutter> Date: Wed Mar 16 14:06:50 2011 -0700 * src/lexer.l (augl_init_lexer): really close the input file on error Test runs have verified that these two fixes solve the issue.