RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 690286 - augeas occasionally leaks fd's and returns errors during initialization of netcf
Summary: augeas occasionally leaks fd's and returns errors during initialization of netcf
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: augeas
Version: 6.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: David Lutterkort
QA Contact: Virtualization Bugs
URL:
Whiteboard:
: 635669 687973 (view as bug list)
Depends On:
Blocks: 670961
TreeView+ depends on / blocked
 
Reported: 2011-03-23 19:35 UTC by Laine Stump
Modified: 2018-11-27 21:17 UTC (History)
15 users (show)

Fixed In Version: augeas-0.7.2-6.el6
Doc Type: Bug Fix
Doc Text:
augeas was not thread safe, and could leak file descriptors when multiple programs attempted to use the libvirt library simultaneously. This resulted in the failure of the calling program. augeas has been modified to remove the global variable that caused this threading issue.
Clone Of:
Environment:
Last Closed: 2011-05-19 13:54:26 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
augeas make check (17.46 KB, application/octet-stream)
2011-03-31 06:00 UTC, Huang Wenlong 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2011:0650 0 normal SHIPPED_LIVE virt-v2v and augeas bug fix update 2011-07-01 15:27:50 UTC

Description Laine Stump 2011-03-23 19:35:54 UTC 
there were reports from a couple of users of a leak of fd's owned by libvirtd, as illustrated by this example output of "lsof | grep aug":

libvirtd  20159      root   42r      REG              253,1     5478    2112374 /usr/share/augeas/lenses/dist/build.aug
libvirtd  20159      root   43r      REG              253,1     3557    2112732 /usr/share/augeas/lenses/dist/util.aug
libvirtd  20159      root   44r      REG              253,1     3557    2112732 /usr/share/augeas/lenses/dist/util.aug
libvirtd  20159      root   45r      REG              253,1     3557    2112732 /usr/share/augeas/lenses/dist/util.aug
libvirtd  20159      root   46r      REG              253,1     3557    2112732 /usr/share/augeas/lenses/dist/util.aug
libvirtd  20159      root   47r      REG              253,1     3557    2112732 /usr/share/augeas/lenses/dist/util.aug
libvirtd  20159      root   48r      REG              253,1     3557    2112732 /usr/share/augeas/lenses/dist/util.aug
libvirtd  20159      root   49r      REG              253,1     5478    2112374 /usr/share/augeas/lenses/dist/build.aug
libvirtd  20159      root   50r      REG              253,1     3557    2112732 /usr/share/augeas/lenses/dist/util.aug
libvirtd  20159      root   51r      REG              253,1     3557    2112732 /usr/share/augeas/lenses/dist/util.aug
libvirtd  20159      root   52r      REG              253,1     3557    2112732 /usr/share/augeas/lenses/dist/util.aug
libvirtd  20159      root   53r      REG              253,1     5478    2112374 /usr/share/augeas/lenses/dist/build.aug
libvirtd  20159      root   54r      REG              253,1     3557    2112732 /usr/share/augeas/lenses/dist/util.aug

This problem can be reproduced by opening two shell windows, and running this command in both:

   while true; do virsh iface-list; done

You will occasionally notice an error scroll past in the windows:

   error: Failed to list active interfaces
   error: internal error failed to get number of interfaces on host (netcf:
          unspecified error - errors in loading some config files)

If you run libvirtd with NETCF_DEBUG defined in its environment, you will be able to see more specifics of the errors reported by augeas. They are always similar (but not identical) to this:
=======================================
warning: augeas initialization had errors
please file a bug with the following lines in the bug report:
/augeas/load/Ifcfg/error = "/usr/share/augeas/lenses/dist/util.aug:149:0: Missing *)\n/usr/share/netcf/lenses/sysconfig.aug:7.12-.20:Could not load module Util for Util.eol\n/usr/share/netcf/lenses/sysconfig.aug:7.12-.20:Undefined variable Util.eol\n/usr/share/netcf/lenses/sysconfig.aug:10.11-.23:Could not load module Util for Util.del_str\n/usr/share/netcf/lenses/sysconfig.aug:10.11-.23:Undefined variable Util.del_str\n/usr/share/netcf/lenses/sysconfig.aug:11.16-.28:Could not load module Util for Util.comment\n/usr/share/netcf/lenses/sysconfig.aug:11.16-.28:Undefined variable Util.comment\n/usr/share/netcf/lenses/sysconfig.aug:12.16-.26:Could not load module Util for Util.empty\n/usr/share/netcf/lenses/sysconfig.aug:12.16-.26:Undefined variable Util.empty\n/usr/share/netcf/lenses/sysconfig.aug:13.16-.27:Could not load module Build for Build.xchgs\n/usr/share/netcf/lenses/sysconfig.aug:13.16-.27:Undefined variable Build.xchgs\n/usr/share/netcf/lenses/sysconfig.aug:14.16-.28:Could not load module Util for Util.del_str\n/usr/share/netcf/lenses/sysconfig.aug:14.16-.28:Undefined variable Util.del_str\n/usr/share/netcf/lenses/sysconfig.aug:33.4-.8:Undefined variable dels\n/usr/share/netcf/lenses/sysconfig.aug:35.32-.47:Could not load module Util for Util.del_ws_spc\n/usr/share/netcf/lenses/sysconfig.aug:35.32-.47:Undefined variable Util.del_ws_spc\n/usr/share/netcf/lenses/sysconfig.aug:36.26-.32:Undefined variable export\n/usr/share/netcf/lenses/sysconfig.aug:37.15-.17:Undefined variable kv\n/usr/share/netcf/lenses/sysconfig.aug:40.6-.11:Undefined variable xchgs\n/usr/share/netcf/lenses/sysconfig.aug:42.14-.24:Undefined variable var_action\n/usr/share/netcf/lenses/sysconfig.aug:43.20-.30:Undefined variable var_action\n/usr/share/netcf/lenses/sysconfig.aug:48.6-.21:Could not load module Util for Util.del_ws_spc\n/usr/share/netcf/lenses/sysconfig.aug:48.6-.21:Undefined variable Util.del_ws_spc\n/usr/share/netcf/lenses/sysconfig.aug:51.12-.20:Undefined variable comment"
==========================================

The leaked fd's are caused by improper error recovery from the parsing errors. The parsing errors are caused by the parser using a global variable to keep track of comment nesting level - if multiple threads in libvirtd are calling augeas at the same time, they can stomp on each others' use of comment nesting level, and the parser gets confused.

Both of these problems (the leaked fd's when recovering from the error, and the use of the global variable that causes this particular error) have been fixed in upstream augeas:

commit 39ad6cd46357961dda67a1c9eda65229a25858ae
Author: David Lutterkort <lutter>
Date:   Tue Mar 22 17:18:44 2011 -0700

    Eliminate global variable 'comment_depth' in lexer

commit 9a5d5834d6a15a0d751233dc7d617b75fd4d7564
Author: David Lutterkort <lutter>
Date:   Wed Mar 16 14:06:50 2011 -0700

    * src/lexer.l (augl_init_lexer): really close the input file on error

Test runs have verified that these two fixes solve the issue.
Comment 1 David Lutterkort 2011-03-23 21:38:58 UTC 
There is one more upstream patch needed to fix this:

commit 35c86e740c268ec7fefa5b32a964419cc5e112c0
Author: David Lutterkort <lutter>
Date:   Tue Mar 8 13:44:51 2011 -0800

    Be more defensive about closing input files for the lexer
Comment 2 David Lutterkort 2011-03-25 16:55:33 UTC 
Built augeas-0.7.2-6.el6 to address this issue; see https://brewweb.devel.redhat.com//taskinfo?taskID=3203845
Comment 3 Laine Stump 2011-03-25 17:31:37 UTC 
*** Bug 687973 has been marked as a duplicate of this bug. ***
Comment 5 Huang Wenlong 2011-03-31 05:22:05 UTC 
verify this bug in RHEL6 with augeas-0.7.2-6.el6

steps : 

1.update augeas to 0.7.2-6  

2.run the "while true; do virsh iface-list; done"  in two shell windows 
No more show error message 

3.run the libvirtd with NETCF_DEBUG=1 
No more show warning message like: "augeas initialization had errors"

4.lsof |grep aug 
libvirtd   2812      root  mem       REG                8,3     301944    1705059 /usr/lib64/libaugeas.so.0.11.0
perl      17548      root  DEL       REG                8,3               1733213 /usr/lib64/libaugeas.so.0.11.0
Comment 6 Huang Wenlong 2011-03-31 06:00:15 UTC 
Created attachment 488983 [details]
augeas make check

augeas make check all tests passed
Comment 9 Laura Bailey 2011-04-27 07:25:26 UTC 
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
augeas was not thread safe, and could leak file descriptors when multiple programs attempted to use the libvirt library simultaneously. This resulted in the failure of the calling program. augeas has been patched to be more threadsafe, so this issue no longer occurs.
Comment 10 David Lutterkort 2011-04-27 17:32:34 UTC 
Change the last sentence to 'Augeas has been modified to remove the global variable that caused this threading issue'

(It is by all accounts not just more threadsafe, but just plain threadsafe now)
Comment 11 Laura Bailey 2011-04-27 23:40:44 UTC 
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1 +1 @@
-augeas was not thread safe, and could leak file descriptors when multiple programs attempted to use the libvirt library simultaneously. This resulted in the failure of the calling program. augeas has been patched to be more threadsafe, so this issue no longer occurs.+augeas was not thread safe, and could leak file descriptors when multiple programs attempted to use the libvirt library simultaneously. This resulted in the failure of the calling program. augeas has been modified to remove the global variable that caused this threading issue.
Comment 12 Dave Allan 2011-04-28 01:04:16 UTC 
*** Bug 635669 has been marked as a duplicate of this bug. ***
Comment 13 errata-xmlrpc 2011-05-19 13:54:26 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0650.html
Note You need to log in before you can comment on or make changes to this bug.