Bug 890039

Summary:

libvirtd memory leak, process grows to 10 GB

Product:

[Fedora] Fedora

Reporter:

Richard W.M. Jones <rjones>

Component:

libvirt

Assignee:

Libvirt Maintainers <libvirt-maint>

Status:

CLOSED CURRENTRELEASE

QA Contact:

Fedora Extras Quality Assurance <extras-qa>

Severity:

unspecified

Docs Contact:

Priority:

unspecified

Version:

CC:

berrange, clalancette, crobinso, dyasny, itamar, jforbes, jyang, laine, libvirt-maint, veillard, virt-maint

Target Milestone:

---

Target Release:

---

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Clones:

891324 (view as bug list)

Environment:

Last Closed:

2013-03-04 17:16:31 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

903203, 903280

Bug Blocks:

891324

Attachments:

Description	Flags
coredump (xz compressed)	none

Description Richard W.M. Jones 2012-12-24 14:18:51 UTC

Description of problem:

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND           
25390 rjones    20   0 10.9g  10g 8.4g S   0.0 66.8  16:34.08 libvirtd          

Note this appears to be a session libvirtd.

I sent this process a SIGBUS and collected the core dump which
is 2.1 GB in size.  I will attempt to compress it and upload it.

Version-Release number of selected component (if applicable):

libvirt-daemon-0.10.2.2-3.fc18.x86_64

How reproducible:

unknown

Comment 1 Richard W.M. Jones 2012-12-24 15:10:00 UTC

Created attachment 668504 [details]
coredump (xz compressed)

A mere 19 MB compressed.  Expands to 2.1 GB.

Comment 2 Richard W.M. Jones 2012-12-24 15:14:58 UTC

I did a 'strings' on the coredump and then sorted it to
show the most common strings.  They all seem to be SELinux
contexts.  (The number in the left hand column is the number
of times each string occurs in the coredump file).

2366430 system_u:object_r:bin_t:s0
1663530 system_u:object_r:textrel_shlib_t:s0
 929390 system_u:object_r:lvm_exec_t:s0
 601370 system_u:object_r:fsadm_exec_t:s0
 421740 system_u:object_r:games_exec_t:s0
 359260 <<none>>
 320210 system_u:object_r:innd_exec_t:s0
 288970 system_u:object_r:fixed_disk_device_t:s0
 273350 system_u:object_r:lib_t:s0
 242110 system_u:object_r:shell_exec_t:s0
 242110 system_u:object_r:nagios_services_plugin_exec_t:s0
 203060 system_u:object_r:rpm_exec_t:s0
 195250 system_u:object_r:etc_runtime_t:s0
 195250 system_u:object_r:cupsd_rw_etc_t:s0
 187440 system_u:object_r:httpd_sys_rw_content_t:s0
 179630 system_u:object_r:sound_device_t:s0
 179630 system_u:object_r:removable_device_t:s0
 171820 system_u:object_r:udev_exec_t:s0
 164010 system_u:object_r:var_log_t:s0
 164010 system_u:object_r:system_munin_plugin_exec_t:s0
 156200 system_u:object_r:tty_device_t:s0
 156200 system_u:object_r:iptables_exec_t:s0
 156200 system_u:object_r:ifconfig_exec_t:s0
 148390 system_u:object_r:initrc_exec_t:s0
 148390 system_u:object_r:httpd_sys_content_t:s0
 140580 system_u:object_r:wine_exec_t:s0
 140580 system_u:object_r:services_munin_plugin_exec_t:s0
 140580 system_u:object_r:mouse_device_t:s0
 140580 system_u:object_r:cobbler_var_lib_t:s0
 132770 system_u:object_r:svc_run_exec_t:s0
 117150 unconfined_u:object_r:mozilla_home_t:s0
 117150 system_u:object_r:vmware_host_exec_t:s0
 117150 system_u:object_r:nagios_system_plugin_exec_t:s0
 117150 system_u:object_r:mozilla_exec_t:s0
 117150 system_u:object_r:httpd_log_t:s0
 109340 system_u:object_r:xdm_var_run_t:s0
 109340 system_u:object_r:lpr_exec_t:s0
 109340 system_u:object_r:etc_t:s0
 101530 system_u:object_r:named_conf_t:s0
 101530 system_u:object_r:httpd_sys_script_exec_t:s0
 101530 system_u:object_r:httpd_cache_t:s0
  93720 system_u:object_r:xserver_misc_device_t:s0
  93720 system_u:object_r:v4l_device_t:s0
  85910 system_u:object_r:xsession_exec_t:s0
  85910 system_u:object_r:thumb_exec_t:s0
  85910 system_u:object_r:syslogd_exec_t:s0
  85910 system_u:object_r:net_conf_t:s0
  85910 system_u:object_r:mnt_t:s0
  85910 system_u:object_r:locale_t:s0
  85910 system_u:object_r:httpd_var_run_t:s0
  78100 system_u:object_r:usr_t:s0
  78100 system_u:object_r:sendmail_exec_t:s0
  78100 system_u:object_r:krb5_host_rcache_t:s0
  78100 system_u:object_r:httpd_exec_t:s0
  70290 system_u:object_r:xserver_exec_t:s0
  70290 system_u:object_r:xdm_exec_t:s0
  70290 system_u:object_r:var_auth_t:s0
  70290 system_u:object_r:traceroute_exec_t:s0
  70290 system_u:object_r:ssh_home_t:s0
  62480 system_u:object_r:xauth_home_t:s0
  62480 system_u:object_r:tape_device_t:s0
  62480 system_u:object_r:smbd_var_run_t:s0
  62480 system_u:object_r:quota_db_t:s0
  62480 system_u:object_r:printer_device_t:s0
  62480 system_u:object_r:postfix_master_exec_t:s0
  62480 system_u:object_r:passwd_file_t:s0
  62480 system_u:object_r:NetworkManager_var_lib_t:s0
  62480 system_u:object_r:mount_exec_t:s0
  62480 system_u:object_r:mdadm_exec_t:s0
  62480 system_u:object_r:likewise_initrc_exec_t:s0
  62480 system_u:object_r:ipsec_exec_t:s0
  62480 system_u:object_r:init_exec_t:s0
  62480 system_u:object_r:httpd_config_t:s0
  62480 system_u:object_r:event_device_t:s0
  62480 system_u:object_r:dhcpc_exec_t:s0
  62480 system_u:object_r:bootloader_exec_t:s0
  62480 system_u:object_r:bluetooth_exec_t:s0
  62480 system_u:object_r:admin_passwd_exec_t:s0
  54670 system_u:object_r:vmware_exec_t:s0
  54670 system_u:object_r:virtd_exec_t:s0
  54670 system_u:object_r:usb_device_t:s0
  54670 system_u:object_r:syslogd_var_run_t:s0
  54670 system_u:object_r:svc_svc_t:s0
  54670 system_u:object_r:sshd_key_t:s0
  54670 system_u:object_r:radiusd_log_t:s0
  54670 system_u:object_r:power_unit_file_t:s0
  54670 system_u:object_r:postgresql_log_t:s0
  54670 system_u:object_r:NetworkManager_exec_t:s0
  54670 system_u:object_r:named_cache_t:s0
  54670 system_u:object_r:mail_munin_plugin_exec_t:s0
  54670 system_u:object_r:lvm_metadata_t:s0
  54670 system_u:object_r:lost_found_t:s0

etc etc etc until ...

   7810 system_u:object_r:abrt_retrace_coredump_exec_t:s0
   7810 system_u:object_r:abrt_initrc_exec_t:s0
   7810 system_u:object_r:abrt_helper_exec_t:s0
   7810 system_u:object_r:abrt_handle_event_exec_t:s0
   7810 system_u:object_r:abrt_etc_t:s0
   7810 system_u:object_r:abrt_dump_oops_exec_t:s0
   4216    5
   2804     5
   1876 LinuxRestoreS5
   1588      5
   1518 rityFileLabel5
   1350 ext on '/home5
   1341 etFilec5
   1279 rSecurit5
   1202       5
   1154 /run/guestfs-%
   1106 ux context5
   1104 ing SELinux c5
   1067 org.freedesktop.DBus.Error.NoReply
   1066 Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
   1043 nes5
   1007 Linux context5
   1000 nuxRestoreSec5
    777 ecurityS5
    737     <5
    719 s/tmp/libgues5
    633  context o5
    629 g SELinux con5
    592   <5
    553 tyFileLabel:95
    552 :s0'
    525    E
    515 /qemu/log/gue%
    506 tmp/libguestf5
    498        5
    495       <5
    490 e_t:s0
    485 le.sock'5
    460 ome/rct_r%
    444     <f5
    435 u:object_%
    427         5
    424 ine>
    417 t on '/home/r5

Comment 3 Richard W.M. Jones 2012-12-24 16:23:03 UTC

I ran a small Perl script over the top "SELinux" strings
and they alone amount to a minimum of 1,392,398,110 bytes
(likely that is a large underestimate when you consider
malloc overheads).

Comment 4 Richard W.M. Jones 2013-01-24 12:52:41 UTC

I'll put this bug in POST so Cole knows that we need
the patches in Fedora 18 and RHEL 7.

http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=05cc03518987fa0f8399930d14c1d635591ca49b
http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=6159710ca1eecefa7c81335612c8141c88fc35a9

Comment 5 Cole Robinson 2013-01-24 22:38:15 UTC

Thanks Rich, I'm planning on new -maint releases on Monday.

Comment 6 Fedora Update System 2013-01-28 20:46:18 UTC

libvirt-0.10.2.3-1.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/libvirt-0.10.2.3-1.fc18

Comment 7 Fedora Update System 2013-02-05 16:53:27 UTC

libvirt-0.10.2.3-1.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.