Bug 785431

Summary: lxc domain log file quickly fills up filesystem
Product: [Fedora] Fedora Reporter: Robin Green <greenrd>
Component: libvirtAssignee: Libvirt Maintainers <libvirt-maint>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 16CC: berrange, clalancette, crobinso, dougsland, itamar, jforbes, laine, libvirt-maint, veillard, virt-maint
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-07-05 19:45:02 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:

Description Robin Green 2012-01-28 17:28:00 EST
Description of problem:
My lxc domain has filled up my root filesystem, by printing this message to /var/log/libvirt/lxc/Rawhide.log over and over in rapid succession:
2012-01-28 21:09:34.997+0000: 15515: error : lxcFdForward:461 : read of fd 9 failed: Input/output error
2012-01-28 21:09:34.997+0000: 15515: error : lxcFdForward:461 : read of fd 9 failed: Input/output error
2012-01-28 21:09:34.997+0000: 15515: error : lxcFdForward:461 : read of fd 9 failed: Input/output error
2012-01-28 21:09:34.997+0000: 15515: error : lxcFdForward:461 : read of fd 9 failed: Input/output error
2012-01-28 21:09:34.997+0000: 15515: error : lxcFdForward:461 : read of fd 9 failed: Input/output error
2012-01-28 21:09:34.997+0000: 15515: error : lxcFdForward:461 : read of fd 9 failed: Input/output error
2012-01-28 21:09:34.997+0000: 15515: error : lxcFdForward:461 : read of fd 9 failed: Input/output error
2012-01-28 21:09:34.997+0000: 15515: error : lxcFdForward:461 : read of fd 9 failed: Input/output error
2012-01-28 21:09:34.997+0000: 15515: error : lxcFdForward:461 : read of fd 9 failed: Input/output error

Version-Release number of selected component (if applicable):
libvirt-0.9.6-4.fc16.x86_64

How reproducible:
Haven't tried

Steps to Reproduce:
1. Install rawhide in disk image
2. Mount root partition from disk image on Fedora 16 host
3. Create LXC domain in virt-manager, using root directory of rawhide root partition as the "install" location, and "install" it. Note: This won't actually work, due to bug 785418. So maybe you'd have to try the attempted workarounds I linked to there (which also don't work) to reproduce this bug - I'm not sure.
  
Actual results:
Log file gets spammed, grows to 5GB within 13 minutes, and silently fills up root filesystem

Expected results:
Log file should not get spammed so much 

Additional info:
Comment 1 Robin Green 2012-01-28 17:38:36 EST
The problem seems to be:

[root@pavilion lxc]# fuser Rawhide.log 
Rawhide.log:         17538
[root@pavilion lxc]# cat /proc/17538/cmdline 
/usr/libexec/libvirt_lxc--nameRawhide--console24--handshake27--background--vethveth1[root@pavilion lxc]# ls -l /proc/17538/fd
total 0
lrwx------. 1 root root 64 Jan 28 22:04 0 -> /dev/null
l-wx------. 1 root root 64 Jan 28 22:04 1 -> /var/log/libvirt/lxc/Rawhide.log
l-wx------. 1 root root 64 Jan 28 22:04 2 -> /var/log/libvirt/lxc/Rawhide.log
lrwx------. 1 root root 64 Jan 28 22:04 24 -> /dev/ptmx
lrwx------. 1 root root 64 Jan 28 22:04 3 -> socket:[1104196]
lrwx------. 1 root root 64 Jan 28 22:04 4 -> socket:[1104999]
lrwx------. 1 root root 64 Jan 28 22:04 5 -> socket:[1102982]
lrwx------. 1 root root 64 Jan 28 22:04 6 -> anon_inode:[eventpoll]
lrwx------. 1 root root 64 Jan 28 22:04 7 -> socket:[1102984]
lrwx------. 1 root root 64 Jan 28 22:04 9 -> /mnt/rawhide/dev/pts/ptmx
[root@pavilion lxc]# ls /mnt/rawhide/dev/pts/ptmx
ls: cannot access /mnt/rawhide/dev/pts/ptmx: No such file or directory
[root@pavilion lxc]# ls /mnt/rawhide/dev/pts/
[root@pavilion lxc]# ls /mnt/rawhide/dev/
fd  full  null  ptmx  pts  random  shm  stderr  stdin  stdout  tty  urandom  zero

- as you can see, the guest /dev/pts is private, so the host libvirt wouldn't be able to read from it. Or maybe I misunderstand.
Comment 2 Daniel Berrange 2012-01-30 05:44:52 EST
> - as you can see, the guest /dev/pts is private, so the host libvirt wouldn't
> be able to read from it. Or maybe I misunderstand.

The libvirt_lxc process is doing special things with mounts & namespaces, so that it can read/write to both the host /dev/pts and the container /dev/pts at the same time.


I have seen this bug before, and I think it is fixed upstream, so I'll check if there's something to backport.
Comment 3 Cole Robinson 2012-06-07 15:59:23 EDT
Moving in to POST, I know there were a bunch of lxc error reporting fixes that need backport.
Comment 4 Fedora Update System 2012-06-24 19:55:16 EDT
libvirt-0.9.6.1-1.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/libvirt-0.9.6.1-1.fc16
Comment 5 Fedora Update System 2012-06-25 20:55:50 EDT
Package libvirt-0.9.6.1-1.fc16:
* should fix your issue,
* was pushed to the Fedora 16 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing libvirt-0.9.6.1-1.fc16'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-9913/libvirt-0.9.6.1-1.fc16
then log in and leave karma (feedback).
Comment 6 Fedora Update System 2012-07-05 19:45:02 EDT
libvirt-0.9.6.1-1.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.