Bug 785431 - lxc domain log file quickly fills up filesystem
lxc domain log file quickly fills up filesystem
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: libvirt (Show other bugs)
16
x86_64 Linux
unspecified Severity medium
: ---
: ---
Assigned To: Libvirt Maintainers
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-01-28 17:28 EST by Robin Green
Modified: 2012-07-05 19:45 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-07-05 19:45:02 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Robin Green 2012-01-28 17:28:00 EST
Description of problem:
My lxc domain has filled up my root filesystem, by printing this message to /var/log/libvirt/lxc/Rawhide.log over and over in rapid succession:
2012-01-28 21:09:34.997+0000: 15515: error : lxcFdForward:461 : read of fd 9 failed: Input/output error
2012-01-28 21:09:34.997+0000: 15515: error : lxcFdForward:461 : read of fd 9 failed: Input/output error
2012-01-28 21:09:34.997+0000: 15515: error : lxcFdForward:461 : read of fd 9 failed: Input/output error
2012-01-28 21:09:34.997+0000: 15515: error : lxcFdForward:461 : read of fd 9 failed: Input/output error
2012-01-28 21:09:34.997+0000: 15515: error : lxcFdForward:461 : read of fd 9 failed: Input/output error
2012-01-28 21:09:34.997+0000: 15515: error : lxcFdForward:461 : read of fd 9 failed: Input/output error
2012-01-28 21:09:34.997+0000: 15515: error : lxcFdForward:461 : read of fd 9 failed: Input/output error
2012-01-28 21:09:34.997+0000: 15515: error : lxcFdForward:461 : read of fd 9 failed: Input/output error
2012-01-28 21:09:34.997+0000: 15515: error : lxcFdForward:461 : read of fd 9 failed: Input/output error

Version-Release number of selected component (if applicable):
libvirt-0.9.6-4.fc16.x86_64

How reproducible:
Haven't tried

Steps to Reproduce:
1. Install rawhide in disk image
2. Mount root partition from disk image on Fedora 16 host
3. Create LXC domain in virt-manager, using root directory of rawhide root partition as the "install" location, and "install" it. Note: This won't actually work, due to bug 785418. So maybe you'd have to try the attempted workarounds I linked to there (which also don't work) to reproduce this bug - I'm not sure.
  
Actual results:
Log file gets spammed, grows to 5GB within 13 minutes, and silently fills up root filesystem

Expected results:
Log file should not get spammed so much 

Additional info:
Comment 1 Robin Green 2012-01-28 17:38:36 EST
The problem seems to be:

[root@pavilion lxc]# fuser Rawhide.log 
Rawhide.log:         17538
[root@pavilion lxc]# cat /proc/17538/cmdline 
/usr/libexec/libvirt_lxc--nameRawhide--console24--handshake27--background--vethveth1[root@pavilion lxc]# ls -l /proc/17538/fd
total 0
lrwx------. 1 root root 64 Jan 28 22:04 0 -> /dev/null
l-wx------. 1 root root 64 Jan 28 22:04 1 -> /var/log/libvirt/lxc/Rawhide.log
l-wx------. 1 root root 64 Jan 28 22:04 2 -> /var/log/libvirt/lxc/Rawhide.log
lrwx------. 1 root root 64 Jan 28 22:04 24 -> /dev/ptmx
lrwx------. 1 root root 64 Jan 28 22:04 3 -> socket:[1104196]
lrwx------. 1 root root 64 Jan 28 22:04 4 -> socket:[1104999]
lrwx------. 1 root root 64 Jan 28 22:04 5 -> socket:[1102982]
lrwx------. 1 root root 64 Jan 28 22:04 6 -> anon_inode:[eventpoll]
lrwx------. 1 root root 64 Jan 28 22:04 7 -> socket:[1102984]
lrwx------. 1 root root 64 Jan 28 22:04 9 -> /mnt/rawhide/dev/pts/ptmx
[root@pavilion lxc]# ls /mnt/rawhide/dev/pts/ptmx
ls: cannot access /mnt/rawhide/dev/pts/ptmx: No such file or directory
[root@pavilion lxc]# ls /mnt/rawhide/dev/pts/
[root@pavilion lxc]# ls /mnt/rawhide/dev/
fd  full  null  ptmx  pts  random  shm  stderr  stdin  stdout  tty  urandom  zero

- as you can see, the guest /dev/pts is private, so the host libvirt wouldn't be able to read from it. Or maybe I misunderstand.
Comment 2 Daniel Berrange 2012-01-30 05:44:52 EST
> - as you can see, the guest /dev/pts is private, so the host libvirt wouldn't
> be able to read from it. Or maybe I misunderstand.

The libvirt_lxc process is doing special things with mounts & namespaces, so that it can read/write to both the host /dev/pts and the container /dev/pts at the same time.


I have seen this bug before, and I think it is fixed upstream, so I'll check if there's something to backport.
Comment 3 Cole Robinson 2012-06-07 15:59:23 EDT
Moving in to POST, I know there were a bunch of lxc error reporting fixes that need backport.
Comment 4 Fedora Update System 2012-06-24 19:55:16 EDT
libvirt-0.9.6.1-1.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/libvirt-0.9.6.1-1.fc16
Comment 5 Fedora Update System 2012-06-25 20:55:50 EDT
Package libvirt-0.9.6.1-1.fc16:
* should fix your issue,
* was pushed to the Fedora 16 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing libvirt-0.9.6.1-1.fc16'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-9913/libvirt-0.9.6.1-1.fc16
then log in and leave karma (feedback).
Comment 6 Fedora Update System 2012-07-05 19:45:02 EDT
libvirt-0.9.6.1-1.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.