Bugzilla will be upgraded to version 5.0 on a still to be determined date in the near future. The original upgrade date has been delayed.
Bug 988335 - [LXC] container die of Input/output error
[LXC] container die of Input/output error
Status: CLOSED DUPLICATE of bug 986826
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt (Show other bugs)
7.0
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Libvirt Maintainers
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-25 06:49 EDT by Monson Shao
Modified: 2014-09-21 18:51 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-07-26 12:33:51 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
libvirtd.log (6.55 MB, application/gzip)
2013-07-25 06:51 EDT, Monson Shao
no flags Details
bash-00015.log (127.97 KB, text/x-log)
2013-07-25 06:52 EDT, Monson Shao
no flags Details

  None (edit)
Description Monson Shao 2013-07-25 06:49:32 EDT
Description of problem:

When running a lot of containers (6000 bash / 2500 httpd), wait for a while (or several hours), some containers will exit sequentially. Memory seems rich and we did nothing during waiting.
Useful logs are below. I will attach the full logs of libvirt-sandbox-service and libvirtd, please focus on bash-00015 (near the end of libvirtd.log)

2013-07-25 10:19:15.136+0000: 2279: debug : virDomainFree:2322 : dom=0x7fac74001400, (VM: name=bash-02093, uuid=ea65ae40-0f30-4db2-95ea-fdd638a4479a)
2013-07-25 10:19:15.244+0000: 2280: debug : virDomainLookupByID:2041 : conn=0x7faa54112f40, id=17217
2013-07-25 10:19:15.248+0000: 2280: debug : virDomainFree:2322 : dom=0x7fac80000d40, (VM: name=bash-02525, uuid=23ecb55e-d474-491b-adff-ef4756a574cd)
2013-07-25 10:19:15.356+0000: 2281: debug : virDomainLookupByID:2041 : conn=0x7faa54112f40, id=14685
2013-07-25 10:19:15.360+0000: 2281: debug : virDomainFree:2322 : dom=0x7fac68000f50, (VM: name=bash-02481, uuid=f196b185-29ee-4764-959c-0eedbb151712)
2013-07-25 10:19:15.468+0000: 4996: debug : virDomainLookupByID:2041 : conn=0x7faa54112f40, id=7891
2013-07-25 10:19:15.472+0000: 4996: debug : virDomainFree:2322 : dom=0x7faa9c048300, (VM: name=bash-01162, uuid=4825ca8b-f676-4526-817d-4da1266e4052)
2013-07-25 10:19:15.872+0000: 2278: debug : virConnectDomainEventDeregisterAny:18812 : conn=0x7faa54112f40, callbackID=6025
2013-07-25 10:19:16.133+0000: 4952: debug : virDomainDestroy:2213 : dom=0x7fab28062420, (VM: name=bash-00015, uuid=eb58bc35-4c32-44b6-a6ad-8498224c546d)
2013-07-25 10:19:16.133+0000: 4952: debug : virLXCProcessStop:749 : Stopping VM name=bash-00015 pid=3385 reason=2
2013-07-25 10:19:16.185+0000: 2271: error : virNetSocketReadWire:1379 : End of file while reading data: Input/output error
2013-07-25 10:19:16.185+0000: 2271: debug : virLXCMonitorEOFNotify:121 : EOF notify mon=0x7fac580c7890
2013-07-25 10:19:16.185+0000: 2271: debug : virLXCMonitorEOFNotify:128 : EOF callback mon=0x7fac580c7890 vm=0x7fac580c44e0
2013-07-25 10:19:16.185+0000: 2271: debug : virLXCProcessMonitorEOFNotify:561 : mon=0x7fac580c7890 vm=0x7fac580c44e0
2013-07-25 10:19:16.335+0000: 4952: debug : virLXCProcessCleanup:229 : Stopping VM name=bash-00015 pid=3385 reason=2
2013-07-25 10:19:16.335+0000: 4952: debug : virLXCProcessAutoDestroyRemove:150 : vm=bash-00015 uuid=eb58bc35-4c32-44b6-a6ad-8498224c546d
2013-07-25 10:19:16.335+0000: 4952: debug : virLXCMonitorClose:210 : mon=0x7fac580c7890
2013-07-25 10:19:16.335+0000: 4952: debug : virLXCMonitorClose:216 : Clear EOF callback mon=0x7fac580c7890
2013-07-25 10:19:16.347+0000: 4952: debug : virDomainFree:2322 : dom=0x7fab28062420, (VM: name=bash-00015, uuid=eb58bc35-4c32-44b6-a6ad-8498224c546d)
2013-07-25 10:19:16.347+0000: 2271: debug : virLXCProcessStop:749 : Stopping VM name=bash-00015 pid=-1 reason=1
2013-07-25 10:19:16.347+0000: 2271: debug : virLXCProcessStop:751 : VM 'bash-00015' not active
2013-07-25 10:19:16.348+0000: 2271: debug : virLXCProcessStop:749 : Stopping VM name=bash-00015 pid=-1 reason=1
2013-07-25 10:19:16.348+0000: 2271: debug : virLXCProcessStop:751 : VM 'bash-00015' not active
2013-07-25 10:19:16.348+0000: 2271: debug : virLXCProcessMonitorEOFNotify:577 : Stop event has already been sent
2013-07-25 10:19:16.386+0000: 2271: debug : virLXCMonitorDispose:201 : mon=0x7fac580c7890
2013-07-25 10:19:16.731+0000: 2271: debug : virConnectClose:1490 : conn=0x7faa54112f40
2013-07-25 10:19:16.732+0000: 2271: debug : virLXCProcessAutoDestroyRun:124 : conn=0x7faa54112f40


Version-Release number of selected component (if applicable):
kernel-3.10.0-0.rc7.64.el7
libvirt-1.1.0-1.el7
libvirt-sandbox-0.2.1-1.el7

How reproducible:
always

Steps to Reproduce:
1. run a log of containers
2. wait

Actual results:
some containers exit unexpectedly.

Expected results:
containers should not exit unless being kill.

Additional info:
Comment 1 Monson Shao 2013-07-25 06:51:28 EDT
Created attachment 778204 [details]
libvirtd.log
Comment 2 Monson Shao 2013-07-25 06:52:07 EDT
Created attachment 778206 [details]
bash-00015.log
Comment 3 Alex Jia 2013-07-25 06:59:39 EDT
(In reply to Monson Shao from comment #0)
> Description of problem:
> 
> When running a lot of containers (6000 bash / 2500 httpd), wait for a while
> (or several hours), some containers will exit sequentially. Memory seems
> rich and we did nothing during waiting.

Did you run "virt-sandbox-serivce list -r"? it exists a bug then will kill containers.
Comment 5 Monson Shao 2013-07-25 21:56:58 EDT
(In reply to Alex Jia from comment #3)
> (In reply to Monson Shao from comment #0)
> 
> Did you run "virt-sandbox-serivce list -r"? it exists a bug then will kill
> containers.

I didn't run that command, but it seems virt-sandbox-service.logrotate did. Could you point out which bug it may related to?

I will turn off logrotate and test again.
Comment 6 Alex Jia 2013-07-26 01:59:12 EDT
(In reply to Monson Shao from comment #5)
> (In reply to Alex Jia from comment #3)
> > (In reply to Monson Shao from comment #0)
> > 
> > Did you run "virt-sandbox-serivce list -r"? it exists a bug then will kill
> > containers.
> 
> I didn't run that command, but it seems virt-sandbox-service.logrotate did.
> Could you point out which bug it may related to?
> 
> I will turn off logrotate and test again.

It may be the same question with bug 986826.

# cat /etc/cron.daily/virt-sandbox-service.logrotate
#!/bin/sh
for i in `/usr/bin/virt-sandbox-service list --running`; do
    /usr/bin/virt-sandbox-service execute $i -- /etc/cron.daily/logrotate
done
exit 0
Comment 7 Daniel Walsh 2013-07-26 12:33:51 EDT

*** This bug has been marked as a duplicate of bug 986826 ***

Note You need to log in before you can comment on or make changes to this bug.