Bug 988335

Summary: [LXC] container die of Input/output error
Product: Red Hat Enterprise Linux 7 Reporter: Monson Shao <jshao>
Component: libvirtAssignee: Libvirt Maintainers <libvirt-maint>
Status: CLOSED DUPLICATE QA Contact: Virtualization Bugs <virt-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.0CC: acathrow, ajia, arozansk, berrange, ccui, dwalsh, dyuan, kzhang, lwang
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-07-26 16:33:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
libvirtd.log
none
bash-00015.log none

Description Monson Shao 2013-07-25 10:49:32 UTC 
Description of problem:

When running a lot of containers (6000 bash / 2500 httpd), wait for a while (or several hours), some containers will exit sequentially. Memory seems rich and we did nothing during waiting.
Useful logs are below. I will attach the full logs of libvirt-sandbox-service and libvirtd, please focus on bash-00015 (near the end of libvirtd.log)

2013-07-25 10:19:15.136+0000: 2279: debug : virDomainFree:2322 : dom=0x7fac74001400, (VM: name=bash-02093, uuid=ea65ae40-0f30-4db2-95ea-fdd638a4479a)
2013-07-25 10:19:15.244+0000: 2280: debug : virDomainLookupByID:2041 : conn=0x7faa54112f40, id=17217
2013-07-25 10:19:15.248+0000: 2280: debug : virDomainFree:2322 : dom=0x7fac80000d40, (VM: name=bash-02525, uuid=23ecb55e-d474-491b-adff-ef4756a574cd)
2013-07-25 10:19:15.356+0000: 2281: debug : virDomainLookupByID:2041 : conn=0x7faa54112f40, id=14685
2013-07-25 10:19:15.360+0000: 2281: debug : virDomainFree:2322 : dom=0x7fac68000f50, (VM: name=bash-02481, uuid=f196b185-29ee-4764-959c-0eedbb151712)
2013-07-25 10:19:15.468+0000: 4996: debug : virDomainLookupByID:2041 : conn=0x7faa54112f40, id=7891
2013-07-25 10:19:15.472+0000: 4996: debug : virDomainFree:2322 : dom=0x7faa9c048300, (VM: name=bash-01162, uuid=4825ca8b-f676-4526-817d-4da1266e4052)
2013-07-25 10:19:15.872+0000: 2278: debug : virConnectDomainEventDeregisterAny:18812 : conn=0x7faa54112f40, callbackID=6025
2013-07-25 10:19:16.133+0000: 4952: debug : virDomainDestroy:2213 : dom=0x7fab28062420, (VM: name=bash-00015, uuid=eb58bc35-4c32-44b6-a6ad-8498224c546d)
2013-07-25 10:19:16.133+0000: 4952: debug : virLXCProcessStop:749 : Stopping VM name=bash-00015 pid=3385 reason=2
2013-07-25 10:19:16.185+0000: 2271: error : virNetSocketReadWire:1379 : End of file while reading data: Input/output error
2013-07-25 10:19:16.185+0000: 2271: debug : virLXCMonitorEOFNotify:121 : EOF notify mon=0x7fac580c7890
2013-07-25 10:19:16.185+0000: 2271: debug : virLXCMonitorEOFNotify:128 : EOF callback mon=0x7fac580c7890 vm=0x7fac580c44e0
2013-07-25 10:19:16.185+0000: 2271: debug : virLXCProcessMonitorEOFNotify:561 : mon=0x7fac580c7890 vm=0x7fac580c44e0
2013-07-25 10:19:16.335+0000: 4952: debug : virLXCProcessCleanup:229 : Stopping VM name=bash-00015 pid=3385 reason=2
2013-07-25 10:19:16.335+0000: 4952: debug : virLXCProcessAutoDestroyRemove:150 : vm=bash-00015 uuid=eb58bc35-4c32-44b6-a6ad-8498224c546d
2013-07-25 10:19:16.335+0000: 4952: debug : virLXCMonitorClose:210 : mon=0x7fac580c7890
2013-07-25 10:19:16.335+0000: 4952: debug : virLXCMonitorClose:216 : Clear EOF callback mon=0x7fac580c7890
2013-07-25 10:19:16.347+0000: 4952: debug : virDomainFree:2322 : dom=0x7fab28062420, (VM: name=bash-00015, uuid=eb58bc35-4c32-44b6-a6ad-8498224c546d)
2013-07-25 10:19:16.347+0000: 2271: debug : virLXCProcessStop:749 : Stopping VM name=bash-00015 pid=-1 reason=1
2013-07-25 10:19:16.347+0000: 2271: debug : virLXCProcessStop:751 : VM 'bash-00015' not active
2013-07-25 10:19:16.348+0000: 2271: debug : virLXCProcessStop:749 : Stopping VM name=bash-00015 pid=-1 reason=1
2013-07-25 10:19:16.348+0000: 2271: debug : virLXCProcessStop:751 : VM 'bash-00015' not active
2013-07-25 10:19:16.348+0000: 2271: debug : virLXCProcessMonitorEOFNotify:577 : Stop event has already been sent
2013-07-25 10:19:16.386+0000: 2271: debug : virLXCMonitorDispose:201 : mon=0x7fac580c7890
2013-07-25 10:19:16.731+0000: 2271: debug : virConnectClose:1490 : conn=0x7faa54112f40
2013-07-25 10:19:16.732+0000: 2271: debug : virLXCProcessAutoDestroyRun:124 : conn=0x7faa54112f40


Version-Release number of selected component (if applicable):
kernel-3.10.0-0.rc7.64.el7
libvirt-1.1.0-1.el7
libvirt-sandbox-0.2.1-1.el7

How reproducible:
always

Steps to Reproduce:
1. run a log of containers
2. wait

Actual results:
some containers exit unexpectedly.

Expected results:
containers should not exit unless being kill.

Additional info:
Comment 1 Monson Shao 2013-07-25 10:51:28 UTC 
Created attachment 778204 [details]
libvirtd.log
Comment 2 Monson Shao 2013-07-25 10:52:07 UTC 
Created attachment 778206 [details]
bash-00015.log
Comment 3 Alex Jia 2013-07-25 10:59:39 UTC 
(In reply to Monson Shao from comment #0)
> Description of problem:
> 
> When running a lot of containers (6000 bash / 2500 httpd), wait for a while
> (or several hours), some containers will exit sequentially. Memory seems
> rich and we did nothing during waiting.

Did you run "virt-sandbox-serivce list -r"? it exists a bug then will kill containers.
Comment 5 Monson Shao 2013-07-26 01:56:58 UTC 
(In reply to Alex Jia from comment #3)
> (In reply to Monson Shao from comment #0)
> 
> Did you run "virt-sandbox-serivce list -r"? it exists a bug then will kill
> containers.

I didn't run that command, but it seems virt-sandbox-service.logrotate did. Could you point out which bug it may related to?

I will turn off logrotate and test again.
Comment 6 Alex Jia 2013-07-26 05:59:12 UTC 
(In reply to Monson Shao from comment #5)
> (In reply to Alex Jia from comment #3)
> > (In reply to Monson Shao from comment #0)
> > 
> > Did you run "virt-sandbox-serivce list -r"? it exists a bug then will kill
> > containers.
> 
> I didn't run that command, but it seems virt-sandbox-service.logrotate did.
> Could you point out which bug it may related to?
> 
> I will turn off logrotate and test again.

It may be the same question with bug 986826.

# cat /etc/cron.daily/virt-sandbox-service.logrotate
#!/bin/sh
for i in `/usr/bin/virt-sandbox-service list --running`; do
    /usr/bin/virt-sandbox-service execute $i -- /etc/cron.daily/logrotate
done
exit 0
Comment 7 Daniel Walsh 2013-07-26 16:33:51 UTC 

*** This bug has been marked as a duplicate of bug 986826 ***