Bug 988335 - [LXC] container die of Input/output error
Summary: [LXC] container die of Input/output error
Status: CLOSED DUPLICATE of bug 986826
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Libvirt Maintainers
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Keywords:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-07-25 10:49 UTC by Monson Shao
Modified: 2014-09-21 22:51 UTC (History)
9 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2013-07-26 16:33:51 UTC


Attachments (Terms of Use)
libvirtd.log (6.55 MB, application/gzip)
2013-07-25 10:51 UTC, Monson Shao
no flags Details
bash-00015.log (127.97 KB, text/x-log)
2013-07-25 10:52 UTC, Monson Shao
no flags Details

Description Monson Shao 2013-07-25 10:49:32 UTC
Description of problem:

When running a lot of containers (6000 bash / 2500 httpd), wait for a while (or several hours), some containers will exit sequentially. Memory seems rich and we did nothing during waiting.
Useful logs are below. I will attach the full logs of libvirt-sandbox-service and libvirtd, please focus on bash-00015 (near the end of libvirtd.log)

2013-07-25 10:19:15.136+0000: 2279: debug : virDomainFree:2322 : dom=0x7fac74001400, (VM: name=bash-02093, uuid=ea65ae40-0f30-4db2-95ea-fdd638a4479a)
2013-07-25 10:19:15.244+0000: 2280: debug : virDomainLookupByID:2041 : conn=0x7faa54112f40, id=17217
2013-07-25 10:19:15.248+0000: 2280: debug : virDomainFree:2322 : dom=0x7fac80000d40, (VM: name=bash-02525, uuid=23ecb55e-d474-491b-adff-ef4756a574cd)
2013-07-25 10:19:15.356+0000: 2281: debug : virDomainLookupByID:2041 : conn=0x7faa54112f40, id=14685
2013-07-25 10:19:15.360+0000: 2281: debug : virDomainFree:2322 : dom=0x7fac68000f50, (VM: name=bash-02481, uuid=f196b185-29ee-4764-959c-0eedbb151712)
2013-07-25 10:19:15.468+0000: 4996: debug : virDomainLookupByID:2041 : conn=0x7faa54112f40, id=7891
2013-07-25 10:19:15.472+0000: 4996: debug : virDomainFree:2322 : dom=0x7faa9c048300, (VM: name=bash-01162, uuid=4825ca8b-f676-4526-817d-4da1266e4052)
2013-07-25 10:19:15.872+0000: 2278: debug : virConnectDomainEventDeregisterAny:18812 : conn=0x7faa54112f40, callbackID=6025
2013-07-25 10:19:16.133+0000: 4952: debug : virDomainDestroy:2213 : dom=0x7fab28062420, (VM: name=bash-00015, uuid=eb58bc35-4c32-44b6-a6ad-8498224c546d)
2013-07-25 10:19:16.133+0000: 4952: debug : virLXCProcessStop:749 : Stopping VM name=bash-00015 pid=3385 reason=2
2013-07-25 10:19:16.185+0000: 2271: error : virNetSocketReadWire:1379 : End of file while reading data: Input/output error
2013-07-25 10:19:16.185+0000: 2271: debug : virLXCMonitorEOFNotify:121 : EOF notify mon=0x7fac580c7890
2013-07-25 10:19:16.185+0000: 2271: debug : virLXCMonitorEOFNotify:128 : EOF callback mon=0x7fac580c7890 vm=0x7fac580c44e0
2013-07-25 10:19:16.185+0000: 2271: debug : virLXCProcessMonitorEOFNotify:561 : mon=0x7fac580c7890 vm=0x7fac580c44e0
2013-07-25 10:19:16.335+0000: 4952: debug : virLXCProcessCleanup:229 : Stopping VM name=bash-00015 pid=3385 reason=2
2013-07-25 10:19:16.335+0000: 4952: debug : virLXCProcessAutoDestroyRemove:150 : vm=bash-00015 uuid=eb58bc35-4c32-44b6-a6ad-8498224c546d
2013-07-25 10:19:16.335+0000: 4952: debug : virLXCMonitorClose:210 : mon=0x7fac580c7890
2013-07-25 10:19:16.335+0000: 4952: debug : virLXCMonitorClose:216 : Clear EOF callback mon=0x7fac580c7890
2013-07-25 10:19:16.347+0000: 4952: debug : virDomainFree:2322 : dom=0x7fab28062420, (VM: name=bash-00015, uuid=eb58bc35-4c32-44b6-a6ad-8498224c546d)
2013-07-25 10:19:16.347+0000: 2271: debug : virLXCProcessStop:749 : Stopping VM name=bash-00015 pid=-1 reason=1
2013-07-25 10:19:16.347+0000: 2271: debug : virLXCProcessStop:751 : VM 'bash-00015' not active
2013-07-25 10:19:16.348+0000: 2271: debug : virLXCProcessStop:749 : Stopping VM name=bash-00015 pid=-1 reason=1
2013-07-25 10:19:16.348+0000: 2271: debug : virLXCProcessStop:751 : VM 'bash-00015' not active
2013-07-25 10:19:16.348+0000: 2271: debug : virLXCProcessMonitorEOFNotify:577 : Stop event has already been sent
2013-07-25 10:19:16.386+0000: 2271: debug : virLXCMonitorDispose:201 : mon=0x7fac580c7890
2013-07-25 10:19:16.731+0000: 2271: debug : virConnectClose:1490 : conn=0x7faa54112f40
2013-07-25 10:19:16.732+0000: 2271: debug : virLXCProcessAutoDestroyRun:124 : conn=0x7faa54112f40


Version-Release number of selected component (if applicable):
kernel-3.10.0-0.rc7.64.el7
libvirt-1.1.0-1.el7
libvirt-sandbox-0.2.1-1.el7

How reproducible:
always

Steps to Reproduce:
1. run a log of containers
2. wait

Actual results:
some containers exit unexpectedly.

Expected results:
containers should not exit unless being kill.

Additional info:

Comment 1 Monson Shao 2013-07-25 10:51:28 UTC
Created attachment 778204 [details]
libvirtd.log

Comment 2 Monson Shao 2013-07-25 10:52:07 UTC
Created attachment 778206 [details]
bash-00015.log

Comment 3 Alex Jia 2013-07-25 10:59:39 UTC
(In reply to Monson Shao from comment #0)
> Description of problem:
> 
> When running a lot of containers (6000 bash / 2500 httpd), wait for a while
> (or several hours), some containers will exit sequentially. Memory seems
> rich and we did nothing during waiting.

Did you run "virt-sandbox-serivce list -r"? it exists a bug then will kill containers.

Comment 5 Monson Shao 2013-07-26 01:56:58 UTC
(In reply to Alex Jia from comment #3)
> (In reply to Monson Shao from comment #0)
> 
> Did you run "virt-sandbox-serivce list -r"? it exists a bug then will kill
> containers.

I didn't run that command, but it seems virt-sandbox-service.logrotate did. Could you point out which bug it may related to?

I will turn off logrotate and test again.

Comment 6 Alex Jia 2013-07-26 05:59:12 UTC
(In reply to Monson Shao from comment #5)
> (In reply to Alex Jia from comment #3)
> > (In reply to Monson Shao from comment #0)
> > 
> > Did you run "virt-sandbox-serivce list -r"? it exists a bug then will kill
> > containers.
> 
> I didn't run that command, but it seems virt-sandbox-service.logrotate did.
> Could you point out which bug it may related to?
> 
> I will turn off logrotate and test again.

It may be the same question with bug 986826.

# cat /etc/cron.daily/virt-sandbox-service.logrotate
#!/bin/sh
for i in `/usr/bin/virt-sandbox-service list --running`; do
    /usr/bin/virt-sandbox-service execute $i -- /etc/cron.daily/logrotate
done
exit 0

Comment 7 Daniel Walsh 2013-07-26 16:33:51 UTC

*** This bug has been marked as a duplicate of bug 986826 ***


Note You need to log in before you can comment on or make changes to this bug.