RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 988335 - [LXC] container die of Input/output error
Summary: [LXC] container die of Input/output error
Keywords:
Status: CLOSED DUPLICATE of bug 986826
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Libvirt Maintainers
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-07-25 10:49 UTC by Monson Shao
Modified: 2014-09-21 22:51 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-07-26 16:33:51 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
libvirtd.log (6.55 MB, application/gzip)
2013-07-25 10:51 UTC, Monson Shao
no flags Details
bash-00015.log (127.97 KB, text/x-log)
2013-07-25 10:52 UTC, Monson Shao
no flags Details

Description Monson Shao 2013-07-25 10:49:32 UTC
Description of problem:

When running a lot of containers (6000 bash / 2500 httpd), wait for a while (or several hours), some containers will exit sequentially. Memory seems rich and we did nothing during waiting.
Useful logs are below. I will attach the full logs of libvirt-sandbox-service and libvirtd, please focus on bash-00015 (near the end of libvirtd.log)

2013-07-25 10:19:15.136+0000: 2279: debug : virDomainFree:2322 : dom=0x7fac74001400, (VM: name=bash-02093, uuid=ea65ae40-0f30-4db2-95ea-fdd638a4479a)
2013-07-25 10:19:15.244+0000: 2280: debug : virDomainLookupByID:2041 : conn=0x7faa54112f40, id=17217
2013-07-25 10:19:15.248+0000: 2280: debug : virDomainFree:2322 : dom=0x7fac80000d40, (VM: name=bash-02525, uuid=23ecb55e-d474-491b-adff-ef4756a574cd)
2013-07-25 10:19:15.356+0000: 2281: debug : virDomainLookupByID:2041 : conn=0x7faa54112f40, id=14685
2013-07-25 10:19:15.360+0000: 2281: debug : virDomainFree:2322 : dom=0x7fac68000f50, (VM: name=bash-02481, uuid=f196b185-29ee-4764-959c-0eedbb151712)
2013-07-25 10:19:15.468+0000: 4996: debug : virDomainLookupByID:2041 : conn=0x7faa54112f40, id=7891
2013-07-25 10:19:15.472+0000: 4996: debug : virDomainFree:2322 : dom=0x7faa9c048300, (VM: name=bash-01162, uuid=4825ca8b-f676-4526-817d-4da1266e4052)
2013-07-25 10:19:15.872+0000: 2278: debug : virConnectDomainEventDeregisterAny:18812 : conn=0x7faa54112f40, callbackID=6025
2013-07-25 10:19:16.133+0000: 4952: debug : virDomainDestroy:2213 : dom=0x7fab28062420, (VM: name=bash-00015, uuid=eb58bc35-4c32-44b6-a6ad-8498224c546d)
2013-07-25 10:19:16.133+0000: 4952: debug : virLXCProcessStop:749 : Stopping VM name=bash-00015 pid=3385 reason=2
2013-07-25 10:19:16.185+0000: 2271: error : virNetSocketReadWire:1379 : End of file while reading data: Input/output error
2013-07-25 10:19:16.185+0000: 2271: debug : virLXCMonitorEOFNotify:121 : EOF notify mon=0x7fac580c7890
2013-07-25 10:19:16.185+0000: 2271: debug : virLXCMonitorEOFNotify:128 : EOF callback mon=0x7fac580c7890 vm=0x7fac580c44e0
2013-07-25 10:19:16.185+0000: 2271: debug : virLXCProcessMonitorEOFNotify:561 : mon=0x7fac580c7890 vm=0x7fac580c44e0
2013-07-25 10:19:16.335+0000: 4952: debug : virLXCProcessCleanup:229 : Stopping VM name=bash-00015 pid=3385 reason=2
2013-07-25 10:19:16.335+0000: 4952: debug : virLXCProcessAutoDestroyRemove:150 : vm=bash-00015 uuid=eb58bc35-4c32-44b6-a6ad-8498224c546d
2013-07-25 10:19:16.335+0000: 4952: debug : virLXCMonitorClose:210 : mon=0x7fac580c7890
2013-07-25 10:19:16.335+0000: 4952: debug : virLXCMonitorClose:216 : Clear EOF callback mon=0x7fac580c7890
2013-07-25 10:19:16.347+0000: 4952: debug : virDomainFree:2322 : dom=0x7fab28062420, (VM: name=bash-00015, uuid=eb58bc35-4c32-44b6-a6ad-8498224c546d)
2013-07-25 10:19:16.347+0000: 2271: debug : virLXCProcessStop:749 : Stopping VM name=bash-00015 pid=-1 reason=1
2013-07-25 10:19:16.347+0000: 2271: debug : virLXCProcessStop:751 : VM 'bash-00015' not active
2013-07-25 10:19:16.348+0000: 2271: debug : virLXCProcessStop:749 : Stopping VM name=bash-00015 pid=-1 reason=1
2013-07-25 10:19:16.348+0000: 2271: debug : virLXCProcessStop:751 : VM 'bash-00015' not active
2013-07-25 10:19:16.348+0000: 2271: debug : virLXCProcessMonitorEOFNotify:577 : Stop event has already been sent
2013-07-25 10:19:16.386+0000: 2271: debug : virLXCMonitorDispose:201 : mon=0x7fac580c7890
2013-07-25 10:19:16.731+0000: 2271: debug : virConnectClose:1490 : conn=0x7faa54112f40
2013-07-25 10:19:16.732+0000: 2271: debug : virLXCProcessAutoDestroyRun:124 : conn=0x7faa54112f40


Version-Release number of selected component (if applicable):
kernel-3.10.0-0.rc7.64.el7
libvirt-1.1.0-1.el7
libvirt-sandbox-0.2.1-1.el7

How reproducible:
always

Steps to Reproduce:
1. run a log of containers
2. wait

Actual results:
some containers exit unexpectedly.

Expected results:
containers should not exit unless being kill.

Additional info:

Comment 1 Monson Shao 2013-07-25 10:51:28 UTC
Created attachment 778204 [details]
libvirtd.log

Comment 2 Monson Shao 2013-07-25 10:52:07 UTC
Created attachment 778206 [details]
bash-00015.log

Comment 3 Alex Jia 2013-07-25 10:59:39 UTC
(In reply to Monson Shao from comment #0)
> Description of problem:
> 
> When running a lot of containers (6000 bash / 2500 httpd), wait for a while
> (or several hours), some containers will exit sequentially. Memory seems
> rich and we did nothing during waiting.

Did you run "virt-sandbox-serivce list -r"? it exists a bug then will kill containers.

Comment 5 Monson Shao 2013-07-26 01:56:58 UTC
(In reply to Alex Jia from comment #3)
> (In reply to Monson Shao from comment #0)
> 
> Did you run "virt-sandbox-serivce list -r"? it exists a bug then will kill
> containers.

I didn't run that command, but it seems virt-sandbox-service.logrotate did. Could you point out which bug it may related to?

I will turn off logrotate and test again.

Comment 6 Alex Jia 2013-07-26 05:59:12 UTC
(In reply to Monson Shao from comment #5)
> (In reply to Alex Jia from comment #3)
> > (In reply to Monson Shao from comment #0)
> > 
> > Did you run "virt-sandbox-serivce list -r"? it exists a bug then will kill
> > containers.
> 
> I didn't run that command, but it seems virt-sandbox-service.logrotate did.
> Could you point out which bug it may related to?
> 
> I will turn off logrotate and test again.

It may be the same question with bug 986826.

# cat /etc/cron.daily/virt-sandbox-service.logrotate
#!/bin/sh
for i in `/usr/bin/virt-sandbox-service list --running`; do
    /usr/bin/virt-sandbox-service execute $i -- /etc/cron.daily/logrotate
done
exit 0

Comment 7 Daniel Walsh 2013-07-26 16:33:51 UTC

*** This bug has been marked as a duplicate of bug 986826 ***


Note You need to log in before you can comment on or make changes to this bug.