Bug 476383 - Better error reporting from xend when dom0 is out of diskspace is needed
Summary: Better error reporting from xend when dom0 is out of diskspace is needed
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: xen
Version: 5.4
Hardware: All
OS: Linux
low
medium
Target Milestone: rc
: ---
Assignee: Michal Novotny
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks: 514498
TreeView+ depends on / blocked
 
Reported: 2008-12-13 22:12 UTC by Gurhan Ozen
Modified: 2014-02-02 22:36 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-07-27 14:38:57 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Patch to introduce dom0-min-space configuration option (3.30 KB, patch)
2010-07-12 17:11 UTC, Michal Novotny
no flags Details | Diff

Description Gurhan Ozen 2008-12-13 22:12:39 UTC
Description of problem:

  I have tried to start a guest and got the following error:
libvir: Xen Daemon error : POST operation failed: (xend.err "Error creating domain: (1, 'Internal error', 'xc_dom_do_gunzip: inflate failed
                    (rc=-5)\\n')")

xend.log said:
 ERROR (XendDomainInfo:212) Domain construction failed
Traceback (most recent call last):
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 205, in create
    vm.initDomain()
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1540, in initDomain
    raise VmError(str(exn))
VmError: (2, 'Invalid kernel', 'xc_dom_find_loader: no loader found\n')
[2008-12-13 16:40:55 xend.XendDomainInfo 6865] DEBUG (XendDomainInfo:1626) XendDomainInfo.destroy: domid=24
[2008-12-13 16:40:55 xend.XendDomainInfo 6865] ERROR (XendDomainInfo:1924) Failed to restart domain 21.
Traceback (most recent call last):
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1907, in restart
    new_dom = XendDomain.instance().domain_create(config)
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 238, in domain_create
    dominfo = XendDomainInfo.create(config)
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 205, in create
    vm.initDomain()
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1540, in initDomain
    raise VmError(str(exn))
VmError: (2, 'Invalid kernel', 'xc_dom_find_loader: no loader found\n')


I was thinking that the guest was corrupted but upon closer inspection, I found out that this error was triggered by the fact that dom0 ran out of harddisk space.

Could the error message be made more concise and specific in this case?


Version-Release number of selected component (if applicable):
# rpm -q xen 
xen-3.0.3-79.el5


How reproducible:
Very

Steps to Reproduce:
1. Try to start a guest after filling up the disk on dom0

Comment 5 Michal Novotny 2010-07-12 15:41:08 UTC
(In reply to comment #0)
> Description of problem:
> 
>   I have tried to start a guest and got the following error:
> libvir: Xen Daemon error : POST operation failed: (xend.err "Error creating
> domain: (1, 'Internal error', 'xc_dom_do_gunzip: inflate failed
>                     (rc=-5)\\n')")
> 
> xend.log said:
>  ERROR (XendDomainInfo:212) Domain construction failed
> Traceback (most recent call last):
>   File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line
> 205, in create
>     vm.initDomain()
>   File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line
> 1540, in initDomain
>     raise VmError(str(exn))
> VmError: (2, 'Invalid kernel', 'xc_dom_find_loader: no loader found\n')
> [2008-12-13 16:40:55 xend.XendDomainInfo 6865] DEBUG (XendDomainInfo:1626)
> XendDomainInfo.destroy: domid=24
> [2008-12-13 16:40:55 xend.XendDomainInfo 6865] ERROR (XendDomainInfo:1924)
> Failed to restart domain 21.
> Traceback (most recent call last):
>   File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line
> 1907, in restart
>     new_dom = XendDomain.instance().domain_create(config)
>   File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 238,
> in domain_create
>     dominfo = XendDomainInfo.create(config)
>   File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line
> 205, in create
>     vm.initDomain()
>   File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line
> 1540, in initDomain
>     raise VmError(str(exn))
> VmError: (2, 'Invalid kernel', 'xc_dom_find_loader: no loader found\n')
> 
> 

I suppose you're trying to run PV guest, right? Nevertheless adding some check for enough space on dom0 on each domain create call could be a good idea.

Michal

Comment 6 Michal Novotny 2010-07-12 17:11:02 UTC
Created attachment 431218 [details]
Patch to introduce dom0-min-space configuration option

It appears to be coming from image.py and X86_Linux_ImageHandler (which is available for PV guests) which is being called from the XendDomainInfo.py on create, most probably from lines 2065 to 2067:

            self.image = image.create(self,
                                      self.info['image'],
                                      self.info['device'])

Also, the extraction of kernel and ramdisk files itself is done by pygrub but we can't rely on this one since the file could be partially extracted. The xc_dom_do_unzip() function initializes the gzip stream and tries to inflate (decompress) it. The problem within the decompression implementations (no matter whether gzip, LZMA or any other) is that when the input data are not correct it just fails with some error code like Z_BUF_ERROR which is code -5 in the zlib.h header file. This means that we can't know in advance that the stream could be OK or not. Since this is because of the insufficient data on the host (dom0) I recommend adding one new dom0-min-* variable, we already have dom0-min-mem in the configuration file which makes domUs *not run* when the dom0 would be having less memory after particular domain starts up.

I was thinking about something similar to this one, called e.g. dom0-min-space or something similar in the configuration file that would be the minimum space on the dom0 and that dom0 will refuse to start up the new guest when this minimum not being met with the warning the dom0 free space dropped below this value. I'm attaching the patch to introduce the dom0-min-space option right now.

Michal

Comment 7 Paolo Bonzini 2010-07-27 14:38:32 UTC
If there is "very little" space (e.g. 1 megabyte) and pygrub fails really early, that's not a problem because python will exit with a status of 1.

I don't think an option like dom0-min-space is worthwhile.  A server running out of disk space will have other issues and anyway the option is inherently racy (what if another program is writing to log files continuously and very fast? that's a common cause of disk full situations).

I think we can close it as WONTFIX, as Daniel suggested.

Comment 8 Paolo Bonzini 2010-07-27 14:38:57 UTC
Closing after talking to Michal.


Note You need to log in before you can comment on or make changes to this bug.