Bug 862815

Summary: imagefactory fails to push to rhevm beta 3.1
Product: [Retired] CloudForms Cloud Engine Reporter: Dave Johnson <dajohnso>
Component: imagefactoryAssignee: Ian McLeod <imcleod>
Status: CLOSED NOTABUG QA Contact: Dave Johnson <dajohnso>
Severity: medium Docs Contact:
Priority: urgent    
Version: 1.1.0CC: jrd, ldelouw, mitch, morazi, mpastern
Target Milestone: rcKeywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-10-12 15:00:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dave Johnson 2012-10-03 15:38:04 UTC
Description of problem:
==============================================================
configure for rhevm 3.1 server and pushes fail to it with nfs mount not found when it is there, mounted, and accessible.


2012-10-02 13:34:21,311 DEBUG imgfac.builders.BaseBuilder.RHEL6_rhevm_Builder thread(c718e3c9) Message: Executing external RHEV-M push command (['/usr/bin/dc-rhev-image', '/tmp/tmpfzVDJq'])
2012-10-02 13:34:21,782 DEBUG paste.httpserver.ThreadPool thread(MainThread) Message: Added task (0 tasks queued)
2012-10-02 13:34:22,415 DEBUG imgfac.builders.BaseBuilder.RHEL6_rhevm_Builder thread(c718e3c9) Message: Exception caught in ImageFactory
2012-10-02 13:34:22,428 DEBUG imgfac.builders.BaseBuilder.RHEL6_rhevm_Builder thread(c718e3c9) Message: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/imgfac/builders/Fedora_rhevm_Builder.py", line 198, in push_image
    self.rhevm_push_image_upload(target_image_id, provider, credentials)
  File "/usr/lib/python2.6/site-packages/imgfac/builders/Fedora_rhevm_Builder.py", line 284, in rhevm_push_image_upload
    (stdout, stderr, retcode) = subprocess_check_output(rhevm_push_command)
  File "/usr/lib/python2.6/site-packages/imgfac/builders/Fedora_rhevm_Builder.py", line 47, in subprocess_check_output
    raise ImageFactoryException("'%s' failed(%d): %s\nstdout: %s" % (cmd, retcode, stderr, stdout))
ImageFactoryException: '/usr/bin/dc-rhev-image /tmp/tmpfzVDJq' failed(1): None
stdout: ERROR NFS storage domain for `qeblade31.xxxxxx.com:/home/rhev/me_export' not found


2012-10-02 13:34:22,429 DEBUG imgfac.BuildJob.BuildJob thread(c718e3c9) Message: Builder (c718e3c9-736b-4bd4-9eef-ff67f32e5e13) changed status from PUSHING to FAILED


[root@tyan-gt24-03 ~]# mount
/dev/sda3 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
/dev/sda1 on /boot type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
qeblade26.xxxxxx.com:/home/rhev/me_expt on /mnt/rhevm-nfs type nfs (rw)
qeblade31.xxxxxx.com:/home/rhev/me_export on /mnt/rhevm-nfs2 type nfs (rw)
[root@tyan-gt24-03 ~]# cd /mnt/rhevm-nfs2
[root@tyan-gt24-03 rhevm-nfs2]# ll
total 4
drwxr-xr-x. 5 nobody nobody 4096 Sep 24 19:29 9b75d300-5f19-4099-ba3c-321db2fc7968
[root@tyan-gt24-03 rhevm-nfs2]#


Version-Release number of selected component (if applicable):
==================================================================
imagefactory-1.0.2-1.el6cf.noarch

Comment 2 Ian McLeod 2012-10-04 17:22:32 UTC
The C program we use to execute this transfer on the v1.0 factory provides very limited debug output in the event of a REST call failure.

I re-ran this exact build and push with our upstream factory code.  This code uses the ovirt-sdk bindings for Python and it also failed.  However, it provided the following error detail:

  File "/usr/lib/python2.7/site-packages/imagefactory_plugins/RHEV /RHEVMHelper.py", line 359, in execute_import
    template = self.storage_domain_object.templates.get(id=str(self.tpl_uuid))

  File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 3374, in get
    raise err

RequestError: 
status: 400
reason: Bad Request
detail: Cannot find master domain

Here is the line that generated that traceback:

https://github.com/aeolusproject/imagefactory/blob/master/imagefactory-plugins/RHEVM/RHEVMHelper.py#L359

It's entirely possible this is a configuration issue with our RHEV-M server.  I would be grateful if someone more familiar with the REST API and its potential error messages could have a look at this.  The full import procedure is represented in the github source file linked above.  We can provide access to both the RHEV-M server and the Image Factory server.

Setting to NEEDINFO from mpastern

Comment 3 Michael Pasternak 2012-10-09 12:53:33 UTC
Ian,

Please add debug=True in sdk .ctr and attach sdk output.

thanks.

Comment 4 Dave Johnson 2012-10-12 15:00:08 UTC
It seems that the storage domains went down for some reason after the rhevm environment was setup.  Re-activating them resolved the push failure.