Bug 862815 - imagefactory fails to push to rhevm beta 3.1
imagefactory fails to push to rhevm beta 3.1
Product: CloudForms Cloud Engine
Classification: Red Hat
Component: imagefactory (Show other bugs)
Unspecified Unspecified
urgent Severity medium
: rc
: ---
Assigned To: Ian McLeod
Dave Johnson
: Triaged
Depends On:
  Show dependency treegraph
Reported: 2012-10-03 11:38 EDT by Dave Johnson
Modified: 2012-10-12 11:00 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2012-10-12 11:00:08 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Dave Johnson 2012-10-03 11:38:04 EDT
Description of problem:
configure for rhevm 3.1 server and pushes fail to it with nfs mount not found when it is there, mounted, and accessible.

2012-10-02 13:34:21,311 DEBUG imgfac.builders.BaseBuilder.RHEL6_rhevm_Builder thread(c718e3c9) Message: Executing external RHEV-M push command (['/usr/bin/dc-rhev-image', '/tmp/tmpfzVDJq'])
2012-10-02 13:34:21,782 DEBUG paste.httpserver.ThreadPool thread(MainThread) Message: Added task (0 tasks queued)
2012-10-02 13:34:22,415 DEBUG imgfac.builders.BaseBuilder.RHEL6_rhevm_Builder thread(c718e3c9) Message: Exception caught in ImageFactory
2012-10-02 13:34:22,428 DEBUG imgfac.builders.BaseBuilder.RHEL6_rhevm_Builder thread(c718e3c9) Message: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/imgfac/builders/Fedora_rhevm_Builder.py", line 198, in push_image
    self.rhevm_push_image_upload(target_image_id, provider, credentials)
  File "/usr/lib/python2.6/site-packages/imgfac/builders/Fedora_rhevm_Builder.py", line 284, in rhevm_push_image_upload
    (stdout, stderr, retcode) = subprocess_check_output(rhevm_push_command)
  File "/usr/lib/python2.6/site-packages/imgfac/builders/Fedora_rhevm_Builder.py", line 47, in subprocess_check_output
    raise ImageFactoryException("'%s' failed(%d): %s\nstdout: %s" % (cmd, retcode, stderr, stdout))
ImageFactoryException: '/usr/bin/dc-rhev-image /tmp/tmpfzVDJq' failed(1): None
stdout: ERROR NFS storage domain for `qeblade31.xxxxxx.com:/home/rhev/me_export' not found

2012-10-02 13:34:22,429 DEBUG imgfac.BuildJob.BuildJob thread(c718e3c9) Message: Builder (c718e3c9-736b-4bd4-9eef-ff67f32e5e13) changed status from PUSHING to FAILED

[root@tyan-gt24-03 ~]# mount
/dev/sda3 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
/dev/sda1 on /boot type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
qeblade26.xxxxxx.com:/home/rhev/me_expt on /mnt/rhevm-nfs type nfs (rw)
qeblade31.xxxxxx.com:/home/rhev/me_export on /mnt/rhevm-nfs2 type nfs (rw)
[root@tyan-gt24-03 ~]# cd /mnt/rhevm-nfs2
[root@tyan-gt24-03 rhevm-nfs2]# ll
total 4
drwxr-xr-x. 5 nobody nobody 4096 Sep 24 19:29 9b75d300-5f19-4099-ba3c-321db2fc7968
[root@tyan-gt24-03 rhevm-nfs2]#

Version-Release number of selected component (if applicable):
Comment 2 Ian McLeod 2012-10-04 13:22:32 EDT
The C program we use to execute this transfer on the v1.0 factory provides very limited debug output in the event of a REST call failure.

I re-ran this exact build and push with our upstream factory code.  This code uses the ovirt-sdk bindings for Python and it also failed.  However, it provided the following error detail:

  File "/usr/lib/python2.7/site-packages/imagefactory_plugins/RHEV /RHEVMHelper.py", line 359, in execute_import
    template = self.storage_domain_object.templates.get(id=str(self.tpl_uuid))

  File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 3374, in get
    raise err

status: 400
reason: Bad Request
detail: Cannot find master domain

Here is the line that generated that traceback:


It's entirely possible this is a configuration issue with our RHEV-M server.  I would be grateful if someone more familiar with the REST API and its potential error messages could have a look at this.  The full import procedure is represented in the github source file linked above.  We can provide access to both the RHEV-M server and the Image Factory server.

Setting to NEEDINFO from mpastern@redhat.com
Comment 3 Michael Pasternak 2012-10-09 08:53:33 EDT

Please add debug=True in sdk .ctr and attach sdk output.

Comment 4 Dave Johnson 2012-10-12 11:00:08 EDT
It seems that the storage domains went down for some reason after the rhevm environment was setup.  Re-activating them resolved the push failure.

Note You need to log in before you can comment on or make changes to this bug.