Bug 797995

Summary: factory push getting permission denied
Product: [Retired] CloudForms Cloud Engine Reporter: Dave Johnson <dajohnso>
Component: imagefactoryAssignee: Ian McLeod <imcleod>
Status: CLOSED CURRENTRELEASE QA Contact: Martin Kočí <mkoci>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 1.0.0CC: akarol, brad, dajohnso, deltacloud-maint, dgao, ssachdev, whayutin
Target Milestone: beta2   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-08-30 17:15:46 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Dave Johnson 2012-02-27 18:25:05 UTC
Description of problem:
============================
Install aeolus-all from the nightly repo which pulls in factory.  Building a image is successful but the push comes back with permission denied.

2012-02-27 13:22:56,203 DEBUG imgfac.BuildDispatcher.BuildDispatcher thread(e09a773f) Message: Testing provider for JSON: No JSON object could be decoded
2012-02-27 13:22:56,206 DEBUG imgfac.builders.BaseBuilder.RHEL6_rhevm_Builder thread(e09a773f) Message: Image file /var/lib/imagefactory/images/rhevm-image-07c1800d-838f-4094-bc16-5f7babf7b421.dsk already present - skipping warehouse download
2012-02-27 13:22:56,208 DEBUG imgfac.builders.BaseBuilder.RHEL6_rhevm_Builder thread(e09a773f) Message: Produced provider json: 
{
    "apipass": "REDACTED", 
    "apiurl": "https://qeblade26.redhat.com:8443/api", 
    "apiuser": "admin@internal", 
    "cluster": "_any_", 
    "image": "/tmp/e09a773f-e753-4dbb-bc2c-5ba4c98f132a", 
    "name": "rhevm-blade27", 
    "nfsdir": "/mnt/rhevm-blade27", 
    "nfshost": "qeblade26.redhat.com", 
    "nfspath": "/home/blade27_export", 
    "password": "REDACTED", 
    "target": "rhevm", 
    "timeout": 1800, 
    "username": "admin@internal"
}
2012-02-27 13:22:56,209 DEBUG imgfac.builders.BaseBuilder.RHEL6_rhevm_Builder thread(e09a773f) Message: Executing external RHEV-M push command (['/usr/bin/dc-rhev-image', '/tmp/tmpauvhoT'])
2012-02-27 13:22:56,292 DEBUG paste.httpserver.ThreadPool thread(MainThread) Message: Added task (0 tasks queued)
2012-02-27 13:22:57,174 DEBUG imgfac.builders.BaseBuilder.RHEL6_rhevm_Builder thread(e09a773f) Message: Exception caught in ImageFactory
2012-02-27 13:22:57,207 DEBUG imgfac.builders.BaseBuilder.RHEL6_rhevm_Builder thread(e09a773f) Message: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/imgfac/builders/Fedora_rhevm_Builder.py", line 197, in push_image
    self.rhevm_push_image_upload(target_image_id, provider, credentials)
  File "/usr/lib/python2.6/site-packages/imgfac/builders/Fedora_rhevm_Builder.py", line 275, in rhevm_push_image_upload
    (stdout, stderr, retcode) = subprocess_check_output(rhevm_push_command)
  File "/usr/lib/python2.6/site-packages/imgfac/builders/Fedora_rhevm_Builder.py", line 46, in subprocess_check_output
    raise ImageFactoryException("'%s' failed(%d): %s\nstdout: %s" % (cmd, retcode, stderr, stdout))
ImageFactoryException: '/usr/bin/dc-rhev-image /tmp/tmpauvhoT' failed(1): None
stdout: dc-rhev-image: error while opening "/tmp/e09a773f-e753-4dbb-bc2c-5ba4c98f132a" for reading: Permission denied


2012-02-27 13:22:57,207 DEBUG imgfac.BuildJob.BuildJob thread(e09a773f) Message: Builder (e09a773f-e753-4dbb-bc2c-5ba4c98f132a) changed status from PUSHING to FAILED
2012-02-27 13:22:57,208 DEBUG imgfac.BuildJob.BuildJob thread(e09a773f) Message: e09a773f-e753-4dbb-bc2c-5ba4c98f132a for rhevm about to exit None queue...



Version-Release number of selected component (if applicable):
imagefactory-1.0.0rc7-1.el6.noarch


How reproducible: 100%


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

Comment 1 Dave Johnson 2012-02-27 18:30:48 UTC
Wes checked vsphere and it is working...

Nothing has changed in the rhevm environment

Comment 2 Ian McLeod 2012-02-27 23:47:11 UTC
This turns out to have been an unusual side effect of a UMASK change we made to protect the log file.

The mechanism we use for RHEV-M pushes requires that the process copying files to the export domain act as the KVM user and group.  Our umask change made the rhev-m image files unreadable to the push tool.

The whole thing is quite messy and we really should find a more elegant fix.  However, for the moment I have worked around it here:

https://github.com/aeolusproject/imagefactory/commit/f36f40b2b27ca83c675824a9ad5edeafbe2e6873

Which is brewed as rc8.

Comment 3 dgao 2012-02-28 14:55:46 UTC
[root@dell-t7400-01 ~]# aeolus image list --images
aID                                       Name                     Environment     OS         OS Version     Arch       Description                   
------------------------------------     --------------------     -----------     ------     ----------     ------     -------------------------     
bbb25f38-6219-11e1-95e2-001e4fb5bf1d     RHEL6_2 configserver     default         RHEL-6     2              x86_64     RHEL 6.2 w/ Audrey Client     

[root@dell-t7400-01 ~]# aeolus image push --account rhevm --image bbb25f38-6219-11e1-95e2-001e4fb5bf1d
Image                                    Provider Image                           Provider          Account     Status      
------------------------------------     ------------------------------------     -------------     -------     -------     
bbb25f38-6219-11e1-95e2-001e4fb5bf1d     bdbae7c2-bdb1-418f-a662-aa1e7d632e2f     rhevm-blade27     rhevm       PUSHING     

[root@dell-t7400-01 ~]# tail /var/log/imagefactory.log
2012-02-28 09:53:40,648 DEBUG imgfac.builders.BaseBuilder.RHEL6_rhevm_Builder thread(bdbae7c2) Message: Command stderr: (None)
2012-02-28 09:53:40,649 DEBUG imgfac.builders.BaseBuilder.RHEL6_rhevm_Builder thread(bdbae7c2) Message: Extracted RHEVM UUID: b27826db-2086-422d-abdc-4058dc154760 
2012-02-28 09:53:40,654 DEBUG imgfac.ImageWarehouse.ImageWarehouse thread(bdbae7c2) Message: Image Warehouse returned status (500) with message: 
2012-02-28 09:53:40,655 INFO imgfac.ImageWarehouse.ImageWarehouse thread(bdbae7c2) Message: Creating a bucket returned status (500), .
2012-02-28 09:53:40,682 DEBUG imgfac.ImageWarehouse.ImageWarehouse thread(bdbae7c2) Message: Setting metadata ({'provider_account_identifier': 'admin@internal', 'target_image': '2617ae01-06a6-42b1-b5c6-e706d5b2af45', 'uuid': 'bdbae7c2-bdb1-418f-a662-aa1e7d632e2f', 'icicle': 'none', 'target_identifier': 'b27826db-2086-422d-abdc-4058dc154760', 'object_type': 'provider_image', 'provider': 'rhevm-blade27'}) for http://localhost:9090/provider_images/bdbae7c2-bdb1-418f-a662-aa1e7d632e2f
2012-02-28 09:53:40,964 DEBUG imgfac.BuildJob.BuildJob thread(bdbae7c2) Message: Builder (bdbae7c2-bdb1-418f-a662-aa1e7d632e2f) changed percent complete from 0 to 100
2012-02-28 09:53:40,964 DEBUG imgfac.BuildJob.BuildJob thread(bdbae7c2) Message: Builder (bdbae7c2-bdb1-418f-a662-aa1e7d632e2f) changed status from PUSHING to COMPLETED
2012-02-28 09:53:40,964 DEBUG imgfac.BuildJob.BuildJob thread(bdbae7c2) Message: bdbae7c2-bdb1-418f-a662-aa1e7d632e2f for rhevm about to exit None queue...
2012-02-28 09:53:40,965 DEBUG imgfac.ImageWarehouse.ImageWarehouse thread(bdbae7c2) Message: Setting metadata ({'latest_build': '7348e1ff-52a1-4c84-bde9-54fd67877d56'}) for http://localhost:9090/images/bbb25f38-6219-11e1-95e2-001e4fb5bf1d
2012-02-28 09:53:40,966 DEBUG imgfac.ImageWarehouse.ImageWarehouse thread(bdbae7c2) Message: Setting metadata ({'latest_unpushed': None}) for http://localhost:9090/images/bbb25f38-6219-11e1-95e2-001e4fb5bf1d

[root@dell-t7400-01 ~]# rpm -qa | grep "imagefactory"
imagefactory-jeosconf-ec2-rhel-1.0.0rc8-1.el6.noarch
imagefactory-1.0.0rc8-1.el6.noarch
rubygem-imagefactory-console-0.4.0-1.el6.noarch
imagefactory-jeosconf-ec2-fedora-1.0.0rc8-1.el6.noarch