Bug 1042932

Summary:	Exporting a yum repository to an iso fails if an export directory is specified
Product:	[Retired] Pulp	Reporter:	Barnaby Court <bcourt>
Component:	rpm-support	Assignee:	Barnaby Court <bcourt>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Preethi Thomas <pthomas>
Severity:	unspecified	Docs Contact:
Priority:	high
Version:	Master	CC:	pthomas, skarmark
Target Milestone:	---	Keywords:	Triaged
Target Release:	2.4.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-08-09 06:54:56 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Barnaby Court 2013-12-13 16:24:31 UTC

Description of problem:
If an export directory is specified the export of a yum repository to an ISO fails.  

This is because the export_distributor is using the export directory as the working directory instead of using the standard working directory and then moving the results to the export directory after the work is completed.  Because of the directory mismatch the generation of the listings file to be placed inside the iso is failing.

Comment 1 Barnaby Court 2014-02-13 18:31:30 UTC

PR: https://github.com/pulp/pulp_rpm/pull/445
PR: https://github.com/pulp/pulp/pull/797

Comment 2 Jeff Ortel 2014-04-03 13:36:05 UTC

build: 2.4.0-0.7.beta

Comment 3 Preethi Thomas 2014-04-22 16:40:49 UTC

fails-qa
[root@pulp-24-server ~]# rpm -qa pulp-server
pulp-server-2.4.0-0.9.beta.el6.noarch
[root@pulp-24-server ~]# 

[root@pulp-24-server ~]# pulp-admin rpm repo export run --repo-id errata --export-dir /root/export/
+----------------------------------------------------------------------+
                     Publishing Repository [errata]
+----------------------------------------------------------------------+

The following publish configuration options will be used:

Export Dir:  /root/export/


This command may be exited via ctrl+c without affecting the request.



Task Failed

Pulp exception occurred: PulpExecutionException

[root@pulp-24-server ~]# 


Apr 22 13:56:57 pulp-24-server pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._reserve_resource[05fcfca8-d8e3-4341-a9e8-5b21f64c7363]
Apr 22 13:56:57 pulp-24-server pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._reserve_resource[05fcfca8-d8e3-4341-a9e8-5b21f64c7363] succeeded in 0.0136211150093s: u'reserved_resource_worker-0.redhat.com'
Apr 22 13:56:57 pulp-24-server pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[029b21af-ed06-49dd-ae70-22d4252cb4b9]
Apr 22 13:56:57 pulp-24-server pulp: pulp.server.managers.repo.publish:ERROR: Exception caught from plugin during publish for repo [errata]
Apr 22 13:56:57 pulp-24-server pulp: pulp.server.managers.repo.publish:ERROR: Traceback (most recent call last):
Apr 22 13:56:57 pulp-24-server pulp: pulp.server.managers.repo.publish:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/publish.py", line 126, in _do_publish
Apr 22 13:56:57 pulp-24-server pulp: pulp.server.managers.repo.publish:ERROR:     publish_report = publish_repo(transfer_repo, conduit, call_config)
Apr 22 13:56:57 pulp-24-server pulp: pulp.server.managers.repo.publish:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 538, in wrap_f
Apr 22 13:56:57 pulp-24-server pulp: pulp.server.managers.repo.publish:ERROR:     return f(*args, **kwargs)
Apr 22 13:56:57 pulp-24-server pulp: pulp.server.managers.repo.publish:ERROR:   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/distributors/export_distributor/distributor.py", line 130, in publish_repo
Apr 22 13:56:57 pulp-24-server pulp: pulp.server.managers.repo.publish:ERROR:     raise PulpDataException(msg)
Apr 22 13:56:57 pulp-24-server pulp: pulp.server.managers.repo.publish:ERROR: PulpDataException: Value for 'export_dir' is not an existing directory: /root/export/
Apr 22 13:56:57 pulp-24-server pulp: celery.worker.job:ERROR: Task pulp.server.managers.repo.publish.publish[029b21af-ed06-49dd-ae70-22d4252cb4b9] raised unexpected: PulpExecutionException()
Apr 22 13:56:57 pulp-24-server pulp: celery.worker.job:ERROR: Traceback (most recent call last):
Apr 22 13:56:57 pulp-24-server pulp: celery.worker.job:ERROR:   File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 238, in trace_task
Apr 22 13:56:57 pulp-24-server pulp: celery.worker.job:ERROR:     R = retval = fun(*args, **kwargs)
Apr 22 13:56:57 pulp-24-server pulp: celery.worker.job:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 391, in __call__
Apr 22 13:56:57 pulp-24-server pulp: celery.worker.job:ERROR:     return super(Task, self).__call__(*args, **kwargs)
Apr 22 13:56:57 pulp-24-server pulp: celery.worker.job:ERROR:   File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 416, in __protected_call__
Apr 22 13:56:57 pulp-24-server pulp: celery.worker.job:ERROR:     return self.run(*args, **kwargs)
Apr 22 13:56:57 pulp-24-server pulp: celery.worker.job:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/publish.py", line 98, in publish
Apr 22 13:56:57 pulp-24-server pulp: celery.worker.job:ERROR:     transfer_repo, conduit, call_config)
Apr 22 13:56:57 pulp-24-server pulp: celery.worker.job:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/publish.py", line 126, in _do_publish
Apr 22 13:56:57 pulp-24-server pulp: celery.worker.job:ERROR:     publish_report = publish_repo(transfer_repo, conduit, call_config)
Apr 22 13:56:57 pulp-24-server pulp: celery.worker.job:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 538, in wrap_f
Apr 22 13:56:57 pulp-24-server pulp: celery.worker.job:ERROR:     return f(*args, **kwargs)
Apr 22 13:56:57 pulp-24-server pulp: celery.worker.job:ERROR:   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/distributors/export_distributor/distributor.py", line 130, in publish_repo
Apr 22 13:56:57 pulp-24-server pulp: celery.worker.job:ERROR:     raise PulpDataException(msg)
Apr 22 13:56:57 pulp-24-server pulp: celery.worker.job:ERROR: PulpExecutionException: Pulp exception occurred: PulpExecutionException
Apr 22 13:56:57 pulp-24-server pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_release_resource[e4c874ed-dd71-449c-9d4a-e41ee0e34eb1]
Apr 22 13:56:57 pulp-24-server pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[431875c6-cde5-4a1b-ad28-b29fc7241c64]
Apr 22 13:56:57 pulp-24-server pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_release_resource[e4c874ed-dd71-449c-9d4a-e41ee0e34eb1] succeeded in 0.0428729699925s: None
Apr 22 13:56:57 pulp-24-server pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[431875c6-cde5-4a1b-ad28-b29fc7241c64] succeeded in 0.0409467179561s: None

Comment 4 Barnaby Court 2014-05-14 20:29:25 UTC

https://github.com/pulp/pulp/pull/964
https://github.com/pulp/pulp_rpm/pull/486
https://github.com/pulp/pulp_docker/pull/17

Comment 5 Randy Barlow 2014-05-16 20:29:43 UTC

Fixed with 2.4.0-0.16.beta.

Comment 6 Preethi Thomas 2014-06-16 13:33:50 UTC

[root@qe-blade-10 ~]# pulp-admin rpm repo export run --repo-id zoo --export-dir  /tmp/export 
+----------------------------------------------------------------------+
                      Publishing Repository [zoo]
+----------------------------------------------------------------------+

The following publish configuration options will be used:

Export Dir:  /tmp/export


This command may be exited via ctrl+c without affecting the request.



Task Failed

Value for 'export_dir' is not an existing directory: /tmp/export

[root@qe-blade-10 ~]# mkdir /tmp/export
[root@qe-blade-10 ~]# 
[root@qe-blade-10 ~]# 
[root@qe-blade-10 ~]# pulp-admin rpm repo export run --repo-id zoo --export-dir  /tmp/export 
+----------------------------------------------------------------------+
                      Publishing Repository [zoo]
+----------------------------------------------------------------------+

The following publish configuration options will be used:

Export Dir:  /tmp/export


This command may be exited via ctrl+c without affecting the request.



Task Failed

Unable to read & write to specified 'export_dir': /tmp/export

[root@qe-blade-10 ~]# chown apache:apache /tmp/export/
[root@qe-blade-10 ~]# pulp-admin rpm repo export run --repo-id zoo --export-dir  /tmp/export 
+----------------------------------------------------------------------+
                      Publishing Repository [zoo]
+----------------------------------------------------------------------+

The following publish configuration options will be used:

Export Dir:  /tmp/export


This command may be exited via ctrl+c without affecting the request.


Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[==================================================] 100%
32 of 32 items
... completed

Publishing Delta RPMs
[-]
... completed

Publishing Errata
[==================================================] 100%
4 of 4 items
... completed

Publishing Comps file
[==================================================] 100%
3 of 3 items
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Copying files
[-]
... completed

Writing Listings File
[-]
... completed


Task Succeeded


[root@qe-blade-10 ~]# ls -l /tmp/export/
total 8
-rw-r--r--. 1 apache apache    3 Jun 16 09:07 listing
drwxr-xr-x. 3 apache apache 4096 Jun 16 09:07 zoo
[root@qe-blade-10 ~]#

Comment 7 Preethi Thomas 2014-06-16 14:43:50 UTC

Fails-qa
[root@qe-blade-10 ~]# rpm -qa pulp-server
pulp-server-2.4.0-0.20.beta.el7.noarch
[root@qe-blade-10 ~]# 

Looks like the export errors out if the export directory does not exist. 

You could either create the directory if not present or add a documentation that the directory needs to be present.

Comment 8 Barnaby Court 2014-06-20 19:40:33 UTC

PR: https://github.com/pulp/pulp_rpm/pull/517
PR: https://github.com/pulp/pulp_rpm/pull/518

Comment 9 Barnaby Court 2014-06-24 13:50:02 UTC

Code has been updated to attempt creation of the directory if it does not exist.

Comment 10 Randy Barlow 2014-06-25 23:24:16 UTC

Fixed in 2.4.0-0.23.beta.

Comment 11 Preethi Thomas 2014-06-27 12:59:20 UTC

verified

[root@yttrium ~]# rpm -qa pulp-server
pulp-server-2.4.0-0.23.beta.el6.noarch
[root@yttrium ~]# 

[root@yttrium ~]# pulp-admin rpm repo export run --repo-id zoo --export-dir  /tmp/export 
+----------------------------------------------------------------------+
                      Publishing Repository [zoo]
+----------------------------------------------------------------------+

The following publish configuration options will be used:

Export Dir:  /tmp/export


This command may be exited via ctrl+c without affecting the request.


Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[==================================================] 100%
32 of 32 items
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[==================================================] 100%
4 of 4 items
... completed

Publishing Comps file
[==================================================] 100%
3 of 3 items
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Copying files
[-]
... completed

Writing Listings File
[-]
... completed


Task Succeeded


[root@yttrium ~]# ls -l /tmp/export/
total 8
-rw-rw-rw-. 1 apache apache    3 Jun 27 08:58 listing
drwxr-xr-x. 3 apache apache 4096 Jun 27 08:58 zoo
[root@yttrium ~]#

Comment 12 Randy Barlow 2014-08-09 06:54:56 UTC

This has been fixed in Pulp 2.4.0-1.