Bug 1021672 - RHEL 5 repository metadata incorrect for anaconda
RHEL 5 repository metadata incorrect for anaconda
Status: CLOSED CURRENTRELEASE
Product: Pulp
Classification: Community
Component: rpm-support (Show other bugs)
Master
Unspecified Unspecified
urgent Severity high
: ---
: 2.3.0
Assigned To: Barnaby Court
Preethi Thomas
: Triaged
Depends On:
Blocks: 950743 1021674
  Show dependency treegraph
 
Reported: 2013-10-21 14:51 EDT by Justin Sherrill
Modified: 2013-12-09 09:30 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1021674 (view as bug list)
Environment:
Last Closed: 2013-12-09 09:30:11 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Justin Sherrill 2013-10-21 14:51:32 EDT
Description of problem:

Metadata for published RHEL 5 repositories are not structured in a way that is usable by anaconda.

Similar to this bugzilla (but slightly different): https://bugzilla.redhat.com/show_bug.cgi?id=805925

Pulp generates:

./repodata/
and
./Server/repodata/

and puts the rpms in ./

The metadata in ./repodata is correct, but the metadta in ./Server/repodata is not, it makes it appear as if the rpms are actually stored in ./Server:


eg:

  <location href="gnome-python2-gtkmozembed-2.14.2-7.el5.x86_64.rpm"/>


Version-Release number of selected component (if applicable):

pulp-server-2.3.0-0.22.beta.el6sat.noarch


How reproducible:
Always

Steps to Reproduce:  
1. Sync a RHEL 5 kickstart tree from the cdn
2.  Try to install/kickstart a machine against it


Actual results:
Failure, system can't find packages

Expected results:
RHEL 5 system can be provisioned.
Comment 1 Michael Hrivnak 2013-10-21 15:13:17 EDT
This looks similar to an issue we've seen before, which resulted in us adding a "Packages" symlink in our repo. Jconnor did that work and can hopefully shed some light on this issue as well.
Comment 2 Barnaby Court 2013-10-22 15:16:58 EDT
Justin, Can you provide a URL for the repo on the CDN that you are running into trouble syncing?
Comment 4 Barnaby Court 2013-10-23 13:42:36 EDT
PR: https://github.com/pulp/pulp_rpm/pull/361

This fix requires a sync to be performed on the affected repo after it has been applied.  After a sync the repo will publish a link to all of the files to the directory specified in [general]->packagedir subdirectory specified in the treeinfo file.
Comment 5 Mike McCune 2013-10-23 16:22:33 EDT
pulp bug is MODIFIED, reflecting accordingly
Comment 6 Mike McCune 2013-10-23 16:23:19 EDT
ignore previous comment, wrong BZ
Comment 7 Jeff Ortel 2013-10-23 19:27:07 EDT
build: 2.3.0-0.24.beta
Comment 8 Preethi Thomas 2013-10-24 15:30:30 EDT
fails-qa
Looks like repo publish is failing



[root@ibm-x3550m3-11 ~]#  pulp-admin rpm repo create --repo-id rhel59-os --feed https://cdn.redhat.com/content/dist/rhel/rhui/server/5/5.9/x86_64/os/ --feed-ca-cert CDN/cdn.redhat.com-chain.crt --feed-cert CDN/1359391926_4512.crt  --feed-key CDN/1359391926_4512.key 
Successfully created repository [rhel59-os]


On the first attempt


[root@ibm-x3550m3-11 ~]#  pulp-admin rpm repo sync run --repo-id rhel59-os
+----------------------------------------------------------------------+
                  Synchronizing Repository [rhel59-os]
+----------------------------------------------------------------------+

This command may be exited by pressing ctrl+c without affecting the actual
operation on the server.

Downloading metadata...
[/]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       15362/15362 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 19/19 items
... completed

Importing errata...
[|]
... completed

Importing package groups/categories...
[\]
... completed

Publishing distributions...
[==================================================] 100%
Distributions: 19/19 items
... completed

Publishing packages...
[==================================================] 100%
Packages: 15362/15362 items
Individual errors encountered during publishing:

An unexpected error has occurred. More information can be found in the client
log file ~/.pulp/admin.log.


On the 2nd attempt


[root@ibm-x3550m3-11 ~]#  pulp-admin rpm repo sync run --repo-id rhel59-os
+----------------------------------------------------------------------+
                  Synchronizing Repository [rhel59-os]
+----------------------------------------------------------------------+

This command may be exited by pressing ctrl+c without affecting the actual
operation on the server.

Downloading metadata...
[|]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       0/0 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 19/19 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[\]
... completed

Publishing distributions...
[==================================================] 100%
Distributions: 0/0 items
[root@ibm-x3550m3-11 ~]#  pulp-admin rpm repo publish run --repo-id rhel59-os
+----------------------------------------------------------------------+
                   Publishing Repository [rhel59-os]
+----------------------------------------------------------------------+

This command may be exited by pressing ctrl+c without affecting the actual
operation on the server.

Publishing distributions...
[==================================================] 100%
Distributions: 0/0 items
[root@ibm-x3550m3-11 ~]# 


From pulp.log on publish

2013-10-24 14:58:58,440 pulp.server.managers.repo.publish:ERROR: Exception caught from plugin during publish for repo [rhel59-os]
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/publish.py", line 121, in _do_publish
    publish_report = distributor_instance.publish_repo(transfer_repo, conduit, call_config)
  File "/usr/lib/pulp/plugins/distributors/yum_distributor/distributor.py", line 328, in publish_repo
    distro_status, distro_errors = self.symlink_distribution_unit_files(distro_units, repo.working_dir, publish_conduit, progress_callback)
  File "/usr/lib/pulp/plugins/distributors/yum_distributor/distributor.py", line 609, in symlink_distribution_unit_files
    os.makedirs(package_path)
  File "/usr/lib64/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 17] File exists: '/var/lib/pulp/working/repos/rhel59-os/distributors/yum_distributor/Server'
2013-10-24 14:58:58,440 pulp.server.dispatch.task:ERROR: Pulp exception occurred: PulpExecutionException
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/dispatch/task.py", line 138, in _run
    result = call(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/publish.py", line 97, in publish
    result = self._do_publish(repo, distributor_id, distributor_instance, transfer_repo, conduit, call_config)
  File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/publish.py", line 121, in _do_publish
    publish_report = distributor_instance.publish_repo(transfer_repo, conduit, call_config)
  File "/usr/lib/pulp/plugins/distributors/yum_distributor/distributor.py", line 328, in publish_repo
    distro_status, distro_errors = self.symlink_distribution_unit_files(distro_units, repo.working_dir, publish_conduit, progress_callback)
  File "/usr/lib/pulp/plugins/distributors/yum_distributor/distributor.py", line 609, in symlink_distribution_unit_files
    os.makedirs(package_path)
  File "/usr/lib64/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
PulpExecutionException: Pulp exception occurred: PulpExecutionException
2013-10-24 14:58:58,441 pulp.server.dispatch.task:INFO: FAILURE: Task 843233ad-5ed1-4f16-9dd2-c9e63cc5f55e: CallRequest: RepoPublishManager.publish(u'rhel59-os', 'yum_distributor', publish_config_override=None)
Comment 9 Barnaby Court 2013-10-24 16:40:00 EDT
PR: https://github.com/pulp/pulp_rpm/pull/362
Comment 10 Jeff Ortel 2013-10-29 11:29:40 EDT
build: 2.3.0-0.26.beta
Comment 11 Preethi Thomas 2013-10-29 13:27:35 EDT
Resynced a rhel5 repo

kickstated a vm with that

 virt-install --vnc --name=rhel5-ks --ram=1024 --vcpu=4 --file=/var/lib/libvirt/images/rhel5-ks --file-size=14 --vnc --location=http://ibm-x3550m3-11.lab.eng.brq.redhat.com/pulp/repos/content/dist/rhel/rhui/server/5/5.9/x86_64/os/ --prompt
Comment 12 Preethi Thomas 2013-10-30 08:18:56 EDT
[root@dell-pe2900-01 ~]# rpm -qa pulp-server
pulp-server-2.3.0-0.26.beta.el6.noarch
[root@dell-pe2900-01 ~]# 


root@dell-pe2900-01 ~]# pulp-admin rpm repo create --repo-id rhel59-os --feed https://cdn.redhat.com/content/dist/rhel/rhui/server/5/5.9/x86_64/os/ --feed-ca-cert CDN/cdn.redhat.com-chain.crt --feed-cert CDN/1359391926_4512.crt  --feed-key CDN/1359391926_4512.key  --proxy-host http://pulp-proxy.usersys.redhat.com --proxy-port 8888 --proxy-user admin --proxy-pass admin 
Successfully created repository [rhel59-os]
[root@dell-pe2900-01 ~]#  pulp-admin rpm repo sync run --repo-id rhel59-os
+----------------------------------------------------------------------+
                  Synchronizing Repository [rhel59-os]
+----------------------------------------------------------------------+

A sync task is already in progress for this repository. Its progress will be
tracked below.

This command may be exited by pressing ctrl+c without affecting the actual
operation on the server.

Downloading metadata...
[-]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       15362/15362 items
Delta RPMs: 0/0 items

... completed

Individual package errors encountered during sync:

An unexpected error has occurred. More information can be found in the client
log file ~/.pulp/admin.log.

[root@dell-pe2900-01 ~]#  pulp-admin rpm repo sync run --repo-id rhel59-os
+----------------------------------------------------------------------+
                  Synchronizing Repository [rhel59-os]
+----------------------------------------------------------------------+

This command may be exited by pressing ctrl+c without affecting the actual
operation on the server.

Downloading metadata...
[\]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       8/8 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 19/19 items
... completed

Importing errata...
[/]
... completed

Importing package groups/categories...
[-]
... completed

Publishing distributions...
[==================================================] 100%
Distributions: 19/19 items
... completed

Publishing packages...
[==================================================] 100%
Packages: 15362/15362 items
... completed

Generating metadata
[|]
... completed

Publishing repository over HTTPS
[-]
... completed

Publishing repository over HTTP
[-]
... completed


Kickstarted a vm

[root@qe-blade-15 ~]#  virt-install --vnc --name=rhel5-ks-new --ram=1024 --vcpu=4 --file=/var/lib/libvirt/images/rhel5-ks-new --file-size=14 --vnc --location=http://dell-pe2900-01.rhts.eng.bos.redhat.com/pulp/repos/content/dist/rhel/rhui/server/5/5.9/x86_64/os/ --prompt
Would you like to use KVM acceleration? (yes or no) yes
Comment 13 Preethi Thomas 2013-12-09 09:30:11 EST
Pulp 2.3 released.

Note You need to log in before you can comment on or make changes to this bug.