Bug 1313801 - unable to sync content to capsule
unable to sync content to capsule
Status: CLOSED ERRATA
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Installer (Show other bugs)
6.2.0
Unspecified Unspecified
unspecified Severity high (vote)
: Beta
: --
Assigned To: Justin Sherrill
Kedar Bidarkar
http://projects.theforeman.org/issues...
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-03-02 06:27 EST by Kedar Bidarkar
Modified: 2016-07-27 05:03 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-07-27 05:03:44 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
sync_to_Capsule_fails (621.66 KB, application/x-xz)
2016-03-02 06:27 EST, Kedar Bidarkar
no flags Details
capsule sync status (58.23 KB, image/png)
2016-03-11 11:03 EST, Kedar Bidarkar
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Foreman Issue Tracker 14081 None None None 2016-04-22 11:52 EDT

  None (edit)
Description Kedar Bidarkar 2016-03-02 06:27:09 EST
Created attachment 1132256 [details]
sync_to_Capsule_fails

Description of problem:

Syncing content to capsule fails with the below error.

~]# hammer -u admin -p changeme capsule content synchronize --id 2
Ignoring ruby-libvirt-0.5.2 because its extensions are not built.  Try: gem pristine ruby-libvirt --version 0.5.2
[...................................................................    ] [95%]
RPM1004: Error retrieving metadata: Not found
RPM1004: Error retrieving metadata: Not found


More details available in the foreman_debug.

Version-Release number of selected component (if applicable):
Sat6.2.0-Beta-SNAP1

How reproducible:


Steps to Reproduce:
1. Import manifest and sync Redhat content
2. Sync "redhat content" which is part of life-cycle env to capsule
3.

Actual results:
unable to sync content to capsule 

Expected results:
Should be able to sync content to capsule.

Additional info:

Looks like the custom content get's synced but the RedHat content does not.

rhel7_sattools_repo]# pwd
/var/lib/pulp/published/yum/http/repos/Default_Organization/Dev/rhel72_cv/custom/rhel7_sattools_katello/rhel7_sattools_repo
[abc@xyz rhel7_sattools_repo]# ls | wc -l 
94
[abc@xyz rhel7_sattools_repo]#
Comment 2 John Mitsch 2016-03-03 09:25:29 EST
Connecting redmine issue http://projects.theforeman.org/issues/14040 from this bug
Comment 3 John Mitsch 2016-03-04 10:26:53 EST
This is actually blocked by a m2crypto issue https://gitlab.com/m2crypto/m2crypto/issues/103 that makes the ueber certs used to verify the repositories not valid if they are valid after 2050.
Comment 4 Mike McCune 2016-03-07 13:50:30 EST
We are going to temporarily work around this via this PR:

https://github.com/Katello/puppet-katello/pull/114
Comment 5 Kedar Bidarkar 2016-03-11 11:03 EST
Created attachment 1135273 [details]
capsule sync status

Capsule sync completes 100 %  as seen from the UI, but content actually couldn't sync. 

I don't see any DIR "rpm" under " /var/lib/pulp/content/units/".

 units]# ls ; pwd
distribution  yum_repo_metadata_file
/var/lib/pulp/content/units
Comment 6 Kedar Bidarkar 2016-03-11 11:19:14 EST
Some more info:

The RedHat content files appear to exist but for some reason, they do not. 
I think the softlinks exist but the actual content is missing. 

os]# ll zsh* ; pwd
lrwxrwxrwx. 1 apache apache 125 Mar 11 10:17 zsh-5.0.2-14.el7.x86_64.rpm -> /var/lib/pulp/content/units/rpm/cf/a0c3b4f43ef5e2866e6b8ea2c4133dd05ae26544f6be9b7756b8ed3134a500/zsh-5.0.2-14.el7.x86_64.rpm
lrwxrwxrwx. 1 apache apache 128 Mar 11 10:17 zsh-5.0.2-7.el7_1.1.x86_64.rpm -> /var/lib/pulp/content/units/rpm/ce/917e77d2633632da302055bb47d1e49dd9892a6f2ef28d5017b63d650c4f91/zsh-5.0.2-7.el7_1.1.x86_64.rpm
lrwxrwxrwx. 1 apache apache 128 Mar 11 10:17 zsh-5.0.2-7.el7_1.2.x86_64.rpm -> /var/lib/pulp/content/units/rpm/13/465cd7f1c3463a312a6573d0111a038d2271dedf8466107910e1691a5a6d09/zsh-5.0.2-7.el7_1.2.x86_64.rpm
lrwxrwxrwx. 1 apache apache 124 Mar 11 10:17 zsh-5.0.2-7.el7.x86_64.rpm -> /var/lib/pulp/content/units/rpm/a3/84dcc455923eb535b4c7a5cec7aec946304f8813e56f1655eceb93a9276894/zsh-5.0.2-7.el7.x86_64.rpm
/var/lib/pulp/published/yum/https/repos/Default_Organization/Dev/rhel7_capsule_cv/content/dist/rhel/server/7/7Server/x86_64/os
Comment 7 Bryan Kearney 2016-03-11 12:03:46 EST
Upstream bug assigned to jsherril@redhat.com
Comment 8 Bryan Kearney 2016-03-11 12:03:49 EST
Upstream bug component is Installer
Comment 9 John Mitsch 2016-03-11 12:10:23 EST
(In reply to Kedar Bidarkar from comment #6)
> Some more info:
> 
> The RedHat content files appear to exist but for some reason, they do not. 
> I think the softlinks exist but the actual content is missing. 
> 
> os]# ll zsh* ; pwd
> lrwxrwxrwx. 1 apache apache 125 Mar 11 10:17 zsh-5.0.2-14.el7.x86_64.rpm ->
> /var/lib/pulp/content/units/rpm/cf/
> a0c3b4f43ef5e2866e6b8ea2c4133dd05ae26544f6be9b7756b8ed3134a500/zsh-5.0.2-14.
> el7.x86_64.rpm
> lrwxrwxrwx. 1 apache apache 128 Mar 11 10:17 zsh-5.0.2-7.el7_1.1.x86_64.rpm
> ->
> /var/lib/pulp/content/units/rpm/ce/
> 917e77d2633632da302055bb47d1e49dd9892a6f2ef28d5017b63d650c4f91/zsh-5.0.2-7.
> el7_1.1.x86_64.rpm
> lrwxrwxrwx. 1 apache apache 128 Mar 11 10:17 zsh-5.0.2-7.el7_1.2.x86_64.rpm
> ->
> /var/lib/pulp/content/units/rpm/13/
> 465cd7f1c3463a312a6573d0111a038d2271dedf8466107910e1691a5a6d09/zsh-5.0.2-7.
> el7_1.2.x86_64.rpm
> lrwxrwxrwx. 1 apache apache 124 Mar 11 10:17 zsh-5.0.2-7.el7.x86_64.rpm ->
> /var/lib/pulp/content/units/rpm/a3/
> 84dcc455923eb535b4c7a5cec7aec946304f8813e56f1655eceb93a9276894/zsh-5.0.2-7.
> el7.x86_64.rpm
> /var/lib/pulp/published/yum/https/repos/Default_Organization/Dev/
> rhel7_capsule_cv/content/dist/rhel/server/7/7Server/x86_64/os

What you are seeing is typical of the new default download policy "on_demand" introduced by the lazy sync feature
Comment 10 Kedar Bidarkar 2016-03-16 14:10:43 EDT
Moving this to ON_QA as per comment9
Comment 11 Kedar Bidarkar 2016-03-21 06:39:25 EDT
VERIFIED With SAT62-SNAP4-compose1

We can now sync content to capsule.
Comment 13 Kedar Bidarkar 2016-04-12 01:15:35 EDT
Syncing Capsule fails, turns red at 98% 

 with the below error as per /v/l/m from capsule.


Apr 12 01:02:20 katello1 pulp: pulp_rpm.plugins.importers.yum.parse.treeinfo:ERROR: some distro file downloads failed
Apr 12 01:02:21 katello1 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (4010-66624)
Apr 12 01:02:21 katello1 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (4010-66624) Traceback (most recent call last):
Apr 12 01:02:21 katello1 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (4010-66624)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 228, in run
Apr 12 01:02:21 katello1 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (4010-66624)     dist_sync.run()
Apr 12 01:02:21 katello1 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (4010-66624)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/parse/treeinfo.py", line 125, in run
Apr 12 01:02:21 katello1 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (4010-66624)     self._run(tmp_dir)
Apr 12 01:02:21 katello1 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (4010-66624)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/parse/treeinfo.py", line 189, in _run
Apr 12 01:02:21 katello1 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (4010-66624)     for destination, location in downloaded:
Apr 12 01:02:21 katello1 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (4010-66624)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/parse/treeinfo.py", line 274, in download_files
Apr 12 01:02:21 katello1 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (4010-66624)     raise DownloadFailed()
Apr 12 01:02:21 katello1 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (4010-66624) DownloadFailed
Apr 12 01:02:21 katello1 pulp: py.warnings:WARNING: (4010-66624) /usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py:804: DeprecationWarning: save is deprecated. Use insert_one or replace_one instead
Apr 12 01:02:21 katello1 pulp: py.warnings:WARNING: (4010-66624)   sync_result_collection.save(sync_result)
Apr 12 01:02:21 katello1 pulp: py.warnings:WARNING: (4010-66624)
Apr 12 01:02:21 katello1 pulp: pulp.server.async.tasks:INFO: Task failed : [303fa070-5a9b-4835-8eed-f52185647efc]
Apr 12 01:02:21 katello1 pulp: celery.worker.job:ERROR: (3693-66624) Task pulp.server.managers.repo.sync.sync[303fa070-5a9b-4835-8eed-f52185647efc] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Apr 12 01:02:21 katello1 pulp: celery.worker.job:ERROR: (3693-66624) Traceback (most recent call last):
Apr 12 01:02:21 katello1 pulp: celery.worker.job:ERROR: (3693-66624)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Apr 12 01:02:21 katello1 pulp: celery.worker.job:ERROR: (3693-66624)     R = retval = fun(*args, **kwargs)
Apr 12 01:02:21 katello1 pulp: celery.worker.job:ERROR: (3693-66624)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 473, in __call__
Apr 12 01:02:21 katello1 pulp: celery.worker.job:ERROR: (3693-66624)     return super(Task, self).__call__(*args, **kwargs)
Apr 12 01:02:21 katello1 pulp: celery.worker.job:ERROR: (3693-66624)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Apr 12 01:02:21 katello1 pulp: celery.worker.job:ERROR: (3693-66624)     return super(PulpTask, self).__call__(*args, **kwargs)
Apr 12 01:02:21 katello1 pulp: celery.worker.job:ERROR: (3693-66624)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Apr 12 01:02:21 katello1 pulp: celery.worker.job:ERROR: (3693-66624)     return self.run(*args, **kwargs)
Apr 12 01:02:21 katello1 pulp: celery.worker.job:ERROR: (3693-66624)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 810, in sync
Apr 12 01:02:21 katello1 pulp: celery.worker.job:ERROR: (3693-66624)     raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))
Apr 12 01:02:21 katello1 pulp: celery.worker.job:ERROR: (3693-66624) PulpExecutionException: Importer indicated a failed response
Apr 12 01:02:21 katello1 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[f1d12da3-980f-4be4-bbcc-53774234b209] succeeded in 0.00668240699451s: None
Comment 14 Kedar Bidarkar 2016-04-12 01:57:55 EDT
Apr 12 01:02:20 katello1 pulp: pulp_rpm.plugins.importers.yum.parse.treeinfo:ERROR: some distro file downloads failed
----------------------------------------------------------------------

Tried provisioning both RHEL7 and RHEL6 repos, 

RHEL7:  provisioning was successful ( boot files could be fetched from isolated capsule by the HOST).

RHEL6 : It was unable to fetch the boot files { vmlinuz and initrd )

May be the above "ERROR: some distro file downloads failed" is being referred to RHEL6 distro files.
Comment 15 Justin Sherrill 2016-04-12 08:49:58 EDT
Kedar, 

Can we open a new bz for this since this is an entirely different issue (in the same area)?  

Also please attach a sosreport from the satellite and capsule to that bz.

Thanks,

-Justin
Comment 16 Kedar Bidarkar 2016-04-12 09:28:18 EDT
ok, will be closing this and raising a new bug with foreman-debug.
Comment 18 errata-xmlrpc 2016-07-27 05:03:44 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1500

Note You need to log in before you can comment on or make changes to this bug.