Bug 1303103

Summary: [RFE] Allow ISO repositories to be added to a content view and published/distributed
Product: Red Hat Satellite Reporter: Brad Buckingham <bbuckingham>
Component: Content ViewsAssignee: Eric Helms <ehelms>
Status: CLOSED ERRATA QA Contact: Sachin Ghai <sghai>
Severity: high Docs Contact:
Priority: medium    
Version: 6.1.6CC: ahumbe, bbuckingham, bkearney, dcaplan, egolov, ehelms, hmore, hshukla, kabbott, mhrivnak, mmccune, pm-sat, sghai, stbenjam, sthirugn, xdmoon
Target Milestone: UnspecifiedKeywords: FutureFeature, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
URL: http://projects.theforeman.org/issues/13661
Whiteboard:
Fixed In Version: katello-installer-base-3.4.5.26-1,tfm-rubygem-runcible-1.12.0.3-1,tfm-rubygem-katello-3.4.5.58-1 Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-02-21 12:33:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1321771    
Attachments:
Description Flags
UI shows 49 files in CV
none
updated CV with RHEL 7Server iso and it includes 49 files. none

Description Brad Buckingham 2016-01-29 14:56:01 UTC 
Description of problem:

Today, when an ISO repository is synced to a Satellite 6 server, the content of the repository is not distributed to Capsules that have requested updates for Library.

The lack of this capability causes issues for users with systems that only have access to the capsule, but also need access to ISOs.

Version-Release number of selected component (if applicable):

Satellite 6.1.6

How reproducible:

Always

Steps to Reproduce:
1. Install Satellite and 1 Capsule
2. Configure the Capsule to get content from Library
3. Enable a Red Hat ISO repository (e.g Red Hat Enterprise Linux 7 Server (ISOs))
4. Sync the repository
5. Create a content view
6. Attempt to add an ISO repository to the content view

Actual results:

ISO repositories are not listed for adding/removing to content views.

Expected results:

ISO repositories should be listed for adding/removing to content views.

Additional info:

As part of this, we need to ensure that those repositories are also published and promoted as part of the content view.  In addition, if there is a capsule associated with the target lifecycle environment, the content should be synced to that capsule.
Comment 2 Michael Hrivnak 2016-01-31 18:29:44 UTC 
Nope, I think everything you need is already available. But an early review from someone on katello would be a good idea to make sure they agree.
Comment 6 Bryan Kearney 2016-03-30 16:05:51 UTC 
Upstream bug component is Content Views
Comment 10 Bryan Kearney 2016-08-08 02:11:37 UTC 
Moving to POST since upstream bug http://projects.theforeman.org/issues/13661 has been closed
Comment 13 Sachin Ghai 2017-12-08 12:06:21 UTC 
Tested with satellite6.3 snap27.

I can add synced RHEL ISOs to CV and publish as well as promote them to next environment. Please see the attached screenshots
Comment 14 Sachin Ghai 2017-12-08 12:06:55 UTC 
Created attachment 1364812 [details]
UI shows 49 files in CV
Comment 15 Sachin Ghai 2017-12-08 12:07:47 UTC 
Created attachment 1364813 [details]
updated CV with RHEL 7Server iso and it includes 49 files.
Comment 16 Sachin Ghai 2017-12-08 13:55:01 UTC 
ISO files are not being synced to capsule. Earlier, I selected the on_demand download policy for capsule and later I changed to "Immediate" but still no luck.

There are 49 ISO files that I synced and published. However, none of them was synced to capsule. 

@Eric: Is there anything specific I need to do to sync ISO files to capsule ?
Comment 17 Evgeni Golov 2017-12-08 14:17:28 UTC 
As Eric is out today, I had a quick look at the setup.

First observation was that Satellite thinks the Capsule never synced, which in a way is true, as each Sync Job has failed.

example of a failed job: https://cloud-qe-14.idmqe.lab.eng.bos.redhat.com/foreman_tasks/tasks/0b399f7c-5395-4288-9665-579f73c14c32
PLP0000: Importer indicated a failed response

looking into dynflow: https://cloud-qe-14.idmqe.lab.eng.bos.redhat.com/foreman_tasks/dynflow/f6a45e70-ac19-44f8-b65e-09bcca24bb7e
"Actions::Pulp::Consumer::SyncCapsule" is skipped as it contains a traceback:

input:
---
capsule_id: 2
repo_pulp_id: 1-cv_rhel73-Dev-f0a46c4b-9b33-4d2b-a126-eda4a8bb5fd2
sync_options:
  remove_missing: false
remote_user: admin
remote_cp_user: admin

output:
---
pulp_tasks:
- exception: 
  task_type: pulp.server.managers.repo.sync.sync
  _href: "/pulp/api/v2/tasks/0cbc8638-54d9-435c-a32f-fa8e30ebdfae/"
  task_id: 0cbc8638-54d9-435c-a32f-fa8e30ebdfae
  tags:
  - pulp:repository:1-cv_rhel73-Dev-f0a46c4b-9b33-4d2b-a126-eda4a8bb5fd2
  - pulp:action:sync
  finish_time: '2017-12-08T12:10:16Z'
  _ns: task_status
  start_time: '2017-12-08T12:10:15Z'
  traceback: |
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
        R = retval = fun(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 527, in __call__
        return super(Task, self).__call__(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__
        return super(PulpTask, self).__call__(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
        return self.run(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 827, in sync
        raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))
    PulpExecutionException: Importer indicated a failed response
  spawned_tasks: []
  progress_report:
    iso_importer:
      error_message:
        response_code: 403
        response_msg: Forbidden
      traceback: 
      finished_bytes: 0
      num_isos: 
      state: manifest_failed
      total_bytes: 
      state_times:
        not_started: '2017-12-08T12:10:15'
        manifest_in_progress: '2017-12-08T12:10:15'
        manifest_failed: '2017-12-08T12:10:15'
      num_isos_finished: 0
      iso_error_messages: []
  queue: reserved_resource_worker-0.lab.eng.bos.redhat.com.dq
  state: error
  worker_name: reserved_resource_worker-0.lab.eng.bos.redhat.com
  result: 
  error:
    code: PLP0000
    data: {}
    description: Importer indicated a failed response
    sub_errors: []
  _id:
    "$oid": 5a2a8127266e6d3221798d1c
  id: 5a2a8127266e6d3221798d1c
poll_attempts:
  total: 1
  failed: 1

the interesting part of that is:
    iso_importer:
      error_message:
        response_code: 403
        response_msg: Forbidden
      state: manifest_failed

the systemd journal on the capsule contains (as expected):
Dec 08 05:57:39 cloud-qe-06.idmqe.lab.eng.bos.redhat.com pulp[1834]: nectar.downloaders.threaded:INFO: Download failed: Download of https://cloud-qe-14.idmqe.lab.eng.bos.redhat.com/pulp/isos/1-cv_rhel73-Dev-f0a46c4b-9b33-4d2b-a126-eda4a8bb5fd2/PULP_MANIFEST failed with code 403: Forbidden
Dec 08 05:57:39 cloud-qe-06.idmqe.lab.eng.bos.redhat.com pulp[1834]: pulp_rpm.plugins.importers.iso.sync:ERROR: Failed to download https://cloud-qe-14.idmqe.lab.eng.bos.redhat.com/pulp/isos/1-cv_rhel73-Dev-f0a46c4b-9b33-4d2b-a126-eda4a8bb5fd2/PULP_MANIFEST: Forbidden.

and on the satellite the httpd log contains:
[Fri Dec 08 05:57:39.876851 2017] [:error] [pid 29888] [client 10.19.34.35:37008] Request denied to destination [/pulp/isos/1-cv_rhel73-Dev-f0a46c4b-9b33-4d2b-a126-eda4a8bb5fd2/PULP_MANIFEST]Client certificate failed extension check for destination: /pulp/isos/1-cv_rhel73-Dev-f0a46c4b-9b33-4d2b-a126-eda4a8bb5fd2/PULP_MANIFEST
[Fri Dec 08 05:57:39.876874 2017] [:error] [pid 29888] [client 10.19.34.35:37008] mod_wsgi (pid=29888): Client denied by server configuration: '/var/www/pub/https/isos/1-cv_rhel73-Dev-f0a46c4b-9b33-4d2b-a126-eda4a8bb5fd2/PULP_MANIFEST'.
[Fri Dec 08 06:27:22.452082 2017] [:error] [pid 10376] [client 10.19.34.35:37128] Request denied to destination [/pulp/isos/1-cv_rhel73-Dev-f0a46c4b-9b33-4d2b-a126-eda4a8bb5fd2/PULP_MANIFEST]Client certificate failed extension check for destination: /pulp/isos/1-cv_rhel73-Dev-f0a46c4b-9b33-4d2b-a126-eda4a8bb5fd2/PULP_MANIFEST

I am not sure what is happening here, so leaving the NI on Eric. Hope my debugging helps :)
Comment 18 Eric Helms 2017-12-21 15:32:58 UTC 
From my investigation, I tested the certs the importer is configured with and was able to curl that Forbidden file with them just fine. So I am not currently sure what is happening when the actual sync operation occurs and will have to dig further.
Comment 20 Eric Helms 2018-01-25 16:39:45 UTC 
*** Bug 1480358 has been marked as a duplicate of this bug. ***
Comment 21 Stephen Benjamin 2018-01-25 17:53:40 UTC 
Testing on upstream, but get the same results.  I don't think the certificate the capsule is getting is configured correctly. Using the certificate in the database, I am not able to retrieve a file from the repo:


[root@centos7-foreman-proxy-3-5 ~]# curl --cert /root/cert.crt --key /root/key.crt https://centos7-katello-3-5.zpm.example.com/pulp/isos/901986be-6f88-4903-930a-2b90dc9bea96/PULP_MANIFEST
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /pulp/isos/901986be-6f88-4903-930a-2b90dc9bea96/PULP_MANIFEST
on this server.</p>
</body></html>


HTTP logs show:

[Thu Jan 25 17:43:02.353200 2018] [:error] [pid 11693] [client 192.168.121.130:40360] Request denied to destination [/pulp/isos/901986be-6f88-4903-930a-2b90dc9bea96/PULP_MANIFEST]Client certificate failed extension check for destination: /pulp/isos/901986be-6f88-4903-930a-2b90dc9bea96/PULP_MANIFEST

Pulp's OID validator seems to be doing some kind of check that determines if the certificate has access to /pulp/isos: /usr/lib/python2.7/site-packages/pulp/oid_validation/oid_validation.py

I'm not sure how to decode the extensions exactly:

[root@centos7-foreman-proxy-3-5 ~]# openssl x509 -in cert.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 2032182190564053392 (0x1c33c2ef73721190)
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, ST=North Carolina, L=Raleigh, O=Katello, OU=SomeOrgUnit, CN=centos7-katello-3-5.zpm.example.com
        Validity
            Not Before: Jan 24 15:32:10 2018 GMT
            Not After : Dec  1 13:00:00 2049 GMT
        Subject: O=Default_Organization
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:80:79:11:9a:71:47:a9:93:fb:be:e1:93:d9:cf:
                    8c:5b:26:a5:f2:59:87:bb:11:ea:a8:75:8a:d9:03:
                    6d:3b:0d:b5:64:1b:c6:cc:5b:b8:8d:dd:2d:67:7d:
                    61:0c:58:b8:64:4b:c2:4d:13:59:3c:fe:36:aa:b4:
                    62:e3:99:3f:19:40:d9:a2:38:84:8c:58:3d:e8:cf:
                    41:ba:8c:d2:33:64:54:8a:45:87:57:66:82:84:56:
                    25:7b:dd:9a:08:eb:1b:b7:59:71:5a:c4:45:b4:8b:
                    91:dc:97:2a:c8:c3:2e:94:03:58:a0:54:bd:1b:38:
                    a5:7c:70:29:a2:20:92:ca:d5:f3:3c:81:91:3c:f6:
                    5c:78:f2:28:fd:0c:63:e7:eb:7b:6f:91:fb:9e:68:
                    14:4d:7c:94:b6:f3:10:09:17:cd:42:b4:83:27:9d:
                    ea:7f:89:30:ac:c0:69:01:96:a2:bc:47:68:9c:6f:
                    fb:ad:a5:4b:ec:39:0b:a3:21:1a:93:11:13:67:f4:
                    2a:6c:b6:84:69:12:07:fc:a0:63:2d:c7:53:bc:bb:
                    aa:2e:2d:4b:55:6d:f1:2e:2c:b2:d4:8b:c4:11:92:
                    19:22:c1:92:3a:85:64:65:08:a2:bd:dd:1f:bd:71:
                    a6:c6:b5:b5:23:9c:f6:b0:ce:71:3f:e4:bc:91:aa:
                    fd:6f
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            Netscape Cert Type: 
                SSL Client, S/MIME
            X509v3 Key Usage: 
                Digital Signature, Key Encipherment, Data Encipherment
            X509v3 Authority Key Identifier: 
                keyid:8B:ED:73:F8:53:4C:76:B2:30:AB:D7:55:EA:85:FC:D3:5E:A7:D9:85
                DirName:/C=US/ST=North Carolina/L=Raleigh/O=Katello/OU=SomeOrgUnit/CN=centos7-katello-3-5.zpm.example.com
                serial:C0:6B:3D:41:B0:98:76:EF

            X509v3 Subject Key Identifier: 
                C9:61:8F:1F:07:99:71:2A:1E:3D:63:C1:D7:64:3A:CE:A7:1F:54:10
            X509v3 Extended Key Usage: 
                TLS Web Client Authentication
            1.3.6.1.4.1.2312.9.1.1516807930959.1: 
                ."Default_Organization_ueber_product
            1.3.6.1.4.1.2312.9.1.1516807930959.3: 
                ..
            1.3.6.1.4.1.2312.9.1.1516807930959.2: 
                ..
            1.3.6.1.4.1.2312.9.1.1516807930959.5: 
                ..
            1.3.6.1.4.1.2312.9.2.1516807930960.1: 
                ..yum
            1.3.6.1.4.1.2312.9.2.1516807930960.1.1: 
ueber_content   .
            1.3.6.1.4.1.2312.9.2.1516807930960.1.2: 
                ..1516807930959_ueber_content
            1.3.6.1.4.1.2312.9.2.1516807930960.1.5: 
                ..Custom
            1.3.6.1.4.1.2312.9.2.1516807930960.1.6: 
                ../Default_Organization
            1.3.6.1.4.1.2312.9.2.1516807930960.1.7: 
                ..
            1.3.6.1.4.1.2312.9.2.1516807930960.1.8: 
                ..1
            1.3.6.1.4.1.2312.9.4.1: 
                ."Default_Organization_ueber_product
            1.3.6.1.4.1.2312.9.4.2: 
                ..
            1.3.6.1.4.1.2312.9.4.3: 
1516807930959   .
            1.3.6.1.4.1.2312.9.4.5: 
                ..1
            1.3.6.1.4.1.2312.9.4.6: 
                ..2018-01-24T15:32:10Z
            1.3.6.1.4.1.2312.9.4.7: 
                ..2049-12-01T13:00:00Z
            1.3.6.1.4.1.2312.9.4.12: 
                ..0
            1.3.6.1.4.1.2312.9.4.10: 
                ..
            1.3.6.1.4.1.2312.9.4.13: 
                ..
            1.3.6.1.4.1.2312.9.4.14: 
                ..0
            1.3.6.1.4.1.2312.9.4.11: 
                ..1
            1.3.6.1.4.1.2312.9.5.1: 
                .$3c1a6371-75de-4a49-9b27-9fa5be87b682



But rhsm seems to confirm the certificate has no rights to this file:

>>> from rhsm import certificate
>>> cert = certificate.create_from_file("/root/cert.crt")
>>> cert.check_path("/pulp/isos/901986be-6f88-4903-930a-2b90dc9bea96/PULP_MANIFEST")
False
Comment 23 Satellite Program 2018-01-25 19:28:24 UTC 
Upstream bug assigned to ehelms
Comment 24 Satellite Program 2018-01-26 01:07:37 UTC 
Upstream bug assigned to ehelms
Comment 25 Stephen Benjamin 2018-01-29 15:47:34 UTC 
Created redmine issue http://projects.theforeman.org/issues/22446 from this bug
Comment 29 Sachin Ghai 2018-02-04 10:33:56 UTC 
Verified with sat6.3 snap35

[root@cloud-qe-17 ~]# hammer -u admin -p changeme capsule content synchronize --id=2
[.......................                                                                                                                                                                                    ] [11%]
[..........................................................................................................................................................................................................] [100%]
[root@cloud-qe-17 ~]# 

I can sync iso repos to capsule by publishing through CV.

synced isos from capsule.

/var/lib/pulp/content/units/iso/6c/93dd9c54d7580b8d48afdc62c0f0ea2273fa523a8495224eccf8a94264d6b4/rhel-workstation-7.3-x86_64-boot.iso
/var/lib/pulp/content/units/iso/a7/df1dc22e4fdc96e083767f4a6c56f80a4da540bb50a58b0de8f2eccacd2064/rhel-server-7.4-x86_64-dvd.iso
/var/lib/pulp/content/units/iso/3b/66a35c07701ed31212ae6000027f70a3b9b49ab865b3dbad979c357e588113/rhel-server-7.1-x86_64-boot.iso
/var/lib/pulp/content/units/iso/4a/3068d44fccb0afda7d199f739ec43c12e9ee0e959a733a381071f601f8ea77/dd-igb-5.2.15_k-1-x86_64.iso
/var/lib/pulp/content/units/iso/2e/04538008082952b11def17dec6f78e44d8a55e0e4a9afd365dee1a8df41272/rhel-workstation-7.2-x86_64-boot.iso
/var/lib/pulp/content/units/iso/32/fd023380edee37e6d040ebc1544eb6a6a720f58495e3debcc45d18ddf617a9/rhel-server-7.2-x86_64-boot.iso
/var/lib/pulp/content/units/iso/ff/8811fa38bdd32f9577f7a89d69f8789e020069890820beb5814ec8bba80077/dd-lpfc-10.2.8021.0-x86_64.iso
/var/lib/pulp/content/units/iso/ff/8fa10a5717370b67a4e965a006386b47adaffb6c167b52a0b23b6df798780f/virt-p2v-1.32.7-2.el7.iso
/var/lib/pulp/content/units/iso/09/8e1a9742de8625e8302a6d5a4dba886219c08877ef818324489706cc376f91/dd-sfc.el7_2.iso
/var/lib/pulp/content/units/iso/6b/9b66bbd3f4065b10c0442e01c442eee047bab463ab5d62760991c6d1566693/dd-ixgbe-4.4.0_k_rh7.4_z-2-ixgbevf-3.2.2_k_rh7.4_z-1.el7_3.iso
/var/lib/pulp/content/units/iso/6b/f6a70bed83a93a123edbc32cd1c49f42b457293e78eaf168905a02c9f97bb3/dd-lpfc.el7_2.iso
/v
Comment 32 errata-xmlrpc 2018-02-21 12:33:41 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0336