Bug 1317047

Summary: with ondemand download policy set cannot provision a HOST
Product: Red Hat Satellite Reporter: Kedar Bidarkar <kbidarka>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Katello QA List <katello-qa-list>
Severity: high Docs Contact:
Priority: high    
Version: 6.2.0CC: bbuckingham, bkearney, bmbouter, cwelton, daviddavis, dkliban, ehelms, ggainey, ipanova, jortel, jsherril, kbidarka, lpramuk, mhrivnak, mmccune, omaciel, pcreech, rchan, ttereshc
Target Milestone: UnspecifiedKeywords: PrioBumpQA, Reopened, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
URL: https://pulp.plan.io/issues/1771
Whiteboard:
Fixed In Version: pulp-2.8.1.2-1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1386671 (view as bug list) Environment:
Last Closed: 2016-10-26 12:34:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1331863, 1386671, 1386672    

Description Kedar Bidarkar 2016-03-11 19:15:30 UTC 
Description of problem:

With the download_policy: set to "ondemand", unable to provision HOST with isolated capsule.

while provisoning a "New Host" for isolated capusle, it says:
"could not find kernel image : boot/RedHat-7.2-x86_64-vmlinuz" ,

On capsule filesystem : /var/lib/tftpboot/boot/

[abc@xyz boot]# ll
total 0
-rw-r--r--. 1 foreman-proxy foreman-proxy 0 Mar 11 13:18 RedHat-7.2-x86_64-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy 0 Mar 11 13:18 RedHat-7.2-x86_64-vmlinuz



Version-Release number of selected component (if applicable):
Sat62-SNAP3-compose1

How reproducible:


Steps to Reproduce:
1. PRovision a New Host on isolated capsule
2. check the Pxeboot process
3.

Actual results:
"could not find kernel image : boot/RedHat-7.2-x86_64-vmlinuz" ,

Expected results:

Should be able to provision a "New Host"

Additional info:

1) Note, Provisioning on Default capsule was **NOT** attempted and was tried directly on isolated capsule.
2) Kickstart repos link appears to contain no pckages or files in LiveOS, images, addons, PAckages, e.t.c
Comment 1 Kedar Bidarkar 2016-03-11 19:16:34 UTC 
Note the size of boot files {kernel, initramfs } to be of zero size.
Comment 3 Mike McCune 2016-03-12 17:51:34 UTC 
This happens on the embedded capsule as well, 403 errors trying to fetch the kernel:

# curl http://localhost/pulp/repos/Default_Organization/Library/content/dist/rhel/server/7/7.2/x86_64/kickstart/images/pxeboot/vmlinuz
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /pulp/repos/Default_Organization/Library/content/dist/rhel/server/7/7.2/x86_64/kickstart/images/pxeboot/vmlinuz
on this server.</p>
</body></html>
Comment 5 Justin Sherrill 2016-03-14 12:16:33 UTC 
Related to:  https://bugzilla.redhat.com/show_bug.cgi?id=1313065
Comment 6 David Davis 2016-03-14 15:08:30 UTC 
Part of the problem looks to be that the 0 files (vmlinuz and an rpm package are 0 bytes). It looks like it's related to https://pulp.plan.io/issues/1734#note-2. This at least explains why the files are 0 bytes but the reason these downloads are failing is not entirely clear. Jeremy is looking into it.
Comment 7 David Davis 2016-03-14 15:09:01 UTC 
s/that the 0 files/that there are 0 byte files/
Comment 8 David Davis 2016-03-15 18:16:06 UTC 
It sounds like https://pulp.plan.io/issues/1734 should fix this problem.

Also Jeremy indicated that https://pulp.plan.io/issues/1737 should make repairing corrupted files easy in case something similar arises.
Comment 9 pulp-infra@redhat.com 2016-03-18 17:13:58 UTC 
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
Comment 10 pulp-infra@redhat.com 2016-03-18 17:14:01 UTC 
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
Comment 11 pulp-infra@redhat.com 2016-03-21 18:30:43 UTC 
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
Comment 12 Mike McCune 2016-03-21 19:10:28 UTC 
We are waiting on this upstream issue as well:

https://pulp.plan.io/issues/1771
Comment 13 Mike McCune 2016-03-21 19:13:15 UTC 
Can't currently link 2 issues in BZ to Redmine, this is a tooling issue that will be addressed in the future
Comment 14 pulp-infra@redhat.com 2016-03-22 19:00:23 UTC 
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.
Comment 15 pulp-infra@redhat.com 2016-03-22 19:00:26 UTC 
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
Comment 16 pulp-infra@redhat.com 2016-03-30 16:03:41 UTC 
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
Comment 17 pulp-infra@redhat.com 2016-03-30 21:33:43 UTC 
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
Comment 18 pulp-infra@redhat.com 2016-03-31 13:33:49 UTC 
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.
Comment 20 Mike McCune 2016-04-04 13:53:43 UTC 
We are now blocked by SELinux:

 SELinux is not allowing access to the certs/keys when trying to read them during content download:

pulp.streamer.server:ERROR: (24248-28544)     return importer_config_to_nectar_config(config, working_dir, download_config_kwargs)
pulp.streamer.server:ERROR: (24248-28544)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/nectar_config.py", line 97, in importer_config_to_nectar_config
pulp.streamer.server:ERROR: (24248-28544)     download_config = DownloaderConfig(**download_config_kwargs)
pulp.streamer.server:ERROR: (24248-28544)   File "/usr/lib/python2.7/site-packages/nectar/config.py", line 136, in __init__
pulp.streamer.server:ERROR: (24248-28544)     self._process_ssl_settings()
pulp.streamer.server:ERROR: (24248-28544)   File "/usr/lib/python2.7/site-packages/nectar/config.py", line 165, in _process_ssl_settings
pulp.streamer.server:ERROR: (24248-28544)     raise AttributeError('Cannot read file: %%s' %% file_arg_value)
pulp.streamer.server:ERROR: (24248-28544) AttributeError: Cannot read file: /var/lib/pulp/importers/Default_Organization-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server-yum_importer/pki/ca.crt
[-] 127.0.0.1 - - [02/Apr/2016:23:17:51 +0000] "GET /var/lib/pulp/content/units/rpm/cf/a0c3b4f43ef5e2866e6b8ea2c4133dd05ae26544f6be9b7756b8ed3134a500/zsh-5.0.2-14.el7.x86_64.rpm HTTP/1.1" 500 - "-" "urlgrabber/3.10 yum/3.4.3"

I swear this worked for me before, I was able to download content without issue.

The files exist on disk:

ls -Z /var/lib/pulp/importers/Default_Organization-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server-yum_importer/pki/ca.crt

-rw-------. apache apache system_u:object_r:httpd_sys_rw_content_t:s0 /var/lib/pulp/importers/Default_Organization-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_7_Server_RPMs_x86_64_7Server-yum_importer/pki/ca.crt

>

denial:

time->Sat Apr  2 19:32:03 2016
type=SYSCALL msg=audit(1459639923.760:2537): arch=c000003e syscall=21 success=no exit=-13 a0=7fe61c00e3d0 a1=4 a2=7fe644bc0fa8 a3=3 items=0 ppid=1 pid=25805 auid=4294967295 uid=48 gid=48 euid=48 suid=48 fsuid=48 egid=48 sgid=48 fsgid=48 tty=(none) ses=4294967295 comm="pulp_streamer" exe="/usr/bin/python2.7" subj=system_u:system_r:streamer_t:s0 key=(null)
type=AVC msg=audit(1459639923.760:2537): avc:  denied  { search } for pid=25805 comm="pulp_streamer" name="pulp" dev="dm-0" ino=68703432 scontext=system_u:system_r:streamer_t:s0 tcontext=system_u:object_r:httpd_sys_rw_content_t:s0 tclass=dir
Comment 21 Og Maciel 2016-04-04 15:46:27 UTC 
Talked to Michale Hrivnak on IRC and he mentioned that this BZ also affects the "Background" download policy. I saw the following errors when synchronizing a YUM repo that had Background download policy:

Apr  4 11:18:50 ibm-x3250m4-01 pulp_streamer: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (3): omaciel.fedorapeople.org
Apr  4 11:18:50 ibm-x3250m4-01 pulp_streamer: nectar.downloaders.threaded:ERROR: (8519-23904) [Errno 13] Permission denied
Apr  4 11:18:50 ibm-x3250m4-01 pulp_streamer: nectar.downloaders.threaded:ERROR: (8519-23904) Traceback (most recent call last):
Apr  4 11:18:50 ibm-x3250m4-01 pulp_streamer: nectar.downloaders.threaded:ERROR: (8519-23904)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 221, in _fetch
Apr  4 11:18:50 ibm-x3250m4-01 pulp_streamer: nectar.downloaders.threaded:ERROR: (8519-23904)     self.config.read_timeout))
Apr  4 11:18:50 ibm-x3250m4-01 pulp_streamer: nectar.downloaders.threaded:ERROR: (8519-23904)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 476, in get
Apr  4 11:18:50 ibm-x3250m4-01 pulp_streamer: nectar.downloaders.threaded:ERROR: (8519-23904)     return self.request('GET', url, **kwargs)
Apr  4 11:18:50 ibm-x3250m4-01 pulp_streamer: nectar.downloaders.threaded:ERROR: (8519-23904)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 464, in request
Apr  4 11:18:50 ibm-x3250m4-01 pulp_streamer: nectar.downloaders.threaded:ERROR: (8519-23904)     resp = self.send(prep, **send_kwargs)
Apr  4 11:18:50 ibm-x3250m4-01 pulp_streamer: nectar.downloaders.threaded:ERROR: (8519-23904)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
Apr  4 11:18:50 ibm-x3250m4-01 pulp_streamer: nectar.downloaders.threaded:ERROR: (8519-23904)     r = adapter.send(request, **kwargs)
Apr  4 11:18:50 ibm-x3250m4-01 pulp_streamer: nectar.downloaders.threaded:ERROR: (8519-23904)   File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
Apr  4 11:18:50 ibm-x3250m4-01 pulp_streamer: nectar.downloaders.threaded:ERROR: (8519-23904)     timeout=timeout
Apr  4 11:18:50 ibm-x3250m4-01 pulp_streamer: nectar.downloaders.threaded:ERROR: (8519-23904)   File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 544, in urlopen
Apr  4 11:18:50 ibm-x3250m4-01 pulp_streamer: nectar.downloaders.threaded:ERROR: (8519-23904)     body=body, headers=headers)
Apr  4 11:18:50 ibm-x3250m4-01 pulp_streamer: nectar.downloaders.threaded:ERROR: (8519-23904)   File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 341, in _make_request
Apr  4 11:18:50 ibm-x3250m4-01 pulp_streamer: nectar.downloaders.threaded:ERROR: (8519-23904)     self._validate_conn(conn)
Apr  4 11:18:50 ibm-x3250m4-01 pulp_streamer: nectar.downloaders.threaded:ERROR: (8519-23904)   File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 762, in _validate_conn
Apr  4 11:18:50 ibm-x3250m4-01 pulp_streamer: nectar.downloaders.threaded:ERROR: (8519-23904)     conn.connect()
Apr  4 11:18:50 ibm-x3250m4-01 pulp_streamer: nectar.downloaders.threaded:ERROR: (8519-23904)   File "/usr/lib/python2.7/site-packages/urllib3/connection.py", line 238, in connect
Apr  4 11:18:50 ibm-x3250m4-01 pulp_streamer: nectar.downloaders.threaded:ERROR: (8519-23904)     ssl_version=resolved_ssl_version)
Apr  4 11:18:50 ibm-x3250m4-01 pulp_streamer: nectar.downloaders.threaded:ERROR: (8519-23904)   File "/usr/lib/python2.7/site-packages/urllib3/util/ssl_.py", line 254, in ssl_wrap_socket
Apr  4 11:18:50 ibm-x3250m4-01 pulp_streamer: nectar.downloaders.threaded:ERROR: (8519-23904)     context.load_cert_chain(certfile, keyfile)
Apr  4 11:18:50 ibm-x3250m4-01 pulp_streamer: nectar.downloaders.threaded:ERROR: (8519-23904) IOError: [Errno 13] Permission denied

Also,

# systemctl status pulp_streamer -l
● pulp_streamer.service - The Pulp lazy content loading streamer
   Loaded: loaded (/usr/lib/systemd/system/pulp_streamer.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2016-04-02 11:35:34 EDT; 1 day 23h ago
 Main PID: 8519 (pulp_streamer)
   CGroup: /system.slice/pulp_streamer.service
           └─8519 /usr/bin/python /usr/bin/pulp_streamer --nodaemon --syslog --prefix=pulp_streamer --pidfile= --python /usr/share/pulp/wsgi/streamer.tac
 
Apr 04 11:18:51 ibm-x3250m4-01.lab.eng.rdu2.redhat.com pulp_streamer[8519]: nectar.downloaders.threaded:ERROR: (8519-12992)   File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 341, in _make_request
Apr 04 11:18:51 ibm-x3250m4-01.lab.eng.rdu2.redhat.com pulp_streamer[8519]: nectar.downloaders.threaded:ERROR: (8519-12992)     self._validate_conn(conn)
Apr 04 11:18:51 ibm-x3250m4-01.lab.eng.rdu2.redhat.com pulp_streamer[8519]: nectar.downloaders.threaded:ERROR: (8519-12992)   File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 762, in _validate_conn
Apr 04 11:18:51 ibm-x3250m4-01.lab.eng.rdu2.redhat.com pulp_streamer[8519]: nectar.downloaders.threaded:ERROR: (8519-12992)     conn.connect()
Apr 04 11:18:51 ibm-x3250m4-01.lab.eng.rdu2.redhat.com pulp_streamer[8519]: nectar.downloaders.threaded:ERROR: (8519-12992)   File "/usr/lib/python2.7/site-packages/urllib3/connection.py", line 238, in connect
Apr 04 11:18:51 ibm-x3250m4-01.lab.eng.rdu2.redhat.com pulp_streamer[8519]: nectar.downloaders.threaded:ERROR: (8519-12992)     ssl_version=resolved_ssl_version)
Apr 04 11:18:51 ibm-x3250m4-01.lab.eng.rdu2.redhat.com pulp_streamer[8519]: nectar.downloaders.threaded:ERROR: (8519-12992)   File "/usr/lib/python2.7/site-packages/urllib3/util/ssl_.py", line 254, in ssl_wrap_socket
Apr 04 11:18:51 ibm-x3250m4-01.lab.eng.rdu2.redhat.com pulp_streamer[8519]: nectar.downloaders.threaded:ERROR: (8519-12992)     context.load_cert_chain(certfile, keyfile)
Apr 04 11:18:51 ibm-x3250m4-01.lab.eng.rdu2.redhat.com pulp_streamer[8519]: nectar.downloaders.threaded:ERROR: (8519-12992) IOError: [Errno 13] Permission denied
Apr 04 11:18:51 ibm-x3250m4-01.lab.eng.rdu2.redhat.com pulp_streamer[8519]: [-] 127.0.0.1 - - [04/Apr/2016:15:18:51 +0000] "GET /var/lib/pulp/content/units/rpm/53/667ebb1eb70b235060e360705d43f9b6b4ba197340b7dd6304838deea21f62/acme-package-1.0.1-1.elfake.noarch.rpm HTTP/1.1" 503 - "-" "python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-327.el7.x86_64"
Comment 22 pulp-infra@redhat.com 2016-04-04 21:03:44 UTC 
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
Comment 25 pulp-infra@redhat.com 2016-04-11 19:33:47 UTC 
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
Comment 26 pulp-infra@redhat.com 2016-04-26 23:04:25 UTC 
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.
Comment 27 pulp-infra@redhat.com 2016-05-09 20:30:49 UTC 
The Pulp upstream bug status is at VERIFIED. Updating the external tracker on this bug.
Comment 28 pulp-infra@redhat.com 2016-05-17 20:01:25 UTC 
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Comment 31 errata-xmlrpc 2016-07-27 09:06:21 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1500
Comment 32 Kedar Bidarkar 2016-10-13 13:57:48 UTC 
provisioning of default capsule fails with the same issue as mentioned in the initial bug report.

# ll
total 216828
lrwxrwxrwx. 1 foreman-proxy root                 40 Oct 12 09:32 fdi-image-rhel_7-img -> foreman-discovery-image-3.1.1-17.iso-img
lrwxrwxrwx. 1 foreman-proxy root                 44 Oct 12 09:32 fdi-image-rhel_7-vmlinuz -> foreman-discovery-image-3.1.1-17.iso-vmlinuz
-rw-r--r--. 1 foreman-proxy root          216873508 Oct 12 09:32 foreman-discovery-image-3.1.1-17.iso-img
-rw-r--r--. 1 foreman-proxy root            5154912 Oct 12 09:32 foreman-discovery-image-3.1.1-17.iso-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy         0 Oct 13 02:12 RedHat-5.11-x86_64-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy         0 Oct 13 02:12 RedHat-5.11-x86_64-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy         0 Oct 13 04:09 RedHat-7.2-x86_64-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy         0 Oct 13 04:09 RedHat-7.2-x86_64-vmlinuz


Tested with Sat6.2.3 SNAP2.
Comment 33 Kedar Bidarkar 2016-10-13 14:00:58 UTC 
Please note, initrd and vmlinuz are of size zero bytes, when we set download_policy as "on_demand" and try to sync kickstart content.
Comment 34 Lukas Pramuk 2016-10-13 14:38:00 UTC 
Does it only fail for the 1st time provisioning for specific kickstart tree (OS)? 

First time provisioning triggers kernel and ramdisk download... and fails.
Comment 35 David Davis 2016-10-13 15:32:37 UTC 
Looks like there are two issues. First, it seems like the pulp streamer is having trouble connecting to the CDN:

Oct 13 07:12:28 qe-testing-rhel7 pulp_streamer: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (7): cdn.redhat.com
Oct 13 07:12:28 qe-testing-rhel7 pulp_streamer: nectar.downloaders.threaded:ERROR: Skipping requests to cdn.redhat.com due to repeated connection failures: [Errno 2] No such file or directory
Oct 13 07:12:28 qe-testing-rhel7 pulp_streamer: [-] 127.0.0.1 - - [13/Oct/2016:11:12:28 +0000] "GET /var/lib/pulp/content/units/distribution/49/da05e1e526ca2cefec9985aea4c85ba2d1a04e167ecf1acbe50b0063fc30c9/images/pxeboot/initrd.img HTTP/1.1" 503 - "-" "Wget/1.17.1 (darwin15.2.0)"


Then it looks like the capsule is using wget to download a 0 byte file (I tried this command manually and got a 0 byte file):

https://git.io/vPVhw


Looking into both now.
Comment 36 Kedar Bidarkar 2016-10-13 15:38:24 UTC 
@lpramuk, I don't get your question?

What is, first time provisioning for specific kickstart tree ?

what is, First time provisioning triggers kernel and ramdisk download... and fails.

No idea.
Comment 37 Kedar Bidarkar 2016-10-13 15:40:37 UTC 
Please note, initrd and vmlinuz are of size zero bytes for both RHEL5 and RHEL7, when we set download_policy as "on_demand" and try to sync kickstart content.

This was intentionally tested using a completely fresh setup, without syncing any content prior to this. 

NOTE: Syncing even once in an ORG using "immediate" may populate the vmlinuz and initrd.img files.
Comment 38 David Davis 2016-10-13 19:18:11 UTC 
For the capsule problem, I opened https://bugzilla.redhat.com/show_bug.cgi?id=1384661. Jeff is looking into the original problem regarding pulp_streamer not downloading the files from CDN.
Comment 39 Lukas Pramuk 2016-10-14 11:11:22 UTC 
@kbidarkar 

We have kickstarts present as "On Demand" here, so for the first time provisioning fails as files are not available but it has triggered to download them...
Next provisioning for the same OS will not fail as files are already downloaded.

Is it clearer?
Comment 41 pulp-infra@redhat.com 2016-10-17 21:03:02 UTC 
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.
Comment 42 pulp-infra@redhat.com 2016-10-17 21:03:07 UTC 
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
Comment 43 pulp-infra@redhat.com 2016-10-17 22:30:51 UTC 
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
Comment 44 Jeff Ortel 2016-10-18 16:58:04 UTC 
Fix merged to pulp 2.10 but safe to cherry pick back to 2.8.x.

commit: b3841b06f54d29fed26cd4a9960fddd4ff0a1748
Comment 45 pulp-infra@redhat.com 2016-10-18 17:00:20 UTC 
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
Comment 47 Kedar Bidarkar 2016-10-25 13:50:58 UTC 
VERIFIED with sat6.2.3 SNAP5
Comment 48 Kedar Bidarkar 2016-10-25 13:58:04 UTC 
1) The vmlinuz and initramd files are not of zero size.

Verified this bug

Steps:

1. On-Demand sync fresh repo 'Red Hat Enterprise Linux 7 Server RPMs x86_64 5Serves' from CDN.
2. On-Demand sync frsh repo 'Red Hat Enterprise Linux 7 Server Kickstart x86_64 7.2' from CDN.
3. On-Demand sync fresh repo 'Red Hat Satellite Tools 6.2 for RHEL 7 Server RPMs x86_64' from CDN.
4. Created CV from these repos, published and promoted to DEV env.
5. Created an Activation key from above CV and enabling all three on-demand synced repos.
6. Created a hostgroup by adding above CV and AK.
7. Provisioned a RHEL7 host by using RHEV hostgroup.

# ll Red*
-rw-r--r--. 1 foreman-proxy foreman-proxy 12597752 Aug 27  2014 RedHat-5.11-x86_64-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  2127884 Aug 19  2014 RedHat-5.11-x86_64-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 39725808 Oct 30  2015 RedHat-7.2-x86_64-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  5154912 Oct 29  2015 RedHat-7.2-x86_64-vmlinuz
Comment 49 pulp-infra@redhat.com 2016-10-25 14:30:59 UTC 
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.
Comment 50 Bryan Kearney 2016-10-26 12:34:37 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:2108
Comment 51 pulp-infra@redhat.com 2016-10-31 20:31:13 UTC 
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.