Bug 1613968

Description of problem:
NOTE: Please click on Unwrap comment for better reading experience. 

It is observed that product-id plugin was failing to download product-id file in to /etc/pki/product after installing a package from a repo (ex : RHSCL in this case)

Version-Release number of selected component (if applicable):
dnf-plugin-subscription-manager-1.21.5-7.el7.x86_64
subscription-manager: 1.21.5-7.el7

How reproducible:
Always

Steps to Reproduce:
1. Before we begin, its worth noting the due to following bugs (Bug 1613454 AND Bug 1613456 )we need to follow few workarounds to get pass these ^ bugs

Workarounds in place 
=====================

# cat /etc/dnf/dnf.conf 
[main]
gpgcheck=1
installonly_limit=3
clean_requirements_on_remove=True
pluginpath=/usr/lib64/python2.7/site-packages/dnf-plugins,/usr/lib/python2.7/site-packages/dnf-plugins

Since yum and dnf can co-exist on RHEL76, make sure the all the yum plugins
are disabled (to make sure the dnf is not using the file that are already
populated by yum ) 

# subscription-manager config --list |grep auto_enable
   auto_enable_yum_plugins = 0

# cat /etc/yum/pluginconf.d/search-disabled-repos.conf | grep enabled
enabled=0

# cat /etc/yum/pluginconf.d/subscription-manager.conf | grep enabled
enabled=0

# cat /etc/yum/pluginconf.d/product-id.conf | grep enabled
enabled=0

2.Register and auto-attach the system 
3.Attach a RHSCL subscription on the system
4.Displaying only RHSCL Subscription below
# subscription-manager list --consumed
+-------------------------------------------+
   Consumed Subscriptions
+-------------------------------------------+
Subscription Name:   Red Hat Enterprise Linux Server, Standard (2 sockets) (Unlimited guests) with Resilient Storage
Provides:            Red Hat Beta
                     Red Hat Developer Tools (for RHEL Server)
                     Oracle Java (for RHEL Server)
                     Red Hat Developer Tools Beta (for RHEL Server)
                     Red Hat Enterprise Linux for x86_64
                     Red Hat Enterprise Linux Resilient Storage (for RHEL Server)
                     Red Hat Software Collections Beta (for RHEL Server)
                     dotNET on RHEL (for RHEL Server)
                     dotNET on RHEL Beta (for RHEL Server)
                     Red Hat Enterprise Linux High Availability (for RHEL Server)
                     Red Hat Ansible Engine
                     Red Hat Enterprise Linux Server
SKU:                 RH0148194
Contract:            11517187
Account:             477931
Serial:              7610270808271145878
Pool ID:             8a99f98160dcc4fc0160dcf003670bb1
Provides Management: No
Active:              True
Quantity Used:       1
Service Level:       Standard
Service Type:        L1-L3
Status Details:      Subscription is current
Subscription Type:   Standard
Starts:              Tuesday 09 January 2018
Ends:                Tuesday 08 January 2019
System Type:         Virtual
<snip>

5.Enable rhscl repo on the system
ex : subscription-manager repos --enable=rhel-server-rhscl-7-rpms

6. List the product certificates

# ll /etc/pki/product*
/etc/pki/product:
total 0

/etc/pki/product-default:
total 4
-rw-r--r--. 1 root root 2167 Aug  1 11:32 69.pem

7.Enable rhscl repos on the system and list

# subscription-manager repos --list-enabled
+----------------------------------------------------------+
    Available Repositories in /etc/yum.repos.d/redhat.repo
+----------------------------------------------------------+
Repo ID:   rhel-server-rhscl-7-rpms
Repo Name: Red Hat Software Collections RPMs for Red Hat Enterprise Linux 7 Server
Repo URL:  https://cdn.redhat.com/content/dist/rhel/server/7/7.4/$basearch/rhscl/1/os
Enabled:   1

Repo ID:   rhel-7-server-rpms
Repo Name: Red Hat Enterprise Linux 7 Server (RPMs)
Repo URL:  https://cdn.redhat.com/content/dist/rhel/server/7/7.4/$basearch/os
Enabled:   1

8.  Delete the redhat.repo file ( only to be tried on rhel76, as the redhat.repo file is already populated by yum)
#rm -rf /etc/yum.repos.d/redhat.repo 

9.Install a package from RHSCL repo
Ex : # dnf install devtoolset-3-apache-commons-el-javadoc-1.0-30* -d3

# dnf install devtoolset-3-apache-commons-el-javadoc-1.0-30* -d3
Loaded plugins: builddep, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, product-id, repoclosure, repograph, repomanage, reposync, subscription-manager
DNF version: 2.7.5
cachedir: /var/cache/dnf
Unknown configuration option: ui_repoid_vars = basearch in /etc/yum.repos.d/redhat.repo
Unknown configuration option: ui_repoid_vars = releasever basearch in /etc/yum.repos.d/redhat.repo

<SNIP> ^^ this  will be addressed in a separate bug 

Updating Subscription Management repositories.
Red Hat Software Collections RPMs for Red Hat Enterprise Linux 7 Server                                                                                                            2.3 MB/s |  17 MB     00:07    
not found deltainfo for: Red Hat Software Collections RPMs for Red Hat Enterprise Linux 7 Server
rhel-server-rhscl-7-rpms: using metadata from Thursday 02 August 2018 04:33:55 AM EDT.
Red Hat Enterprise Linux 7 Server (RPMs)                                                                                                                                           4.1 MB/s |  51 MB     00:12    
not found deltainfo for: Red Hat Enterprise Linux 7 Server (RPMs)
rhel-7-server-rpms: using metadata from Thursday 02 August 2018 04:35:21 AM EDT.
Last metadata expiration check: 0:00:13 ago on Wednesday 08 August 2018 07:35:10 AM EDT.
Missing file *modules.yaml in metadata cache dir: /var/cache/dnf/rhel-server-rhscl-7-rpms-d6513f5c7529d87c
Missing file *modules.yaml in metadata cache dir: /var/cache/dnf/rhel-7-server-rpms-007572f6a49efdad
Completion plugin: Generating completion cache...
--> Starting dependency resolution
---> Package devtoolset-3-apache-commons-el-javadoc.noarch 1.0-30.el7 will be installed
--> Finished dependency resolution
Dependencies resolved.
===================================================================================================================================================================================================================
 Package                                                              Arch                                 Version                                    Repository                                              Size
===================================================================================================================================================================================================================
Installing:
 devtoolset-3-apache-commons-el-javadoc                               noarch                               1.0-30.el7                                 rhel-server-rhscl-7-rpms                               107 k

Transaction Summary
===================================================================================================================================================================================================================
Install  1 Package

Total download size: 107 k
Installed size: 1.6 M
Is this ok [y/N]: y
Downloading Packages:
devtoolset-3-apache-commons-el-javadoc-1.0-30.el7.noarch.rpm                                                                                                                       104 kB/s | 107 kB     00:01    
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                              103 kB/s | 107 kB     00:01     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                           1/1 
  Installing       : devtoolset-3-apache-commons-el-javadoc-1.0-30.el7.noarch                                                                                                                                  1/1 
  Verifying        : devtoolset-3-apache-commons-el-javadoc-1.0-30.el7.noarch                                                                                                                                  1/1 
Installed products updated.
Completion plugin: Generating completion cache...

Installed:
  devtoolset-3-apache-commons-el-javadoc.noarch 1.0-30.el7                                                                                                                                                         

Complete!

10. Check the install product 

Actual results:
Observed that product cert 201.pem was not installed on the system 

Expected results:
Product cert should be installed 

Additional info:
If the same scanrio was performed by yum on the same system the product certificate gets installed 

rhsm.log
--------
2018-08-08 07:37:14,554 [WARNING] dnf:3637:MainThread @product-id.py:102 - Error loading productid metadata for [rhel-server-rhscl-7-rpms]
ip_resolve: None
enabled_metadata: None
proxy_password: None
deltarpm: True
bandwidth: 0
cost: 1000
exclude: []
priority: 99
max_parallel_downloads: None
repo_gpgcheck: False
sslverify: True
failovermethod: priority
gpgkey: [u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release', u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release']
minrate: 1000
fastestmirror: False
excludepkgs: []
mirrorlist: None
gpgcheck: True
type: None
sslcacert: /etc/rhsm/ca/redhat-uep.pem
username: None
metadata_expire: 86400
sslclientcert: /etc/pki/entitlement/2001309781034992442.pem
baseurl: [u'https://cdn.qa.redhat.com/content/dist/rhel/server/7/7Server/x86_64/rhscl/1/os']
includepkgs: []
throttle: 0
password: None
basecachedir: /var/cache/dnf
retries: 10
mediaid: None
name: Red Hat Software Collections RPMs for Red Hat Enterprise Linux 7 Server
deltarpm_percentage: 75
sslclientkey: /etc/pki/entitlement/2001309781034992442-key.pem
metalink: None
enabled: True
skip_if_unavailable: True
protected_packages: [u'dnf', u'systemd', u'dnf']
proxy_username: None
timeout: 30
enablegroups: True
hotfixes: False
proxy: https://auto-services.usersys.redhat.com:3129.
2018-08-08 07:37:14,555 [ERROR] dnf:3637:MainThread @product-id.py:103 - [Errno 2] No such file or directory: u'/tmp/dnf-rD4lLt/repodata/46cc2fe682fb760d661e101a9d0629c91ef257cf55ad566a630d6f5a1244cca4'
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/dnf-plugins/product-id.py", line 93, in get_enabled
    cert = self._get_cert(fn)
  File "/usr/lib64/python2.7/site-packages/subscription_manager/productid.py", line 712, in _get_cert
    f = open(fn)
IOError: [Errno 2] No such file or directory: u'/tmp/dnf-rD4lLt/repodata/46cc2fe682fb760d661e101a9d0629c91ef257cf55ad566a630d6f5a1244cca4'
2018-08-08 07:37:16,027 [WARNING] dnf:3637:MainThread @product-id.py:102 - Error loading productid metadata for [rhel-7-server-rpms]
ip_resolve: None
enabled_metadata: None
proxy_password: None
deltarpm: True
bandwidth: 0
cost: 1000
exclude: []
priority: 99
max_parallel_downloads: None
repo_gpgcheck: False
sslverify: True
failovermethod: priority
gpgkey: [u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release']
minrate: 1000
fastestmirror: False
excludepkgs: []
mirrorlist: None
gpgcheck: True
type: None
sslcacert: /etc/rhsm/ca/redhat-uep.pem
username: None
metadata_expire: 86400
sslclientcert: /etc/pki/entitlement/7610270808271145878.pem
baseurl: [u'https://cdn.qa.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os']
includepkgs: []
throttle: 0
password: None
basecachedir: /var/cache/dnf
retries: 10
mediaid: None
name: Red Hat Enterprise Linux 7 Server (RPMs)
deltarpm_percentage: 75
sslclientkey: /etc/pki/entitlement/7610270808271145878-key.pem
metalink: None
enabled: True
skip_if_unavailable: True
protected_packages: [u'dnf', u'systemd', u'dnf']
proxy_username: None
timeout: 30
enablegroups: True
hotfixes: False
proxy: https://auto-services.usersys.redhat.com:3129.
2018-08-08 07:37:16,027 [ERROR] dnf:3637:MainThread @product-id.py:103 - [Errno 2] No such file or directory: u'/tmp/dnf-3xM2lS/repodata/d79954bc5f3f069740740db4c1cce5aeaa8e8328f7bfb6203b8cd7fdd966c803'
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/dnf-plugins/product-id.py", line 93, in get_enabled
    cert = self._get_cert(fn)
  File "/usr/lib64/python2.7/site-packages/subscription_manager/productid.py", line 712, in _get_cert
    f = open(fn)
IOError: [Errno 2] No such file or directory: u'/tmp/dnf-3xM2lS/repodata/d79954bc5f3f069740740db4c1cce5aeaa8e8328f7bfb6203b8cd7fdd966c803'
2018-08-08 07:37:16,029 [DEBUG] dnf:3637:MainThread @product-id.py:108 - Unable to load productid metadata for repos: [u'rhel-server-rhscl-7-rpms', u'rhel-7-server-rpms']
2018-08-08 07:37:17,098 [DEBUG] dnf:3637:MainThread @productid.py:418 - Checking for product id certs to install or update.
2018-08-08 07:37:17,098 [DEBUG] dnf:3637:MainThread @productid.py:423 - active set([u'rhel-7-server-rpms'])
2018-08-08 07:37:17,099 [DEBUG] dnf:3637:MainThread @productid.py:424 - enabled []
2018-08-08 07:37:17,099 [DEBUG] dnf:3637:MainThread @productid.py:556 - about to run post_product_id_install
2018-08-08 07:37:17,099 [DEBUG] dnf:3637:MainThread @productid.py:567 - about to run post_product_id_update