Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2214133

Summary: RHUI_CLOUD_MAP for Azure in the leapp is not compatible with the leapp-rhui-azure-sap-1.0.0-6.el7
Product: Red Hat Enterprise Linux 7 Reporter: Ameya Patil <amepatil>
Component: leappAssignee: Michal Hečko <mhecko>
Status: CLOSED CURRENTRELEASE QA Contact: upgrades-and-conversions
Severity: urgent Docs Contact:
Priority: urgent    
Version: 7.9CC: dsirrine, igulina, mhecko, mkarnik, nchoudhu, rhandlin
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-08-22 12:40:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ameya Patil 2023-06-12 03:41:51 UTC
Description of problem:

Customer while following the SAP HANA inplace upgrade path using the latest version of leapp currently available i.e. (leapp-upgrade-el7toel8-0.18.0-1.el7_9.noarch) fail with the error
~~~
Process Process-502:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/site-packages/leapp/repository/actor_definition.py", line 74, in _do_run
    actor_instance.run(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/leapp/actors/__init__.py", line 289, in run
    self.process(*args)
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/actor.py", line 58, in process
    userspacegen.perform()
  File "/usr/lib/python2.7/site-packages/leapp/utils/deprecation.py", line 42, in process_wrapper
    return target_item(*args, **kwargs)
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py", line 77
4, in perform
    target_repoids = _gather_target_repositories(context, indata, prod_cert_path)
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py", line 71
4, in _gather_target_repositories
    rhui.copy_rhui_data(context, indata.rhui_info.provider)
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/rhui.py", line 259, in copy_rhui_data
    context.copy_to(os.path.join(data_dir, path_[0]), path_[1])
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/mounting.py", line 232, in copy_to
    shutil.copy2(src, self.full_path(dst))
  File "/usr/lib64/python2.7/shutil.py", line 130, in copy2
    copyfile(src, dst)
  File "/usr/lib64/python2.7/shutil.py", line 82, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: u'/etc/leapp/repos.d/system_upgrade/common/files/rhui/azure-sap-ha/content-sap-ha.crt'
~~~


Upon further investigation we found that the Certificate leapp is referring to here in the above error comes from the RHUI_CLOUD_MAP file.

And we observed that the latest version of leapp, (leapp-upgrade-el7toel8-0.18.0-1.el7_9.noarch) contains the RHUI_CLOUD_MAP as below,
File - /usr/share/leapp-repository/repositories/system_upgrade/common/libraries/rhui.py
~~~
The files in 'files_map' are provided by special Leapp rpms (per cloud) and # are delivered into "repos/system_upgrade/common/files/rhui/<PROVIDER>

RHUI_CLOUD_MAP = {
    '7to8': {

<...output skipped...>

        'azure': {
            'src_pkg': 'rhui-azure-rhel7',
            'target_pkg': 'rhui-azure-rhel8',
            'agent_pkg': 'WALinuxAgent',
            'leapp_pkg': 'leapp-rhui-azure',
            'leapp_pkg_repo': 'leapp-azure.repo',
            'files_map': [
                ('content.crt', RHUI_PKI_PRODUCT_DIR),
                ('key.pem', RHUI_PKI_PRIVATE_DIR),
                ('leapp-azure.repo', YUM_REPOS_PATH)
            ],
        },
        'azure-sap-apps': {
            'src_pkg': 'rhui-azure-rhel7-base-sap-apps',
            'target_pkg': 'rhui-azure-rhel8-sapapps',
            'agent_pkg': 'WALinuxAgent',
            'leapp_pkg': 'leapp-rhui-azure-sap',
            'leapp_pkg_repo': 'leapp-azure-sap-apps.repo',
            'files_map': [
                ('content-sapapps.crt', RHUI_PKI_PRODUCT_DIR),   <---
                ('key-sapapps.pem', RHUI_PKI_PRIVATE_DIR),       <---
                ('leapp-azure-sap-apps.repo', YUM_REPOS_PATH),   <---
            ],
        },
        'azure-sap-ha': {
            'src_pkg': 'rhui-azure-rhel7-base-sap-ha',
            'target_pkg': 'rhui-azure-rhel8-sap-ha',
            'agent_pkg': 'WALinuxAgent',
            'leapp_pkg': 'leapp-rhui-azure-sap',
            'leapp_pkg_repo': 'leapp-azure-sap-ha.repo',
            'files_map': [
                ('content-sap-ha.crt', RHUI_PKI_PRODUCT_DIR),     <---
                ('key-sap-ha.pem', RHUI_PKI_PRIVATE_DIR),         <---
                ('leapp-azure-sap-ha.repo', YUM_REPOS_PATH)       <---
            ],
        },

<...output skipped...>
~~~

While "leapp-rhui-azure-sap-1.0.0-6.el7.noarch" provides below files which are different from the above. 
~~~
# rpm -qlv leapp-rhui-azure-sap
-rw-r--r--    1 root    root                     3537 Feb 21 10:33 /usr/share/leapp-repository/repositories/system_upgrade/common/files/rhui/azure-sap-apps/content-rhel8-eus.crt
-rw-r--r--    1 root    root                     3642 Feb 21 10:33 /usr/share/leapp-repository/repositories/system_upgrade/common/files/rhui/azure-sap-apps/content-rhel8-sapapps.crt
-rw-r--r--    1 root    root                     1917 Feb 21 10:33 /usr/share/leapp-repository/repositories/system_upgrade/common/files/rhui/azure-sap-apps/key-rhel8-eus.pem
-rw-r--r--    1 root    root                     1917 Feb 21 10:33 /usr/share/leapp-repository/repositories/system_upgrade/common/files/rhui/azure-sap-apps/key-rhel8-sapapps.pem
-rw-r--r--    1 root    root                     9274 Feb 21 10:33 /usr/share/leapp-repository/repositories/system_upgrade/common/files/rhui/azure-sap-apps/leapp-azure-sap-apps.repo
-rw-r--r--    1 root    root                     8082 Feb 21 10:33 /usr/share/leapp-repository/repositories/system_upgrade/common/files/rhui/azure-sap/content-rhel8-sap-ha.crt
-rw-r--r--    1 root    root                     1917 Feb 21 10:33 /usr/share/leapp-repository/repositories/system_upgrade/common/files/rhui/azure-sap/key-rhel8-sap-ha.pem
-rw-r--r--    1 root    root                    12689 Feb 21 10:33 /usr/share/leapp-repository/repositories/system_upgrade/common/files/rhui/azure-sap/leapp-azure-sap.repo
~~~

I looked up old version of leapp (leapp-upgrade-el7toel8-0.17.0-10.el7_9.noarch) and the file is looks for matches this above "leapp-rhui-azure-sap" package.
~~~
# The files in 'files_map' are provided by special Leapp rpms (per cloud) and # are delivered into "repos/system_upgrade/common/files/rhui/<PROVIDER>


RHUI_CLOUD_MAP = {
    '7to8': {

<...output skipped...>

        'azure': {
            'src_pkg': 'rhui-azure-rhel7',
            'target_pkg': 'rhui-azure-rhel8',
            'agent_pkg': 'WALinuxAgent',
            'leapp_pkg': 'leapp-rhui-azure',
            'leapp_pkg_repo': 'leapp-azure.repo',
            'files_map': [
                ('content.crt', RHUI_PKI_PRODUCT_DIR),
                ('key.pem', RHUI_PKI_PRIVATE_DIR),
                ('leapp-azure.repo', YUM_REPOS_PATH)
            ],
        },
        'azure-sap-apps': {
            'src_pkg': 'rhui-azure-rhel7-base-sap-apps',
            'target_pkg': 'rhui-azure-rhel8-sapapps',
            'agent_pkg': 'WALinuxAgent',
            'leapp_pkg': 'leapp-rhui-azure-sap',
            'leapp_pkg_repo': 'leapp-azure-sap-apps.repo',
            'files_map': [
                ('content-rhel8-eus.crt', RHUI_PKI_PRODUCT_DIR),     <---
                ('content-rhel8-sapapps.crt', RHUI_PKI_PRODUCT_DIR), <---
                ('key-rhel8-eus.pem', RHUI_PKI_DIR),                 <---
                ('key-rhel8-sapapps.pem', RHUI_PKI_DIR),             <---
                ('leapp-azure-sap-apps.repo', YUM_REPOS_PATH),       <---
            ],
        },
        'azure-sap': {
            'src_pkg': 'rhui-azure-rhel7-base-sap-ha',
            'target_pkg': 'rhui-azure-rhel8-sap-ha',
            'agent_pkg': 'WALinuxAgent',
            'leapp_pkg': 'leapp-rhui-azure-sap',
            'leapp_pkg_repo': 'leapp-azure-sap.repo',
            'files_map': [
                ('content-rhel8-sap-ha.crt', RHUI_PKI_PRODUCT_DIR),  <---
                ('key-rhel8-sap-ha.pem', RHUI_PKI_DIR),              <---
                ('leapp-azure-sap.repo', YUM_REPOS_PATH)             <---
            ],
        },
~~~



This must mean that with the latest version of leapp (leapp-upgrade-el7toel8-0.18.0-1.el7_9.noarch) the files which are required are content-sapapps.crt , content-sap-ha.crt , etc
But that "leapp-rhui-azure-sap-1.0.0-6.el7.noarch" is providing the file compatible with old version of leapp (leapp-upgrade-el7toel8-0.17.0-10.el7_9.noarch)
This  must mean that there should be a new version of "leapp-rhui-azure-sap" which would be compatible with latest version of leapp mapping
However we confirmed with Customer and Microsoft over the support case that the "leapp-rhui-azure-sap-1.0.0-6.el7.noarch" is already the latest available.
~~~
# yum list --showduplicates leapp-rhui-azure-sap
Loaded plugins: product-id, search-disabled-repos, subscription-manager

This system is not registered with an entitlement server. You can use subscription-manager to register.
Installed Packages
leapp-rhui-azure-sap.noarch                             1.0.0-6.el7                              @rhui-microsoft-azure-rhel7-base-sap-ha
Available Packages
leapp-rhui-azure-sap.noarch                             1.0.0-1.el7                              rhui-microsoft-azure-rhel7-base-sap-ha
leapp-rhui-azure-sap.noarch                             1.0.0-3.el7                              rhui-microsoft-azure-rhel7-base-sap-ha
leapp-rhui-azure-sap.noarch                             1.0.0-5.el7                              rhui-microsoft-azure-rhel7-base-sap-ha
leapp-rhui-azure-sap.noarch                             1.0.0-6.el7                              rhui-microsoft-azure-rhel7-base-sap-ha


This lists the repo:
repo id                                                   repo name      status
rhel-ha-for-rhel-7-server-rhui-rpms/7Server/x86_64        Red Hat Enterp    865
rhel-sap-for-rhel-7-server-rhui-rpms/7Server/x86_64       Red Hat Enterp    145
rhel-sap-hana-for-rhel-7-server-rhui-rpms/7Server/x86_64  Red Hat Enterp    113
rhui-microsoft-azure-rhel7                                Microsoft Azur     13
rhui-microsoft-azure-rhel7-base-sap-ha                    Microsoft Azur     10
rhui-rhel-7-server-dotnet-rhui-rpms/x86_64                dotNET on RHEL  1,261
rhui-rhel-7-server-rhui-extras-rpms/x86_64                Red Hat Enterp  1,459
rhui-rhel-7-server-rhui-rh-common-rpms/7Server/x86_64     Red Hat Enterp    243
rhui-rhel-7-server-rhui-rpms/7Server/x86_64               Red Hat Enterp 33,760
rhui-rhel-7-server-rhui-supplementary-rpms/7Server/x86_64 Red Hat Enterp    479
rhui-rhel-server-rhui-rhscl-7-rpms/7Server/x86_64         Red Hat Softwa 14,520
repolist: 52,868
 
And it's showing the latest version: 1.0.0.6.el7
# yum info leapp-rhui-azure-sap
Loaded plugins: langpacks, product-id, search-disabled-repos
Available Packages
Name        : leapp-rhui-azure-sap
Arch        : noarch
Version     : 1.0.0
Release     : 6.el7
Size        : 14 k
Repo        : rhui-microsoft-azure-rhel7-base-sap-ha
Summary     : Leapp in-place upgrade cloud specific package (SAP)
URL         : http://redhat.com
License     : LGPLv2+
Description : Leapp in-place upgrade of SAP on Azure
~~~





Version-Release number of selected component (if applicable):
leapp-upgrade-el7toel8-0.18.0-1.el7_9.noarch
leapp-rhui-azure-sap-1.0.0.6.el7.noarch


How reproducible:
Every time


Steps to Reproduce:
Following the inplace upgrade path for SAP HANA - https://access.redhat.com/solutions/5154031

     
1. Install the latest version of leapp-rhui-azure-sap package

      # yum install leapp-rhui-azure-sap

2. Enable the rhui-rhel-7-server-rhui-extras-rpms repository and install latest version of leapp

      # yum-config-manager --enable rhui-rhel-7-server-rhui-extras-rpms

      # yum install leapp-upgrade

3. Perform Preupgrade on the machine and clear any inhibitor which might be occuring like download of leapp-data-22.tar.gz or any other inhibitor

      # leapp preupgrade --debug
  

4. After clearing all the inhibitors when we finally run the preupgrade again it fails during the "targetuserspacecreator" actor

      # leapp preupgrade --debug



Actual results:
Leapp preupgrade fails with this error while copying a non-existent cert into the temporary upgrade environment its building due to the files in the RHUI_CLOULD_MAP mapping change in the recent version of leapp

Expected results:
Leapp preupgrade complete successfully

Additional info:

Comment 4 Michal Hečko 2023-06-12 10:58:40 UTC
Hi Ameya,

thank you for investigating the issue and producing a very helpful bug report. We are aware of the root cause of the issue and we are already working
with the Azure RHUI team on deploying a new version of the leapp-rhui-azure-sap to production. I will update this BZ once the new version is deployed.

Comment 6 Novonil Choudhuri 2023-06-14 16:30:41 UTC
Michal,

This is highly escalated from Microsoft now. Below is the email we received today. Can you please throw some light on this and update this bug. 

////
From Simrat.Satia:

Looping in RedHat Leapp Team: @Michal Hecko @Michal Reznik @Irina Gulina @Martin Kluson

 

I understand that a new version of leapp is underway but how is the existing functionality of previous leapp package affected?
Was there any change made in leapp utility recently that affected the in-place SAP upgrade of RHEL7 to RHEL8?
 

Bz: https://bugzilla.redhat.com/show_bug.cgi?id=2214133

As per Michal: “We are aware of the root cause of the issue and we are already working with the Azure RHUI team on deploying a new version of the leapp-rhui-azure-sap to production.”
Our team has not been made aware of any such issue. 
Please provide insight into the same and how it will be fixed.
///

thanks,
novo

Comment 10 Irina Gulina 2023-08-22 12:40:17 UTC
That has been fixed with the RHUI rpms of version 602, and leapp-rhui-azure.*-1.0.0-10.el* packages back in mid June.