Bug 2214133

Summary: RHUI_CLOUD_MAP for Azure in the leapp is not compatible with the leapp-rhui-azure-sap-1.0.0-6.el7
Product: Red Hat Enterprise Linux 7 Reporter: Ameya Patil <amepatil>
Component: leappAssignee: Michal Hečko <mhecko>
Status: NEW --- QA Contact: upgrades-and-conversions
Severity: urgent Docs Contact:
Priority: urgent    
Version: 7.9CC: dsirrine, mhecko, mkarnik, nchoudhu, rhandlin
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ameya Patil 2023-06-12 03:41:51 UTC
Description of problem:

Customer while following the SAP HANA inplace upgrade path using the latest version of leapp currently available i.e. (leapp-upgrade-el7toel8-0.18.0-1.el7_9.noarch) fail with the error
~~~
Process Process-502:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/site-packages/leapp/repository/actor_definition.py", line 74, in _do_run
    actor_instance.run(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/leapp/actors/__init__.py", line 289, in run
    self.process(*args)
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/actor.py", line 58, in process
    userspacegen.perform()
  File "/usr/lib/python2.7/site-packages/leapp/utils/deprecation.py", line 42, in process_wrapper
    return target_item(*args, **kwargs)
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py", line 77
4, in perform
    target_repoids = _gather_target_repositories(context, indata, prod_cert_path)
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py", line 71
4, in _gather_target_repositories
    rhui.copy_rhui_data(context, indata.rhui_info.provider)
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/rhui.py", line 259, in copy_rhui_data
    context.copy_to(os.path.join(data_dir, path_[0]), path_[1])
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/mounting.py", line 232, in copy_to
    shutil.copy2(src, self.full_path(dst))
  File "/usr/lib64/python2.7/shutil.py", line 130, in copy2
    copyfile(src, dst)
  File "/usr/lib64/python2.7/shutil.py", line 82, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: u'/etc/leapp/repos.d/system_upgrade/common/files/rhui/azure-sap-ha/content-sap-ha.crt'
~~~


Upon further investigation we found that the Certificate leapp is referring to here in the above error comes from the RHUI_CLOUD_MAP file.

And we observed that the latest version of leapp, (leapp-upgrade-el7toel8-0.18.0-1.el7_9.noarch) contains the RHUI_CLOUD_MAP as below,
File - /usr/share/leapp-repository/repositories/system_upgrade/common/libraries/rhui.py
~~~
The files in 'files_map' are provided by special Leapp rpms (per cloud) and # are delivered into "repos/system_upgrade/common/files/rhui/<PROVIDER>

RHUI_CLOUD_MAP = {
    '7to8': {

<...output skipped...>

        'azure': {
            'src_pkg': 'rhui-azure-rhel7',
            'target_pkg': 'rhui-azure-rhel8',
            'agent_pkg': 'WALinuxAgent',
            'leapp_pkg': 'leapp-rhui-azure',
            'leapp_pkg_repo': 'leapp-azure.repo',
            'files_map': [
                ('content.crt', RHUI_PKI_PRODUCT_DIR),
                ('key.pem', RHUI_PKI_PRIVATE_DIR),
                ('leapp-azure.repo', YUM_REPOS_PATH)
            ],
        },
        'azure-sap-apps': {
            'src_pkg': 'rhui-azure-rhel7-base-sap-apps',
            'target_pkg': 'rhui-azure-rhel8-sapapps',
            'agent_pkg': 'WALinuxAgent',
            'leapp_pkg': 'leapp-rhui-azure-sap',
            'leapp_pkg_repo': 'leapp-azure-sap-apps.repo',
            'files_map': [
                ('content-sapapps.crt', RHUI_PKI_PRODUCT_DIR),   <---
                ('key-sapapps.pem', RHUI_PKI_PRIVATE_DIR),       <---
                ('leapp-azure-sap-apps.repo', YUM_REPOS_PATH),   <---
            ],
        },
        'azure-sap-ha': {
            'src_pkg': 'rhui-azure-rhel7-base-sap-ha',
            'target_pkg': 'rhui-azure-rhel8-sap-ha',
            'agent_pkg': 'WALinuxAgent',
            'leapp_pkg': 'leapp-rhui-azure-sap',
            'leapp_pkg_repo': 'leapp-azure-sap-ha.repo',
            'files_map': [
                ('content-sap-ha.crt', RHUI_PKI_PRODUCT_DIR),     <---
                ('key-sap-ha.pem', RHUI_PKI_PRIVATE_DIR),         <---
                ('leapp-azure-sap-ha.repo', YUM_REPOS_PATH)       <---
            ],
        },

<...output skipped...>
~~~

While "leapp-rhui-azure-sap-1.0.0-6.el7.noarch" provides below files which are different from the above. 
~~~
# rpm -qlv leapp-rhui-azure-sap
-rw-r--r--    1 root    root                     3537 Feb 21 10:33 /usr/share/leapp-repository/repositories/system_upgrade/common/files/rhui/azure-sap-apps/content-rhel8-eus.crt
-rw-r--r--    1 root    root                     3642 Feb 21 10:33 /usr/share/leapp-repository/repositories/system_upgrade/common/files/rhui/azure-sap-apps/content-rhel8-sapapps.crt
-rw-r--r--    1 root    root                     1917 Feb 21 10:33 /usr/share/leapp-repository/repositories/system_upgrade/common/files/rhui/azure-sap-apps/key-rhel8-eus.pem
-rw-r--r--    1 root    root                     1917 Feb 21 10:33 /usr/share/leapp-repository/repositories/system_upgrade/common/files/rhui/azure-sap-apps/key-rhel8-sapapps.pem
-rw-r--r--    1 root    root                     9274 Feb 21 10:33 /usr/share/leapp-repository/repositories/system_upgrade/common/files/rhui/azure-sap-apps/leapp-azure-sap-apps.repo
-rw-r--r--    1 root    root                     8082 Feb 21 10:33 /usr/share/leapp-repository/repositories/system_upgrade/common/files/rhui/azure-sap/content-rhel8-sap-ha.crt
-rw-r--r--    1 root    root                     1917 Feb 21 10:33 /usr/share/leapp-repository/repositories/system_upgrade/common/files/rhui/azure-sap/key-rhel8-sap-ha.pem
-rw-r--r--    1 root    root                    12689 Feb 21 10:33 /usr/share/leapp-repository/repositories/system_upgrade/common/files/rhui/azure-sap/leapp-azure-sap.repo
~~~

I looked up old version of leapp (leapp-upgrade-el7toel8-0.17.0-10.el7_9.noarch) and the file is looks for matches this above "leapp-rhui-azure-sap" package.
~~~
# The files in 'files_map' are provided by special Leapp rpms (per cloud) and # are delivered into "repos/system_upgrade/common/files/rhui/<PROVIDER>


RHUI_CLOUD_MAP = {
    '7to8': {

<...output skipped...>

        'azure': {
            'src_pkg': 'rhui-azure-rhel7',
            'target_pkg': 'rhui-azure-rhel8',
            'agent_pkg': 'WALinuxAgent',
            'leapp_pkg': 'leapp-rhui-azure',
            'leapp_pkg_repo': 'leapp-azure.repo',
            'files_map': [
                ('content.crt', RHUI_PKI_PRODUCT_DIR),
                ('key.pem', RHUI_PKI_PRIVATE_DIR),
                ('leapp-azure.repo', YUM_REPOS_PATH)
            ],
        },
        'azure-sap-apps': {
            'src_pkg': 'rhui-azure-rhel7-base-sap-apps',
            'target_pkg': 'rhui-azure-rhel8-sapapps',
            'agent_pkg': 'WALinuxAgent',
            'leapp_pkg': 'leapp-rhui-azure-sap',
            'leapp_pkg_repo': 'leapp-azure-sap-apps.repo',
            'files_map': [
                ('content-rhel8-eus.crt', RHUI_PKI_PRODUCT_DIR),     <---
                ('content-rhel8-sapapps.crt', RHUI_PKI_PRODUCT_DIR), <---
                ('key-rhel8-eus.pem', RHUI_PKI_DIR),                 <---
                ('key-rhel8-sapapps.pem', RHUI_PKI_DIR),             <---
                ('leapp-azure-sap-apps.repo', YUM_REPOS_PATH),       <---
            ],
        },
        'azure-sap': {
            'src_pkg': 'rhui-azure-rhel7-base-sap-ha',
            'target_pkg': 'rhui-azure-rhel8-sap-ha',
            'agent_pkg': 'WALinuxAgent',
            'leapp_pkg': 'leapp-rhui-azure-sap',
            'leapp_pkg_repo': 'leapp-azure-sap.repo',
            'files_map': [
                ('content-rhel8-sap-ha.crt', RHUI_PKI_PRODUCT_DIR),  <---
                ('key-rhel8-sap-ha.pem', RHUI_PKI_DIR),              <---
                ('leapp-azure-sap.repo', YUM_REPOS_PATH)             <---
            ],
        },
~~~



This must mean that with the latest version of leapp (leapp-upgrade-el7toel8-0.18.0-1.el7_9.noarch) the files which are required are content-sapapps.crt , content-sap-ha.crt , etc
But that "leapp-rhui-azure-sap-1.0.0-6.el7.noarch" is providing the file compatible with old version of leapp (leapp-upgrade-el7toel8-0.17.0-10.el7_9.noarch)
This  must mean that there should be a new version of "leapp-rhui-azure-sap" which would be compatible with latest version of leapp mapping
However we confirmed with Customer and Microsoft over the support case that the "leapp-rhui-azure-sap-1.0.0-6.el7.noarch" is already the latest available.
~~~
# yum list --showduplicates leapp-rhui-azure-sap
Loaded plugins: product-id, search-disabled-repos, subscription-manager

This system is not registered with an entitlement server. You can use subscription-manager to register.
Installed Packages
leapp-rhui-azure-sap.noarch                             1.0.0-6.el7                              @rhui-microsoft-azure-rhel7-base-sap-ha
Available Packages
leapp-rhui-azure-sap.noarch                             1.0.0-1.el7                              rhui-microsoft-azure-rhel7-base-sap-ha
leapp-rhui-azure-sap.noarch                             1.0.0-3.el7                              rhui-microsoft-azure-rhel7-base-sap-ha
leapp-rhui-azure-sap.noarch                             1.0.0-5.el7                              rhui-microsoft-azure-rhel7-base-sap-ha
leapp-rhui-azure-sap.noarch                             1.0.0-6.el7                              rhui-microsoft-azure-rhel7-base-sap-ha


This lists the repo:
repo id                                                   repo name      status
rhel-ha-for-rhel-7-server-rhui-rpms/7Server/x86_64        Red Hat Enterp    865
rhel-sap-for-rhel-7-server-rhui-rpms/7Server/x86_64       Red Hat Enterp    145
rhel-sap-hana-for-rhel-7-server-rhui-rpms/7Server/x86_64  Red Hat Enterp    113
rhui-microsoft-azure-rhel7                                Microsoft Azur     13
rhui-microsoft-azure-rhel7-base-sap-ha                    Microsoft Azur     10
rhui-rhel-7-server-dotnet-rhui-rpms/x86_64                dotNET on RHEL  1,261
rhui-rhel-7-server-rhui-extras-rpms/x86_64                Red Hat Enterp  1,459
rhui-rhel-7-server-rhui-rh-common-rpms/7Server/x86_64     Red Hat Enterp    243
rhui-rhel-7-server-rhui-rpms/7Server/x86_64               Red Hat Enterp 33,760
rhui-rhel-7-server-rhui-supplementary-rpms/7Server/x86_64 Red Hat Enterp    479
rhui-rhel-server-rhui-rhscl-7-rpms/7Server/x86_64         Red Hat Softwa 14,520
repolist: 52,868
 
And it's showing the latest version: 1.0.0.6.el7
# yum info leapp-rhui-azure-sap
Loaded plugins: langpacks, product-id, search-disabled-repos
Available Packages
Name        : leapp-rhui-azure-sap
Arch        : noarch
Version     : 1.0.0
Release     : 6.el7
Size        : 14 k
Repo        : rhui-microsoft-azure-rhel7-base-sap-ha
Summary     : Leapp in-place upgrade cloud specific package (SAP)
URL         : http://redhat.com
License     : LGPLv2+
Description : Leapp in-place upgrade of SAP on Azure
~~~





Version-Release number of selected component (if applicable):
leapp-upgrade-el7toel8-0.18.0-1.el7_9.noarch
leapp-rhui-azure-sap-1.0.0.6.el7.noarch


How reproducible:
Every time


Steps to Reproduce:
Following the inplace upgrade path for SAP HANA - https://access.redhat.com/solutions/5154031

     
1. Install the latest version of leapp-rhui-azure-sap package

      # yum install leapp-rhui-azure-sap

2. Enable the rhui-rhel-7-server-rhui-extras-rpms repository and install latest version of leapp

      # yum-config-manager --enable rhui-rhel-7-server-rhui-extras-rpms

      # yum install leapp-upgrade

3. Perform Preupgrade on the machine and clear any inhibitor which might be occuring like download of leapp-data-22.tar.gz or any other inhibitor

      # leapp preupgrade --debug
  

4. After clearing all the inhibitors when we finally run the preupgrade again it fails during the "targetuserspacecreator" actor

      # leapp preupgrade --debug



Actual results:
Leapp preupgrade fails with this error while copying a non-existent cert into the temporary upgrade environment its building due to the files in the RHUI_CLOULD_MAP mapping change in the recent version of leapp

Expected results:
Leapp preupgrade complete successfully

Additional info:

Comment 4 Michal Hečko 2023-06-12 10:58:40 UTC
Hi Ameya,

thank you for investigating the issue and producing a very helpful bug report. We are aware of the root cause of the issue and we are already working
with the Azure RHUI team on deploying a new version of the leapp-rhui-azure-sap to production. I will update this BZ once the new version is deployed.

Comment 6 Novonil Choudhuri 2023-06-14 16:30:41 UTC
Michal,

This is highly escalated from Microsoft now. Below is the email we received today. Can you please throw some light on this and update this bug. 

////
From Simrat.Satia:

Looping in RedHat Leapp Team: @Michal Hecko @Michal Reznik @Irina Gulina @Martin Kluson

 

I understand that a new version of leapp is underway but how is the existing functionality of previous leapp package affected?
Was there any change made in leapp utility recently that affected the in-place SAP upgrade of RHEL7 to RHEL8?
 

Bz: https://bugzilla.redhat.com/show_bug.cgi?id=2214133

As per Michal: “We are aware of the root cause of the issue and we are already working with the Azure RHUI team on deploying a new version of the leapp-rhui-azure-sap to production.”
Our team has not been made aware of any such issue. 
Please provide insight into the same and how it will be fixed.
///

thanks,
novo