Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1978779

Summary:

SkipRhelEnforcement appears to be ignored when upgrading controller from 16.1 to 16.2

Product:

Red Hat OpenStack

Reporter:

James Parker <jparker>

Component:

openstack-heat-templates

Assignee:

Sofer Athlan-Guyot <sathlang>

Status:

CLOSED NOTABUG

QA Contact:

James Parker <jparker>

Severity:

medium

Docs Contact:

Priority:

high

Version:

16.2 (Train)

CC:

amodi, apevec, jfrancoa, jpretori, kchamart, lhh, mbultel, pkesavar, pkomarov, sathlang

Target Milestone:

---

Keywords:

Triaged

Target Release:

---

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2021-07-06 09:17:49 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

Bug Blocks:

1965124

Attachments:

Description	Flags
Upgrade procedure utilized	none

Description James Parker 2021-07-02 17:41:32 UTC

Created attachment 1797275 [details]
Upgrade procedure utilized

Description of problem:
While attempting to verify [1], upgrade is failing due to the system not having a RHEL subscription.  This follow up verification is following the same procedure described in [2], setting SkipRhelEnforcement for both the Undercloud and Overcloud.  During controller upgrade it still attempts to check for a subscription.

    (undercloud) [stack@undercloud-0 ~]$ cat virt/custom-undercloud-params.yaml
    parameter_defaults:
        SkipRhelEnforcement: true
        DnfStreams: [{'module':'container-tools', 'stream':'2.0'}]
     
    (undercloud) [stack@undercloud-0 ~]$ cat virt/config_heat.yaml
    parameter_defaults:
        SkipRhelEnforcement: true
        DnfStreams: [{'module':'container-tools', 'stream':'2.0'}]
        ServiceNetMap:
            NovaLibvirtNetwork: ctlplane
            NovaVncProxyNetwork: ctlplane
     
    (undercloud) [stack@undercloud-0 ~]$ cat overcloud_update_prepare.sh
    #!/bin/bash
     
    openstack overcloud update prepare -y \
    --timeout 100 \
    --templates /usr/share/openstack-tripleo-heat-templates \
    --stack overcloud \
    --libvirt-type kvm \
    --ntp-server clock1.rdu2.redhat.com \
    -e /home/stack/virt/config_lvm.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
    -e /home/stack/virt/network/network-environment.yaml \
    -e /home/stack/virt/network/dvr-override.yaml \
    -e /home/stack/virt/inject-trust-anchor.yaml \
    -e /home/stack/virt/hostnames.yml \
    -e /home/stack/virt/debug.yaml \
    -e /home/stack/virt/config_heat.yaml \
    -e /home/stack/virt/nodes_data.yaml \
    -e ~/containers-prepare-parameter.yaml \
    -e /home/stack/virt/set-nova-scheduler-filter.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-everywhere-endpoints-dns.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-internal-tls.yaml \
    -e /home/stack/virt/cloud-names.yaml \
    -e /home/stack/virt/public_vip.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/services/haproxy-public-tls-certmonger.yaml \
    --log-file overcloud_deployment_90.log
     
    (undercloud) [stack@undercloud-0 ~]$ bash ./overcloud_update_prepare.sh
    (undercloud) [stack@undercloud-0 ~]$ openstack overcloud external-update run -y --tags container_image_prepare
    (undercloud) [stack@undercloud-0 ~]$ openstack overcloud update run -y --stack overcloud --limit Controller --playbook all
    ...
    2021-07-02 15:35:23.689308 | 52540080-0b2d-4813-ecc9-000000000446 |    SKIPPED | Delete if symlink | controller-0
    2021-07-02 15:35:23.689905 | 52540080-0b2d-4813-ecc9-000000000446 |     TIMING | Delete if symlink | controller-0 | 0:00:24.476780 | 0.06s
    2021-07-02 15:35:23.703961 | 52540080-0b2d-4813-ecc9-000000000447 |       TASK | Enforce RHOSP rules regarding subscription.
    2021-07-02 15:35:23.759324 | 52540080-0b2d-4813-ecc9-000000000447 |     TIMING | Enforce RHOSP rules regarding subscription. | controller-0 | 0:00:24.546213 | 0.05s
    2021-07-02 15:35:23.805582 | 52540080-0b2d-4813-ecc9-00000000059f |       TASK | Gather variables for each operating system
    2021-07-02 15:35:23.861421 | 52540080-0b2d-4813-ecc9-00000000059f |         OK | Gather variables for each operating system | controller-0
    2021-07-02 15:35:23.862132 | 52540080-0b2d-4813-ecc9-00000000059f |     TIMING | tripleo-redhat-enforce : Gather variables for each operating system | controller-0 | 0:00:24.649020 | 0.06s
    2021-07-02 15:35:23.875776 | 52540080-0b2d-4813-ecc9-0000000005a0 |       TASK | Enforce RHEL/OSP version pair
    2021-07-02 15:35:23.931395 | 52540080-0b2d-4813-ecc9-0000000005a0 |     TIMING | tripleo-redhat-enforce : Enforce RHEL/OSP version pair | controller-0 | 0:00:24.718278 | 0.06s
    2021-07-02 15:35:23.949701 | 361e6fb7-e7e2-415d-b67e-5cebe503a1a5 |   INCLUDED | /usr/share/ansible/roles/tripleo-redhat-enforce/tasks/enforce_release.yml | controller-0
    2021-07-02 15:35:23.965105 | 52540080-0b2d-4813-ecc9-0000000005fc |       TASK | get current release settings
    2021-07-02 15:35:25.428673 | 52540080-0b2d-4813-ecc9-0000000005fc |    IGNORED | get current release settings | controller-0 | error={"attempts": 1, "changed": true, "cmd": ["subscription-manager", "release", "--show"], "delta": "0:00:01.034028", "end": "2021-07-02 15:35:25.374359", "msg": "non-zero return code", "rc": 1, "start": "2021-07-02 15:35:24.340331", "stderr": "This system is not yet registered. Try 'subscription-manager register --help' for more information.", "stderr_lines": ["This system is not yet registered. Try 'subscription-manager register --help' for more information."], "stdout": "", "stdout_lines": []}
    2021-07-02 15:35:25.429742 | 52540080-0b2d-4813-ecc9-0000000005fc |     TIMING | tripleo-redhat-enforce : get current release settings | controller-0 | 0:00:26.216632 | 1.46s
    2021-07-02 15:35:25.445735 | 52540080-0b2d-4813-ecc9-0000000005fd |       TASK | fails if not registered
    2021-07-02 15:35:25.501796 | 52540080-0b2d-4813-ecc9-0000000005fd |      FATAL | fails if not registered | controller-0 | error={"changed": false, "msg": "Your environment is not subscribed! If it is expected, please set SkipRhelEnforcement to true. For Director the documentation is there https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.0/html-single/ director_installation_and_usage/index#configuring-the-undercloud-with-environment-files, for the Overcloud you need to add a new parameter file to your deploy command with that parameter set. You can also disable it in the role, see rhsm_enforce role parameter. If this is unexpected, you have to subscribe this node and ensure that RHEL is pinned to 8.4 as this is the only version supported for 16.2."}
    2021-07-02 15:35:25.502610 | 52540080-0b2d-4813-ecc9-0000000005fd |     TIMING | tripleo-redhat-enforce : fails if not registered | controller-0 | 0:00:26.289474 | 0.06s


[1] https://bugzilla.redhat.com/show_bug.cgi?id=1965124
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1965124#c0

Version-Release number of selected component (if applicable):
(undercloud) [stack@undercloud-0 ~]$ cat core_puddle_version 
RHOS-16.1-RHEL-8-20210604.n.0

How reproducible:
100%

Steps to Reproduce:
1. Deploy a 16.1 TLS-e environment
2. Upgrade the deployment based on procedure listed in [2]


Actual results:
Controller upgrade fails due to rhel subscription check

Expected results:
Upgrade should skip rhel subscription check

Additional info:
Full procedure, minus output is attached

Comment 1 Sofer Athlan-Guyot 2021-07-05 10:51:04 UTC

Hi James,

so I check on one of my env and in ci and the parameter seems to work
just fine.

So let's try to check what's different.

In your environment according to your description we have:

> (undercloud) [stack@undercloud-0 ~]$ cat core_puddle_version 
> RHOS-16.1-RHEL-8-20210604.n.0

That shouldn't be, it should be:

  [stack@undercloud-0 ~]$ cat core_puddle_version
  RHOS-16.2-RHEL-8-20210630.n.0

So it looks like the undercloud wasn't updated, or did you log the
wrong puddle ? The templates version should be:

  openstack-tripleo-heat-templates-11.5.1-2.20210603174816.el8ost.4.noarch

Here a couple of commands that would help to make sure everything is
alright before running an update run.

Just before you run (which is what I call "update run")

  openstack overcloud update run -y --stack overcloud --limit Controller --playbook all

could you run:

  openstack --os-cloud undercloud stack environment show overcloud |grep -i skiprhel

So we make sure the parameter is indeed there and taken into account by heat.

Also we can check what config-download has to say about the generated
ansibles tasks, so at the same time (after prepare and before the
update run)

  openstack --os-cloud undercloud overcloud config download --config-dir tripleo-conf
  grep -i -A6 -r 'Enforce RHOSP rules regarding subscription' tripleo-conf/

The output should look like this (edited here)

  tripleo-conf/qe-Cloud-0/Controller/update_tasks.yaml:  name: Enforce RHOSP rules regarding subscription.
  tripleo-conf/qe-Cloud-0/Controller/update_tasks.yaml-  vars:
  tripleo-conf/qe-Cloud-0/Controller/update_tasks.yaml-    skip_rhel_enforcement: 'True'
  tripleo-conf/qe-Cloud-0/Controller/update_tasks.yaml-  when:
  tripleo-conf/qe-Cloud-0/Controller/update_tasks.yaml-  - step|int == 0
  tripleo-conf/qe-Cloud-0/Controller/update_tasks.yaml-  - ansible_facts['distribution'] == 'RedHat'
  tripleo-conf/qe-Cloud-0/Controller/update_tasks.yaml-  - not (skip_rhel_enforcement | bool)

See how the skip_rhel_enforcement should be 'True'

For reference here's the CI setup (but it looks like your setup, so
nothing really new here):

  [stack@undercloud-0 ~]$ cat /home/stack/skip_rhel_release.yaml
  parameter_defaults:
      SkipRhelEnforcement: true
  
  openstack overcloud update prepare ${PREPARE_ANSWER} \
      --templates /usr/share/openstack-tripleo-heat-templates \
      --stack qe-Cloud-0 \
      -e /usr/share/openstack-tripleo-heat-templates/environments/cinder-backup.yaml \
      -e /home/stack/virt/config_lvm.yaml \
      -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
      -e /home/stack/virt/network/network-environment.yaml \
      -e /home/stack/virt/network/dvr-override.yaml \
      -e /home/stack/virt/enable-tls.yaml \
      -e /home/stack/virt/inject-trust-anchor.yaml \
      -e /home/stack/virt/public_vip.yaml \
      -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \
      -e /home/stack/virt/hostnames.yml \
      -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-dvr-ha.yaml \
      -e /home/stack/virt/debug.yaml \
      -e /home/stack/virt/nodes_data.yaml \
      -e ~/containers-prepare-parameter.yaml \
      -e /home/stack/cli_opts_params.yaml \
      -e /home/stack/skip_rhel_release.yaml \
      -e /home/stack/virt/config_heat_extra.yaml \
      2>&1

Comment 4 Sofer Athlan-Guyot 2021-07-06 09:17:49 UTC

Hi,

so I looked at the env and did:

openstack --os-cloud undercloud stack environment show overcloud > env.txt
[stack@undercloud-0 ~]$ grep Skip env.txt                                                                                                                                                                           │···········································
  SkipRhelEnforcement: true   

So that part is good, heat environment know about it.

Then I config-download the resulting playbook:

openstack --os-cloud undercloud overcloud config download --config-dir foobar

stack@undercloud-0 ~]$ grep -r skip_rhel_enf foobar/overcloud/*/update_*                         
foobar/overcloud/BlockStorage/update_tasks_step0.yaml:    skip_rhel_enforcement: 'True'          
foobar/overcloud/BlockStorage/update_tasks_step0.yaml:  - not (skip_rhel_enforcement | bool)     
foobar/overcloud/BlockStorage/update_tasks.yaml:    skip_rhel_enforcement: 'True'                
foobar/overcloud/BlockStorage/update_tasks.yaml:  - not (skip_rhel_enforcement | bool)           
foobar/overcloud/CephStorage/update_tasks_step0.yaml:    skip_rhel_enforcement: 'True'           
foobar/overcloud/CephStorage/update_tasks_step0.yaml:  - not (skip_rhel_enforcement | bool)      
foobar/overcloud/CephStorage/update_tasks.yaml:    skip_rhel_enforcement: 'True'                 
foobar/overcloud/CephStorage/update_tasks.yaml:  - not (skip_rhel_enforcement | bool)            
foobar/overcloud/Compute/update_tasks_step0.yaml:    skip_rhel_enforcement: 'True'               
foobar/overcloud/Compute/update_tasks_step0.yaml:  - not (skip_rhel_enforcement | bool)          
foobar/overcloud/Compute/update_tasks.yaml:    skip_rhel_enforcement: 'True'                     
foobar/overcloud/Compute/update_tasks.yaml:  - not (skip_rhel_enforcement | bool)                
foobar/overcloud/Controller/update_tasks_step0.yaml:    skip_rhel_enforcement: 'True'            
foobar/overcloud/Controller/update_tasks_step0.yaml:  - not (skip_rhel_enforcement | bool)       
foobar/overcloud/Controller/update_tasks.yaml:    skip_rhel_enforcement: 'True'                  
foobar/overcloud/Controller/update_tasks.yaml:  - not (skip_rhel_enforcement | bool)             
foobar/overcloud/ObjectStorage/update_tasks_step0.yaml:    skip_rhel_enforcement: 'True'         
foobar/overcloud/ObjectStorage/update_tasks_step0.yaml:  - not (skip_rhel_enforcement | bool)    
foobar/overcloud/ObjectStorage/update_tasks.yaml:    skip_rhel_enforcement: 'True'               
foobar/overcloud/ObjectStorage/update_tasks.yaml:  - not (skip_rhel_enforcement | bool)          

All settings are correct here.

But when I look at what was run in /var/lib/mistral:

root@undercloud-0 3328b781-c012-485f-a0a7-871842e71809]# grep -r skip_rhel_enf */update_*         
BlockStorage/update_tasks_step0.yaml:    skip_rhel_enforcement: 'false'                         
BlockStorage/update_tasks_step0.yaml:  - not (skip_rhel_enforcement | bool)                     
BlockStorage/update_tasks.yaml:    skip_rhel_enforcement: 'false'                               
BlockStorage/update_tasks.yaml:  - not (skip_rhel_enforcement | bool)                           
CephStorage/update_tasks_step0.yaml:    skip_rhel_enforcement: 'false'                          
CephStorage/update_tasks_step0.yaml:  - not (skip_rhel_enforcement | bool)                      
CephStorage/update_tasks.yaml:    skip_rhel_enforcement: 'false'                                
CephStorage/update_tasks.yaml:  - not (skip_rhel_enforcement | bool)                            
Compute/update_tasks_step0.yaml:    skip_rhel_enforcement: 'false'                              
Compute/update_tasks_step0.yaml:  - not (skip_rhel_enforcement | bool)                          
Compute/update_tasks.yaml:    skip_rhel_enforcement: 'false'                                    
Compute/update_tasks.yaml:  - not (skip_rhel_enforcement | bool)                                
Controller/update_tasks_step0.yaml:    skip_rhel_enforcement: 'false'                           
Controller/update_tasks_step0.yaml:  - not (skip_rhel_enforcement | bool)                       
Controller/update_tasks.yaml:    skip_rhel_enforcement: 'false'                                 
Controller/update_tasks.yaml:  - not (skip_rhel_enforcement | bool)                             
ObjectStorage/update_tasks_step0.yaml:    skip_rhel_enforcement: 'false'                        
ObjectStorage/update_tasks_step0.yaml:  - not (skip_rhel_enforcement | bool)                    
ObjectStorage/update_tasks.yaml:    skip_rhel_enforcement: 'false'                              
ObjectStorage/update_tasks.yaml:  - not (skip_rhel_enforcement | bool)                          

The can be explained if the prepare was run after the update run or the initial prepare didn't run to completion.

Note that at the end of the update prepare we had this issue:

2021-07-05 19:51:31.692 16458 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare [-] Roles in the following list are expected to have a TSX flag configured in their KernelArgs parameter. For more informat
ion on why we must explicitly define the TSX flag, please visit: https://access.redhat.com/solutions/6036141ESC[00m                                                                                                 
2021-07-05 19:51:31.693 16458 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare [-] You can also skip this validation by setting ForceNoTsx parameter for the desired role(s)ESC[00m                       
2021-07-05 19:51:31.693 16458 ERROR tripleoclient.v1.overcloud_update.MinorUpdatePrepare [-] Impacted roles: ComputeESC[00m                                                                                         
2021-07-05 19:51:31.693 16458 ERROR openstack [-] : tripleoclient.exceptions.PostStackValidationErrorESC[00m                                                                                                        
2021-07-05 19:51:31.696 16458 INFO osc_lib.shell [-] END return value: 1ESC[00m                                                                  

It's a new validation for the TSX flag that must be enabled, see https://bugzilla.redhat.com/show_bug.cgi?id=1975240 for more.

There is another other issue with the env. The repositories were not updated to rhel 8.4:
both 16.1 and 16.2 are still enabled and we have rhel which is still on 8.2 instead of 8.4.


To make extra sure that the skiprhelenforcement is indeed taken into account I re-run:

openstack overcloud update run -y --stack overcloud --limit Controller --playbook all

and Ctl-C it after the playbook were generated.

Then:

cd /var/lib/mistral/ec821ccf-5f49-4330-9b20-b8c5457b5fef  

[root@undercloud-0 ec821ccf-5f49-4330-9b20-b8c5457b5fef]# grep -r skip_rhel_enf */update_*        
BlockStorage/update_tasks_step0.yaml:    skip_rhel_enforcement: 'True'                          
BlockStorage/update_tasks_step0.yaml:  - not (skip_rhel_enforcement | bool)                     
BlockStorage/update_tasks.yaml:    skip_rhel_enforcement: 'True'                                
BlockStorage/update_tasks.yaml:  - not (skip_rhel_enforcement | bool)                           
CephStorage/update_tasks_step0.yaml:    skip_rhel_enforcement: 'True'                           
CephStorage/update_tasks_step0.yaml:  - not (skip_rhel_enforcement | bool)                      
CephStorage/update_tasks.yaml:    skip_rhel_enforcement: 'True'                                 
CephStorage/update_tasks.yaml:  - not (skip_rhel_enforcement | bool)                            
Compute/update_tasks_step0.yaml:    skip_rhel_enforcement: 'True'                               
Compute/update_tasks_step0.yaml:  - not (skip_rhel_enforcement | bool)                          
Compute/update_tasks.yaml:    skip_rhel_enforcement: 'True'                                     
Compute/update_tasks.yaml:  - not (skip_rhel_enforcement | bool)                                
Controller/update_tasks_step0.yaml:    skip_rhel_enforcement: 'True'                            
Controller/update_tasks_step0.yaml:  - not (skip_rhel_enforcement | bool)                       
Controller/update_tasks.yaml:    skip_rhel_enforcement: 'True'                                  
Controller/update_tasks.yaml:  - not (skip_rhel_enforcement | bool)                             
ObjectStorage/update_tasks_step0.yaml:    skip_rhel_enforcement: 'True'                         
ObjectStorage/update_tasks_step0.yaml:  - not (skip_rhel_enforcement | bool)                    
ObjectStorage/update_tasks.yaml:    skip_rhel_enforcement: 'True'                               
ObjectStorage/update_tasks.yaml:  - not (skip_rhel_enforcement | bool)


The parameter is correctly taken into account.

This confirms that the first run happened without the parameter taken into account by heat. 

So, the skip parameter works fine, but the update procedure is incomplete, we can help setting up a jenkins job in the future for 161-162 update if needed.

For the original problem in bz https://bugzilla.redhat.com/show_bug.cgi?id=1965124, PIDONE say it's safe to move it to verified.