Bug 1668774 - OSP Director - handle lowercase uuid change for dmidecode >= 3.1 for ceph-ansible per-node customizations
Summary: OSP Director - handle lowercase uuid change for dmidecode >= 3.1 for ceph-ans...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-common
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: z5
: 13.0 (Queens)
Assignee: Francesco Pantano
QA Contact: Yogev Rabl
URL:
Whiteboard:
Depends On: 1671049
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-01-23 14:59 UTC by Matt Flusche
Modified: 2022-03-13 16:50 UTC (History)
11 users (show)

Fixed In Version: openstack-tripleo-common-8.6.6-16.el7ost
Doc Type: Bug Fix
Doc Text:
This bug was caused by updated versions of dmidecode 3.1 or later that returned system UUIDs in lowercase. As a consequence, systems deployed with per node ceph-ansible customization prior to this version can break if UUID case mismatches and cause deployment failures. This fix updates the `openstack-tripleo-common` package to accept uppercase or lowercase UUIDs. Forced lowercase on dmidecode output make the code case insensitive.
Clone Of:
: 1671049 1671065 (view as bug list)
Environment:
Last Closed: 2019-03-14 13:55:11 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1814070 0 None None None 2019-01-31 10:35:54 UTC
OpenStack gerrit 635193 0 None MERGED Handle case change for dmidecode >= 3.1 in ceph-ansible workbook 2020-08-12 10:57:21 UTC
Red Hat Product Errata RHBA-2019:0448 0 None None None 2019-03-14 13:55:16 UTC

Description Matt Flusche 2019-01-23 14:59:51 UTC
Description of problem:
dmidecode >= 3.1 now displays uuid in lowercase.

https://github.com/mirror/dmidecode/blob/master/NEWS

Systems deployed with per node ceph-ansible customization prior this version may break during patching and cause deployment failures.  An example would be this failure:

openstack stack resource list -n5 overcloud | grep -v COMPLETE 
+-----------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
| resource_name                           | physical_resource_id                                                                                                                                                                 | resource_type                                                                                                                             | resource_status | updated_time         | stack_name                                                                                                                                               |
+-----------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
| AllNodesDeploySteps                     | 129027a6-c80c-43dd-ad8e-8a847a023a3a                                                                                                                                                 | OS::TripleO::PostDeploySteps                                                                                                              | UPDATE_FAILED   | 2019-01-17T20:50:17Z | overcloud                                                                                                                                                |
| WorkflowTasks_Step2_Execution           | 5451ca1d-de57-42c7-9107-24af27a75a66                                                                                                                                                 | OS::TripleO::WorkflowSteps                                                                                                                | CREATE_FAILED   | 2019-01-17T20:53:07Z | overcloud-AllNodesDeploySteps-zh5jioeqopmc                                                                                                               |
+-----------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+

ceph-install-workflow.log
2019-01-17 19:09:35,976 p=22329 u=mistral |  failed: [192.168.24.15] (item=/dev/vdb) => {"changed": false, "err": "Error: Could not stat device /dev/vdb - No such file or directory.\n", "failed": true, "item": "/dev/vdb", "msg": "Error while getting device information with parted script: '/sbin/parted -s -m /dev/vdb -- unit 'MiB' print'", "out": "", "rc": 1}


The puppet hiera data per node issue is addressed here so the same is needed for ceph-ansible:

https://bugzilla.redhat.com/show_bug.cgi?id=1654449


Version-Release number of selected component (if applicable):
Current versions

How reproducible:
100%

Steps to Reproduce:
1. Deploy with dmidecode <= 3.1 with per node customizations
2. upgrade systems to dmidecode >= 3.1
3. observe deployment failure or global config applied where per node config was configured.


Additional info:

$ rpm -q dmidecode
dmidecode-3.0-5.el7.x86_64
$ sudo dmidecode -s system-uuid
A6F05964-0E55-42C9-938E-1E657C49EFDB
$ sudo yum -y update dmidecode
...
Updated:
  dmidecode.x86_64 1:3.1-2.el7                                                                                                                                                 

Complete!
$ sudo dmidecode -s system-uuid
a6f05964-0e55-42c9-938e-1e657c49efdb

Comment 1 Giulio Fidente 2019-01-30 16:09:26 UTC
it looks like this was an issue introduced in dmidecode 3.1 and fixed starting from 3.2, see https://github.com/mirror/dmidecode/commit/aec83995082070c47edf394c4b7f9f17fea7fd16 and https://savannah.nongnu.org/bugs/index.php?53569

we'll probably need to ensure our code makes case-insesitive matching anyway

Comment 10 Eliad Cohen 2019-03-01 15:39:46 UTC
Seems like the relevant fix is not in the code base for osp 13.
Puddle used was 2019-02-25.2
The file /usr/share/openstack-tripleo-heat-templates/docker/services/ceph-ansible/ceph-base.yaml is missing the new code as per https://github.com/openstack/tripleo-heat-templates/commit/c01d9d8475d7b256fed944227a06f028e8d487f8

Comment 11 Eliad Cohen 2019-03-01 16:35:01 UTC
Clarification. Fix is in code base, other code may be missing, resulting in the following error upon deployment:

2019-02-28 19:36:53Z [overcloud]: CREATE_FAILED  Resource CREATE failed: resources.WorkflowTasks_Step2_Execution: resources.AllNodesDeploySteps.Failure caused by error in tasks: ceph_base_ansible_workflow

  ceph_base_ansible_workflow [task_ex_id=d7318d1f-6f92-4f63-bd8a-8a72a6872eae] -> Failed to handle

 Stack overcloud CREATE_FAILED

overcloud.AllNodesDeploySteps.WorkflowTasks_Step2_Execution:
  resource_type: OS::TripleO::WorkflowSteps
  physical_resource_id: e2c5a650-2f64-4b0c-99a5-73eaac687aec
  status: CREATE_FAILED
  status_reason: |
    ...

      File "/usr/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 79, in _evaluate_item
        return evaluate(item, context)
      File "/usr/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 71, in evaluate
        return evaluator.evaluate(expression, context)
      File "/usr/lib/python2.7/site-packages/mistral/expressions/yaql_expression.py", line 119, in evaluate
        cls).evaluate(trim_expr, data_context)
      File "/usr/lib/python2.7/site-packages/mistral/expressions/yaql_expression.py", line 73, in evaluate
        ", data=%s]" % (expression, str(e), data_context)
    YaqlEvaluationException: Can not evaluate YAQL expression [expression=json_parse($.node_data_lookup), error=Expecting property name enclosed in double quotes: line 1 column 2 (char 1), data={}]

Comment 12 Eliad Cohen 2019-03-01 17:31:37 UTC
Returned to QA - Deployment should be done with NodeDataLookup as json string in osp13, not json.
Retest

Comment 13 Eliad Cohen 2019-03-01 20:00:16 UTC
Tested with NodeDataLookup passed as json string and verified as fixed

Comment 16 errata-xmlrpc 2019-03-14 13:55:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0448


Note You need to log in before you can comment on or make changes to this bug.