Bug 2079264 - Upgrade [OSP16.2 -> OSP17.1] openstack overcloud network provision fails
Summary: Upgrade [OSP16.2 -> OSP17.1] openstack overcloud network provision fails
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: tripleo-ansible
Version: 17.1 (Wallaby)
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: beta
: 17.1
Assignee: Harald Jensås
QA Contact: Archana Singh
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-04-27 10:12 UTC by Juan Badia Payno
Modified: 2023-08-16 01:11 UTC (History)
4 users (show)

Fixed In Version: tripleo-ansible-3.3.1-0.20220706003125.fa5422f.el8ost openstack-tripleo-heat-templates-14.3.1-0.20220518231249.3965818.el9ost
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-08-16 01:11:06 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1970604 0 None Closed Cannot Run a VM 2022-06-03 12:44:19 UTC
OpenStack gerrit 839512 0 None MERGED Don't extract dns_domain when not set 2022-06-22 14:16:23 UTC
OpenStack gerrit 839911 0 None MERGED Set dns_name propery on ports and networks 2022-07-04 14:04:27 UTC
OpenStack gerrit 848599 0 None MERGED Don't extract dns_domain when not set 2022-07-11 12:43:23 UTC
Red Hat Issue Tracker OSP-14899 0 None None None 2022-04-27 10:15:09 UTC
Red Hat Product Errata RHEA-2023:4577 0 None None None 2023-08-16 01:11:35 UTC

Description Juan Badia Payno 2022-04-27 10:12:46 UTC
Executing the "openstack overcloud network provision" failed with the following output "Failed schema validation at dns_domain:\n    None is not of type 'string'"

How to reproduce:
 Deploy osp16.2
 Upgrade the undercloud to osp17
 Provisioning the network

Comment 1 Juan Badia Payno 2022-04-27 10:13:42 UTC
openstack --debug overcloud -vvv network provision --output jbp/generated-networks-deployed.yaml ~/jbp/tripleo-qe-Cloud-0-network-data.yaml 
START with options: --debug overcloud -vvv network provision --output jbp/generated-networks-deployed.yaml /home/stack/jbp/tripleo-qe-Cloud-0-network-data.yaml
...
...
...
2022-04-27 09:39:39.167880 | 525400bf-b550-cdbb-e797-00000000000e |     TIMING | Create/Update composable networks | localhost | 0:00:06.290028 | 5.88s
2022-04-27 09:39:40.327160 | 525400bf-b550-cdbb-e797-00000000000e |      FATAL | Create/Update composable networks | localhost | item={'dns_domain': None, 'name': 'Management', 'name_lower': 'management', 'subnets': {'management_subnet': {'allocation_pools': [{'end': '10.0.1.250', 'start': '10.0.1.4'}], 'gateway_ip': '10.0.1.1', 'ip_subnet': '10.0.1.0/24', 'physical_network': 'management', 'routes': [], 'vlan': 60}}} | error={
    "ansible_index_var": "idx",
    "ansible_loop_var": "item",
    "changed": false,
    "idx": 5,
    "invocation": {
        "module_args": {
            "api_timeout": null,
            "auth": null,
            "auth_type": null,
            "availability_zone": null,
            "ca_cert": null,
            "client_cert": null,
            "client_key": null,
            "cloud": null,
            "idx": 5,
            "interface": "public",
            "net_data": {
                "dns_domain": null,
                "name": "Management",
                "name_lower": "management",
                "subnets": {
                    "management_subnet": {
                        "allocation_pools": [
                            {
                                "end": "10.0.1.250",
                                "start": "10.0.1.4"
                            }
                        ],
                        "gateway_ip": "10.0.1.1",
                        "ip_subnet": "10.0.1.0/24",
                        "physical_network": "management",
                        "routes": [],
                        "vlan": 60
                    }
                }
            },
            "region_name": null,
            "timeout": 180,
            "validate_certs": null,
            "wait": true
        }
    },
    "item": {
        "dns_domain": null,
        "name": "Management",
        "name_lower": "management",
        "subnets": {
            "management_subnet": {
                "allocation_pools": [
                    {
                        "end": "10.0.1.250",
                        "start": "10.0.1.4"
                    }
                ],
                "gateway_ip": "10.0.1.1",
                "ip_subnet": "10.0.1.0/24",
                "physical_network": "management",
                "routes": [],
                "vlan": 60
            }
        }
    },
    "msg": "Failed schema validation at dns_domain:\n    None is not of type 'string'"
}
2022-04-27 09:39:40.328325 | 525400bf-b550-cdbb-e797-00000000000e |     TIMING | Create/Update composable networks | localhost | 0:00:07.450475 | 7.04s
2022-04-27 09:39:40.333688 | 525400bf-b550-cdbb-e797-00000000000e |     TIMING | Create/Update composable networks | localhost | 0:00:07.455838 | 7.05s

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=1    skipped=3    rescued=0    ignored=0   
2022-04-27 09:39:40.339016 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2022-04-27 09:39:40.339311 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 6          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2022-04-27 09:39:40.339558 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:00:07.461714 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2022-04-27 09:39:40.339801 |                                 UUID |       Info |       Host |   Task Name |   Run Time
2022-04-27 09:39:40.340063 | 525400bf-b550-cdbb-e797-00000000000e |    SUMMARY |  localhost | Create/Update composable networks | 7.05s
2022-04-27 09:39:40.340385 | 525400bf-b550-cdbb-e797-00000000000a |    SUMMARY |  localhost | Check if network deployment file already exist | 0.23s
2022-04-27 09:39:40.340652 | 525400bf-b550-cdbb-e797-00000000000d |    SUMMARY |  localhost | Load config from file | 0.04s
2022-04-27 09:39:40.340956 | 525400bf-b550-cdbb-e797-000000000008 |    SUMMARY |  localhost | fail | 0.03s
2022-04-27 09:39:40.341214 | 525400bf-b550-cdbb-e797-00000000000b |    SUMMARY |  localhost | fail | 0.02s
2022-04-27 09:39:40.341467 | 525400bf-b550-cdbb-e797-000000000009 |    SUMMARY |  localhost | fail | 0.02s


(undercloud) [stack@undercloud-0 jbp]$ head -n 14  tripleo-qe-Cloud-0-network-data.yaml 
- dns_domain: null
  name: Storage
  name_lower: storage
  subnets:
    storage_subnet:
      allocation_pools:
      - end: 172.17.3.149
        start: 172.17.3.10
      ip_subnet: 172.17.3.0/24
      physical_network: storage
      routes: []
      vlan: 30
  vip: true
- dns_domain: null

(undercloud) [stack@undercloud-0 jbp]$ openstack network list 
/usr/lib64/python3.6/site-packages/_yaml/__init__.py:23: DeprecationWarning: The _yaml extension module is now located at yaml._yaml and its location is subject to change.  To use the LibYAML-based parser and emitter, import from `yaml`: `from yaml import CLoader as Loader, CDumper as Dumper`.
  DeprecationWarning
/usr/lib/python3.6/site-packages/barbicanclient/__init__.py:61: UserWarning: The secrets module is moved to barbicanclient/v1 directory, direct import of barbicanclient.secrets will be deprecated. Please import barbicanclient.v1.secrets instead.
  % (name, name, name))
+--------------------------------------+------------------+--------------------------------------+
| ID                                   | Name             | Subnets                              |
+--------------------------------------+------------------+--------------------------------------+
| 2d578e41-34cf-4507-b252-879ac28506a9 | storage          | ab6fd639-b0f7-4157-b48e-00ec00f63cca |
| 4b0aed86-bcf2-459d-9c36-7655ec55a9a5 | external         | e4d644d4-45d4-43c4-a326-ac2c3566e32f |
| 66a62fc1-677c-42dd-bc99-33983efdc42c | storage_mgmt     | cdf9d597-aee4-4029-a97a-cd85f7835e59 |
| 761bf950-c22a-4faf-be54-a471cd0a6fec | management       | d040eae8-e5cd-4dd2-a222-c209016e9410 |
| 815314d4-5460-4662-9754-fae0f0936611 | ctlplane         | 32e28c03-aa50-4f0a-ba45-90fa95abfb7f |
| 9b324b23-2365-475c-8588-11da492337d5 | internal_api     | 3a78e8e6-a391-42be-937a-1e8268629e68 |
| a88842d3-3197-4a87-879f-8f093a210807 | tenant           | 8c407a33-0509-4bf4-8d07-46ae9c24cf80 |
| e73d1c64-c535-4ebb-bf86-716d9ee69225 | ovn_mac_addr_net |                                      |
+--------------------------------------+------------------+--------------------------------------+
(undercloud) [stack@undercloud-0 jbp]$ openstack network show  2d578e41-34cf-4507-b252-879ac28506a9 -f json 
/usr/lib64/python3.6/site-packages/_yaml/__init__.py:23: DeprecationWarning: The _yaml extension module is now located at yaml._yaml and its location is subject to change.  To use the LibYAML-based parser and emitter, import from `yaml`: `from yaml import CLoader as Loader, CDumper as Dumper`.
  DeprecationWarning
/usr/lib/python3.6/site-packages/barbicanclient/__init__.py:61: UserWarning: The secrets module is moved to barbicanclient/v1 directory, direct import of barbicanclient.secrets will be deprecated. Please import barbicanclient.v1.secrets instead.
  % (name, name, name))
{
  "admin_state_up": false,
  "availability_zone_hints": [],
  "availability_zones": [],
  "created_at": "2022-03-25T03:23:31Z",
  "description": "",
  "dns_domain": "",
  "id": "2d578e41-34cf-4507-b252-879ac28506a9",
  "ipv4_address_scope": null,
  "ipv6_address_scope": null,
  "is_default": null,
  "is_vlan_transparent": null,
  "mtu": 1500,
  "name": "storage",
  "port_security_enabled": true,
  "project_id": "f39a94664d1a4714a861ad9a9c8f7696",
  "provider:network_type": "flat",
  "provider:physical_network": "storage",
  "provider:segmentation_id": null,
  "qos_policy_id": null,
  "revision_number": 3,
  "router:external": false,
  "segments": null,
  "shared": false,
  "status": "ACTIVE",
  "subnets": [
    "ab6fd639-b0f7-4157-b48e-00ec00f63cca"
  ],
  "tags": [
    "tripleo_network_name=Storage",
    "tripleo_net_idx=0",
    "tripleo_vip=True"
  ],
  "updated_at": "2022-03-25T03:23:32Z"
}

Comment 2 Juan Badia Payno 2022-04-27 10:14:34 UTC
WORKAROUND which didnt work

Modifed the dns_domain to ''

2022-04-27 09:45:52.681033 | 525400bf-b550-5c21-9b85-00000000000e |     TIMING | Create/Update composable networks | localhost | 0:00:06.096935 | 5.69s
Using module file /usr/share/ansible/plugins/modules/tripleo_composable_network.py
Pipelining is enabled.
<127.0.0.1> EXEC /bin/sh -c '/usr/libexec/platform-python && sleep 0'
2022-04-27 09:45:53.890521 | 525400bf-b550-5c21-9b85-00000000000e |      FATAL | Create/Update composable networks | localhost | item={'dns_domain': '', 'name': 'Management', 'name_lower': 'management', 'subnets': {'management_subnet': {'allocation_pools': [{'end': '10.0.1.250', 'start': '10.0.1.4'}], 'gateway_ip': '10.0.1.1', 'ip_subnet': '10.0.1.0/24', 'physical_network': 'management', 'routes': [], 'vlan': 60}}} | error={
    "ansible_index_var": "idx",
    "ansible_loop_var": "item",
    "changed": false,
    "idx": 5,
    "invocation": {
        "module_args": {
            "api_timeout": null,
            "auth": null,
            "auth_type": null,
            "availability_zone": null,
            "ca_cert": null,
            "client_cert": null,
            "client_key": null,
            "cloud": null,
            "idx": 5,
            "interface": "public",
            "net_data": {
                "dns_domain": "",
                "name": "Management",
                "name_lower": "management",
                "subnets": {
                    "management_subnet": {
                        "allocation_pools": [
                            {
                                "end": "10.0.1.250",
                                "start": "10.0.1.4"
                            }
                        ],
                        "gateway_ip": "10.0.1.1",
                        "ip_subnet": "10.0.1.0/24",
                        "physical_network": "management",
                        "routes": [],
                        "vlan": 60
                    }
                }
            },
            "region_name": null,
            "timeout": 180,
            "validate_certs": null,
            "wait": true
        }
    },
    "item": {
        "dns_domain": "",
        "name": "Management",
        "name_lower": "management",
        "subnets": {
            "management_subnet": {
                "allocation_pools": [
                    {
                        "end": "10.0.1.250",
                        "start": "10.0.1.4"
                    }
                ],
                "gateway_ip": "10.0.1.1",
                "ip_subnet": "10.0.1.0/24",
                "physical_network": "management",
                "routes": [],
                "vlan": 60
            }
        }
    },
    "msg": "Failed schema validation at dns_domain:\n    '' does not match '^(?=^.{1,255}$)(?!.*\\\\.\\\\..*)(.{1,63}\\\\.)+(.{0,63}\\\\.?)|(?!\\\\.)(?!.*\\\\.\\\\..*)(^.{1,63}$)|(^\\\\.$)$'"
}
2022-04-27 09:45:53.891667 | 525400bf-b550-5c21-9b85-00000000000e |     TIMING | Create/Update composable networks | localhost | 0:00:07.307570 | 6.90s
2022-04-27 09:45:53.896917 | 525400bf-b550-5c21-9b85-00000000000e |     TIMING | Create/Update composable networks | localhost | 0:00:07.312823 | 6.91s

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=1    skipped=3    rescued=0    ignored=0   
2022-04-27 09:45:53.902134 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2022-04-27 09:45:53.902458 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 6          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2022-04-27 09:45:53.902740 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:00:07.318646 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2022-04-27 09:45:53.903056 |                                 UUID |       Info |       Host |   Task Name |   Run Time
2022-04-27 09:45:53.903346 | 525400bf-b550-5c21-9b85-00000000000e |    SUMMARY |  localhost | Create/Update composable networks | 6.91s
2022-04-27 09:45:53.903661 | 525400bf-b550-5c21-9b85-00000000000a |    SUMMARY |  localhost | Check if network deployment file already exist | 0.23s
2022-04-27 09:45:53.903978 | 525400bf-b550-5c21-9b85-00000000000d |    SUMMARY |  localhost | Load config from file | 0.04s
2022-04-27 09:45:53.904260 | 525400bf-b550-5c21-9b85-000000000008 |    SUMMARY |  localhost | fail | 0.02s
2022-04-27 09:45:53.904582 | 525400bf-b550-5c21-9b85-000000000009 |    SUMMARY |  localhost | fail | 0.02s
2022-04-27 09:45:53.904854 | 525400bf-b550-5c21-9b85-00000000000b |    SUMMARY |  localhost | fail | 0.02s
2022-04-27 09:45:53.905142 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2022-04-27 09:45:53.905481 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2022-04-27 09:45:53.905776 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~
2022-04-27 09:45:53.906037 |  The following node(s) had failures: localhost
2022-04-27 09:45:53.906369 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Comment 3 Juan Badia Payno 2022-04-27 10:16:44 UTC
WORKAROUND that  worked
- delete the "dns_domain: null" parameter of all the networks

IMHO, the dns_domain shouldn't be extracted

Comment 20 errata-xmlrpc 2023-08-16 01:11:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2023:4577


Note You need to log in before you can comment on or make changes to this bug.