Bug 2079264

Summary: Upgrade [OSP16.2 -> OSP17.1] openstack overcloud network provision fails
Product: Red Hat OpenStack Reporter: Juan Badia Payno <jbadiapa>
Component: tripleo-ansibleAssignee: Harald Jensås <hjensas>
Status: CLOSED ERRATA QA Contact: Archana Singh <arcsingh>
Severity: medium Docs Contact:
Priority: high    
Version: 17.1 (Wallaby)CC: hjensas, jpretori, kthakre, sbaker
Target Milestone: betaKeywords: Triaged
Target Release: 17.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: tripleo-ansible-3.3.1-0.20220706003125.fa5422f.el8ost openstack-tripleo-heat-templates-14.3.1-0.20220518231249.3965818.el9ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-08-16 01:11:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Juan Badia Payno 2022-04-27 10:12:46 UTC 
Executing the "openstack overcloud network provision" failed with the following output "Failed schema validation at dns_domain:\n    None is not of type 'string'"

How to reproduce:
 Deploy osp16.2
 Upgrade the undercloud to osp17
 Provisioning the network
Comment 1 Juan Badia Payno 2022-04-27 10:13:42 UTC 
openstack --debug overcloud -vvv network provision --output jbp/generated-networks-deployed.yaml ~/jbp/tripleo-qe-Cloud-0-network-data.yaml 
START with options: --debug overcloud -vvv network provision --output jbp/generated-networks-deployed.yaml /home/stack/jbp/tripleo-qe-Cloud-0-network-data.yaml
...
...
...
2022-04-27 09:39:39.167880 | 525400bf-b550-cdbb-e797-00000000000e |     TIMING | Create/Update composable networks | localhost | 0:00:06.290028 | 5.88s
2022-04-27 09:39:40.327160 | 525400bf-b550-cdbb-e797-00000000000e |      FATAL | Create/Update composable networks | localhost | item={'dns_domain': None, 'name': 'Management', 'name_lower': 'management', 'subnets': {'management_subnet': {'allocation_pools': [{'end': '10.0.1.250', 'start': '10.0.1.4'}], 'gateway_ip': '10.0.1.1', 'ip_subnet': '10.0.1.0/24', 'physical_network': 'management', 'routes': [], 'vlan': 60}}} | error={
    "ansible_index_var": "idx",
    "ansible_loop_var": "item",
    "changed": false,
    "idx": 5,
    "invocation": {
        "module_args": {
            "api_timeout": null,
            "auth": null,
            "auth_type": null,
            "availability_zone": null,
            "ca_cert": null,
            "client_cert": null,
            "client_key": null,
            "cloud": null,
            "idx": 5,
            "interface": "public",
            "net_data": {
                "dns_domain": null,
                "name": "Management",
                "name_lower": "management",
                "subnets": {
                    "management_subnet": {
                        "allocation_pools": [
                            {
                                "end": "10.0.1.250",
                                "start": "10.0.1.4"
                            }
                        ],
                        "gateway_ip": "10.0.1.1",
                        "ip_subnet": "10.0.1.0/24",
                        "physical_network": "management",
                        "routes": [],
                        "vlan": 60
                    }
                }
            },
            "region_name": null,
            "timeout": 180,
            "validate_certs": null,
            "wait": true
        }
    },
    "item": {
        "dns_domain": null,
        "name": "Management",
        "name_lower": "management",
        "subnets": {
            "management_subnet": {
                "allocation_pools": [
                    {
                        "end": "10.0.1.250",
                        "start": "10.0.1.4"
                    }
                ],
                "gateway_ip": "10.0.1.1",
                "ip_subnet": "10.0.1.0/24",
                "physical_network": "management",
                "routes": [],
                "vlan": 60
            }
        }
    },
    "msg": "Failed schema validation at dns_domain:\n    None is not of type 'string'"
}
2022-04-27 09:39:40.328325 | 525400bf-b550-cdbb-e797-00000000000e |     TIMING | Create/Update composable networks | localhost | 0:00:07.450475 | 7.04s
2022-04-27 09:39:40.333688 | 525400bf-b550-cdbb-e797-00000000000e |     TIMING | Create/Update composable networks | localhost | 0:00:07.455838 | 7.05s

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=1    skipped=3    rescued=0    ignored=0   
2022-04-27 09:39:40.339016 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2022-04-27 09:39:40.339311 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 6          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2022-04-27 09:39:40.339558 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:00:07.461714 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2022-04-27 09:39:40.339801 |                                 UUID |       Info |       Host |   Task Name |   Run Time
2022-04-27 09:39:40.340063 | 525400bf-b550-cdbb-e797-00000000000e |    SUMMARY |  localhost | Create/Update composable networks | 7.05s
2022-04-27 09:39:40.340385 | 525400bf-b550-cdbb-e797-00000000000a |    SUMMARY |  localhost | Check if network deployment file already exist | 0.23s
2022-04-27 09:39:40.340652 | 525400bf-b550-cdbb-e797-00000000000d |    SUMMARY |  localhost | Load config from file | 0.04s
2022-04-27 09:39:40.340956 | 525400bf-b550-cdbb-e797-000000000008 |    SUMMARY |  localhost | fail | 0.03s
2022-04-27 09:39:40.341214 | 525400bf-b550-cdbb-e797-00000000000b |    SUMMARY |  localhost | fail | 0.02s
2022-04-27 09:39:40.341467 | 525400bf-b550-cdbb-e797-000000000009 |    SUMMARY |  localhost | fail | 0.02s


(undercloud) [stack@undercloud-0 jbp]$ head -n 14  tripleo-qe-Cloud-0-network-data.yaml 
- dns_domain: null
  name: Storage
  name_lower: storage
  subnets:
    storage_subnet:
      allocation_pools:
      - end: 172.17.3.149
        start: 172.17.3.10
      ip_subnet: 172.17.3.0/24
      physical_network: storage
      routes: []
      vlan: 30
  vip: true
- dns_domain: null

(undercloud) [stack@undercloud-0 jbp]$ openstack network list 
/usr/lib64/python3.6/site-packages/_yaml/__init__.py:23: DeprecationWarning: The _yaml extension module is now located at yaml._yaml and its location is subject to change.  To use the LibYAML-based parser and emitter, import from `yaml`: `from yaml import CLoader as Loader, CDumper as Dumper`.
  DeprecationWarning
/usr/lib/python3.6/site-packages/barbicanclient/__init__.py:61: UserWarning: The secrets module is moved to barbicanclient/v1 directory, direct import of barbicanclient.secrets will be deprecated. Please import barbicanclient.v1.secrets instead.
  % (name, name, name))
+--------------------------------------+------------------+--------------------------------------+
| ID                                   | Name             | Subnets                              |
+--------------------------------------+------------------+--------------------------------------+
| 2d578e41-34cf-4507-b252-879ac28506a9 | storage          | ab6fd639-b0f7-4157-b48e-00ec00f63cca |
| 4b0aed86-bcf2-459d-9c36-7655ec55a9a5 | external         | e4d644d4-45d4-43c4-a326-ac2c3566e32f |
| 66a62fc1-677c-42dd-bc99-33983efdc42c | storage_mgmt     | cdf9d597-aee4-4029-a97a-cd85f7835e59 |
| 761bf950-c22a-4faf-be54-a471cd0a6fec | management       | d040eae8-e5cd-4dd2-a222-c209016e9410 |
| 815314d4-5460-4662-9754-fae0f0936611 | ctlplane         | 32e28c03-aa50-4f0a-ba45-90fa95abfb7f |
| 9b324b23-2365-475c-8588-11da492337d5 | internal_api     | 3a78e8e6-a391-42be-937a-1e8268629e68 |
| a88842d3-3197-4a87-879f-8f093a210807 | tenant           | 8c407a33-0509-4bf4-8d07-46ae9c24cf80 |
| e73d1c64-c535-4ebb-bf86-716d9ee69225 | ovn_mac_addr_net |                                      |
+--------------------------------------+------------------+--------------------------------------+
(undercloud) [stack@undercloud-0 jbp]$ openstack network show  2d578e41-34cf-4507-b252-879ac28506a9 -f json 
/usr/lib64/python3.6/site-packages/_yaml/__init__.py:23: DeprecationWarning: The _yaml extension module is now located at yaml._yaml and its location is subject to change.  To use the LibYAML-based parser and emitter, import from `yaml`: `from yaml import CLoader as Loader, CDumper as Dumper`.
  DeprecationWarning
/usr/lib/python3.6/site-packages/barbicanclient/__init__.py:61: UserWarning: The secrets module is moved to barbicanclient/v1 directory, direct import of barbicanclient.secrets will be deprecated. Please import barbicanclient.v1.secrets instead.
  % (name, name, name))
{
  "admin_state_up": false,
  "availability_zone_hints": [],
  "availability_zones": [],
  "created_at": "2022-03-25T03:23:31Z",
  "description": "",
  "dns_domain": "",
  "id": "2d578e41-34cf-4507-b252-879ac28506a9",
  "ipv4_address_scope": null,
  "ipv6_address_scope": null,
  "is_default": null,
  "is_vlan_transparent": null,
  "mtu": 1500,
  "name": "storage",
  "port_security_enabled": true,
  "project_id": "f39a94664d1a4714a861ad9a9c8f7696",
  "provider:network_type": "flat",
  "provider:physical_network": "storage",
  "provider:segmentation_id": null,
  "qos_policy_id": null,
  "revision_number": 3,
  "router:external": false,
  "segments": null,
  "shared": false,
  "status": "ACTIVE",
  "subnets": [
    "ab6fd639-b0f7-4157-b48e-00ec00f63cca"
  ],
  "tags": [
    "tripleo_network_name=Storage",
    "tripleo_net_idx=0",
    "tripleo_vip=True"
  ],
  "updated_at": "2022-03-25T03:23:32Z"
}
Comment 2 Juan Badia Payno 2022-04-27 10:14:34 UTC 
WORKAROUND which didnt work

Modifed the dns_domain to ''

2022-04-27 09:45:52.681033 | 525400bf-b550-5c21-9b85-00000000000e |     TIMING | Create/Update composable networks | localhost | 0:00:06.096935 | 5.69s
Using module file /usr/share/ansible/plugins/modules/tripleo_composable_network.py
Pipelining is enabled.
<127.0.0.1> EXEC /bin/sh -c '/usr/libexec/platform-python && sleep 0'
2022-04-27 09:45:53.890521 | 525400bf-b550-5c21-9b85-00000000000e |      FATAL | Create/Update composable networks | localhost | item={'dns_domain': '', 'name': 'Management', 'name_lower': 'management', 'subnets': {'management_subnet': {'allocation_pools': [{'end': '10.0.1.250', 'start': '10.0.1.4'}], 'gateway_ip': '10.0.1.1', 'ip_subnet': '10.0.1.0/24', 'physical_network': 'management', 'routes': [], 'vlan': 60}}} | error={
    "ansible_index_var": "idx",
    "ansible_loop_var": "item",
    "changed": false,
    "idx": 5,
    "invocation": {
        "module_args": {
            "api_timeout": null,
            "auth": null,
            "auth_type": null,
            "availability_zone": null,
            "ca_cert": null,
            "client_cert": null,
            "client_key": null,
            "cloud": null,
            "idx": 5,
            "interface": "public",
            "net_data": {
                "dns_domain": "",
                "name": "Management",
                "name_lower": "management",
                "subnets": {
                    "management_subnet": {
                        "allocation_pools": [
                            {
                                "end": "10.0.1.250",
                                "start": "10.0.1.4"
                            }
                        ],
                        "gateway_ip": "10.0.1.1",
                        "ip_subnet": "10.0.1.0/24",
                        "physical_network": "management",
                        "routes": [],
                        "vlan": 60
                    }
                }
            },
            "region_name": null,
            "timeout": 180,
            "validate_certs": null,
            "wait": true
        }
    },
    "item": {
        "dns_domain": "",
        "name": "Management",
        "name_lower": "management",
        "subnets": {
            "management_subnet": {
                "allocation_pools": [
                    {
                        "end": "10.0.1.250",
                        "start": "10.0.1.4"
                    }
                ],
                "gateway_ip": "10.0.1.1",
                "ip_subnet": "10.0.1.0/24",
                "physical_network": "management",
                "routes": [],
                "vlan": 60
            }
        }
    },
    "msg": "Failed schema validation at dns_domain:\n    '' does not match '^(?=^.{1,255}$)(?!.*\\\\.\\\\..*)(.{1,63}\\\\.)+(.{0,63}\\\\.?)|(?!\\\\.)(?!.*\\\\.\\\\..*)(^.{1,63}$)|(^\\\\.$)$'"
}
2022-04-27 09:45:53.891667 | 525400bf-b550-5c21-9b85-00000000000e |     TIMING | Create/Update composable networks | localhost | 0:00:07.307570 | 6.90s
2022-04-27 09:45:53.896917 | 525400bf-b550-5c21-9b85-00000000000e |     TIMING | Create/Update composable networks | localhost | 0:00:07.312823 | 6.91s

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=1    skipped=3    rescued=0    ignored=0   
2022-04-27 09:45:53.902134 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2022-04-27 09:45:53.902458 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 6          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2022-04-27 09:45:53.902740 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:00:07.318646 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2022-04-27 09:45:53.903056 |                                 UUID |       Info |       Host |   Task Name |   Run Time
2022-04-27 09:45:53.903346 | 525400bf-b550-5c21-9b85-00000000000e |    SUMMARY |  localhost | Create/Update composable networks | 6.91s
2022-04-27 09:45:53.903661 | 525400bf-b550-5c21-9b85-00000000000a |    SUMMARY |  localhost | Check if network deployment file already exist | 0.23s
2022-04-27 09:45:53.903978 | 525400bf-b550-5c21-9b85-00000000000d |    SUMMARY |  localhost | Load config from file | 0.04s
2022-04-27 09:45:53.904260 | 525400bf-b550-5c21-9b85-000000000008 |    SUMMARY |  localhost | fail | 0.02s
2022-04-27 09:45:53.904582 | 525400bf-b550-5c21-9b85-000000000009 |    SUMMARY |  localhost | fail | 0.02s
2022-04-27 09:45:53.904854 | 525400bf-b550-5c21-9b85-00000000000b |    SUMMARY |  localhost | fail | 0.02s
2022-04-27 09:45:53.905142 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2022-04-27 09:45:53.905481 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2022-04-27 09:45:53.905776 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~
2022-04-27 09:45:53.906037 |  The following node(s) had failures: localhost
2022-04-27 09:45:53.906369 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Comment 3 Juan Badia Payno 2022-04-27 10:16:44 UTC 
WORKAROUND that  worked
- delete the "dns_domain: null" parameter of all the networks

IMHO, the dns_domain shouldn't be extracted
Comment 20 errata-xmlrpc 2023-08-16 01:11:06 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2023:4577