Bug 1576572 - yaml-nic-config-2-script.py fails when template to be converted contains a blank line followed by a comment
Summary: yaml-nic-config-2-script.py fails when template to be converted contains a bl...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: z2
: 13.0 (Queens)
Assignee: Bob Fournier
QA Contact: mlammon
URL:
Whiteboard:
: 1594068 1596824 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-05-09 19:44 UTC by Marius Cornea
Modified: 2018-12-24 11:40 UTC (History)
9 users (show)

Fixed In Version: openstack-tripleo-heat-templates-8.0.4-3.el7ost
Doc Type: Bug Fix
Doc Text:
If a nic config template contained a blank line followed by a line starting with a comma, the yaml-nic-config-2-script.py did not reset the starting column of the next row. The nic config template converted by the script is invalid and caused a deployment failure. The script now properly sets the value for the column when the blank line is detected. Scripts that have a blank line followed by a line with a comma get converted correctly.
Clone Of:
Environment:
Last Closed: 2018-08-29 16:36:37 UTC
Target Upstream Version:


Attachments (Terms of Use)
controller.yaml (5.58 KB, text/plain)
2018-05-09 19:44 UTC, Marius Cornea
no flags Details


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 578956 0 None master: MERGED tripleo-heat-templates: Handle blank lines followed by comment in nic conversion script (I286da578c42df24242ba38df12d42c... 2018-07-19 01:47:51 UTC
OpenStack gerrit 580371 0 None stable/queens: MERGED tripleo-heat-templates: Handle blank lines followed by comment in nic conversion script (I286da578c42df24242ba38df12d42c... 2018-07-19 01:47:46 UTC
Red Hat Product Errata RHBA-2018:2574 0 None None None 2018-08-29 16:37:23 UTC

Description Marius Cornea 2018-05-09 19:44:46 UTC
Created attachment 1434001 [details]
controller.yaml

Description of problem:

yaml-nic-config-2-script.py fails when template to be converted contains a commented line:

[stack@undercloud-0 ~]$ /usr/share/openstack-tripleo-heat-templates/tools/yaml-nic-config-2-script.py nic-configs/controller.yaml
Using script at /usr/share/openstack-tripleo-heat-templates/network/scripts/run-os-net-config.sh
The yaml file will be overwritten and the original saved as /home/stack/nic-configs/controller.yaml.20180509154119
Overwrite nic-configs/controller.yaml? [y/n] y
Converting nic-configs/controller.yaml
Traceback (most recent call last):
  File "/usr/share/openstack-tripleo-heat-templates/tools/yaml-nic-config-2-script.py", line 155, in convert
    tpl = yaml.load(open(filename).read(), Loader=TemplateLoader)
  File "/usr/lib64/python2.7/site-packages/yaml/__init__.py", line 71, in load
    return loader.get_single_data()
  File "/usr/lib64/python2.7/site-packages/yaml/constructor.py", line 37, in get_single_data
    node = self.get_single_node()
  File "/usr/lib64/python2.7/site-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/usr/lib64/python2.7/site-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
  File "/usr/lib64/python2.7/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/usr/lib64/python2.7/site-packages/yaml/composer.py", line 127, in compose_mapping_node
    while not self.check_event(MappingEndEvent):
  File "/usr/lib64/python2.7/site-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/usr/lib64/python2.7/site-packages/yaml/parser.py", line 439, in parse_block_mapping_key
    "expected <block end>, but found %r" % token.id, token.start_mark)
ParserError: while parsing a block mapping
  in "<string>", line 1, column 1:
    heat_template_version: 2015-04-30
    ^
expected <block end>, but found '<block sequence start>'
  in "<string>", line 171, column 13:


Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-8.0.2-14.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Run yaml-nic-config-2-script.py on the attached templated.

Actual results:
Fails.

Expected results:
Doesn't fail.

Additional info:
Attaching nic template where the script is failing. If I remove the commented line(# Uncomment when including environments/network-management.yaml
) then the script works as expected.

Comment 1 Marius Cornea 2018-05-09 20:09:10 UTC
Workaround:

sed -i -E "s/^[ ]{0,}#.*//" controller.yaml

Comment 2 Bob Fournier 2018-06-22 15:21:48 UTC
*** Bug 1594068 has been marked as a duplicate of this bug. ***

Comment 3 Bob Fournier 2018-06-28 17:54:21 UTC
The problem with the script isn't that it can't handle all comments.  All example nic comments have comments both standalone and embedded, and the script handles them fine.

The problem is that it can't handle a blank line followed by a comment, like this:

               -
                  type: vlan
                  vlan_id: {get_param: StorageMgmtNetworkVlanID}
                  addresses:
                    -
                      ip_netmask: {get_param: StorageMgmtIpSubnet}

          # Uncomment when including environments/network-management.yaml
            -
              type: interface
              name: nic5

Its the blank like before "# Uncomment when..." that causes the first pass of the script to write the line with no indentation (last_non_comment_spaces is set to ''):
                -
                  type: vlan
                  vlan_id: {get_param: TenantNetworkVlanID}
                  addresses:
                    -
                      ip_netmask: {get_param: TenantIpSubnet}

comment7_16: ' Uncomment when including environments/network-management.yaml'
                -
                  type: vlan

The comment line with no indentation causes a problem when passing the file to the yaml parser.

The solution looks to be to add a check for blank lines.

Comment 4 Bob Fournier 2018-07-03 20:24:41 UTC
*** Bug 1596824 has been marked as a duplicate of this bug. ***

Comment 14 mlammon 2018-08-07 21:45:18 UTC
Steps:
1. Installed latest osp13 UC
2. copy attached file as /nic-configs/controller.yaml
3. execute /usr/share/openstack-tripleo-heat-templates/tools/yaml-nic-config-2-script.py nic-configs/controller.yaml


Environment:
openstack-tripleo-heat-templates-8.0.4-15.el7ost.noarchcd


Below is test output. I did not see any error.  It appears to convert and preserved previous template.

[stack@undercloud-0 ~]$ /usr/share/openstack-tripleo-heat-templates/tools/yaml-nic-config-2-script.py nic-configs/controller.yaml
Using script at /usr/share/openstack-tripleo-heat-templates/network/scripts/run-os-net-config.sh
The yaml file will be overwritten and the original saved as /home/stack/nic-configs/controller.yaml.20180807174112
Overwrite nic-configs/controller.yaml? [y/n] y
Converting nic-configs/controller.yaml
[stack@undercloud-0 ~]$ cd nic-configs/
[stack@undercloud-0 nic-configs]$ ll
total 16
-rw-rw-r--. 1 stack stack 5775 Aug  7 17:41 controller.yaml
-rw-rw-r--. 1 stack stack 5709 Aug  7 17:41 controller.yaml.20180807174112
[stack@undercloud-0 nic-configs]$ diff controller.yaml controller.yaml.20180807174112
1c1,2
< heat_template_version: queens
---
> heat_template_version: 2015-04-30
>
56c57
<     default: 10.0.1.1
---
>     default: '10.0.1.1'
60c61
<     default: 10.0.0.1
---
>     default: '10.0.0.1'
76a78
>
79d80
<     type: OS::Heat::SoftwareConfig
81d81
<       group: script
83,95c83,92
<         str_replace:
<           template:
<             get_file: /usr/share/openstack-tripleo-heat-templates/network/scripts/run-os-net-config.sh
<           params:
<             $network_config:
<               network_config:
<               - type: interface
<                 name: nic1
<                 use_dhcp: false
<                 dns_servers:
<                   get_param: DnsServers
<                 addresses:
<                 - ip_netmask:
---
>         os_net_config:
>           network_config:
>             -
>               type: interface
>               name: nic1
>               use_dhcp: false
>               dns_servers: {get_param: DnsServers}
>               addresses:
>                 -
>                   ip_netmask:
97,109c94,107
<                     - /
<                     - - get_param: ControlPlaneIp
<                       - get_param: ControlPlaneSubnetCidr
<                 routes:
<                 - ip_netmask: 169.254.169.254/32
<                   next_hop:
<                     get_param: EC2MetadataIp
<               - type: ovs_bridge
<                 name:
<                   bridge_name
<                 use_dhcp: false
<                 members:
<                 - type: interface
---
>                       - '/'
>                       - - {get_param: ControlPlaneIp}
>                         - {get_param: ControlPlaneSubnetCidr}
>               routes:
>                 -
>                   ip_netmask: 169.254.169.254/32
>                   next_hop: {get_param: EC2MetadataIp}
>             -
>               type: ovs_bridge
>               name: {get_input: bridge_name}
>               use_dhcp: false
>               members:
>                 -
>                   type: interface
112,116c110,113
<                 - type: vlan
<                   vlan_id:
<                     get_param: ExternalNetworkVlanID
<                   dns_servers:
<                     get_param: DnsServers
---
>                 -
>                   type: vlan
>                   vlan_id: {get_param: ExternalNetworkVlanID}
>                   dns_servers: {get_param: DnsServers}
118,119c115,116
<                   - ip_netmask:
<                       get_param: ExternalIpSubnet
---
>                   -
>                     ip_netmask: {get_param: ExternalIpSubnet}
121,131c118,131
<                   - default: true
<                     next_hop:
<                       get_param: ExternalInterfaceDefaultRoute
<               - type: ovs_bridge
<                 name: br-infra
<                 use_dhcp: false
<                 addresses:
<                 - ip_netmask:
<                     get_param: TenantIpSubnet
<                 members:
<                 - type: interface
---
>                     -
>                       default: true
>                       next_hop: {get_param: ExternalInterfaceDefaultRoute}
>
>             -
>               type: ovs_bridge
>               name: br-infra
>               use_dhcp: false
>               addresses:
>                 -
>                   ip_netmask: {get_param: TenantIpSubnet}
>               members:
>                 -
>                   type: interface
135,137c135,137
<                 - type: vlan
<                   vlan_id:
<                     get_param: InternalApiNetworkVlanID
---
>                 -
>                   type: vlan
>                   vlan_id: {get_param: InternalApiNetworkVlanID}
139,145c139,148
<                   - ip_netmask:
<                       get_param: InternalApiIpSubnet
<               - type: ovs_bridge
<                 name: br-storage
<                 use_dhcp: false
<                 members:
<                 - type: interface
---
>                     -
>                       ip_netmask: {get_param: InternalApiIpSubnet}
>
>             -
>               type: ovs_bridge
>               name: br-storage
>               use_dhcp: false
>               members:
>                 -
>                   type: interface
149,151c152,154
<                 - type: vlan
<                   vlan_id:
<                     get_param: StorageNetworkVlanID
---
>                 -
>                   type: vlan
>                   vlan_id: {get_param: StorageNetworkVlanID}
153,157c156,160
<                   - ip_netmask:
<                       get_param: StorageIpSubnet
<                 - type: vlan
<                   vlan_id:
<                     get_param: StorageMgmtNetworkVlanID
---
>                     -
>                       ip_netmask: {get_param: StorageIpSubnet}
>                 -
>                   type: vlan
>                   vlan_id: {get_param: StorageMgmtNetworkVlanID}
159,160c162,164
<                   - ip_netmask:
<                       get_param: StorageMgmtIpSubnet
---
>                     -
>                       ip_netmask: {get_param: StorageMgmtIpSubnet}
>
162,168c166,177
<               - type: interface
<                 name: nic5
<                 use_dhcp: false
<                 use_dhcpv6: false
<                 addresses:
<                 - ip_netmask:
<                     get_param: ManagementIpSubnet
---
>             -
>               type: interface
>               name: nic5
>               use_dhcp: false
>               use_dhcpv6: false
>               addresses:
>                 -
>                   ip_netmask: {get_param: ManagementIpSubnet}
>
>       group: os-apply-config
>     type: OS::Heat::StructuredConfig
>
172,174c181
<     value:
<       get_resource: OsNetConfigImpl
<
---
>     value: {get_resource: OsNetConfigImpl}
[stack@undercloud-0 nic-configs]$

Comment 15 Joanne O'Flynn 2018-08-15 08:07:01 UTC
This bug is marked for inclusion in the errata but does not currently contain draft documentation text. To ensure the timely release of this advisory please provide draft documentation text for this bug as soon as possible.

If you do not think this bug requires errata documentation, set the requires_doc_text flag to "-".


To add draft documentation text:

* Select the documentation type from the "Doc Type" drop down field.

* A template will be provided in the "Doc Text" field based on the "Doc Type" value selected. Enter draft text in the "Doc Text" field.

Comment 17 errata-xmlrpc 2018-08-29 16:36:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2574


Note You need to log in before you can comment on or make changes to this bug.