Unable to apply MTU updates (and other updates) to deployed overcloud * Description of problem: If I want to apply a configuration change to an already deployed overcloud, then I should be able to update the Heat templates and re-run the same `openstack overcloud deploy` I used to deploy my overcloud (as per our docs [1]). An example of this not working for an MTU change is described below, though I have seen it not work for changing the timezone too. However, if I do a new deployment with the exact same modified Heat templates the desired change is deployed the first time. * Version-Release number of selected component (if applicable): openstack-tripleo-0.0.7-1.el7ost.noarch openstack-tripleo-common-0.3.1-1.el7ost.noarch openstack-tripleo-heat-templates-0.8.14-9.el7ost.noarch openstack-tripleo-heat-templates-kilo-0.8.14-9.el7ost.noarch openstack-tripleo-image-elements-0.9.9-2.el7ost.noarch openstack-tripleo-puppet-elements-0.0.5-1.el7ost.noarch * How reproducible: - I've reproduced the problem in OSP8 with three different installs (but with the same Heat templates) - I've reproduced the problem in OSP7 too (want me to open another BZ or clone this?) * Steps to Reproduce: 1. Install OSP8 with Network Isolation as per our docs and take note of the MTU; e.g. 1500 2. Update the Interface Templates (e.g. nic-configs/controller-nics.yaml) to set the MTU to a different value; e.g. 9000 3. Re-run the original `openstack overcloud deploy` command 4. Observe that the MTU is unchanged Additional Details listed in footnote [2] * Actual results: MTU does not change. * Expected results: MTU does change. * Footnotes: [1] https://access.redhat.com/documentation/en/red-hat-openstack-platform/8/director-installation-and-usage/87-modifying-the-overcloud-environment [2] Additional Details: OSP8: -- Deployment and Update command: time openstack overcloud deploy --templates ~/templates/ -e ~/templates/clean_osd.yaml -e ~/templates/environments/puppet-pacemaker.yaml -e ~/templates/advanced-networking.yaml -e ~/templates/environments/puppet-ceph-external.yaml -e ~/templates/extraconfig/pre_deploy/rhel-registration/environment-rhel-registration.yaml -e ~/templates/extraconfig/pre_deploy/rhel-registration/rhel-registration-resource-registry.yaml -e ~/templates/timezone.yaml --control-flavor control --control-scale 3 --compute-flavor compute --compute-scale 3 --log-file overcloud_deployment.log --ntp-server 10.5.26.10 --timeout 90 --neutron-bridge-mappings datacentre:br-ex,tenant:br-tenant --neutron-network-type vlan --neutron-network-vlan-ranges tenant:4051:4060 --neutron-disable-tunneling -- Heat Templates used for initial Deployment https://github.com/RHsyseng/conv-osp-ceph/tree/osp8_ceph1.3/templates-diff If a template is not in the above then it is from /usr/share/openstack-tripleo-heat-templates -- Desired Change https://github.com/RHsyseng/conv-osp-ceph/commit/ce587bec1b8bd654777a503f21d8f201e2441ce9 -- MTU remains at 1500 [stack@hci-director ~]$ ansible all -b -m shell -a "ip a | grep -i mtu " 192.168.1.48 | success | rc=0 >> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 2: em3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 3: em4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000 4: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP qlen 1000 5: em2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 7: br-tenant: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 8: vlan4050: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 9: vlan4047@em2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 10: vlan4046@em2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 11: vlan4049@em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 12: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 13: br-int: <BROADCAST,MULTICAST> mtu 1400 qdisc noop state DOWN 78: qbr2610e771-27: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP 79: qvo2610e771-27@qvb2610e771-27: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1400 qdisc pfifo_fast master ovs-system state UP qlen 1000 80: qvb2610e771-27@qvo2610e771-27: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1400 qdisc pfifo_fast master qbr2610e771-27 state UP qlen 1000 81: tap2610e771-27: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc pfifo_fast master qbr2610e771-27 state UNKNOWN qlen 500 ... 192.168.1.50 | success | rc=0 >> 2: em3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 3: em4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000 4: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP qlen 1000 5: em2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 7: br-tenant: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 8: vlan4050: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 9: vlan4047@em2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 10: vlan4046@em2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 11: vlan4049@em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 12: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN [stack@hci-director ~]$ -- MTU is not updated to 9000 for the storage network [stack@hci-director ~]$ ansible all -b -m shell -a "ip a | grep -i mtu | grep 9000 " 192.168.1.52 | FAILED | rc=1 >> 192.168.1.53 | FAILED | rc=1 >> 192.168.1.48 | FAILED | rc=1 >> 192.168.1.51 | FAILED | rc=1 >> 192.168.1.50 | FAILED | rc=1 >> 192.168.1.49 | FAILED | rc=1 >> [stack@hci-director ~]$ Deployment finishes without errors: 2016-05-13 17:27:57 [VolumeDeployment_Step1]: UPDATE_COMPLETE state changed 2016-05-13 17:27:57 [overcloud-BlockStorageNodesPostDeployment-rhit2yvmfaha-VolumeDeployment_Step1-dqqyv2kqwlil]: UPDATE_COMPLETE Stack UPDATE completed successfully 2016-05-13 17:27:57 [CephStorageDeployment_Step1]: UPDATE_COMPLETE state changed 2016-05-13 17:27:57 [ExtraConfig]: UPDATE_IN_PROGRESS state changed 2016-05-13 17:27:57 [overcloud-CephStorageNodesPostDeployment-kgrc6i7kn2qy-ExtraConfig-of2k4ihvuobo]: UPDATE_IN_PROGRESS Stack UPDATE started 2016-05-13 17:27:57 [overcloud-CephStorageNodesPostDeployment-kgrc6i7kn2qy-ExtraConfig-of2k4ihvuobo]: UPDATE_COMPLETE Stack UPDATE completed successfully 2016-05-13 17:27:58 [ExtraConfig]: UPDATE_IN_PROGRESS state changed 2016-05-13 17:27:58 [overcloud-BlockStorageNodesPostDeployment-rhit2yvmfaha-ExtraConfig-4eynvdgy3nan]: UPDATE_IN_PROGRESS Stack UPDATE started 2016-05-13 17:27:58 [overcloud-BlockStorageNodesPostDeployment-rhit2yvmfaha-ExtraConfig-4eynvdgy3nan]: UPDATE_COMPLETE Stack UPDATE completed successfully 2016-05-13 17:27:58 [ExtraConfig]: UPDATE_COMPLETE state changed 2016-05-13 17:27:59 [ExtraConfig]: UPDATE_COMPLETE state changed 2016-05-13 17:28:02 [overcloud-CephStorageNodesPostDeployment-kgrc6i7kn2qy]: UPDATE_COMPLETE Stack UPDATE completed successfully 2016-05-13 17:28:04 [overcloud-BlockStorageNodesPostDeployment-rhit2yvmfaha]: UPDATE_COMPLETE Stack UPDATE completed successfully 2016-05-13 17:28:05 [BlockStorageNodesPostDeployment]: UPDATE_COMPLETE state changed Stack overcloud UPDATE_COMPLETE Overcloud Endpoint: http://10.19.139.37:5000/v2.0 Overcloud Deployed real 11m5.153s user 0m14.000s sys 0m1.266s [stack@hci-director ~]$ -- If I do a NEW deployment, not an update, with the same Heat template, then I get the desired result: [root@overcloud-controller-0 ~]# ip a s vlan4046 15: vlan4046: mtu 9000 qdisc noqueue state UNKNOWN link/ether 7e:58:bc:09:8f:5a brd ff:ff:ff:ff:ff:ff inet 172.16.1.12/24 brd 172.16.1.255 scope global vlan4046 valid_lft forever preferred_lft forever inet6 fe80::7c58:bcff:fe09:8f5a/64 scope link valid_lft forever preferred_lft forever [root@overcloud-controller-0 ~]# ip a s vlan4047 14: vlan4047: mtu 9000 qdisc noqueue state UNKNOWN link/ether 06:dd:69:38:0c:d1 brd ff:ff:ff:ff:ff:ff inet 172.16.2.12/24 brd 172.16.2.255 scope global vlan4047 valid_lft forever preferred_lft forever inet6 fe80::4dd:69ff:fe38:cd1/64 scope link valid_lft forever preferred_lft forever [root@overcloud-controller-0 ~]# ---------------------------------------------------- OSP7: (same problem) -- Deployment and Update command: time openstack overcloud deploy --templates ~/templates/ -e ~/templates/clean_osd.yaml -e ~/templates/environments/puppet-pacemaker.yaml -e ~/templates/advanced-networking.yaml -e ~/templates/environments/puppet-ceph-external.yaml -e ~/templates/extraconfig/pre_deploy/rhel-registration/environment-rhel-registration.yaml -e ~/templates/extraconfig/pre_deploy/rhel-registration/rhel-registration-resource-registry.yaml -e ~/templates/timezone.yaml --control-flavor control --control-scale 3 --compute-flavor compute --compute-scale 3 --log-file overcloud_deployment.log --ntp-server 10.5.26.10 --timeout 90 --neutron-bridge-mappings datacentre:br-ex,tenant:br-tenant --neutron-network-type vlan --neutron-network-vlan-ranges tenant:4051:4060 --neutron-disable-tunneling -- Heat Templates used for intial Deployment https://github.com/RHsyseng/conv-osp-ceph/tree/osp7_ceph1.3/templates-diff If a template is not in the above then it is from /usr/share/openstack-tripleo-heat-templates -- Desired Change: [stack@hci-director nic-configs]$ grep 1500 *.yaml compute-nics.yaml: mtu: 1500 compute-nics.yaml: mtu: 1500 compute-nics.yaml: mtu: 1500 compute-nics.yaml: mtu: 1500 compute-nics.yaml: mtu: 1500 compute-nics.yaml: mtu: 1500 controller-nics.yaml: mtu: 1500 controller-nics.yaml: mtu: 1500 controller-nics.yaml: mtu: 1500 controller-nics.yaml: mtu: 1500 controller-nics.yaml: mtu: 1500 controller-nics.yaml: mtu: 1500 [stack@hci-director nic-configs]$ [stack@hci-director nic-configs]$ grep mtu *.yaml compute-nics.yaml: mtu: 9000 compute-nics.yaml: mtu: 9000 compute-nics.yaml: mtu: 1500 compute-nics.yaml: mtu: 1500 compute-nics.yaml: mtu: 1500 compute-nics.yaml: mtu: 1500 controller-nics.yaml: mtu: 1500 controller-nics.yaml: mtu: 9000 controller-nics.yaml: mtu: 9000 controller-nics.yaml: mtu: 1500 controller-nics.yaml: mtu: 1500 controller-nics.yaml: mtu: 1500 [stack@hci-director nic-configs]$ -- Before re-running `openstack overcloud deploy ...` [stack@hci-director nic-configs]$ ansible all -b -m shell -a "ip a | grep -i mtu | grep 9000 " 192.168.1.22 | FAILED | rc=1 >> 192.168.1.24 | FAILED | rc=1 >> 192.168.1.26 | FAILED | rc=1 >> 192.168.1.23 | FAILED | rc=1 >> 192.168.1.25 | FAILED | rc=1 >> 192.168.1.27 | FAILED | rc=1 >> [stack@hci-director nic-configs]$ -- After re-running `openstack overcloud deploy ...` [stack@hci-director nic-configs]$ ansible all -b -m shell -a "ip a | grep -i mtu | grep 9000 " 192.168.1.26 | FAILED | rc=1 >> 192.168.1.24 | FAILED | rc=1 >> 192.168.1.22 | FAILED | rc=1 >> 192.168.1.23 | FAILED | rc=1 >> 192.168.1.25 | FAILED | rc=1 >> 192.168.1.27 | FAILED | rc=1 >> [stack@hci-director nic-configs]$
My initial example [0] lacked an interface entry, but after further testing, the bug remains. Even with an updated change, [0] and [1], the setting is applied during an install but not during an update. During an update, if I run `grep NET_CONFIG /var/log/messages` from a controller node and strip the quotes, then the dictionary I get [2] does not contain an entry for the MTU of the device, even after adding entries to set it [4]. [0] https://github.com/RHsyseng/conv-osp-ceph/commit/ce587bec1b8bd654777a503f21d8f201e2441ce9 [1] https://github.com/RHsyseng/conv-osp-ceph/commit/54a26764845929b461da7158cb68431c7d0ab8fa [2] In [14]: NET_CONFIG Out[14]: {'network_config': [{'addresses': [{'ip_netmask': '192.168.1.26/24'}], 'dns_servers': ['10.19.143.247', '10.19.143.248'], 'name': 'p2p1', 'routes': [{'ip_netmask': '169.254.169.254/32', 'next_hop': '192.168.1.1'}], 'type': 'interface', 'use_dhcp': 0}, {'addresses': [{'ip_netmask': '10.19.139.38/21'}], 'members': [{'name': 'p2p2', 'primary': 0, 'type': 'interface'}], 'name': 'br-ex', 'routes': [{'ip_netmask': '0.0.0.0/0', 'next_hop': '10.19.143.254'}], 'type': 'ovs_bridge', 'use_dhcp': 0}, {'addresses': [{'ip_netmask': '192.168.2.12/24'}], 'device': 'em1', 'mtu': 1500, 'type': 'vlan', 'use_dhcp': 0, 'vlan_id': 4049}, {'defroute': 0, 'name': 'em3', 'type': 'interface', 'use_dhcp': 0}, {'defroute': 0, 'name': 'em4', 'type': 'interface', 'use_dhcp': 0}, {'defroute': 0, 'name': 'p2p3', 'type': 'interface', 'use_dhcp': 0}, {'defroute': 0, 'name': 'p2p4', 'type': 'interface', 'use_dhcp': 0}, {'addresses': [{'ip_netmask': '172.16.2.11/24'}], 'device': 'em2', 'mtu': 9000, 'type': 'vlan', 'use_dhcp': 0, 'vlan_id': 4047}, {'addresses': [{'ip_netmask': '172.16.1.11/24'}], 'device': 'em2', 'mtu': 9000, 'type': 'vlan', 'use_dhcp': 0, 'vlan_id': 4046}, {'members': [{'mtu': 1500, 'name': 'em1', 'primary': 0, 'type': 'interface', 'use_dhcp': 0}, {'addresses': [{'ip_netmask': '192.168.3.10/24'}], 'mtu': 1500, 'type': 'vlan', 'vlan_id': 4050}], 'mtu': 1500, 'name': 'br-tenant', 'type': 'ovs_bridge', 'use_dhcp': 0}]} In [15]: [4] [stack@hci-director ~]$ time openstack overcloud deploy --templates ~/templates/ -e ~/templates/clean_osd.yaml -e ~/templates/advanced-networking.yaml -e ~/templates/environments/puppet-ceph-external.yaml -e ~/templates/timezone.yaml --control-flavor control --control-scale 3 --compute-flavor compute --compute-scale 3 --log-file overcloud_deployment.log --ntp-server 10.5.26.10 --timeout 90 --neutron-bridge-mappings datacentre:br-ex,tenant:br-tenant --neutron-network-type vlan --neutron-network-vlan-ranges tenant:4051:4060 --neutron-disable-tunneling ... 2016-05-19 14:12:24 [overcloud-BlockStorageNodesPostDeployment-etq2oap3s6ae]: UPDATE_COMPLETE Stack UPDATE completed successfully 2016-05-19 14:12:25 [BlockStorageNodesPostDeployment]: UPDATE_COMPLETE state changed Stack overcloud UPDATE_COMPLETE Overcloud Endpoint: http://10.19.139.37:5000/v2.0 Overcloud Deployed real 10m21.714s user 0m12.965s sys 0m1.178s [stack@hci-director ~]$ grep em2 -B 2 -A 2 ~/templates/nic-configs/* /home/stack/templates/nic-configs/compute-nics.yaml- - /home/stack/templates/nic-configs/compute-nics.yaml- type: interface /home/stack/templates/nic-configs/compute-nics.yaml: name: em2 /home/stack/templates/nic-configs/compute-nics.yaml- use_dhcp: false /home/stack/templates/nic-configs/compute-nics.yaml- mtu: 9000 /home/stack/templates/nic-configs/compute-nics.yaml- - /home/stack/templates/nic-configs/compute-nics.yaml- type: vlan /home/stack/templates/nic-configs/compute-nics.yaml: device: em2 /home/stack/templates/nic-configs/compute-nics.yaml- mtu: 9000 /home/stack/templates/nic-configs/compute-nics.yaml- use_dhcp: false -- /home/stack/templates/nic-configs/compute-nics.yaml- - /home/stack/templates/nic-configs/compute-nics.yaml- type: vlan /home/stack/templates/nic-configs/compute-nics.yaml: device: em2 /home/stack/templates/nic-configs/compute-nics.yaml- mtu: 9000 /home/stack/templates/nic-configs/compute-nics.yaml- use_dhcp: false -- /home/stack/templates/nic-configs/controller-nics.yaml- - /home/stack/templates/nic-configs/controller-nics.yaml- type: interface /home/stack/templates/nic-configs/controller-nics.yaml: name: em2 /home/stack/templates/nic-configs/controller-nics.yaml- use_dhcp: false /home/stack/templates/nic-configs/controller-nics.yaml- mtu: 9000 /home/stack/templates/nic-configs/controller-nics.yaml- - /home/stack/templates/nic-configs/controller-nics.yaml- type: vlan /home/stack/templates/nic-configs/controller-nics.yaml: device: em2 /home/stack/templates/nic-configs/controller-nics.yaml- mtu: 9000 /home/stack/templates/nic-configs/controller-nics.yaml- use_dhcp: false -- /home/stack/templates/nic-configs/controller-nics.yaml- - /home/stack/templates/nic-configs/controller-nics.yaml- type: vlan /home/stack/templates/nic-configs/controller-nics.yaml: device: em2 /home/stack/templates/nic-configs/controller-nics.yaml- mtu: 9000 /home/stack/templates/nic-configs/controller-nics.yaml- use_dhcp: false [stack@hci-director ~]$ grep nic-configs ~/templates/advanced-networking.yaml OS::TripleO::Compute::Net::SoftwareConfig: /home/stack/templates/nic-configs/compute-nics.yaml OS::TripleO::Controller::Net::SoftwareConfig: /home/stack/templates/nic-configs/controller-nics.yaml [stack@hci-director ~]$
Hi John, as of https://access.redhat.com/solutions/2213711 you this is normal and resolvable as there mentioned. Can you try this and come back with a result for your tests?
I opened this BZ assuming that NetworkDeploymentActions should take effect on update, not just on create. I see now that the THT was defined so that network changes would happen on create only [1] and that this was likely intentional as it increases stability and avoids the risk of taking a production cloud off of the network. Thus, the default is correct but the user may override it if they really want to as described in the kbase article [2]. I am closing this NOTABUG. [1] [root@hci-director ~]# cd /usr/share/openstack-tripleo-heat-templates [root@hci-director openstack-tripleo-heat-templates]# grep -R NetworkDeploymentActions: -A 4 puppet/blockstorage-role.yaml: NetworkDeploymentActions: puppet/blockstorage-role.yaml- type: comma_delimited_list puppet/blockstorage-role.yaml- description: > puppet/blockstorage-role.yaml- Heat action when to apply network configuration changes puppet/blockstorage-role.yaml- default: ['CREATE'] -- puppet/cephstorage-role.yaml: NetworkDeploymentActions: puppet/cephstorage-role.yaml- type: comma_delimited_list puppet/cephstorage-role.yaml- description: > puppet/cephstorage-role.yaml- Heat action when to apply network configuration changes puppet/cephstorage-role.yaml- default: ['CREATE'] -- puppet/compute-role.yaml: NetworkDeploymentActions: puppet/compute-role.yaml- type: comma_delimited_list puppet/compute-role.yaml- description: > puppet/compute-role.yaml- Heat action when to apply network configuration changes puppet/compute-role.yaml- default: ['CREATE'] -- puppet/controller-role.yaml: NetworkDeploymentActions: puppet/controller-role.yaml- type: comma_delimited_list puppet/controller-role.yaml- description: > puppet/controller-role.yaml- Heat action when to apply network configuration changes puppet/controller-role.yaml- default: ['CREATE'] -- puppet/objectstorage-role.yaml: NetworkDeploymentActions: puppet/objectstorage-role.yaml- type: comma_delimited_list puppet/objectstorage-role.yaml- description: > puppet/objectstorage-role.yaml- Heat action when to apply network configuration changes puppet/objectstorage-role.yaml- default: ['CREATE'] -- puppet/role.role.j2.yaml: NetworkDeploymentActions: puppet/role.role.j2.yaml- type: comma_delimited_list puppet/role.role.j2.yaml- description: > puppet/role.role.j2.yaml- Heat action when to apply network configuration changes puppet/role.role.j2.yaml- default: ['CREATE'] [root@hci-director openstack-tripleo-heat-templates]# [2] https://access.redhat.com/solutions/2213711