Hi OVS team,
Please suggest, I am waiting for your suggestions, do let me know if you need any other information.

can you attach the NIC config templates; this looks like a configuration issue or an environmental issue because the logs suggest there was an error with em2 and em3 not getting the IP assigned by DHCP

        "[2021/04/29 02:03:51 PM] [ERROR] Failure(s) occurred when applying configuration",
        "[2021/04/29 02:03:51 PM] [ERROR] stdout: ",
        "Determining IP information for em2... failed.",
        ", stderr: WARN      : [ifup] You are using 'ifup' script provided by 'network-scripts', which are now deprecated.",
        "WARN      : [ifup] 'network-scripts' will be removed in one of the next major releases of RHEL.",
        "WARN      : [ifup] It is advised to switch to 'NetworkManager' instead - it provides 'ifup/ifdown' scripts as well.",
        "",
        "[2021/04/29 02:03:51 PM] [ERROR] stdout: ",
        "Determining IP information for em3... failed.",
        ", stderr: WARN      : [ifup] You are using 'ifup' script provided by 'network-scripts', which are now deprecated.",
        "WARN      : [ifup] 'network-scripts' will be removed in one of the next major releases of RHEL.",
        "WARN      : [ifup] It is advised to switch to 'NetworkManager' instead - it provides 'ifup/ifdown' scripts as well.",
        "",
        "Traceback (most recent call last):",
        "  File \"/bin/os-net-config\", line 10, in <module>",
        "    sys.exit(main())",
        "  File \"/usr/lib/python3.6/site-packages/os_net_config/cli.py\", line 349, in main",
        "    activate=not opts.no_activate)",
        "  File \"/usr/lib/python3.6/site-packages/os_net_config/impl_ifcfg.py\", line 1806, in apply",
        "    raise os_net_config.ConfigurationError(message)",
        "os_net_config.ConfigurationError: Failure(s) occurred when applying configuration",
        "+ RETVAL=1",
        "+ set -e",
        "+ [[ 1 == 2 ]]",
        "+ [[ 1 != 0 ]]",
        "+ echo 'ERROR: configuration of safe defaults failed.'"

Created attachment 1784160 [details]
ceph-storage.yaml file

Created attachment 1784161 [details]
controller.yaml file

Created attachment 1784162 [details]
computesriov.yaml file

(In reply to Giulio Fidente from comment #3)
> can you attach the NIC config templates; this looks like a configuration
> issue or an environmental issue because the logs suggest there was an error
> with em2 and em3 not getting the IP assigned by DHCP
> 
>         "[2021/04/29 02:03:51 PM] [ERROR] Failure(s) occurred when applying
> configuration",
>         "[2021/04/29 02:03:51 PM] [ERROR] stdout: ",
>         "Determining IP information for em2... failed.",
>         ", stderr: WARN      : [ifup] You are using 'ifup' script provided
> by 'network-scripts', which are now deprecated.",
>         "WARN      : [ifup] 'network-scripts' will be removed in one of the
> next major releases of RHEL.",
>         "WARN      : [ifup] It is advised to switch to 'NetworkManager'
> instead - it provides 'ifup/ifdown' scripts as well.",
>         "",
>         "[2021/04/29 02:03:51 PM] [ERROR] stdout: ",
>         "Determining IP information for em3... failed.",
>         ", stderr: WARN      : [ifup] You are using 'ifup' script provided
> by 'network-scripts', which are now deprecated.",
>         "WARN      : [ifup] 'network-scripts' will be removed in one of the
> next major releases of RHEL.",
>         "WARN      : [ifup] It is advised to switch to 'NetworkManager'
> instead - it provides 'ifup/ifdown' scripts as well.",
>         "",
>         "Traceback (most recent call last):",
>         "  File \"/bin/os-net-config\", line 10, in <module>",
>         "    sys.exit(main())",
>         "  File \"/usr/lib/python3.6/site-packages/os_net_config/cli.py\",
> line 349, in main",
>         "    activate=not opts.no_activate)",
>         "  File
> \"/usr/lib/python3.6/site-packages/os_net_config/impl_ifcfg.py\", line 1806,
> in apply",
>         "    raise os_net_config.ConfigurationError(message)",
>         "os_net_config.ConfigurationError: Failure(s) occurred when applying
> configuration",
>         "+ RETVAL=1",
>         "+ set -e",
>         "+ [[ 1 == 2 ]]",
>         "+ [[ 1 != 0 ]]",
>         "+ echo 'ERROR: configuration of safe defaults failed.'"

Hi Giulio Fidente,
I have attached nic-configs file for controller,computesriov and ceph-storage, do let me know if any other info required from my side.

hi there seems to be a validation issue for the nic templates config/indentation

        "[2021/04/29 02:00:28 PM] [WARNING] Config file failed schema validation at network_config/1:",
        "    {'dns_servers': ['8.8.8.8', '8.8.4.4'], 'domain': [], 'members': [{'bonding_options': 'bond_mode=active-backup', 'members': [{'name': 'em2', 'primary': True, 'type': 'interface'}, {'name': 'em3', 'type': 'interface'}], 'name': 'bond1', 'ovs_options': None, 'type': 'ovs_bond'}, {'addresses': [{'ip_netmask': '192.168.23.161/24'}], 'type': 'vlan', 'vlan_id': 23}, {'addresses': [{'ip_netmask': '192.168.24.200/24'}], 'type': 'vlan', 'vlan_id': 24}], 'name': 'br-bond', 'type': 'ovs_bridge', 'nic_mapping': None, 'persist_mapping': False} is not valid under any of the given schemas",
        "  Sub-schemas tested and not matching:",
        "  - items/oneOf/ovs_bridge/members/items/oneOf: {'bonding_options': 'bond_mode=active-backup', 'members': [{'name': 'em2', 'primary': True, 'type': 'interface'}, {'name': 'em3', 'type': 'interface'}], 'name': 'bond1', 'ovs_options': None, 'type': 'ovs_bond'} is not valid under any of the given schemas",
        "  -- items/oneOf/ovs_bridge/members/items/oneOf/ovs_bond/additionalProperties: Additional properties are not allowed ('bonding_options' was unexpected)",
        "  -- items/oneOf/ovs_bridge/members/items/oneOf/ovs_bond/ovs_options/oneOf: None is not valid under any of the given schemas",
        "  --- items/oneOf/ovs_bridge/members/items/oneOf/ovs_bond/ovs_options/oneOf/ovs_options_string/type: 'None' is not of type 'string'",
        "  --- items/oneOf/ovs_bridge/members/items/oneOf/ovs_bond/ovs_options/oneOf/param/oneOf: None is not valid under any of the given schemas",
        "  ---- items/oneOf/ovs_bridge/members/items/oneOf/ovs_bond/ovs_options/oneOf/param/oneOf/0/type: 'None' is not of type 'object'",
        "  ---- items/oneOf/ovs_bridge/members/items/oneOf/ovs_bond/ovs_options/oneOf/param/oneOf/1/type: 'None' is not of type 'object'",

we're looking into that to try find the exact problem

Hi,

Looking at the ceph-storage.yaml file, it appears you're using bonding_options - which is intended to be used with Linux Bonds:
              - type: ovs_bridge
                name: br-bond
                dns_servers:
                  get_param: DnsServers
                domain:
                  get_param: DnsSearchDomains
                members:
                - type: ovs_bond
                  name: bond1
                  ovs_options: null
                  bonding_options:
                    get_param: BondInterfaceOvsOptions
                  members:
                  - type: interface
                    name: em2
                    primary: true
                  - type: interface
                    name: em3

The correct way to do this with a ovs_bond would be:
              - type: ovs_bridge
                name: br-bond
                dns_servers:
                  get_param: DnsServers
                domain:
                  get_param: DnsSearchDomains
                members:
                - type: ovs_bond
                  name: bond1
                  ovs_options:
                    get_param: BondInterfaceOvsOptions
                  members:
                  - type: interface
                    name: em2
                    primary: true
                  - type: interface
                    name: em3


The schema is defined here and we can see that bonding_options is only used for the linux_bond  and linux_team interface types:
https://github.com/openstack/os-net-config/blob/stable/train/os_net_config/schema.yaml#L1165-L1181

Whereas for ovs_bond it uses bonding_options:
https://github.com/openstack/os-net-config/blob/stable/train/os_net_config/schema.yaml#L606-L623

Here's some examples from the default network config files for reference:
linux_bond example:
https://github.com/openstack/tripleo-heat-templates/blob/stable/train/network/config/bond-with-vlans/role.role.j2.yaml#L195-L200

ovs_bond example:
https://github.com/openstack/tripleo-heat-templates/blob/stable/train/network/config/bond-with-vlans/role.role.j2.yaml#L159-L164

I believe that is the reason the schemas are not matching and you're getting those errors.

I would recommend the following changes:

- Convert from OVS bond to Linux bond

- Add "device: bond1" to the VLANs that are attached to the bond

- Remove OVS bridge (there is no reason for it if you are using Linux bonds)

- Remove line that has: "ovs_options: null"


Alternately, you can remove the "bonding_options:" line and put the "get_param: BondInterfaceOvsOptions" under the ovs_options: line (without the "null").

So it would look like this:

              network_config:
              - type: interface
                name: em1
                use_dhcp: false
                dns_servers:
                  get_param: DnsServers
                domain:
                  get_param: DnsSearchDomains
                addresses:
                - ip_netmask:
                    list_join:
                    - /
                    - - get_param: ControlPlaneIp
                      - get_param: ControlPlaneSubnetCidr
                routes:
                - ip_netmask: 169.254.169.254/32
                  next_hop:
                    get_param: EC2MetadataIp
                - default: true
                  next_hop:
                    get_param: ControlPlaneDefaultRoute
              - type: linux_bond
                name: bond1
                bonding_options:
                  get_param: BondInterfaceOvsOptions
                members:
                - type: interface
                  name: em2
                  primary: true
                - type: interface
                  name: em3
              - type: vlan
                vlan_id:
                  get_param: StorageNetworkVlanID
                addresses:
                - ip_netmask:
                    get_param: StorageIpSubnet
              - type: vlan
                vlan_id:
                  get_param: StorageMgmtNetworkVlanID
                addresses:
                - ip_netmask:
                    get_param: StorageMgmtIpSubnet

Note that in order to apply the network configuration, you will have to have NetworkDeploymentActions set with "UPDATE" in the list in an environment file:

parameter_defaults:
  NetworkDeploymentActions: ["CREATE","UPDATE"]

Then run a stack update, and on subsequent stack updates you shouldn't need to set NetworkDeploymentActions.

(In reply to Dan Sneddon from comment #10)
> I would recommend the following changes:
> 
> - Convert from OVS bond to Linux bond
> 
> - Add "device: bond1" to the VLANs that are attached to the bond
> 
> - Remove OVS bridge (there is no reason for it if you are using Linux bonds)
> 
> - Remove line that has: "ovs_options: null"
> 
> 
> Alternately, you can remove the "bonding_options:" line and put the
> "get_param: BondInterfaceOvsOptions" under the ovs_options: line (without
> the "null").
> 
> So it would look like this:
> 
>               network_config:
>               - type: interface
>                 name: em1
>                 use_dhcp: false
>                 dns_servers:
>                   get_param: DnsServers
>                 domain:
>                   get_param: DnsSearchDomains
>                 addresses:
>                 - ip_netmask:
>                     list_join:
>                     - /
>                     - - get_param: ControlPlaneIp
>                       - get_param: ControlPlaneSubnetCidr
>                 routes:
>                 - ip_netmask: 169.254.169.254/32
>                   next_hop:
>                     get_param: EC2MetadataIp
>                 - default: true
>                   next_hop:
>                     get_param: ControlPlaneDefaultRoute
>               - type: linux_bond
>                 name: bond1
>                 bonding_options:
>                   get_param: BondInterfaceOvsOptions
>                 members:
>                 - type: interface
>                   name: em2
>                   primary: true
>                 - type: interface
>                   name: em3
>               - type: vlan
>                 vlan_id:
>                   get_param: StorageNetworkVlanID
>                 addresses:
>                 - ip_netmask:
>                     get_param: StorageIpSubnet
>               - type: vlan
>                 vlan_id:
>                   get_param: StorageMgmtNetworkVlanID
>                 addresses:
>                 - ip_netmask:
>                     get_param: StorageMgmtIpSubnet

Hi Dan,
I will apply these changes suggested by you and verify once again during rhosp fast forward upgrade, but one thing that comes to mind is why I am not getting error for controller nodes because I am using the same bonding_options parameter in nic-configs file for controller nodes also like below:
              - type: ovs_bridge
                name: br-ex
                dns_servers:
                  get_param: DnsServers
                domain:
                  get_param: DnsSearchDomains
                members:
                - type: ovs_bond
                  name: bond1
                  ovs_options: null
                  bonding_options:
                    get_param: BondInterfaceOvsOptions
                  members:
                  - type: interface
                    name: em2
                    primary: true
                  - type: interface
                    name: em3