Hide Forgot
Description of problem: Documentation states how to configure predictable IPs here: https://access.redhat.com/documentation/en/red-hat-openstack-platform/version-8/director-installation-and-usage/#sect-Predictable_IPs However, if the IPs you pick are at the start of your usable range, the deployment will fail because the VIPs will automatically take the first IPs even if they are specified for use in the ips-from-pool-all.yaml. This results in things like galera or redis not starting up properly in the deployment but otherwise doesn't show the conflict. Version-Release number of selected component (if applicable): RHEL OSP 8 GA How reproducible: It is possible to reproduce every time your IPs in ips-from-pool-all.yaml are at the start of your IP range. Steps to Reproduce: 1. Create a network-isolation.yaml file with allocation pools like this: InternalApiNetCidr: 192.168.221.0/24 InternalApiAllocationPools: [{'start': '192.168.221.20', 'end': '192.168.221.200'}] #InternalApiNetworkVlanID: "" 2. Create an ips-from-pool-all.yaml file with hosts starting at the beginning of the pool. For example: ControllerIPs: # Each controller will get an IP from the lists below, first controller, first IP internal_api: - 192.168.221.20 - 192.168.221.21 - 192.168.221.22 3. Deploy OpenStack with these files openstack overcloud deploy --templates ~/templates/ --ntp-server 0.fedora.pool.ntp.org --libvirt-type kvm --control-flavor control --compute-flavor compute --ceph-storage-flavor ceph-storage --control-scale 3 --compute-scale 2 --ceph-storage-scale 3 --neutron-tunnel-types vxlan --neutron-network-type vxlan -e ~/templates/environments/storage-environment.yaml -e ~/templates/environments/network-isolation.yaml -e ~/templates/environments/ips-from-pool-all.yaml Actual results: The deploy will set a VIP to 192.168.221.20, but a controller will also have the IP 192.168.221.20. This results in some services failing and strange behavior due to the IP conflict Expected results: Ideally our documentation would call out this risk and point out how to avoid this. Additional info: We should update our documentation to include a warning to NOT set the hosts in the ips-from-pool-all.yaml file to the beginning of the range. This will avoid IP conflicts. In addition, we should include the following to also predictably assign VIPs: To predictably assign VIPs, update your network isolation file to include the required details: vi ~/templates/environments/network-isolation.yaml # In the resource_registry section update the port assignment values as follows: # Port assignments for the VIPs OS::TripleO::Network::Ports::NetVipMap: ../network/ports/net_vip_map_external.yaml OS::TripleO::Network::Ports::ExternalVipPort: ../network/ports/noop.yaml OS::TripleO::Network::Ports::InternalApiVipPort: ../network/ports/noop.yaml OS::TripleO::Network::Ports::StorageVipPort: ../network/ports/noop.yaml OS::TripleO::Network::Ports::StorageMgmtVipPort: ../network/ports/noop.yaml OS::TripleO::Network::Ports::RedisVipPort: ../network/ports/from_service.yaml # In the parameter_defaults section add VIPs for each network as follows (with IPs updated to align with your network): ControlPlaneIP: 192.168.200.100 ExternalNetworkVip: 192.168.225.100 InternalApiNetworkVip: 192.168.221.100 StorageNetworkVip: 192.168.223.100 StorageMgmtNetworkVip: 192.168.224.100 ServiceVips: redis: 192.168.221.101
I can confirm I have the same behavior in OSP10. Deployment fails with VIPs set in the beginning of the pool The VIP is the same as a regular IP assigned form the Pool to nodes, i.e. a storage node has the same IP as the VIP of the controller for the StorageNetworkVIP I am redeploying with an IP in the middle of the range, to see if this is an edge case. It's a bug, nonetheless, and a workaround could be described in the documentation page https://access.redhat.com/documentation/en/red-hat-openstack-platform/10/single/advanced-overcloud-customization/#sect-Predictable_VIPs NOTE: once the deployment have failed, I inspect all nodes for all their IPs with the following command (first do source stackrc) # ansible overcloud -i /bin/tripleo-ansible-inventory -u heat-admin --ssh-extra-args='-o StrictHostKeyChecking=no' -m shell -a "ip a | grep inet\ “ > output.txt Then look for duplicate IPs in output.txt [stack@undercloud ~]$ sort output.txt |grep global inet 10.12.48.100/32 brd 10.12.48.255 scope global br-ex inet 10.12.48.105/24 brd 10.12.48.255 scope global br-ex inet 10.12.48.109/24 brd 10.12.48.255 scope global br-ex inet 10.12.48.112/24 brd 10.12.48.255 scope global br-ex inet 172.16.3.20/24 brd 172.16.3.255 scope global br-mgmt inet 172.16.3.20/32 brd 172.16.3.255 scope global br-mgmt inet 172.16.3.23/24 brd 172.16.3.255 scope global br-mgmt inet 172.16.3.25/32 brd 172.16.3.255 scope global br-mgmt inet 172.16.3.26/24 brd 172.16.3.255 scope global br-mgmt inet 172.16.3.27/24 brd 172.16.3.255 scope global br-mgmt inet 172.16.3.28/24 brd 172.16.3.255 scope global br-mgmt inet 172.16.3.31/24 brd 172.16.3.255 scope global br-mgmt inet 172.16.3.37/24 brd 172.16.3.255 scope global br-mgmt inet 172.16.4.20/24 brd 172.16.4.255 scope global enp8s0 inet 172.16.4.20/32 brd 172.16.4.255 scope global enp8s0 inet 172.16.4.21/24 brd 172.16.4.255 scope global enp8s0 inet 172.16.4.24/24 brd 172.16.4.255 scope global enp8s0 inet 172.16.4.27/24 brd 172.16.4.255 scope global enp8s0 inet 172.16.4.29/24 brd 172.16.4.255 scope global enp8s0 inet 172.16.4.30/24 brd 172.16.4.255 scope global enp8s0 inet 172.16.4.38/24 brd 172.16.4.255 scope global enp8s0 inet 172.16.5.20/32 brd 172.16.5.255 scope global enp13s0 inet 172.16.5.28/24 brd 172.16.5.255 scope global enp13s0 inet 172.16.5.33/24 brd 172.16.5.255 scope global enp13s0 inet 172.16.5.35/24 brd 172.16.5.255 scope global enp13s0 inet 172.16.6.20/24 brd 172.16.6.255 scope global enp15s0 inet 172.16.6.23/24 brd 172.16.6.255 scope global enp15s0 inet 172.16.6.25/24 brd 172.16.6.255 scope global enp15s0 inet 172.16.6.26/24 brd 172.16.6.255 scope global enp14s0 inet 172.16.6.29/24 brd 172.16.6.255 scope global enp14s0 inet 172.16.6.30/24 brd 172.16.6.255 scope global enp14s0 inet 172.16.6.32/24 brd 172.16.6.255 scope global enp14s0 inet 192.168.0.21/24 brd 192.168.0.255 scope global enp6s0 inet 192.168.0.2/32 brd 192.168.0.255 scope global enp6s0 inet 192.168.0.25/24 brd 192.168.0.255 scope global enp6s0 inet 192.168.0.28/24 brd 192.168.0.255 scope global enp6s0 inet 192.168.0.31/24 brd 192.168.0.255 scope global enp6s0 inet 192.168.0.32/24 brd 192.168.0.255 scope global enp6s0 inet 192.168.0.34/24 brd 192.168.0.255 scope global enp6s0 inet 192.168.0.37/24 brd 192.168.0.255 scope global enp6s0 As you can see, some of the IPs ending in .20 appear in /24 (regular IP of a NIC) and /32 (VIP in a controlleR) - like 172.16.4.20 and 172.16.3.20 [stack@undercloud ~]$ tail templates/network-config.yaml -n 15 # Customize bonding options, e.g. "mode=4 lacp_rate=1 updelay=1000 miimon=100" BondInterfaceOvsOptions: "mode=active-backup" ExternalNetworkVip: 10.12.48.100 ControlPlaneIP: 192.168.0.2 InternalApiNetworkVip: 172.16.3.20 StorageNetworkVip: 172.16.4.20 StorageMgmtNetworkVip: 172.16.5.20 ServiceVips: redis: 172.16.3.21
The documentation states : Predictable IPs : "Make sure the chosen IP addresses fall outside the allocation pools for each network defined in your network environment file” [1] Predictable VIPs : "Select these IPs from their respective allocation pool ranges. For example, select the InternalApiNetworkVip from the InternalApiAllocationPools range.(...) This avoids any conflicts with the VIPs chosen for each network." [2] I do confirm setting predictable IPs from the allocation pools ends up in a conflict nevertheless the documentation looks fine to me. [1]https://access.redhat.com/documentation/en/red-hat-openstack-platform/10/single/advanced-overcloud-customization/#sect-Predictable_IPs [2]https://access.redhat.com/documentation/en/red-hat-openstack-platform/10/single/advanced-overcloud-customization/#sect-Predictable_VIPs
Confirmed that using Predictable VIPs that fall on the first part of the allocation block results in a failed deployment. I retried with a PredictableVIP at the end of the block, and it worked OK It makes me think that PredictableVIPs do not play nice with regular dinamic IPs. As you can see, the VIPs I want are not in neutron port-list (undercloud) [stack@undercloud ~]$ neutron port-list | grep virtual_ip | 17d4be25-a921-42e4-9da5-3912525110a0 | internal_api_virtual_ip | fa:16:3e:c1:a7:6c | {"subnet_id": "856ad1cb-38be-4253-a7d4-b2a73260999c", "ip_address": "172.16.3.27"} | | 3f7461bc-8349-4e65-a457-1495416441c5 | storage_management_virtual_ip | fa:16:3e:de:e0:c7 | {"subnet_id": "8899d7e6-099e-475b-b348-197002d18266", "ip_address": "172.16.5.21"} | | 47eb4883-0dfa-4c2f-94c1-7a5258a2ed0e | storage_virtual_ip | fa:16:3e:54:92:e7 | {"subnet_id": "4b3bcc4f-25e1-4907-91b1-05660b7f211b", "ip_address": "172.16.4.20"} | | 534a9d3d-18a4-440b-9bab-cb10c8a190a6 | public_virtual_ip | fa:16:3e:ee:bc:fc | {"subnet_id": "a83fb401-3036-4e9f-b233-7c522550c5e0", "ip_address": "10.12.48.106"} | | bec0047a-eaea-4971-b12e-a062821503e4 | control_virtual_ip | fa:16:3e:ac:76:5f | {"subnet_id": "2afc404d-94d2-4479-bbef-ee38e1bb0a9a", "ip_address": "192.168.0.26"} | | c5b8be1c-bacb-4f90-9748-490fe8a0fda2 | redis_virtual_ip | fa:16:3e:7c:72:66 | {"subnet_id": "856ad1cb-38be-4253-a7d4-b2a73260999c", "ip_address": "172.16.3.20"} | [stack@undercloud ~]$ cat templates/network-config.yaml #This file is an example of an environment file for defining the isolated #networks and related parameters. resource_registry: # Network Interface templates to use (these files must exist) OS::TripleO::Compute::Net::SoftwareConfig: /home/stack/templates/nic-configs/compute.yaml OS::TripleO::Controller::Net::SoftwareConfig: /home/stack/templates/nic-configs/controller.yaml OS::TripleO::CephStorage::Net::SoftwareConfig: /home/stack/templates/nic-configs/ceph-storage.yaml OS::TripleO::Network::Ports::NetVipMap: /usr/share/openstack-tripleo-heat-templates/network/ports/net_vip_map_external.yaml OS::TripleO::Network::Ports::ExternalVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::Network::Ports::InternalApiVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::Network::Ports::StorageVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::Network::Ports::StorageMgmtVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::Network::Ports::RedisVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/from_service.yaml parameter_defaults: InternalApiNetCidr: 172.16.3.0/24 StorageNetCidr: 172.16.4.0/24 StorageMgmtNetCidr: 172.16.5.0/24 TenantNetCidr: 172.16.6.0/24 ExternalNetCidr: 10.12.48.0/24 # CIDR subnet mask length for provisioning network ControlPlaneSubnetCidr: '24' # Customize the IP ranges on each network to use for static IPs and VIPs InternalApiAllocationPools: [{'start': '172.16.3.20', 'end': '172.16.3.100'}] StorageAllocationPools: [{'start': '172.16.4.20', 'end': '172.16.4.100'}] StorageMgmtAllocationPools: [{'start': '172.16.5.20', 'end': '172.16.5.100'}] TenantAllocationPools: [{'start': '172.16.6.20', 'end': '172.16.6.100'}] # Leave room if the external network is also used for floating IPs ExternalAllocationPools: [{'start': '10.12.48.100', 'end': '10.12.48.120'}] # Gateway router for the external network ExternalInterfaceDefaultRoute: 10.12.48.254 # Gateway router for the provisioning network (or Undercloud IP) ControlPlaneDefaultRoute: 192.168.0.1 # Generally the IP of the Undercloud EC2MetadataIp: 192.168.0.1 # Define the DNS servers (maximum 2) for the overcloud nodes DnsServers: ["10.12.48.11","8.8.8.8"] # Customize the VLAN IDs to match the local environment InternalApiNetworkVlanID: 3 #NOTE: we do not use tags, all vNICs are in native mode StorageNetworkVlanID: 4 #NOTE: we do not use tags, all vNICs are in native mode StorageMgmtNetworkVlanID: 5 #NOTE: we do not use tags, all vNICs are in native mode TenantNetworkVlanID: 6 #NOTE: we do not use tags, all vNICs are in native mode # Set to empty string to enable multiple external networks or VLANs NeutronExternalNetworkBridge: "br-ex" # Customize bonding options, e.g. "mode=4 lacp_rate=1 updelay=1000 miimon=100" BondInterfaceOvsOptions: "mode=active-backup" ExternalNetworkVip: 10.12.48.100 #must be 100 so DNS can work ControlPlaneIP: 192.168.0.4 InternalApiNetworkVip: 172.16.3.99 StorageNetworkVip: 172.16.4.99 StorageMgmtNetworkVip: 172.16.5.99 ServiceVips: redis: 172.16.3.98
My tests are not the same as Jon, as he tried BOTH predictableIP and predictableVIP, whereas I am only testing predictableVIPs, which does not work as documented. (In reply to Gregory Charot from comment #3) > The documentation states : > > Predictable IPs : > > "Make sure the chosen IP addresses fall outside the allocation pools for > each network defined in your network environment file” [1] > > Predictable VIPs : > > "Select these IPs from their respective allocation pool ranges. For example, > select the InternalApiNetworkVip from the InternalApiAllocationPools > range.(...) This avoids any conflicts with the VIPs chosen for each > network." [2] > > > I do confirm setting predictable IPs from the allocation pools ends up in a > conflict nevertheless the documentation looks fine to me. > > [1]https://access.redhat.com/documentation/en/red-hat-openstack-platform/10/ > single/advanced-overcloud-customization/#sect-Predictable_IPs > > [2]https://access.redhat.com/documentation/en/red-hat-openstack-platform/10/ > single/advanced-overcloud-customization/#sect-Predictable_VIPs
Testing a PredictableVIP inside the allocation pool, 2nd place to be exact ( InternalApiNetworkVip: 172.16.3.21 , where InternalApiAllocationPools: [{'start': '172.16.3.20', 'end': '172.16.3.100'}]), makes the deployment fail here: [stack@undercloud ~]$ openstack stack failures list overcloud overcloud.AllNodesDeploySteps.ControllerDeployment_Step2.0: resource_type: OS::Heat::StructuredDeployment physical_resource_id: 696c261b-0ddf-45f7-b92e-ab960d3047eb status: CREATE_FAILED status_reason: | Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1 deploy_stdout: | Matching apachectl 'Server version: Apache/2.4.6 (Red Hat Enterprise Linux) Server built: Aug 3 2016 08:33:27' Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked. deploy_stderr: | exception: connect failed Warning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided. Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications. Error: Duplicate declaration: Pcmk_resource[ip-172.16.3.21] is already declared in file /etc/puppet/modules/pacemaker/manifests/resource/ip.pp:106; cannot redeclare at /etc/puppet/modules/pacemaker/manifests/resource/ip.pp:106 on node overcloud-controller-0.localdomain Error: Duplicate declaration: Pcmk_resource[ip-172.16.3.21] is already declared in file /etc/puppet/modules/pacemaker/manifests/resource/ip.pp:106; cannot redeclare at /etc/puppet/modules/pacemaker/manifests/resource/ip.pp:106 on node overcloud-controller-0.localdomain [stack@undercloud ~]$ cat templates/network-config.yaml #This file is an example of an environment file for defining the isolated #networks and related parameters. resource_registry: # Network Interface templates to use (these files must exist) OS::TripleO::Compute::Net::SoftwareConfig: /home/stack/templates/nic-configs/compute.yaml OS::TripleO::Controller::Net::SoftwareConfig: /home/stack/templates/nic-configs/controller.yaml OS::TripleO::CephStorage::Net::SoftwareConfig: /home/stack/templates/nic-configs/ceph-storage.yaml OS::TripleO::Network::Ports::NetVipMap: /usr/share/openstack-tripleo-heat-templates/network/ports/net_vip_map_external.yaml OS::TripleO::Network::Ports::ExternalVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::Network::Ports::InternalApiVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::Network::Ports::StorageVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::Network::Ports::StorageMgmtVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::Network::Ports::RedisVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/from_service.yaml parameter_defaults: InternalApiNetCidr: 172.16.3.0/24 StorageNetCidr: 172.16.4.0/24 StorageMgmtNetCidr: 172.16.5.0/24 TenantNetCidr: 172.16.6.0/24 ExternalNetCidr: 10.12.48.0/24 # CIDR subnet mask length for provisioning network ControlPlaneSubnetCidr: '24' # Customize the IP ranges on each network to use for static IPs and VIPs InternalApiAllocationPools: [{'start': '172.16.3.20', 'end': '172.16.3.100'}] StorageAllocationPools: [{'start': '172.16.4.20', 'end': '172.16.4.100'}] StorageMgmtAllocationPools: [{'start': '172.16.5.20', 'end': '172.16.5.100'}] TenantAllocationPools: [{'start': '172.16.6.20', 'end': '172.16.6.100'}] # Leave room if the external network is also used for floating IPs ExternalAllocationPools: [{'start': '10.12.48.100', 'end': '10.12.48.120'}] # Gateway router for the external network ExternalInterfaceDefaultRoute: 10.12.48.254 # Gateway router for the provisioning network (or Undercloud IP) ControlPlaneDefaultRoute: 192.168.0.1 # Generally the IP of the Undercloud EC2MetadataIp: 192.168.0.1 # Define the DNS servers (maximum 2) for the overcloud nodes DnsServers: ["10.12.48.11","8.8.8.8"] # Customize the VLAN IDs to match the local environment InternalApiNetworkVlanID: 3 #NOTE: we do not use tags, all vNICs are in native mode StorageNetworkVlanID: 4 #NOTE: we do not use tags, all vNICs are in native mode StorageMgmtNetworkVlanID: 5 #NOTE: we do not use tags, all vNICs are in native mode TenantNetworkVlanID: 6 #NOTE: we do not use tags, all vNICs are in native mode # Set to empty string to enable multiple external networks or VLANs NeutronExternalNetworkBridge: "br-ex" # Customize bonding options, e.g. "mode=4 lacp_rate=1 updelay=1000 miimon=100" BondInterfaceOvsOptions: "mode=active-backup" ExternalNetworkVip: 10.12.48.100 #must be 100 so DNS can work ControlPlaneIP: 192.168.0.4 InternalApiNetworkVip: 172.16.3.21 StorageNetworkVip: 172.16.4.21 StorageMgmtNetworkVip: 172.16.5.21 ServiceVips: redis: 172.16.3.98
I have now tried with VIPs outside of the allocation pool, and the deployment worked OK. To be perfectly accurate, ExternalNetworkVIP is actually inside of the allocation pool, whereas all the other ones are now outside of the pool [stack@undercloud ~]$ cat templates/network-config.yaml #This file is an example of an environment file for defining the isolated #networks and related parameters. resource_registry: # Network Interface templates to use (these files must exist) OS::TripleO::Compute::Net::SoftwareConfig: /home/stack/templates/nic-configs/compute.yaml OS::TripleO::Controller::Net::SoftwareConfig: /home/stack/templates/nic-configs/controller.yaml OS::TripleO::CephStorage::Net::SoftwareConfig: /home/stack/templates/nic-configs/ceph-storage.yaml OS::TripleO::Network::Ports::NetVipMap: /usr/share/openstack-tripleo-heat-templates/network/ports/net_vip_map_external.yaml OS::TripleO::Network::Ports::ExternalVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::Network::Ports::InternalApiVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::Network::Ports::StorageVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::Network::Ports::StorageMgmtVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::Network::Ports::RedisVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/from_service.yaml parameter_defaults: InternalApiNetCidr: 172.16.3.0/24 StorageNetCidr: 172.16.4.0/24 StorageMgmtNetCidr: 172.16.5.0/24 TenantNetCidr: 172.16.6.0/24 ExternalNetCidr: 10.12.48.0/24 # CIDR subnet mask length for provisioning network ControlPlaneSubnetCidr: '24' # Customize the IP ranges on each network to use for static IPs and VIPs InternalApiAllocationPools: [{'start': '172.16.3.20', 'end': '172.16.3.100'}] StorageAllocationPools: [{'start': '172.16.4.20', 'end': '172.16.4.100'}] StorageMgmtAllocationPools: [{'start': '172.16.5.20', 'end': '172.16.5.100'}] TenantAllocationPools: [{'start': '172.16.6.20', 'end': '172.16.6.100'}] # Leave room if the external network is also used for floating IPs ExternalAllocationPools: [{'start': '10.12.48.100', 'end': '10.12.48.120'}] # Gateway router for the external network ExternalInterfaceDefaultRoute: 10.12.48.254 # Gateway router for the provisioning network (or Undercloud IP) ControlPlaneDefaultRoute: 192.168.0.1 # Generally the IP of the Undercloud EC2MetadataIp: 192.168.0.1 # Define the DNS servers (maximum 2) for the overcloud nodes DnsServers: ["10.12.48.11","8.8.8.8"] # Customize the VLAN IDs to match the local environment InternalApiNetworkVlanID: 3 #NOTE: we do not use tags, all vNICs are in native mode StorageNetworkVlanID: 4 #NOTE: we do not use tags, all vNICs are in native mode StorageMgmtNetworkVlanID: 5 #NOTE: we do not use tags, all vNICs are in native mode TenantNetworkVlanID: 6 #NOTE: we do not use tags, all vNICs are in native mode # Set to empty string to enable multiple external networks or VLANs NeutronExternalNetworkBridge: "br-ex" # Customize bonding options, e.g. "mode=4 lacp_rate=1 updelay=1000 miimon=100" BondInterfaceOvsOptions: "mode=active-backup" ExternalNetworkVip: 10.12.48.100 #must be 100 so DNS can work ControlPlaneIP: 192.168.0.4 InternalApiNetworkVip: 172.16.3.19 StorageNetworkVip: 172.16.4.19 StorageMgmtNetworkVip: 172.16.5.19 ServiceVips: redis: 172.16.3.98
I have also encountered this bug.
Documentation has been provided to not hit this bug: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/html-single/advanced_overcloud_customization/#sect-Predictable_VIPs Feel free to re-open if there is anything that needs to be fixed in the code.