Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2012222

Summary: 16.2 undercloud needs better heat tuning by default on large VMs.
Product: Red Hat OpenStack Reporter: Vincent S. Cojot <vcojot>
Component: python-tripleoclientAssignee: OSP Team <rhos-maint>
Status: CLOSED INSUFFICIENT_DATA QA Contact: David Rosenfeld <drosenfe>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 16.2 (Train)CC: aschultz, bdobreli, hbrock, jslagle, mburns, sbaker, sputhenp, steve
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-10-26 20:12:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Vincent S. Cojot 2021-10-08 15:13:42 UTC 
Description of problem:

On 16.2 it is a lot harder to get a successful deploy on the 1st attempt than on previous versions.

The undercloud I am using on 10/13/15/16.1/16.2 is always spec'ed the same:

- 32G RAM
- 12vcpus
- virtio-scsi disk (450-500Mb/sec as reported by 'hdparm -t').
- CPU passthrough and 2mb hugepages (Xeon Platinum SP Gen2)

On 16.2, most of the time a simple deploy (3controllers, 2computes, 1 or 3ceph) is not successful the first time.
Comment 1 Vincent S. Cojot 2021-10-08 15:17:12 UTC 
Hypervisor is running UC + Overcloud (as usual). All VMs are using CPU passthroug and dedicated hugepages:

# virsh list
 Id    Name                  State
--------------------------------------
 2     idm                   running
 28    vom7                  running
 99    bdc414x               running
 164   instack-16.2          running
 173   overcloud-ceph-0      running
 174   overcloud-control-2   running
 179   overcloud-ceph-1      running
 180   overcloud-compute-5   running
 181   overcloud-ceph-3      running
 182   overcloud-compute-3   running
 184   overcloud-control-0   running

The undercloud is 'instack-16.2':


[root@osp16d ~]# free -g
              total        used        free      shared  buff/cache   available
Mem:             31          13           1           0          15          17
Swap:             7           0           7
[root@osp16d ~]# lscpu |grep Model
Model:               85
Model name:          Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz
[root@osp16d ~]# lscpu |grep CPU.s
CPU(s):              12
On-line CPU(s) list: 0-11
NUMA node0 CPU(s):   0-11
[root@osp16d ~]# hdparm -t /dev/vda

/dev/vda:
 Timing buffered disk reads: 1066 MB in  3.00 seconds = 354.83 MB/sec
Comment 2 Vincent S. Cojot 2021-10-08 15:19:08 UTC 
On a fresh deploy (no overcloud deploy, here's what I am seeing) - sometimes the 3 controllers don't even become ACTIVE:
(undercloud) [stack@osp16d ~]$ nova list 
+--------------------------------------+--------------------+--------+------------+-------------+----------------------+
| ID                                   | Name               | Status | Task State | Power State | Networks             |
+--------------------------------------+--------------------+--------+------------+-------------+----------------------+
| 0f80bb07-f67e-4854-9766-71ed49f5c9a4 | krynn-osp16-ceph-0 | ACTIVE | -          | Running     | ctlplane=10.20.0.113 |
| 984980f6-5912-4186-a3c5-372ff89cd6e0 | krynn-osp16-ceph-1 | ACTIVE | -          | Running     | ctlplane=10.20.0.122 |
| 7ef6546a-7eaf-41d2-8a99-785bef6c10a1 | krynn-osp16-ceph-2 | ACTIVE | -          | Running     | ctlplane=10.20.0.132 |
| 73913049-9437-4f47-8fd4-5c582cdd9b94 | krynn-osp16-cmpt-0 | ACTIVE | -          | Running     | ctlplane=10.20.0.131 |
| 906222c1-28d1-4140-95e0-be12d0d40286 | krynn-osp16-cmpt-1 | ACTIVE | -          | Running     | ctlplane=10.20.0.129 |
| 4f103afb-3e58-4923-8eed-b3ad867eaf62 | krynn-osp16-ctrl-0 | ERROR  | -          | NOSTATE     |                      |
| 1cda61ca-da21-4d50-8d49-ca5bf9e722b1 | krynn-osp16-ctrl-1 | ACTIVE | -          | Running     | ctlplane=10.20.0.116 |
| c1a6f67b-8dbc-4eee-ad03-73ca49bc9124 | krynn-osp16-ctrl-2 | ACTIVE | -          | Running     | ctlplane=10.20.0.103 |
+--------------------------------------+--------------------+--------+------------+-------------+----------------------+
(undercloud) [stack@osp16d ~]$ openstack baremetal node list
+--------------------------------------+-------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name  | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+-------+--------------------------------------+-------------+--------------------+-------------+
| ac489bcb-dab5-498a-88fa-d9c9e202a36a | bm-0  | 7ef6546a-7eaf-41d2-8a99-785bef6c10a1 | power on    | active             | False       |
| 341b8249-a8ec-4ba2-81e0-cc75638c954f | bm-1  | 984980f6-5912-4186-a3c5-372ff89cd6e0 | power on    | active             | False       |
| aeefb7ab-3ffc-455d-acad-3daee0c4b5ee | bm-2  | None                                 | power off   | available          | False       |
| 5fe3e686-2a5e-4e70-8dc2-a2dbb2a13984 | bm-3  | 0f80bb07-f67e-4854-9766-71ed49f5c9a4 | power on    | active             | False       |
| c25f70b3-e963-4026-8b33-d9f09a13b3d8 | bm-4  | None                                 | power off   | available          | False       |
| 3933d1dc-b481-40ba-a33c-bb96b53a3f51 | bm-5  | None                                 | power off   | available          | False       |
| fb3ea0b3-dbf8-4cdc-9e9f-c7dd628e9e32 | bm-6  | None                                 | power off   | available          | False       |
| 63040ddf-e640-449d-9f70-b65b6315030b | bm-7  | 1cda61ca-da21-4d50-8d49-ca5bf9e722b1 | power on    | active             | False       |
| 09c49ff3-bd06-4b72-abd6-4c33504d221a | bm-8  | None                                 | power off   | available          | False       |
| cf970d12-a3f9-435e-8e7f-7dafa4d38228 | bm-9  | c1a6f67b-8dbc-4eee-ad03-73ca49bc9124 | power on    | active             | False       |
| 4ca9d062-8c0c-4524-bbd3-4802de3d30f1 | bm-10 | None                                 | power off   | available          | False       |
| cd57f708-b004-4038-93e8-238916bc3c4b | bm-11 | None                                 | power off   | available          | False       |
| 756f4bcf-da7b-4ca1-a358-d9eedd484245 | bm-12 | None                                 | power off   | available          | False       |
| 99b49740-221a-49f9-8a79-107e3617c784 | bm-13 | 906222c1-28d1-4140-95e0-be12d0d40286 | power on    | active             | False       |
| 0ac3252b-caaa-43f6-a1b1-296dd3a84571 | bm-14 | None                                 | power off   | available          | False       |
| f046bfa5-9cce-40cf-905b-ce62aa6e26b8 | bm-15 | 73913049-9437-4f47-8fd4-5c582cdd9b94 | power on    | active             | False       |
+--------------------------------------+-------+--------------------------------------+-------------+--------------------+-------------+
Comment 3 Vincent S. Cojot 2021-10-08 15:20:25 UTC 
(undercloud) [stack@osp16d ~]$ openstack stack list
+--------------------------------------+------------+----------------------------------+---------------+----------------------+--------------+
| ID                                   | Stack Name | Project                          | Stack Status  | Creation Time        | Updated Time |
+--------------------------------------+------------+----------------------------------+---------------+----------------------+--------------+
| 1f0a41d1-cfde-4fc9-9a74-cfb53e44a1d4 | overcloud  | 7320ad71457e42d09d65464f33027e9f | CREATE_FAILED | 2021-10-07T22:46:25Z | None         |
+--------------------------------------+------------+----------------------------------+---------------+----------------------+--------------+

(undercloud) [stack@osp16d ~]$ openstack stack list --nested --hidden --property status=CREATE_FAILED   
+--------------------------------------+---------------------------------------------------+----------------------------------+---------------+----------------------+--------------+--------------------------------------+
| ID                                   | Stack Name                                        | Project                          | Stack Status  | Creation Time        | Updated Time | Parent                               |
+--------------------------------------+---------------------------------------------------+----------------------------------+---------------+----------------------+--------------+--------------------------------------+
| 400d9159-41c3-498c-8541-c31e761bc438 | overcloud-Controller-zmqbamsujrby-2-e66qczgyavmv  | 7320ad71457e42d09d65464f33027e9f | CREATE_FAILED | 2021-10-07T22:50:27Z | None         | d38a51e4-b503-4dae-8cc8-3f7075b1cb03 |
| 1a5d8e3b-d70a-43aa-80ec-00d4f4ad46ea | overcloud-Controller-zmqbamsujrby-1-igd3lhwqp7vr  | 7320ad71457e42d09d65464f33027e9f | CREATE_FAILED | 2021-10-07T22:50:24Z | None         | d38a51e4-b503-4dae-8cc8-3f7075b1cb03 |
| 32b3ee28-d268-4ae1-9b88-7000c88b0415 | overcloud-Controller-zmqbamsujrby-0-eodzjuekhevy  | 7320ad71457e42d09d65464f33027e9f | CREATE_FAILED | 2021-10-07T22:50:23Z | None         | d38a51e4-b503-4dae-8cc8-3f7075b1cb03 |
| 6323c2d5-1dbd-4acb-8437-39b9cb9d02a8 | overcloud-Compute-ndcjmlauhnrs-1-kzg7cpnnr7u5     | 7320ad71457e42d09d65464f33027e9f | CREATE_FAILED | 2021-10-07T22:47:51Z | None         | b7eba772-7afd-455d-b321-02aaac3cce31 |
| 663e57f7-a396-40fa-89a8-9b3806ac9862 | overcloud-Compute-ndcjmlauhnrs-0-kapr2kjxmk2r     | 7320ad71457e42d09d65464f33027e9f | CREATE_FAILED | 2021-10-07T22:47:50Z | None         | b7eba772-7afd-455d-b321-02aaac3cce31 |
| 9c142d4e-1e57-4a31-85f0-3b71a64ef17b | overcloud-CephStorage-fxtzo2p7x3lh-1-zlcegpmh2gxa | 7320ad71457e42d09d65464f33027e9f | CREATE_FAILED | 2021-10-07T22:47:37Z | None         | 5982917e-950b-4f00-9ece-63582d4adc46 |
| f53680a1-d631-4daa-9b34-ee5f6a6df43b | overcloud-CephStorage-fxtzo2p7x3lh-0-3au2vzqkvvmh | 7320ad71457e42d09d65464f33027e9f | CREATE_FAILED | 2021-10-07T22:47:36Z | None         | 5982917e-950b-4f00-9ece-63582d4adc46 |
| 6a378c86-502a-4877-bf08-fae9880d8a1e | overcloud-CephStorage-fxtzo2p7x3lh-2-adet3ztergsp | 7320ad71457e42d09d65464f33027e9f | CREATE_FAILED | 2021-10-07T22:47:35Z | None         | 5982917e-950b-4f00-9ece-63582d4adc46 |
| 1f0a41d1-cfde-4fc9-9a74-cfb53e44a1d4 | overcloud                                         | 7320ad71457e42d09d65464f33027e9f | CREATE_FAILED | 2021-10-07T22:46:25Z | None         | None                                 |
+--------------------------------------+---------------------------------------------------+----------------------------------+---------------+----------------------+--------------+--------------------------------------+

(undercloud) [stack@osp16d ~]$  openstack stack resource list --nested-depth 5 --filter status=CREATE_FAILED overcloud
+---------------+--------------------------------------+--------------------------------+-----------------+----------------------+---------------------------------------------------+
| resource_name | physical_resource_id                 | resource_type                  | resource_status | updated_time         | stack_name                                        |
+---------------+--------------------------------------+--------------------------------+-----------------+----------------------+---------------------------------------------------+
| CephStorage   | 5982917e-950b-4f00-9ece-63582d4adc46 | OS::Heat::ResourceGroup        | CREATE_FAILED   | 2021-10-07T22:46:27Z | overcloud                                         |
| Controller    | d38a51e4-b503-4dae-8cc8-3f7075b1cb03 | OS::Heat::ResourceGroup        | CREATE_FAILED   | 2021-10-07T22:46:27Z | overcloud                                         |
| Compute       | b7eba772-7afd-455d-b321-02aaac3cce31 | OS::Heat::ResourceGroup        | CREATE_FAILED   | 2021-10-07T22:46:27Z | overcloud                                         |
| 0             | f53680a1-d631-4daa-9b34-ee5f6a6df43b | OS::TripleO::CephStorage       | CREATE_FAILED   | 2021-10-07T22:47:32Z | overcloud-CephStorage-fxtzo2p7x3lh                |
| 1             | 9c142d4e-1e57-4a31-85f0-3b71a64ef17b | OS::TripleO::CephStorage       | CREATE_FAILED   | 2021-10-07T22:47:32Z | overcloud-CephStorage-fxtzo2p7x3lh                |
| 2             | 6a378c86-502a-4877-bf08-fae9880d8a1e | OS::TripleO::CephStorage       | CREATE_FAILED   | 2021-10-07T22:47:32Z | overcloud-CephStorage-fxtzo2p7x3lh                |
| CephStorage   | 0f80bb07-f67e-4854-9766-71ed49f5c9a4 | OS::TripleO::CephStorageServer | CREATE_FAILED   | 2021-10-07T22:47:39Z | overcloud-CephStorage-fxtzo2p7x3lh-0-3au2vzqkvvmh |
| CephStorage   | 984980f6-5912-4186-a3c5-372ff89cd6e0 | OS::TripleO::CephStorageServer | CREATE_FAILED   | 2021-10-07T22:47:39Z | overcloud-CephStorage-fxtzo2p7x3lh-1-zlcegpmh2gxa |
| CephStorage   | 7ef6546a-7eaf-41d2-8a99-785bef6c10a1 | OS::TripleO::CephStorageServer | CREATE_FAILED   | 2021-10-07T22:47:38Z | overcloud-CephStorage-fxtzo2p7x3lh-2-adet3ztergsp |
| 0             | 32b3ee28-d268-4ae1-9b88-7000c88b0415 | OS::TripleO::Controller        | CREATE_FAILED   | 2021-10-07T22:50:13Z | overcloud-Controller-zmqbamsujrby                 |
| 1             | 1a5d8e3b-d70a-43aa-80ec-00d4f4ad46ea | OS::TripleO::Controller        | CREATE_FAILED   | 2021-10-07T22:50:13Z | overcloud-Controller-zmqbamsujrby                 |
| 2             | 400d9159-41c3-498c-8541-c31e761bc438 | OS::TripleO::Controller        | CREATE_FAILED   | 2021-10-07T22:50:13Z | overcloud-Controller-zmqbamsujrby                 |
| Controller    | 4f103afb-3e58-4923-8eed-b3ad867eaf62 | OS::TripleO::ControllerServer  | CREATE_FAILED   | 2021-10-07T22:50:59Z | overcloud-Controller-zmqbamsujrby-0-eodzjuekhevy  |
| Controller    | 1cda61ca-da21-4d50-8d49-ca5bf9e722b1 | OS::TripleO::ControllerServer  | CREATE_FAILED   | 2021-10-07T22:50:59Z | overcloud-Controller-zmqbamsujrby-1-igd3lhwqp7vr  |
| Controller    | c1a6f67b-8dbc-4eee-ad03-73ca49bc9124 | OS::TripleO::ControllerServer  | CREATE_FAILED   | 2021-10-07T22:51:27Z | overcloud-Controller-zmqbamsujrby-2-e66qczgyavmv  |
| 0             | 663e57f7-a396-40fa-89a8-9b3806ac9862 | OS::TripleO::Compute           | CREATE_FAILED   | 2021-10-07T22:47:47Z | overcloud-Compute-ndcjmlauhnrs                    |
| 1             | 6323c2d5-1dbd-4acb-8437-39b9cb9d02a8 | OS::TripleO::Compute           | CREATE_FAILED   | 2021-10-07T22:47:47Z | overcloud-Compute-ndcjmlauhnrs                    |
| NovaCompute   | 73913049-9437-4f47-8fd4-5c582cdd9b94 | OS::TripleO::ComputeServer     | CREATE_FAILED   | 2021-10-07T22:47:51Z | overcloud-Compute-ndcjmlauhnrs-0-kapr2kjxmk2r     |
| NovaCompute   | 906222c1-28d1-4140-95e0-be12d0d40286 | OS::TripleO::ComputeServer     | CREATE_FAILED   | 2021-10-07T22:47:52Z | overcloud-Compute-ndcjmlauhnrs-1-kzg7cpnnr7u5     |
+---------------+--------------------------------------+--------------------------------+-----------------+----------------------+---------------------------------------------------+
Comment 4 Vincent S. Cojot 2021-10-08 15:21:08 UTC 
In the containers logs, here's what I see:

[root@osp16d ~]# sudo grep -r 'HTTP 504' /var/log/containers/neutron/
/var/log/containers/neutron/ironic-neutron-agent.log.1:2021-10-07 19:01:19.315 74435 ERROR networking_baremetal.agent.ironic_neutron_agent [-] Failed to get ironic port data!: ironicclient.common.apiclient.exceptions.GatewayTimeout: Gateway Timeout (HTTP 504)
/var/log/containers/neutron/ironic-neutron-agent.log.1:2021-10-07 19:01:19.315 74435 ERROR networking_baremetal.agent.ironic_neutron_agent ironicclient.common.apiclient.exceptions.GatewayTimeout: Gateway Timeout (HTTP 504)
/var/log/containers/neutron/ironic-neutron-agent.log.1:2021-10-07 19:03:20.472 74435 ERROR networking_baremetal.agent.ironic_neutron_agent [-] Failed to get ironic port data!: ironicclient.common.apiclient.exceptions.GatewayTimeout: Gateway Timeout (HTTP 504)
/var/log/containers/neutron/ironic-neutron-agent.log.1:2021-10-07 19:03:20.472 74435 ERROR networking_baremetal.agent.ironic_neutron_agent ironicclient.common.apiclient.exceptions.GatewayTimeout: Gateway Timeout (HTTP 504)
Comment 5 Vincent S. Cojot 2021-10-08 15:22:25 UTC 
My undercloud what configured with this:

[DEFAULT]
image_path = /home/stack/images/
container_images_file = /home/stack/OSP/osp16.2/containers-prepare-parameter.yaml
enabled_hardware_types = ipmi,redfish,idrac
enabled_boot_interfaces = pxe
enabled_deploy_interfaces = iscsi,direct
enabled_management_interfaces = ipmitool
enabled_power_interfaces = ipmitool
discovery_default_driver = ipmi

# Local site
# undercloud_hostname = osp16h.lasthome.solace.krynn
overcloud_domain_name = lasthome.solace.krynn
subnets = ctlplane-subnet
local_subnet = ctlplane-subnet
undercloud_nameservers = 10.0.128.234,10.0.128.236,10.0.128.254
undercloud_ntp_servers = 10.20.0.1,10.0.128.234,10.0.128.236,10.0.128.254
hieradata_override = /home/stack/OSP/osp16.2/undercloud-override.yaml

# Certificates
generate_service_certificate = true
certificate_generation_ca = local
undercloud_service_certificate = /etc/pki/instack-certs/undercloud.pem

# Network
local_ip = 10.20.0.2/24
# undercloud_public_host = 10.0.129.42
undercloud_public_host = 10.20.0.3
undercloud_admin_host = 10.20.0.4
local_interface = bond0
inspection_interface = br-ctlplane
inspection_iprange = 10.20.0.200,10.20.0.240

# Features
enable_mistral = true
enable_zaqar = true
enable_telemetry = false
enable_tempest = true
inspection_extras = true
enable_validations = true
clean_nodes = false
enable_node_discovery = true
ipxe_enabled = true
# inspection_enable_uefi = true

[ctlplane-subnet]
cidr = 10.20.0.0/24
dhcp_start = 10.20.0.101
dhcp_end = 10.20.0.164
inspection_iprange = 10.20.0.200,10.20.0.240
gateway = 10.20.0.1
masquerade = true

And these overrides:
$ cat OSP/osp16.2/undercloud-override.yaml 
# HAProxy timeouts
tripleo::haproxy::ssl_cipher_suite: "!SSLv2:kEECDH:kRSA:kEDH:kPSK:+3DES:!aNULL:!eNULL:!MD5:!EXP:!RC4:!SEED:!IDEA:!DES:!MEDIUM"
tripleo::haproxy::ssl_options: 'no-sslv3 no-tls-tickets'
tripleo::haproxy::haproxy_global_maxconn: 32768
tripleo::haproxy::haproxy_default_maxconn: 8192
# Keystone token expiry (for updates on large clouds):
keystone::token_expiration: 14400
# MySQL
mysql_max_connections: '8192'
#
nova::compute::ironic::max_concurrent_builds: 8
nova::rpc_response_timeout: '600'
ironic::rpc_response_timeout: 600
# IPMI timeouts
ironic::drivers::ipmi::command_retry_timeout: 120
ironic::drivers::ipmi::min_command_interval: 10
# Ironic cleaning
ironic::conductor::cleaning_disk_erase: metadata
ironic::conductor::cleaning_network: ctlplane
Comment 6 Vincent S. Cojot 2021-10-08 16:42:00 UTC 
My undercloud is a VM.
With 16.2 I found it configured with the tuned profile 'throughput-performance':

[root@osp16d /home/stack/undercloud-ansible-1arxw8x6]# grep through *
ansible.log:2021-10-08 11:44:51,258 p=7960 u=root n=ansible | 2021-10-08 11:44:51.258637 | 560d00af-e102-5bfd-6af0-000000000d53 |       TASK | Configure isolated cores 
for profile throughput-performance
ansible.log:2021-10-08 11:44:51,312 p=7960 u=root n=ansible | 2021-10-08 11:44:51.312062 | 560d00af-e102-5bfd-6af0-000000000d53 |    SKIPPED | Configure isolated cores 
for profile throughput-performance | osp16d


For underclouds as VMs, I usually (since OSP7) configured the tuned profile as 'latency-performance' or 'virtual-guest'.
For a message passing and Database systems such as the undercloud, 'throughput-performance' seemed to be a performance killer for me.
I've switched to latency-performance before the deploy and will report here.
Comment 7 Vincent S. Cojot 2021-10-08 16:57:45 UTC 
In the initial phase of the deploy (before ansible and before ironic hosts are launched), I'm seeing this:

[stack@osp16d ~]$ . stackrc
(undercloud) [stack@osp16d ~]$ nova list
ERROR (GatewayTimeout): Gateway Timeout (HTTP 504)
(undercloud) [stack@osp16d ~]$ nova list

Broadcast message from systemd-journald.solace.krynn (Fri 2021-10-08 12:55:54 EDT):

haproxy[37920]: proxy keystone_public has no server available!


Broadcast message from systemd-journald.solace.krynn (Fri 2021-10-08 12:55:54 EDT):

haproxy[37920]: proxy keystone_admin has no server available!

ERROR (GatewayTimeout): Gateway Timeout (HTTP 504)
Comment 8 Vincent S. Cojot 2021-10-08 17:00:44 UTC 
(undercloud) [stack@osp16d ~]$ nova list
+--------------------------------------+--------------------+--------+------------+-------------+----------+
| ID                                   | Name               | Status | Task State | Power State | Networks |
+--------------------------------------+--------------------+--------+------------+-------------+----------+
| 3acba8a8-59d4-40e3-a554-9ef12a416a23 | krynn-osp16-ceph-0 | ERROR  | -          | NOSTATE     |          |
| ae900367-acef-46aa-8660-0ae90e203e24 | krynn-osp16-cmpt-0 | BUILD  | -          | NOSTATE     |          |
| 04711966-956f-4c6d-a661-5d6855fdec1f | krynn-osp16-cmpt-1 | BUILD  | -          | NOSTATE     |          |
| b5800fe4-2af0-48d1-a290-eafb0f45dfc7 | krynn-osp16-ctrl-1 | BUILD  | spawning   | NOSTATE     |          |
| 7c429291-a674-4a17-93f0-4c3e8882040d | krynn-osp16-ctrl-2 | ERROR  | -          | NOSTATE     |          |
+--------------------------------------+--------------------+--------+------------+-------------+----------+

(this is a smaller deploy)
Comment 9 Vincent S. Cojot 2021-10-08 17:01:35 UTC 
(undercloud) [stack@osp16d ~]$ openstack baremetal node list
+--------------------------------------+-------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name  | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+-------+--------------------------------------+-------------+--------------------+-------------+
| 66b5ee08-22aa-4823-8c67-44729b013597 | bm-0  | None                                 | power off   | available          | False       |
| bf9dd6b1-a084-4e06-af62-8c294e50aef0 | bm-1  | None                                 | power on    | deploy failed      | False       |
| a4dd8443-56a7-4355-9a58-3881c059cf05 | bm-2  | None                                 | power off   | available          | False       |
| 722d7c67-524c-4f67-87b8-3e8203e967d4 | bm-3  | None                                 | power off   | available          | False       |
| e9679ff3-6074-40fa-ac32-3ceae44967e2 | bm-4  | None                                 | power off   | available          | False       |
| 40ec54d0-5ae3-45f6-b119-397d0bfcea27 | bm-5  | None                                 | power off   | available          | False       |
| c1dcbe24-b073-49c0-829d-b915690ee4fc | bm-6  | None                                 | power off   | available          | False       |
| a21b9601-66d1-4b52-9b69-b02984701103 | bm-7  | None                                 | power off   | available          | False       |
| d5184bba-170a-46a6-b057-4d329c1d7b87 | bm-8  | None                                 | power off   | available          | False       |
| 9455d5c3-b34a-4113-a01b-1c8f57f589a7 | bm-9  | b5800fe4-2af0-48d1-a290-eafb0f45dfc7 | power on    | deploying          | False       |
| 9ec63890-e46f-41cf-a88a-465c9b59097b | bm-10 | None                                 | power off   | available          | False       |
| 5643505b-c89c-4d2d-be68-c7b979fc9195 | bm-11 | 04711966-956f-4c6d-a661-5d6855fdec1f | power on    | wait call-back     | False       |
| 26a430ad-dca7-403b-a4ca-6db51ef4e3cd | bm-12 | None                                 | power off   | available          | False       |
| 4f4ddf4b-e07a-4f55-b1fa-4ae008acfa38 | bm-13 | None                                 | power off   | available          | False       |
| 3bd7f534-5d5b-4381-88a5-3c698133646b | bm-14 | None                                 | power off   | available          | False       |
| 4d7241c5-f986-41c5-aedd-0bc86a18b68b | bm-15 | ae900367-acef-46aa-8660-0ae90e203e24 | power on    | wait call-back     | False       |
+--------------------------------------+-------+--------------------------------------+-------------+--------------------+-------------+
Comment 10 Vincent S. Cojot 2021-10-08 18:14:21 UTC 
Might be related to the podman downloads of many containers by tripleo when they care aren't cached

on the 2nd attempt, I'm never seeing load avg spike (stays around 3) and podman images works immmediately.. I wonder if the job of pre-caching the containers is not getting fully executed and complete before the heat stack starts creating..

I deleted my previous failed stack and re-launched.
I will delete that stack again and then I will also delete the cached containers on my undercloud (I am using Satellite on my LAN).
I'll report here...
Comment 11 Vincent S. Cojot 2021-10-08 18:15:39 UTC 
At any case, I find it wrong to enforce a tuned profile of 'thoughput-performance' on a large VM when it should be using 'virtual-guest'
Comment 12 Alex Schultz 2021-10-08 18:48:51 UTC 
The tuning of the undercloud is up to the end user especially when on an external hypervisor system.  I've been able to deploy larger clouds with less so there's likely some other performance related problem.
Comment 13 Alex Schultz 2021-10-08 21:57:56 UTC 
I was able to deploy 3ctl+5comp on a 16G 4vcpu undercloud with all the nodes as VMs on the same hypervisors. However since the provided errors seem to be related to node provisioning, I am moving over to ironic for their review. From a deployment standpoint, this is not an officially supported architecture for customers and we don't really recommend a VM for the undercloud (thought it is technically supported) primarily due to the resource contention issues.


[stack@undercloud ~]$ lscpu | grep Model
Model:               2
Model name:          AMD Opteron(tm) Processor 6376
[stack@undercloud ~]$ lscpu | grep CPU.s
CPU(s):              4
On-line CPU(s) list: 0-3
NUMA node0 CPU(s):   0-3
[stack@undercloud ~]$ free
              total        used        free      shared  buff/cache   available
Mem:       16212188     7173936     1363212       62552     7675040     8470032
Swap:       4194300       33664     4160636
[stack@undercloud ~]$ sudo hdparm -t /dev/vda

/dev/vda:
 Timing buffered disk reads: 620 MB in  3.04 seconds = 203.87 MB/sec

[stack@undercloud ~]$ source stackrc
(undercloud) [stack@undercloud ~]$ openstack baremetal node list
+--------------------------------------+-------------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name        | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+-------------+--------------------------------------+-------------+--------------------+-------------+
| 9ad33d98-1c0d-4695-98c2-985b09da3854 | overcloud-2 | 08f78f66-8dba-4265-ac90-b74198bd7f9d | power on    | active             | False       |
| 3b8f5da1-6fa3-4657-8740-21929e3ad6cb | overcloud-4 | a925b33c-446e-414f-a42c-f9483610fbf6 | power on    | active             | False       |
| 33b5bb51-d50f-4292-a29f-c9ded39ed0cc | overcloud-1 | 48f810e0-adcd-4357-81a5-ea60b05e4eee | power on    | active             | False       |
| 69a2bb77-ec81-40e9-b3e0-523bc451b0d1 | overcloud-6 | 1ab0f102-5b45-4ca8-a969-53e71475c979 | power on    | active             | False       |
| 4cb7aa43-3bbc-4731-acdb-6252a2240358 | overcloud-7 | 2a575692-c48c-43ad-b76a-909da1d1b28c | power on    | active             | False       |
| 0ab40ebd-0dcd-42bf-89de-57049ed93e1e | overcloud-0 | 204a0915-e437-496b-9efe-364ec3015b5a | power on    | active             | False       |
| 0bf5ff3b-ef40-484b-8e1a-e3e61a532328 | overcloud-5 | 1377faf8-0a95-4b5d-a126-0af32bcff519 | power on    | active             | False       |
| 18b545e3-810d-4298-a4ed-3d01e4d3bb57 | overcloud-8 | None                                 | power off   | available          | False       |
| 02ad8e19-8419-4816-8e57-ed873e59951d | overcloud-3 | b5dc4c10-1c47-4d5f-9baf-cf12f70ce972 | power on    | active             | False       |
+--------------------------------------+-------------+--------------------------------------+-------------+--------------------+-------------+
(undercloud) [stack@undercloud ~]$
Comment 16 Vincent S. Cojot 2021-10-09 01:03:44 UTC 
BTW, I think the podman download was a red herring but I did see lots of qemu-img processes during the heat phase of deployment which were causing the 30-35% iowait.
Comment 18 Vincent S. Cojot 2021-10-11 18:46:51 UTC 
I'm having a clearer picture of what might have happened:

My server (A Dell T640) was last rebooted on Oct 2nd. Everything was fine. Small Deploys of 16.2 over the week-end (Oct 3rd/4th) were fine.

On October 6th, my server pulled down a big qemu-kvm/libvirtd update (see below).

Shortly after that (Oct 7th/8th), I started experiencing 504 http errors on the undercloud during a fresh deploy due to high iowait.

I ended up rebooting the server on Oct 11th.
What worries me is this: the server is running RHEL 8.4 with the same qemu-kvm from advanced-virt (like RHV 4.4 and RHOSP 16.2 computes), does that mean that RHOSP computes -must- be rebooted even if there is no kernel update and 'needs-restarting -r' doesn't say so?
Comment 19 Vincent S. Cojot 2021-10-12 00:17:03 UTC 
On the 8.4 kernel elevator=noop or elevator=none in GRUB_CMDLINE_LINUX do not work at all. 
Even if 'elevator=noop' is added to the 8.4 kernel, the virtio disks still show up on 'deadline'.

I am getting better results on 16.2 after adding this to my undercloud.

(undercloud) [stack@osp16d ~]$ cat /etc/udev/rules.d/60-ssd-scheduler.rules
ACTION=="add|change", KERNEL=="[sv]d[a-z]", ATTR{queue/scheduler}="none"

I will continue testing but with the above a 3controllers + 3 ceph + 2 computes deploy was launched successsfully from my undercloud on the first attempt.
this points at a possible issue in the 8.4 kernel used on the undercloud.

I will continue testing and report here.
Comment 20 Steve Baker 2021-10-12 19:43:44 UTC 
When you've done more testing can you make the full undercloud logs available for download?
Comment 21 Steve Baker 2021-10-12 19:51:03 UTC 
It looks like the issue is that overcloud deploy is started before all of the undercloud services have started. This isn't necessarily a realistic scenario, and is more a symptom of automation going too fast on an undercloud with particular tuning that has faster throughput than we test for.

Can the automation add a step before deploy to poll undercloud endpoints to confirm all services are up before attempting deployment?
Comment 22 Vincent S. Cojot 2021-10-13 00:54:28 UTC 
@steve I am pretty sure the undercloud services are up before the deploy is launched because the next step after undercloud install is to import the ironic director images and the instackenv.json.

Also. tonight I am testing this:
1) fresh 16.2 undercloud with 4.18.0-305.19.1.el8_4
2) first deploy attempt fails with 504 http errors.

3) then I installed the latest 8.2 kernel: 4.18.0-193.65.2.el8_2

4) Rebooted into that kernel and it is looking a lot better (no failures so far)

I will add the 4.18.0-193.65.2.el8_2 kernel injection into the pipeline so I can reproduce it with a full pipeline.

What undercloud logs did you want, Steve? /var/log/containers and friends?
Comment 23 Vincent S. Cojot 2021-10-13 00:55:53 UTC 
the 16.2 undercloud with the 8.2 kernel succeeded the first time:

(undercloud) [stack@osp16d ~]$ openstack stack list
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
| ID                                   | Stack Name | Project                          | Stack Status    | Creation Time        | Updated Time |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
| e0a0cf34-c9b7-4f8f-ad03-3e780e140300 | overcloud  | 8f925b56a2d94577a54f694fbb7cca5a | CREATE_COMPLETE | 2021-10-13T00:40:13Z | None         |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
(undercloud) [stack@osp16d ~]$ nova list
+--------------------------------------+--------------------+--------+------------+-------------+----------------------+
| ID                                   | Name               | Status | Task State | Power State | Networks             |
+--------------------------------------+--------------------+--------+------------+-------------+----------------------+
| bd8a1c10-6c73-47aa-acce-d1f231bd71fc | krynn-osp16-ceph-0 | ACTIVE | -          | Running     | ctlplane=10.20.0.139 |
| b4ad84ad-1684-43f6-b636-de47d1dcc023 | krynn-osp16-ceph-1 | ACTIVE | -          | Running     | ctlplane=10.20.0.145 |
| df212984-38ae-4159-86d2-7c66c25cb8a3 | krynn-osp16-ceph-2 | ACTIVE | -          | Running     | ctlplane=10.20.0.156 |
| 6ee1af29-db3f-4bef-a755-25b6dbea086b | krynn-osp16-cmpt-0 | ACTIVE | -          | Running     | ctlplane=10.20.0.126 |
| c89d320f-f5bc-45b9-924c-98e785889ef2 | krynn-osp16-cmpt-1 | ACTIVE | -          | Running     | ctlplane=10.20.0.140 |
| fb041e4b-2279-4a60-95a7-823b52f02284 | krynn-osp16-ctrl-0 | ACTIVE | -          | Running     | ctlplane=10.20.0.150 |
| 533f827a-d3c9-4847-9e37-cade4465e885 | krynn-osp16-ctrl-1 | ACTIVE | -          | Running     | ctlplane=10.20.0.134 |
| de6aab9b-2117-4350-963e-eb180dd71ca1 | krynn-osp16-ctrl-2 | ACTIVE | -          | Running     | ctlplane=10.20.0.128 |
+--------------------------------------+--------------------+--------+------------+-------------+----------------------+
(undercloud) [stack@osp16d ~]$ uname -r
4.18.0-193.65.2.el8_2.x86_64
Comment 24 Vincent S. Cojot 2021-10-18 19:29:52 UTC 
I've looked everywhere and I cannot find a reliable reason why the 16.2 undercloud would hit the 504 more often than the others.

All of my underclouds are currently using these:
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' io='native' discard='ignore' iothread='2' detect_zeroes='off'/>
      <source file='/shared/kvm0/images/instack-16.2-boot.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>


Most deploys work right the first time on all versions but it is only on 16.2 that I am seeing reproductible 504 failures.. the -first- time as deploy is attempted (after restoring the undercloud fresh from its base RHEL snapshot based on RHEL 8.1)

- I'm using the same RHEL 8.1 qcow2 to seed all of my RHEL8-underclouds: 15, 16.0, 16.1 and 16.2
- All of my underclouds are hosted on the same filesystem on an 700k iops NVMe device from HGST.

Most of my deploys on any version work 100% right the first time the pipeline is launched and the hypervisor is clearly 99.9% idle.
It is only on the 16.2 deploys that I get almost 100% failures on the first 'overcloud deploy'. If I let it fail, delete the failed overcloud and re-run the same deploy script from the undercloud, in most cases it works fine... as if it needed to fail one time initially to be successful later.

I no longer have time to research this issue and I hope it gets fixed in subsequent patches (RHEL, RHV or RHOSP, not idea where it is coming from).

I even tried running the undercloud with emulated skylake IBRS instead of the CPU passthrough I'm always using and it made no difference.
Comment 25 Steve Baker 2021-10-26 19:52:15 UTC 
I'm going to kick this back to DFG:DF. It looks like overcloud deploy is being started before all of the undercloud endpoints are active. This could be mitigated by a final undercloud task which confirms endpoint are responding before exiting the undercloud install.
Comment 26 Alex Schultz 2021-10-26 20:12:43 UTC 
I don't think that's possible because you wouldn't be able to setup overcloud image or introspect the hosts if that was true.  We also talk to neutron during post-deployment so something else happened.  Honestly i haven't seen this reproduced in any of our environments afaik. The only thing that might be remotely similar is the upstream https://bugs.launchpad.net/tripleo/+bug/1947403 but that's with introspection.  Given this seemed to be related to the vm host image, i'm going to close this for now unless there's something specific we're doing during installation that's causing the wait.  I think this likely needs to be reported to RHEV