Bug 1909587

Summary: [OCP4] all of the OCP master nodes with soft-anti-affinity run on the same OSP node
Product: OpenShift Container Platform Reporter: Hideshi Fukumoto <hfukumot>
Component: InstallerAssignee: Pierre Prinetti <pprinett>
Installer sub component: OpenShift on OpenStack QA Contact: weiwei jiang <wjiang>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: crawford, m.andre, mbooth, pprinett, wjiang
Version: 4.6.zKeywords: UpcomingSprint
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-24 15:47:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1916297    

Comment 5 weiwei jiang 2021-01-11 09:18:05 UTC
Checked with 4.7.0-0.nightly-2021-01-10-070949

$ ./openshift-install-4.7 version
./openshift-install-4.7 4.7.0-0.nightly-2021-01-10-070949
built from commit 69f0bbc18e8c6b1a6e278c54efa2def9b210033a
release image registry.ci.openshift.org/ocp/release@sha256:62768f2e8bb5850c469c1d15f6cd4337bd0f3f7ecb560b009c61e6d36a134fe8


# openstack server group show wj47ios111a-7xdqh-master
+----------+------------------------------------------------------------------------------------------------------------------+
| Field    | Value                                                                                                            |
+----------+------------------------------------------------------------------------------------------------------------------+
| id       | 35109bc2-8cab-4dc7-8b90-4c2c10dddfcb                                                                             |
| members  | 10fe715e-47d7-4662-aaf0-6aa6f11ceff5, cef21ccc-0c61-43ef-869c-33f38ae8bc63, ed0c8910-abcb-4e42-bf6a-7d6eb659e2e8 |
| name     | wj47ios111a-7xdqh-master                                                                                         |
| policies | soft-anti-affinity                                                                                               |
+----------+------------------------------------------------------------------------------------------------------------------+

# for i in 10fe715e-47d7-4662-aaf0-6aa6f11ceff5 cef21ccc-0c61-43ef-869c-33f38ae8bc63 ed0c8910-abcb-4e42-bf6a-7d6eb659e2e8; do openstack server show $i; done
+-----------------------------+-------------------------------------------------------------------------+
| Field                       | Value                                                                   |
+-----------------------------+-------------------------------------------------------------------------+
| OS-DCF:diskConfig           | MANUAL                                                                  |
| OS-EXT-AZ:availability_zone | nova                                                                    |
| OS-EXT-STS:power_state      | Running                                                                 |
| OS-EXT-STS:task_state       | None                                                                    |
| OS-EXT-STS:vm_state         | active                                                                  |
| OS-SRV-USG:launched_at      | 2021-01-11T09:09:47.000000                                              |
| OS-SRV-USG:terminated_at    | None                                                                    |
| accessIPv4                  |                                                                         |
| accessIPv6                  |                                                                         |
| addresses                   | wj47ios111a-7xdqh-openshift=192.168.3.66                                |
| config_drive                |                                                                         |
| created                     | 2021-01-11T09:08:49Z                                                    |
| flavor                      | m1.xlarge (3f183920-6cba-4bfb-ab3a-599559cf0f97)                        |
| hostId                      | c85e36acb06e96d58a53887aee729f705815e9b3e455b861662f9055                |
| id                          | 10fe715e-47d7-4662-aaf0-6aa6f11ceff5                                    |
| image                       | rhcos-47.83.202012030221-0 (6aaac14e-2758-4537-bd98-fa0e3c027bf2)       |
| key_name                    | None                                                                    |
| name                        | wj47ios111a-7xdqh-master-0                                              |
| progress                    | 0                                                                       |
| project_id                  | 542c6ebd48bf40fa857fc245c7572e30                                        |
| properties                  | Name='wj47ios111a-7xdqh-master', openshiftClusterID='wj47ios111a-7xdqh' |
| security_groups             | name='wj47ios111a-7xdqh-master'                                         |
| status                      | ACTIVE                                                                  |
| updated                     | 2021-01-11T09:09:47Z                                                    |
| user_id                     | d57c957cb3c7010f14a049b72bbfd93d22c1cba255c3c2aa34be30ce5f83da7e        |
| volumes_attached            |                                                                         |
+-----------------------------+-------------------------------------------------------------------------+
+-----------------------------+-------------------------------------------------------------------------+
| Field                       | Value                                                                   |
+-----------------------------+-------------------------------------------------------------------------+
| OS-DCF:diskConfig           | MANUAL                                                                  |
| OS-EXT-AZ:availability_zone | nova                                                                    |
| OS-EXT-STS:power_state      | Running                                                                 |
| OS-EXT-STS:task_state       | None                                                                    |
| OS-EXT-STS:vm_state         | active                                                                  |
| OS-SRV-USG:launched_at      | 2021-01-11T09:11:10.000000                                              |
| OS-SRV-USG:terminated_at    | None                                                                    |
| accessIPv4                  |                                                                         |
| accessIPv6                  |                                                                         |
| addresses                   | wj47ios111a-7xdqh-openshift=192.168.0.21                                |
| config_drive                |                                                                         |
| created                     | 2021-01-11T09:10:56Z                                                    |
| flavor                      | m1.xlarge (3f183920-6cba-4bfb-ab3a-599559cf0f97)                        |
| hostId                      | c87ae7d173f622214a8847c34d7125c70107c767a25656e15ad636b9                |
| id                          | cef21ccc-0c61-43ef-869c-33f38ae8bc63                                    |
| image                       | rhcos-47.83.202012030221-0 (6aaac14e-2758-4537-bd98-fa0e3c027bf2)       |
| key_name                    | None                                                                    |
| name                        | wj47ios111a-7xdqh-master-2                                              |
| progress                    | 0                                                                       |
| project_id                  | 542c6ebd48bf40fa857fc245c7572e30                                        |
| properties                  | Name='wj47ios111a-7xdqh-master', openshiftClusterID='wj47ios111a-7xdqh' |
| security_groups             | name='wj47ios111a-7xdqh-master'                                         |
| status                      | ACTIVE                                                                  |
| updated                     | 2021-01-11T09:11:10Z                                                    |
| user_id                     | d57c957cb3c7010f14a049b72bbfd93d22c1cba255c3c2aa34be30ce5f83da7e        |
| volumes_attached            |                                                                         |
+-----------------------------+-------------------------------------------------------------------------+
+-----------------------------+-------------------------------------------------------------------------+
| Field                       | Value                                                                   |
+-----------------------------+-------------------------------------------------------------------------+
| OS-DCF:diskConfig           | MANUAL                                                                  |
| OS-EXT-AZ:availability_zone | nova                                                                    |
| OS-EXT-STS:power_state      | Running                                                                 |
| OS-EXT-STS:task_state       | None                                                                    |
| OS-EXT-STS:vm_state         | active                                                                  |
| OS-SRV-USG:launched_at      | 2021-01-11T09:10:43.000000                                              |
| OS-SRV-USG:terminated_at    | None                                                                    |
| accessIPv4                  |                                                                         |
| accessIPv6                  |                                                                         |
| addresses                   | wj47ios111a-7xdqh-openshift=192.168.3.217                               |
| config_drive                |                                                                         |
| created                     | 2021-01-11T09:10:00Z                                                    |
| flavor                      | m1.xlarge (3f183920-6cba-4bfb-ab3a-599559cf0f97)                        |
| hostId                      | b2f18f9ba5a0e0abfc899bf65ecf794753c61ffc82c287aedcffce27                |
| id                          | ed0c8910-abcb-4e42-bf6a-7d6eb659e2e8                                    |
| image                       | rhcos-47.83.202012030221-0 (6aaac14e-2758-4537-bd98-fa0e3c027bf2)       |
| key_name                    | None                                                                    |
| name                        | wj47ios111a-7xdqh-master-1                                              |
| progress                    | 0                                                                       |
| project_id                  | 542c6ebd48bf40fa857fc245c7572e30                                        |
| properties                  | Name='wj47ios111a-7xdqh-master', openshiftClusterID='wj47ios111a-7xdqh' |
| security_groups             | name='wj47ios111a-7xdqh-master'                                         |
| status                      | ACTIVE                                                                  |
| updated                     | 2021-01-11T09:10:43Z                                                    |
| user_id                     | d57c957cb3c7010f14a049b72bbfd93d22c1cba255c3c2aa34be30ce5f83da7e        |
| volumes_attached            |                                                                         |
+-----------------------------+-------------------------------------------------------------------------+

We can see masters now on different hosts.

Comment 7 Pierre Prinetti 2021-01-18 07:40:51 UTC
The 4.6.z backport is currenly awaiting approval from the Release manager. No further action is expected on your (or our) side at this time. Thank you!

Bug 1916297

Comment 8 weiwei jiang 2021-01-22 10:49:14 UTC
Checked with more than 3 master again with 4.7.0-0.nightly-2021-01-22-063949, and it works well.

./openshift-install 4.7.0-0.nightly-2021-01-22-063949
built from commit f0f10f6086f2e0446362ed7378b13ba6ed88e7e1
release image registry.ci.openshift.org/ocp/release@sha256:7e62d6eced986e77be37153e8e38069163e0457e29859664ff6d016dbdca0b3b

# openstack server group show wj47ios122aw-tzsnx-master
+----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field    | Value                                                                                                                                                                                        |
+----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id       | 3db57d3f-7517-48f6-b26a-ace8430b94b5                                                                                                                                                         |
| members  | 3ba2c989-76a9-47ff-bda6-080dd8941e61, ba0f3775-f416-48c9-ba5c-f46051742d71, dc267f08-8567-4777-89e4-6f30d79f924f, ebc042a1-0228-4f2d-bc79-36a1450edab1, f4650415-ff24-497d-b682-7961fbf27358 |
| name     | wj47ios122aw-tzsnx-master                                                                                                                                                                    |
| policies | soft-anti-affinity                                                                                                                                                                           |
+----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


# for i  in `openstack server list --name wj4 -f value -c Name |grep -i master`; do openstack server show $i ; done |grep -i hostid
| hostId                      | 6a2ee645e9f14b9f0e702891f333307b2f5a18562fc3ea7e7f52607e                  |
| hostId                      | 2bdc41f55878483467141fba5dec5dd18a5b6e5935c29d536cce6296                  |
| hostId                      | b2819cfacdc3cc8ea3735a801f5c44ae30b20eae3009e4fd8e2e9917                  |
| hostId                      | 90d74f34572f5cf1523a3b6490e03fe63e05e03b44cdfe2b7d00a964                  |
| hostId                      | b71246d7dab294fde8273ee4322f60abf5c7f8978726e2892d008a97                  |

Comment 9 Alex Crawford 2021-01-22 23:25:04 UTC
How about from OpenShift's perspective, do you see five control plane nodes in your cluster?

Comment 10 Alex Crawford 2021-01-22 23:41:43 UTC
Scratch that. I misread the previous command you were running and now I'm very confused. The fix for this issue hard codes the number of control nodes at three. How is it that you ended up with five nodes? What created the other two?

Comment 11 Martin André 2021-01-23 11:16:27 UTC
(In reply to Alex Crawford from comment #10)
> Scratch that. I misread the previous command you were running and now I'm
> very confused. The fix for this issue hard codes the number of control nodes
> at three. How is it that you ended up with five nodes? What created the
> other two?

The patch indeed changed the terraform code to always provision 3 control plane instances. However, we're still creating 'n' machine objects, where 'n' is the number of replicas specified in the control plane machine pool. Wouldn't then MAO notice these objects once the control plane is up and provisions the missing instances?

Comment 12 Alex Crawford 2021-01-25 16:56:26 UTC
The control plane isn't represented by MachineSets and Machines though. Only the workers make use of that right now. It's been on our backlog for a long time, but we haven't gotten around to managing the control nodes the same way that we manage the compute nodes. Maybe my understanding is dated.

Comment 13 Alex Crawford 2021-01-26 00:44:15 UTC
I had some time to dig into this further and was able to reproduce the behavior described, but on AWS. My understanding was dated. Sorry for the noise.

Comment 16 errata-xmlrpc 2021-02-24 15:47:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633