Bug 1909587 - [OCP4] all of the OCP master nodes with soft-anti-affinity run on the same OSP node
Summary: [OCP4] all of the OCP master nodes with soft-anti-affinity run on the same OS...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.6.z
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.7.0
Assignee: Pierre Prinetti
QA Contact: weiwei jiang
URL:
Whiteboard:
Depends On:
Blocks: 1916297
TreeView+ depends on / blocked
 
Reported: 2020-12-21 04:52 UTC by Hideshi Fukumoto
Modified: 2021-02-24 15:47 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-02-24 15:47:16 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 4522 0 None closed Bug 1909587: openstack: Create Control plane nodes sequentially 2021-02-17 11:00:35 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:47:59 UTC

Comment 5 weiwei jiang 2021-01-11 09:18:05 UTC
Checked with 4.7.0-0.nightly-2021-01-10-070949

$ ./openshift-install-4.7 version
./openshift-install-4.7 4.7.0-0.nightly-2021-01-10-070949
built from commit 69f0bbc18e8c6b1a6e278c54efa2def9b210033a
release image registry.ci.openshift.org/ocp/release@sha256:62768f2e8bb5850c469c1d15f6cd4337bd0f3f7ecb560b009c61e6d36a134fe8


# openstack server group show wj47ios111a-7xdqh-master
+----------+------------------------------------------------------------------------------------------------------------------+
| Field    | Value                                                                                                            |
+----------+------------------------------------------------------------------------------------------------------------------+
| id       | 35109bc2-8cab-4dc7-8b90-4c2c10dddfcb                                                                             |
| members  | 10fe715e-47d7-4662-aaf0-6aa6f11ceff5, cef21ccc-0c61-43ef-869c-33f38ae8bc63, ed0c8910-abcb-4e42-bf6a-7d6eb659e2e8 |
| name     | wj47ios111a-7xdqh-master                                                                                         |
| policies | soft-anti-affinity                                                                                               |
+----------+------------------------------------------------------------------------------------------------------------------+

# for i in 10fe715e-47d7-4662-aaf0-6aa6f11ceff5 cef21ccc-0c61-43ef-869c-33f38ae8bc63 ed0c8910-abcb-4e42-bf6a-7d6eb659e2e8; do openstack server show $i; done
+-----------------------------+-------------------------------------------------------------------------+
| Field                       | Value                                                                   |
+-----------------------------+-------------------------------------------------------------------------+
| OS-DCF:diskConfig           | MANUAL                                                                  |
| OS-EXT-AZ:availability_zone | nova                                                                    |
| OS-EXT-STS:power_state      | Running                                                                 |
| OS-EXT-STS:task_state       | None                                                                    |
| OS-EXT-STS:vm_state         | active                                                                  |
| OS-SRV-USG:launched_at      | 2021-01-11T09:09:47.000000                                              |
| OS-SRV-USG:terminated_at    | None                                                                    |
| accessIPv4                  |                                                                         |
| accessIPv6                  |                                                                         |
| addresses                   | wj47ios111a-7xdqh-openshift=192.168.3.66                                |
| config_drive                |                                                                         |
| created                     | 2021-01-11T09:08:49Z                                                    |
| flavor                      | m1.xlarge (3f183920-6cba-4bfb-ab3a-599559cf0f97)                        |
| hostId                      | c85e36acb06e96d58a53887aee729f705815e9b3e455b861662f9055                |
| id                          | 10fe715e-47d7-4662-aaf0-6aa6f11ceff5                                    |
| image                       | rhcos-47.83.202012030221-0 (6aaac14e-2758-4537-bd98-fa0e3c027bf2)       |
| key_name                    | None                                                                    |
| name                        | wj47ios111a-7xdqh-master-0                                              |
| progress                    | 0                                                                       |
| project_id                  | 542c6ebd48bf40fa857fc245c7572e30                                        |
| properties                  | Name='wj47ios111a-7xdqh-master', openshiftClusterID='wj47ios111a-7xdqh' |
| security_groups             | name='wj47ios111a-7xdqh-master'                                         |
| status                      | ACTIVE                                                                  |
| updated                     | 2021-01-11T09:09:47Z                                                    |
| user_id                     | d57c957cb3c7010f14a049b72bbfd93d22c1cba255c3c2aa34be30ce5f83da7e        |
| volumes_attached            |                                                                         |
+-----------------------------+-------------------------------------------------------------------------+
+-----------------------------+-------------------------------------------------------------------------+
| Field                       | Value                                                                   |
+-----------------------------+-------------------------------------------------------------------------+
| OS-DCF:diskConfig           | MANUAL                                                                  |
| OS-EXT-AZ:availability_zone | nova                                                                    |
| OS-EXT-STS:power_state      | Running                                                                 |
| OS-EXT-STS:task_state       | None                                                                    |
| OS-EXT-STS:vm_state         | active                                                                  |
| OS-SRV-USG:launched_at      | 2021-01-11T09:11:10.000000                                              |
| OS-SRV-USG:terminated_at    | None                                                                    |
| accessIPv4                  |                                                                         |
| accessIPv6                  |                                                                         |
| addresses                   | wj47ios111a-7xdqh-openshift=192.168.0.21                                |
| config_drive                |                                                                         |
| created                     | 2021-01-11T09:10:56Z                                                    |
| flavor                      | m1.xlarge (3f183920-6cba-4bfb-ab3a-599559cf0f97)                        |
| hostId                      | c87ae7d173f622214a8847c34d7125c70107c767a25656e15ad636b9                |
| id                          | cef21ccc-0c61-43ef-869c-33f38ae8bc63                                    |
| image                       | rhcos-47.83.202012030221-0 (6aaac14e-2758-4537-bd98-fa0e3c027bf2)       |
| key_name                    | None                                                                    |
| name                        | wj47ios111a-7xdqh-master-2                                              |
| progress                    | 0                                                                       |
| project_id                  | 542c6ebd48bf40fa857fc245c7572e30                                        |
| properties                  | Name='wj47ios111a-7xdqh-master', openshiftClusterID='wj47ios111a-7xdqh' |
| security_groups             | name='wj47ios111a-7xdqh-master'                                         |
| status                      | ACTIVE                                                                  |
| updated                     | 2021-01-11T09:11:10Z                                                    |
| user_id                     | d57c957cb3c7010f14a049b72bbfd93d22c1cba255c3c2aa34be30ce5f83da7e        |
| volumes_attached            |                                                                         |
+-----------------------------+-------------------------------------------------------------------------+
+-----------------------------+-------------------------------------------------------------------------+
| Field                       | Value                                                                   |
+-----------------------------+-------------------------------------------------------------------------+
| OS-DCF:diskConfig           | MANUAL                                                                  |
| OS-EXT-AZ:availability_zone | nova                                                                    |
| OS-EXT-STS:power_state      | Running                                                                 |
| OS-EXT-STS:task_state       | None                                                                    |
| OS-EXT-STS:vm_state         | active                                                                  |
| OS-SRV-USG:launched_at      | 2021-01-11T09:10:43.000000                                              |
| OS-SRV-USG:terminated_at    | None                                                                    |
| accessIPv4                  |                                                                         |
| accessIPv6                  |                                                                         |
| addresses                   | wj47ios111a-7xdqh-openshift=192.168.3.217                               |
| config_drive                |                                                                         |
| created                     | 2021-01-11T09:10:00Z                                                    |
| flavor                      | m1.xlarge (3f183920-6cba-4bfb-ab3a-599559cf0f97)                        |
| hostId                      | b2f18f9ba5a0e0abfc899bf65ecf794753c61ffc82c287aedcffce27                |
| id                          | ed0c8910-abcb-4e42-bf6a-7d6eb659e2e8                                    |
| image                       | rhcos-47.83.202012030221-0 (6aaac14e-2758-4537-bd98-fa0e3c027bf2)       |
| key_name                    | None                                                                    |
| name                        | wj47ios111a-7xdqh-master-1                                              |
| progress                    | 0                                                                       |
| project_id                  | 542c6ebd48bf40fa857fc245c7572e30                                        |
| properties                  | Name='wj47ios111a-7xdqh-master', openshiftClusterID='wj47ios111a-7xdqh' |
| security_groups             | name='wj47ios111a-7xdqh-master'                                         |
| status                      | ACTIVE                                                                  |
| updated                     | 2021-01-11T09:10:43Z                                                    |
| user_id                     | d57c957cb3c7010f14a049b72bbfd93d22c1cba255c3c2aa34be30ce5f83da7e        |
| volumes_attached            |                                                                         |
+-----------------------------+-------------------------------------------------------------------------+

We can see masters now on different hosts.

Comment 7 Pierre Prinetti 2021-01-18 07:40:51 UTC
The 4.6.z backport is currenly awaiting approval from the Release manager. No further action is expected on your (or our) side at this time. Thank you!

Bug 1916297

Comment 8 weiwei jiang 2021-01-22 10:49:14 UTC
Checked with more than 3 master again with 4.7.0-0.nightly-2021-01-22-063949, and it works well.

./openshift-install 4.7.0-0.nightly-2021-01-22-063949
built from commit f0f10f6086f2e0446362ed7378b13ba6ed88e7e1
release image registry.ci.openshift.org/ocp/release@sha256:7e62d6eced986e77be37153e8e38069163e0457e29859664ff6d016dbdca0b3b

# openstack server group show wj47ios122aw-tzsnx-master
+----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field    | Value                                                                                                                                                                                        |
+----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id       | 3db57d3f-7517-48f6-b26a-ace8430b94b5                                                                                                                                                         |
| members  | 3ba2c989-76a9-47ff-bda6-080dd8941e61, ba0f3775-f416-48c9-ba5c-f46051742d71, dc267f08-8567-4777-89e4-6f30d79f924f, ebc042a1-0228-4f2d-bc79-36a1450edab1, f4650415-ff24-497d-b682-7961fbf27358 |
| name     | wj47ios122aw-tzsnx-master                                                                                                                                                                    |
| policies | soft-anti-affinity                                                                                                                                                                           |
+----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


# for i  in `openstack server list --name wj4 -f value -c Name |grep -i master`; do openstack server show $i ; done |grep -i hostid
| hostId                      | 6a2ee645e9f14b9f0e702891f333307b2f5a18562fc3ea7e7f52607e                  |
| hostId                      | 2bdc41f55878483467141fba5dec5dd18a5b6e5935c29d536cce6296                  |
| hostId                      | b2819cfacdc3cc8ea3735a801f5c44ae30b20eae3009e4fd8e2e9917                  |
| hostId                      | 90d74f34572f5cf1523a3b6490e03fe63e05e03b44cdfe2b7d00a964                  |
| hostId                      | b71246d7dab294fde8273ee4322f60abf5c7f8978726e2892d008a97                  |

Comment 9 Alex Crawford 2021-01-22 23:25:04 UTC
How about from OpenShift's perspective, do you see five control plane nodes in your cluster?

Comment 10 Alex Crawford 2021-01-22 23:41:43 UTC
Scratch that. I misread the previous command you were running and now I'm very confused. The fix for this issue hard codes the number of control nodes at three. How is it that you ended up with five nodes? What created the other two?

Comment 11 Martin André 2021-01-23 11:16:27 UTC
(In reply to Alex Crawford from comment #10)
> Scratch that. I misread the previous command you were running and now I'm
> very confused. The fix for this issue hard codes the number of control nodes
> at three. How is it that you ended up with five nodes? What created the
> other two?

The patch indeed changed the terraform code to always provision 3 control plane instances. However, we're still creating 'n' machine objects, where 'n' is the number of replicas specified in the control plane machine pool. Wouldn't then MAO notice these objects once the control plane is up and provisions the missing instances?

Comment 12 Alex Crawford 2021-01-25 16:56:26 UTC
The control plane isn't represented by MachineSets and Machines though. Only the workers make use of that right now. It's been on our backlog for a long time, but we haven't gotten around to managing the control nodes the same way that we manage the compute nodes. Maybe my understanding is dated.

Comment 13 Alex Crawford 2021-01-26 00:44:15 UTC
I had some time to dig into this further and was able to reproduce the behavior described, but on AWS. My understanding was dated. Sorry for the noise.

Comment 16 errata-xmlrpc 2021-02-24 15:47:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.