Bug 1876153 - Using split-stack leads to large number of duplicates in /etc/hosts for undercloud.ctlplane over time
Summary: Using split-stack leads to large number of duplicates in /etc/hosts for under...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-tripleoclient
Version: 16.1 (Train)
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: ---
: ---
Assignee: Emilien Macchi
QA Contact: David Rosenfeld
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-09-05 21:00 UTC by Kellen Gattis
Modified: 2021-02-01 14:43 UTC (History)
10 users (show)

Fixed In Version: python-tripleoclient-12.3.2-1.20200815213404.29686a6.el8ost
Doc Type: If docs needed, set a value
Doc Text:
The entry in `/etc/hosts` for the undercloud duplicates anytime the Compute stack is updated on the undercloud and overcloud nodes. This occurs for split-stack deployments where the Controllers and Compute nodes are divided into multiple stacks. + Other indications of this problem are the following: + ** mysql reporting errors about packets exceeding their maximum size. ** The Orchestration service (heat) warning that templates are exceeding their maximum size. ** The Workflow service (mistral) warning that fields are exceeding their maximum size. As a workaround, in the file generated by running the `openstack overcloud export` command that is included in the Compute stack, under `ExtraHostFileEntries`, remove the erroneous entry for the undercloud.
Clone Of:
Environment:
Last Closed: 2020-10-28 15:39:36 UTC
Target Upstream Version:
Embargoed:
kgattis: needinfo-


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 740543 0 None MERGED Cleanup UndercloudHostsEntries 2021-02-01 14:32:39 UTC
Red Hat Product Errata RHEA-2020:4284 0 None None None 2020-10-28 15:39:57 UTC

Description Kellen Gattis 2020-09-05 21:00:05 UTC
Description of problem:
When using a split-stack deployment where the Controllers and Computes are split into multiple stacks, the entry in /etc/hosts for the undercloud starts duplicating with each subsequent compute stack update on the undercloud and overcloud nodes.

Some symptoms of this include an entry in /etc/hosts that could look like this:
192.168.10.11 undercloud.ctlplane.example.org undercloud.ctlplane undercloud.ctlplane undercloud.ctlplane undercloud.ctlplane undercloud.ctlplane undercloud.ctlplane undercloud.ctlplane undercloud.ctlplane undercloud.ctlplane undercloud.ctlplane undercloud.ctlplane undercloud.ctlplane undercloud.ctlplane undercloud.ctlplane undercloud.ctlplane undercloud.ctlplane undercloud.ctlplane undercloud.ctlplane 

Other signs that this is happening include mysql complaining about packets exceeding max size, heat complaining about templates exceeding max size, and mistral complaining about fields exceeding max size.

When looking at the file generated by running "openstack overcloud export" that is included in the compute stack, I noticed that there was an entry in "ExtraHostFileEntries" for the undercloud.  Removing this entry from the file stopped the /etc/hosts entry storm.

As a workaround, one can simply delete the undercloud entry in ExtraHostFileEntries whenever the central site export file is generated and everything works as expected.


Version-Release number of selected component (if applicable):
python3-tripleoclient-12.3.2-0.20200615103427.6f877f6.el8ost.noarch

How reproducible:
Very

Steps to Reproduce:
1. Deploy a controller stack
2. Export controller (central) stack to template file
Docs: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/distributed_compute_node_and_storage_deployment/configuring_and_installing_the_hub_undercloud#deploying-edge-nodes-without-storage
3. Deploy a compute stack using exported file from step #2

Actual results:
/etc/hosts grows with each compute stack update

Expected results:
/etc/hosts has one entry for the undercloud that is stable

Additional info:
https://bugzilla.redhat.com/show_bug.cgi?id=1842919 seems like a similar user experience

Comment 1 Michele Baldessari 2020-09-07 06:46:08 UTC
Does applying https://review.opendev.org/#/c/740543/ to your undercloud fix it in your case?

Comment 3 Kellen Gattis 2020-09-10 01:32:55 UTC
(In reply to Michele Baldessari from comment #1)
> Does applying https://review.opendev.org/#/c/740543/ to your undercloud fix
> it in your case?

I applied the patch and it definitely helped to stop undercloud.ctlplane from multiplying in /etc/hosts - thanks for the recommendation.

The only oddity after applying the patch was that I ended up with two identical lines in /etc/hosts for the undercloud entry on both the undercloud and the overcloud compute stack nodes after I ran the compute stack creation/update.  I never saw more than two undercloud lines appear regardless of how many stack updates I ran and I didn't notice any misbehaving operation as a result, so pretty harmless overall.  I removed the undercloud entry from the central export file's ExtraHostFileEntries (same procedure as the workaround mentioned in the bugzilla's description of the problem) and the double entry went away when I performed a subsequent stack update.

Comment 11 errata-xmlrpc 2020-10-28 15:39:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:4284


Note You need to log in before you can comment on or make changes to this bug.