Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1718696

Summary: Check IPs specified in undercloud.conf aren't going to cause issues or conflict
Product: Red Hat OpenStack Reporter: Brendan Shephard <bshephar>
Component: python-tripleoclientAssignee: RHOS Maint <rhos-maint>
Status: CLOSED ERRATA QA Contact: Sasha Smolyak <ssmolyak>
Severity: medium Docs Contact:
Priority: medium    
Version: 15.0 (Stein)CC: apetrich, beth.white, cjeanner, hbrock, jbuchta, jslagle, mburns, ramishra
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-tripleoclient-11.5.2-0.20191005040431.1f577e4.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-03-05 11:59:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Brendan Shephard 2019-06-09 23:58:20 UTC
Description of problem:
If the same IP is used for any of these options in undercloud.conf:
local_ip undercloud_admin_host undercloud_public_host

It will cause issues with the endpoints and haproxy for services. In this specific case, the zaqar_ws haproxy service was using the same bind and server IP address.

Version-Release number of selected component (if applicable):
Stein

How reproducible:
Very

Steps to Reproduce:
1. Define the same IP address for local_ip and undercloud_public_host as the same address:
local_ip = 192.168.24.2/24
undercloud_public_host = 192.168.24.2
undercloud_admin_host = 192.168.24.3

2. openstack undercloud install
3. Try to deploy the overcloud.

Actual results:
Traceback:
clean_up DeployOvercloud: Handshake status 502 Bad Gateway
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 136, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 281, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 176, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 401, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 32, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 925, in take_action
    self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 374, in _deploy_tripleo_heat_templates_tmpdir
    new_tht_root, tht_root)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 386, in _deploy_tripleo_heat_templates
    plans = plan_management.list_deployment_plans(self.clients)
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/plan_management.py", line 129, in list_deployment_plans
    with tripleoclients.messaging_websocket() as ws:
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 211, in messaging_websocket
    cacert=self._instance.cacert)
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 89, in __init__
    sslopt=OS_CACERT)
  File "/usr/lib/python2.7/site-packages/websocket/_core.py", line 511, in create_connection
    websock.connect(url, **options)
  File "/usr/lib/python2.7/site-packages/websocket/_core.py", line 223, in connect
    self.handshake_response = handshake(self.sock, *addrs, **options)
  File "/usr/lib/python2.7/site-packages/websocket/_handshake.py", line 79, in handshake
    status, resp = _get_resp_headers(sock)
  File "/usr/lib/python2.7/site-packages/websocket/_handshake.py", line 152, in _get_resp_headers
    raise WebSocketBadStatusException("Handshake status %d %s", status, status_message)
WebSocketBadStatusException: Handshake status 502 Bad Gateway

Check haproxy.cfg
listen zaqar_ws
  bind 192.168.24.2:9000 ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 192.168.24.3:9000 
  mode http
  option forwardfor
  redirect scheme https code 301 if { hdr(host) -i 192.168.24.2 } !{ ssl_fc }
  rsprep ^Location:\ http://(.*) Location:\ https://\1
  timeout connect 5s
  timeout client 25s
  timeout server 25s
  timeout tunnel 14400s
  server undercloud 192.168.24.2:9000 check fall 5 inter 2000 rise 2

We have configured the same bind IP and server IP here. Breaking Zaqar

Expected results:
Undercloud install should throw an error advising that this can't be done.


Additional info:
We have the following to validate the IP is real: https://github.com/openstack/python-tripleoclient/blob/stable/stein/tripleoclient/v1/undercloud_preflight.py#L159-L169

We could add a check here. But it appears we skip these validations? 
https://github.com/openstack/python-tripleoclient/blob/stable/stein/tripleoclient/v1/tripleo_deploy.py#L123-L125

Comment 1 Brendan Shephard 2019-06-10 00:09:37 UTC
Should we do something like this and call the _validate_ips() function:


def _validate_ips():
    def is_ip(value, param_name):
        try:
            netaddr.IPAddress(value)
        except netaddr.core.AddrFormatError:
            msg = (_('{0} "{1}" must be a valid IP address')
                   .format(param_name, value))
            LOG.error(msg)
            raise FailedValidation(msg)

    def check_ip_clash():
       local_ip = CONF.local_ip
       admin_ip = CONF.undercloud_admin_host
       public_ip = CONF.undercloud_public_host
       msg = ("The same IP cannot be used for multiple endpoints.")
       if local_ip == admin_ip or local_ip == public_ip or public_ip == admin_ip:
          LOG.error("The same IP was used for multiple endpoints. Defined IPs are:"
                    "local_ip: %{local}, undercloud_admin_host: %{admin}, undercloud_public_host: ${public}",
                    {"local": local_ip, "admin": admin_ip, "public": public_ip} 
          raise FailedValidation(msg)
       else:
          pass

    for ip in CONF.undercloud_nameservers:
        is_ip(ip, 'undercloud_nameservers')
    check_ip_clash


This is untested; just throwing idea's out there.

Comment 2 Brendan Shephard 2019-06-10 05:42:59 UTC
This works, 


    def _run_preflight_checks(self, parsed_args):
        """Run preflight deployment checks

        Perform any pre-deployment checks that we want to run when deploying
        standalone deployments. This is skipped when in output only mode or
        when used with an undercloud. The undercloud has it's own set of
        deployment preflight requirements.

        :param parsed_args: parsed arguments from the cli
        """

        # We need to make sure unique IP's have been used in undercloud.conf
        ip_list = [
                parsed_args.local_ip.split('/')[0],
                parsed_args.control_virtual_ip,
                parsed_args.public_virtual_ip
        ]
        msg = ("The same IP cannot be used for multiple endpoints.")
        try:
            if len(ip_list) != len(set(ip_list)):
               LOG.error(
                        "The same IP was used for multiple endpoints. Defined IPs are:"
                        "local_ip: %{local}, undercloud_admin_host: %{admin}",
                        "undercloud_public_host: %{public}",
                        {"local": ip_list[0], "admin": ip_list[1], "public": ip_list[2]}
                        )
        except Exception as e:
                raise exceptions.InvalidConfiguration(msg)

        # we skip preflight checks for output only and undercloud
        if parsed_args.output_only or self._is_undercloud_deploy(parsed_args):
            return

        # in standalone we don't want to fixup the /etc/hosts as we'll be
        # managing that elsewhere during the deployment
        utils.check_hostname(fix_etc_hosts=False, logger=self.log)

        # Users can use http_proxy and https_proxy as part of the deployment,
        # however we need localhost to not be proxied because we use it to talk
        # to our heat api.
        utils.check_env_for_proxy(no_proxy_hosts=['127.0.0.1'])

    # NOTE(cjeanner) Quick'n'dirty way before we have proper
    # escalation support through oslo.privsep



Output:

The same IP cannot be used for multiple endpoints.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

An error has occured while deploying the Undercloud.

See the previous output for details about what went wrong.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!


Thoughts?

Comment 8 errata-xmlrpc 2020-03-05 11:59:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0643