Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1827545

Summary:

[OCP-4.5] Can not add rhel-7 workers for IPI-RHOS deployed clusters

Product:

OpenShift Container Platform

Reporter:

Lukas Bednar <lbednar>

Component:

Installer

Assignee:

Adolfo Duarte <adduarte>

Installer sub component:

OpenShift on OpenStack

QA Contact:

David Sanz <dsanzmor>

Status:

CLOSED DUPLICATE

Docs Contact:

Severity:

medium

Priority:

medium

CC:

jialiu, lbednar, pprinett, yunjiang

Version:

4.5

Keywords:

UpcomingSprint

Target Milestone:

---

Target Release:

4.6.0

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2020-06-18 14:37:24 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
openshift-ansible.log	none
openshift-ansible-vvv.log	none
openshift-ansible-vvv.log	none

Description Lukas Bednar 2020-04-24 07:16:12 UTC

Created attachment 1681374 [details]
openshift-ansible.log

Description of problem:

When running openshift-ansible to add rhel7 workers to my existing IPI-RHOS deployed cluster, tt seems like worker can not reach machine config at https://api.lbednar-ocp45.cnv-qe.rhcloud.com:22623/config/worker .

fatal: [lbednar-ocp45-242kc-rhel-worker-0]: FAILED! => {"attempts": 60, "changed": false, "content": "", "elapsed": 30, "msg": "Status code was -1 and not [200]: Request failed: <urlopen error timed out>", "redirected": false, "status": -1, "url": "https://api.lbednar-ocp45.cnv-qe.rhcloud.com:22623/config/worker"}
 

My rhel-worker machine is attached to same network as cluster is, and also associated worker security group. 

Version-Release number of the following components:
rpm -q openshift-ansible
openshift-ansible-4.5.0-202004180718.git.1.db9b4b0.el7.noarch
rpm -q ansible
ansible-2.9.5-1.el7ae.noarch
ansible --version
ansible 2.9.5
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /bin/ansible
  python version = 2.7.5 (default, Jun 11 2019, 14:33:56) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]


How reproducible: 100

Steps to Reproduce:
1. Deploy OCP-4.5 IPI cluster on RHOS
2. Deploy RHEL7 worker( https://access.redhat.com/documentation/en-us/openshift_container_platform/4.3/html/machine_management/adding-rhel-compute)
3. 

Actual results: 
fatal: [lbednar-ocp45-242kc-rhel-worker-0]: FAILED! => {"attempts": 60, "changed": false, "content": "", "elapsed": 30, "msg": "Status code was -1 and not [200]: Request failed: <urlopen error timed out>", "redirected": false, "status": -1, "url": "https://api.lbednar-ocp45.cnv-qe.rhcloud.com:22623/config/worker"}

Expected results: RHEL7 worker join cluster

Additional info:
see openshift-ansible.log attachement

Comment 1 Russell Teague 2020-04-24 12:15:25 UTC

From the Description I see it was trying to connect to "https://api.lbednar-ocp45.cnv-qe.rhcloud.com:22623/config/worker".

It should be using api-int, which was a change merged in February.

From the attached log I see "https://api-int.lbednar-ocp45.cnv-qe.rhcloud.com:22623/config/worker"

Please attach Ansible logs with '-vvv' output.

Comment 2 Lukas Bednar 2020-04-27 09:09:31 UTC

Created attachment 1682061 [details]
openshift-ansible-vvv.log

Comment 3 Lukas Bednar 2020-04-29 15:21:35 UTC

Log attached

Comment 4 Russell Teague 2020-04-29 15:28:33 UTC

Attached log does not look complete.

Comment 5 Lukas Bednar 2020-04-29 16:43:10 UTC

I am sorry, but this is all what I have got from openshift-ansible execution.

Comment 6 Lukas Bednar 2020-05-04 12:16:20 UTC

Created attachment 1684792 [details]
openshift-ansible-vvv.log

I reuploaded logs

Comment 7 Russell Teague 2020-05-14 20:17:51 UTC

I am unable to determine any further issues from the log.  The failing task is simply trying to connect to the given endpoint (https://api-int.lbednar-ocp45.cnv-qe.rhcloud.com:22623/config/worker).  You could test connectivity by using curl from the RHEL host to confirm network connectivity.  If you are not able to curl that endpoint, there is a network issue outside the realm of openshift-ansible.

From the RHEL host:
# curl --insecure https://api-int.lbednar-ocp45.cnv-qe.rhcloud.com:22623/config/worker

Comment 8 Lukas Bednar 2020-05-15 08:19:38 UTC

That is right, it can not connect to given endpoint.
I didn't find any doc how am I supposed to create rhel VM in RHOS IPI deployment.
openshift-installer creates everything, including network, security groups, network trunks, ports and more.

Here it how I create VM:

```
- name: Create rhel-7 worker instance
  os_server:
    state: present
    name: "lbednar-ocp45-5x5p5-rhel-worker-0"
    image: "rhel-7"
    key_name: "cnv-qe-jenkins-key"
    network: "lbednar-ocp45-5x5p5-openshift"
    flavor: "ci.nested.virt.w1.xlarge"
    security_groups:
      - "lbednar-ocp45-5x5p5-worker"
    wait: yes
    timeout: 300
```

I believe that I am missing something here.

Comment 11 Adolfo Duarte 2020-05-22 22:54:18 UTC

@Lbednar. Did youmodify the security groups in any way before installation? I am thinking this is similar if not the same problem seen here: https://bugzilla.redhat.com/show_bug.cgi?id=1804083

The problem in that defect is caused because although the rhel vm being added as a worker is in the same location as the other worker vms, the ip address of the api, https://api-int.lbednar-ocp45.cnv-qe.rhcloud.com, is being resolved to the FIP ip address of the load balancer. This is the ip address which is used to access it from outside of the openstack cloud. so the new rehl vm being added thinks the ip address is in the outside world.
It sends the ip packet to that address. In doing so the source ip address of the traffic from rhel vm to cluster api is changed. Either due to openstack SNAT process or a FIP mapping. When it hits the security groups it does not match.

By default when a cluster is deployed the security groups match on both, source and destination ip address. I am thinking that in this case, the source address is not matching. That is what I am trying to described in the previous paragraph.

One possible solution is to assign a FIP to the rhel vm and add that FIP to the cluster security groups.
Another solution, is to hardcode the correct (internal ip address) of the cluster api in the new rhel vm. For example on the /etc/hosts file.

but before attempting any of this, you can check what is going on. If you have access to the rhel vm before adding it, try doing a dns resolve for https://api-int.lbednar-ocp45.cnv-qe.rhcloud.com.

Then do the the curl --insecure https://<ip-address-you-found-before>:22623

before you can proceed, with adding the new rhel vm, that command should NOT get a timeout. May be a 404, or access deny. Anything that shows the ip:port is accessible from the new rhel vm.

If you know the internal (inside openstack cloud) ip address of api-int.lbednar-ocp45.cnv-qe.rhcloud.com you can try adding that to the new rhel vm dnsresolution process. like I said above, perhaps through /etc/hosts, or pointing it to a dns which resolves the correct ip address (like the internal cluster dns).
That would allow you to not to have to edit the security groups. But if you have access to the security groups and can edit them, then adding a fip mapping to the new rhel vm and using that to allow access to the api from the new vm, would be the fastest way. After installation you may reset the security groups. (as long as the vm resolvs to the correct ip address, internal ip address of the cluster api).

Comment 12 Lukas Bednar 2020-05-25 13:30:59 UTC

Hi,

> @Lbednar. Did you modify the security groups in any way before installation?

No I don't touch existing security groups, I just assign existing worker sec-group to my RHEL worker (see https://bugzilla.redhat.com/show_bug.cgi?id=1827545#c8 how I create it).

> I am thinking this is similar if not the same problem seen here: https://bugzilla.redhat.com/show_bug.cgi?id=1804083

That bug is about UPI, and as far as I remember we were able to add RHEL worker to UPI deployment.

> but before attempting any of this, you can check what is going on. If you have access to the rhel vm before adding it, try doing a dns resolve for  https://api-int.lbednar-ocp45.cnv-qe.rhcloud.com. 

api-int.lbednar-ocp45.cnv-qe.rhcloud.com doesn't resolve to IP.
I have only two resolvable records there:
 api.lbednar-ocp45.cnv-qe.rhcloud.com = 10.0.96.30
 *.apps.lbednar-ocp45.cnv-qe.rhcloud.com = 10.0.97.134

> curl --insecure https://10.0.96.30:22623

curl: (7) Failed to connect to 10.0.96.30 port 22623: Connection refused

I know internal IP of api.lbednar-ocp45.cnv-qe.rhcloud.com endpoint, it is master-1 and internal IP is 192.168.0.13 .

[core@lbednar-ocp45-f265s-master-1 ~]$ curl --insecure https://192.168.0.13:22623
curl: (7) Failed to connect to 192.168.0.13 port 22623: Connection refused

I don't have a problem adding record to /etc/hosts file.
I am just confused now, why am I missing this api-ini end point ??

Comment 18 Adolfo Duarte 2020-06-04 05:11:45 UTC

@lbednar
the error logs you attached are pretty dificult to read because they got their scape charaters replaced. but at the end you can see this: 

"url\": \"https://api-int.lbednar-ocp45.cnv-qe.rhcloud.com:22623/config/worker\", \u001b[0m", "\u001b[0;31m            \"url_password\": null, \u001b[0m", "\u001b[0;31m            \"url_username\": null, \u001b[0m", "\u001b[0;31m            \"use_proxy\": true, \u001b[0m", "\u001b[0;31m            \"validate_certs\": false\u001b[0m", "\u001b[0;31m        }\u001b[0m", "\u001b[0;31m    }, \u001b[0m", "\u001b[0;31m    \"msg\": \"Status code was -1 and not [200]: Request failed: <urlopen error [Errno -2] Name or service not known>\", \u001b[0m", "\u001b[0;31m    \"redirected\": false, \u001b[0m", "\u001b[0;31m    \"status\": -1, \u001b[0m", 

Which as @rteague pointed out it means that the host can't reach https://api-int.lbednar-ocp45.cnv-qe.rhcloud.com:22623

looks like it failed with: 

 Request failed: <urlopen error [Errno -2] Name or service not known>\", \u001b[0m", "\u001b[0;31m    \"redirected\": false, \u001b[0m", "\u001b[0;31m    \"status\": -1, \u001b[0m", 

but that does not give much info really. 

I see that you add the dns entry for api in your playbook here: 
https://gitlab.cee.redhat.com/contra/cnv-qe-automation/merge_requests/96/diffs#9ede9188d12404de95d49e1b70bedb0a80650aae_13_12

do you add one for api-int anywhere. 

like @rteague said, this command should work from the rhel worker being added 
curl --insecure https://api-int.lbednar-ocp45.cnv-qe.rhcloud.com:22623/config/worker

what happens when you execute it?

Comment 20 Lukas Bednar 2020-06-05 07:42:23 UTC

> I see that you add the dns entry for api in your playbook here: 
> https://gitlab.cee.redhat.com/contra/cnv-qe-automation/merge_requests/96/diffs#9ede9188d12404de95d49e1b70bedb0a80650aae_13_12
> 
> do you add one for api-int anywhere. 
> 
> like @rteague said, this command should work from the rhel worker being added 
> curl --insecure https://api-int.lbednar-ocp45.cnv-qe.rhcloud.com:22623/config/worker
> 
> what happens when you execute it?

We never added dns record for api-int explicitly, and it is working for UPI deployment (at least in OCP-4.4).

Anyway I will try to add `api-int` record to dns, pointing to same IP as `api` record. I will let you know ...

Comment 21 Lukas Bednar 2020-06-08 10:19:13 UTC

Just FYI: I am trying to deploy OCP-4.5-rc.1 cluster to give another loop with rhel-7 workers, unfortunately I am facing this customer case https://access.redhat.com/support/cases/internal/#/case/02669256/overview failed already 5x times.
I will give an update this bug once I have a cluster to work on.

Comment 23 Johnny Liu 2020-06-11 06:14:10 UTC

This should be a dup with https://bugzilla.redhat.com/show_bug.cgi?id=1804083.

This a known issues whatever you are scaling up a rhel worker against a UPI on OSP or IPI on OSP.

The rhel worker must be in the same subnet with rhcos worker, and a additional dns record for api-int is needed to ensure api-int to be resolved to the fifth ip address (like 192.168.0.5).

Comment 25 Lukas Bednar 2020-07-22 09:37:31 UTC

I was trying again, and openshift-ansible run failed to install device-mapper-multipath-0.4.9-123.el7.x86_64.
I am trying rhel-7.9 worker. What version of rhel am I supposed to use for that worker?

Error: Package: device-mapper-multipath-0.4.9-123.el7.x86_64 (rhel-7-server-rpms)\\\
           Requires: kpartx = 0.4.9-123.el7\\\
           Installed: kpartx-0.4.9-133.el7.x86_64 (installed)\\\
               kpartx = 0.4.9-133.el7\\\
           Available: kpartx-0.4.9-66.el7.x86_64 (rhel-7-server-rpms)\\\
               kpartx = 0.4.9-66.el7\\\
           Available: kpartx-0.4.9-77.el7.x86_64 (rhel-7-server-rpms)\\\
               kpartx = 0.4.9-77.el7\\\
           Available: kpartx-0.4.9-77.el7_1.1.x86_64 (rhel-7-server-rpms)\\\
               kpartx = 0.4.9-77.el7_1.1\\\
           Available: kpartx-0.4.9-77.el7_1.2.x86_64 (rhel-7-server-rpms)\\\
               kpartx = 0.4.9-77.el7_1.2\\\
           Available: kpartx-0.4.9-85.el7.x86_64 (rhel-7-server-rpms)\\\
               kpartx = 0.4.9-85.el7\\\
           Available: kpartx-0.4.9-85.el7_2.1.x86_64 (rhel-7-server-rpms)\\\
               kpartx = 0.4.9-85.el7_2.1\\\
           Available: kpartx-0.4.9-85.el7_2.2.x86_64 (rhel-7-server-rpms)\\\
               kpartx = 0.4.9-85.el7_2.2\\\
           Available: kpartx-0.4.9-85.el7_2.3.x86_64 (rhel-7-server-rpms)\\\
               kpartx = 0.4.9-85.el7_2.3\\\
           Available: kpartx-0.4.9-85.el7_2.4.x86_64 (rhel-7-server-rpms)\\\
               kpartx = 0.4.9-85.el7_2.4\\\
           Available: kpartx-0.4.9-85.el7_2.5.x86_64 (rhel-7-server-rpms)\\\
               kpartx = 0.4.9-85.el7_2.5\\\
           Available: kpartx-0.4.9-85.el7_2.6.x86_64 (rhel-7-server-rpms)\\\
               kpartx = 0.4.9-85.el7_2.6\\\
           Available: kpartx-0.4.9-99.el7.x86_64 (rhel-7-server-rpms)\\\
               kpartx = 0.4.9-99.el7\\\
           Available: kpartx-0.4.9-99.el7_3.1.x86_64 (rhel-7-server-rpms)\\\
               kpartx = 0.4.9-99.el7_3.1\\\
           Available: kpartx-0.4.9-99.el7_3.3.x86_64 (rhel-7-server-rpms)\\\
               kpartx = 0.4.9-99.el7_3.3\\\
           Available: kpartx-0.4.9-111.el7.x86_64 (rhel-7-server-rpms)\\\
               kpartx = 0.4.9-111.el7\\\
           Available: kpartx-0.4.9-111.el7_4.2.x86_64 (rhel-7-server-rpms)\\\
               kpartx = 0.4.9-111.el7_4.2\\\
           Available: kpartx-0.4.9-119.el7.x86_64 (rhel-7-server-rpms)\\\
               kpartx = 0.4.9-119.el7\\\
           Available: kpartx-0.4.9-119.el7_5.1.x86_64 (rhel-7-server-rpms)\\\
               kpartx = 0.4.9-119.el7_5.1\\\
           Available: kpartx-0.4.9-123.el7.x86_64 (rhel-7-server-rpms)\\\
               kpartx = 0.4.9-123.el7\\\
Error: Package: glibc-2.17-260.el7_6.6.i686 (rhel-7-server-rpms)\\\
           Requires: glibc-common = 2.17-260.el7_6.6\\\
           Installed: glibc-common-2.17-316.el7.x86_64 (installed)\\\
               glibc-common = 2.17-316.el7\\\
           Available: glibc-common-2.17-55.el7.x86_64 (rhel-7-server-rpms)\\\
               glibc-common = 2.17-55.el7\\\
           Available: glibc-common-2.17-55.el7_0.1.x86_64 (rhel-7-server-rpms)\\\
               glibc-common = 2.17-55.el7_0.1\\\
           Available: glibc-common-2.17-55.el7_0.3.x86_64 (rhel-7-server-rpms)\\\
               glibc-common = 2.17-55.el7_0.3\\\
           Available: glibc-common-2.17-55.el7_0.5.x86_64 (rhel-7-server-rpms)\\\
               glibc-common = 2.17-55.el7_0.5\\\
           Available: glibc-common-2.17-78.el7.x86_64 (rhel-7-server-rpms)\\\
               glibc-common = 2.17-78.el7\\\
           Available: glibc-common-2.17-105.el7.x86_64 (rhel-7-server-rpms)\\\
               glibc-common = 2.17-105.el7\\\
           Available: glibc-common-2.17-106.el7_2.1.x86_64 (rhel-7-server-rpms)\\\
               glibc-common = 2.17-106.el7_2.1\\\
           Available: glibc-common-2.17-106.el7_2.4.x86_64 (rhel-7-server-rpms)\\\
               glibc-common = 2.17-106.el7_2.4\\\
           Available: glibc-common-2.17-106.el7_2.6.x86_64 (rhel-7-server-rpms)\\\
               glibc-common = 2.17-106.el7_2.6\\\
           Available: glibc-common-2.17-106.el7_2.8.x86_64 (rhel-7-server-rpms)\\\
               glibc-common = 2.17-106.el7_2.8\\\
           Available: glibc-common-2.17-157.el7.x86_64 (rhel-7-server-rpms)\\\
               glibc-common = 2.17-157.el7\\\
           Available: glibc-common-2.17-157.el7_3.1.x86_64 (rhel-7-server-rpms)\\\
               glibc-common = 2.17-157.el7_3.1\\\
           Available: glibc-common-2.17-157.el7_3.2.x86_64 (rhel-7-server-rpms)\\\
               glibc-common = 2.17-157.el7_3.2\\\
           Available: glibc-common-2.17-157.el7_3.4.x86_64 (rhel-7-server-rpms)\\\
               glibc-common = 2.17-157.el7_3.4\\\
           Available: glibc-common-2.17-157.el7_3.5.x86_64 (rhel-7-server-rpms)\\\
               glibc-common = 2.17-157.el7_3.5\\\
           Available: glibc-common-2.17-196.el7.x86_64 (rhel-7-server-rpms)\\\
               glibc-common = 2.17-196.el7\\\
           Available: glibc-common-2.17-196.el7_4.2.x86_64 (rhel-7-server-rpms)\\\
               glibc-common = 2.17-196.el7_4.2\\\
           Available: glibc-common-2.17-222.el7.x86_64 (rhel-7-server-rpms)\\\
               glibc-common = 2.17-222.el7\\\
           Available: glibc-common-2.17-260.el7.x86_64 (rhel-7-server-rpms)\\\
               glibc-common = 2.17-260.el7\\\
           Available: glibc-common-2.17-260.el7_6.3.x86_64 (rhel-7-server-rpms)\\\
               glibc-common = 2.17-260.el7_6.3\\\
           Available: glibc-common-2.17-260.el7_6.4.x86_64 (rhel-7-server-rpms)\\\
               glibc-common = 2.17-260.el7_6.4\\\
           Available: glibc-common-2.17-260.el7_6.5.x86_64 (rhel-7-server-rpms)\\\
               glibc-common = 2.17-260.el7_6.5\\\
           Available: glibc-common-2.17-260.el7_6.6.x86_64 (rhel-7-server-rpms)\\\
               glibc-common = 2.17-260.el7_6.6