Bug 1825823 - [OSP] haproxy pod from openshift-openstack-infra is crashlooping
Summary: [OSP] haproxy pod from openshift-openstack-infra is crashlooping
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Machine Config Operator
Version: 4.5
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.5.0
Assignee: Mike Fedosin
QA Contact: weiwei jiang
Depends On:
TreeView+ depends on / blocked
Reported: 2020-04-20 10:38 UTC by Mike Fedosin
Modified: 2020-07-13 17:28 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2020-07-13 17:28:40 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift machine-config-operator pull 1665 0 None closed Bug 1825823: Add support for reading API LB backends from KUBE-API for OpenStack and Ovirt 2020-12-14 13:55:37 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:28:54 UTC

Description Mike Fedosin 2020-04-20 10:38:50 UTC
During the installation of OpenShift on OpenStack I noticed that all haproxy pods are crashlooping:

haproxy-mfedosin-rwkfm-master-0              1/2     CrashLoopBackOff   13         30m
haproxy-mfedosin-rwkfm-master-1              1/2     CrashLoopBackOff   13         30m
haproxy-mfedosin-rwkfm-master-2              1/2     CrashLoopBackOff   13         30m

But the cluster was deployed successfully anyway.

From the pods logs:

$ oc logs -n openshift-openstack-infra haproxy-mfedosin-rwkfm-master-0 -c haproxy-monitor
time="2020-04-20T09:55:30Z" level=info msg="API is not reachable through HAProxy"
time="2020-04-20T09:55:30Z" level=info msg="Failed to get master Nodes list" err="nodes is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot list resource \"nodes\" in API group \"\" at the cluster scope"
time="2020-04-20T09:55:30Z" level=error msg="Failed to retrieve API members information" kubeconfigPath=/etc/kubernetes/kubeconfig
time="2020-04-20T09:55:30Z" level=info msg="GetLBConfig failed, sleep half of interval and retry" kubeconfigPath=/etc/kubernetes/kubeconfig
time="2020-04-20T09:55:33Z" level=info msg="Failed to get master Nodes list" err="nodes is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot list resource \"nodes\" in API group \"\" at the cluster scope"

$ oc logs -n openshift-openstack-infra haproxy-mfedosin-rwkfm-master-0 -c haproxy
+ declare -r haproxy_sock=/var/run/haproxy/haproxy-master.sock
+ declare -r haproxy_log_sock=/var/run/haproxy/haproxy-log.sock
+ export -f msg_handler
+ export -f reload_haproxy
+ export -f verify_old_haproxy_ps_being_deleted
+ rm -f /var/run/haproxy/haproxy-master.sock /var/run/haproxy/haproxy-log.sock
+ '[' -s /etc/haproxy/haproxy.cfg ']'
+ socat UNIX-LISTEN:/var/run/haproxy/haproxy-master.sock,fork 'system:bash -c msg_handler'
+ socat UNIX-RECV:/var/run/haproxy/haproxy-log.sock STDOUT

Comment 4 weiwei jiang 2020-05-18 02:06:30 UTC
Checked with 4.5.0-0.nightly-2020-05-17-220731, moved to verified.

$ oc get pods -n openshift-openstack-infra -l app=openstack-infra-api-lb -o wide 
NAME                                 READY   STATUS    RESTARTS   AGE   IP             NODE                         NOMINATED NODE   READINESS GATES
haproxy-wj45ios518a-qwztx-master-0   2/2     Running   2          31m   wj45ios518a-qwztx-master-0   <none>           <none>
haproxy-wj45ios518a-qwztx-master-1   2/2     Running   0          31m   wj45ios518a-qwztx-master-1   <none>           <none>
haproxy-wj45ios518a-qwztx-master-2   2/2     Running   0          31m   wj45ios518a-qwztx-master-2   <none>           <none>
$ oc get clusterversion 
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.0-0.nightly-2020-05-17-220731   True        False         12m     Cluster version is 4.5.0-0.nightly-2020-05-17-220731

Comment 5 errata-xmlrpc 2020-07-13 17:28:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.