Bug 1573497

Summary: Octavia healthmonitor HTTPS - haproxy backend has no server available
Product: Red Hat OpenStack Reporter: Noam Manos <nmanos>
Component: openstack-octaviaAssignee: Nir Magnezi <nmagnezi>
Status: CLOSED DUPLICATE QA Contact: Alexander Stafeyev <astafeye>
Severity: medium Docs Contact:
Priority: medium    
Version: 13.0 (Queens)CC: astafeye, bperkins, cgoncalves, ihrachys, lpeer, majopela, nmanos, tfreger
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: 14.0 (Rocky)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-03-18 14:58:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Health Monitor HTTPS creation
none
workaround_creating_https_pool none

Description Noam Manos 2018-05-01 14:17:48 UTC 
Description of problem:
Creating Octavia healthmonitor type HTTPS - Loadbalancer service is unavailable. 
(When switching to healthmonitor of PING type, Loadbalancer works as expected).


Version-Release number of selected component (if applicable):
OSP: 13   
Puddle: 2018-04-10.2

How reproducible:
Always


Steps to Reproduce:

(tester) [stack@undercloud-0 ~]$ openstack loadbalancer healthmonitor create --delay 5 --max-retries 4 --timeout 10 --type HTTPS pool1 --name https_monitor


Actual results:
[root@amphora-9b983ed7-5a3b-4197-8981-1695cc8a0897 ~]# tail -f /var/log/**/*.log


Broadcast message from systemd-journald@amphora-9b983ed7-5a3b-4197-8981-1695cc8a0897 (Tue 2018-05-01 07:38:47 EDT):

haproxy[22916]: backend d001cc2c-a349-4495-a9b2-13b865676245 has no server available!


Expected results:


Additional info:
Adding console output and amphora log.
Comment 1 Noam Manos 2018-05-01 14:19:30 UTC 
Created attachment 1429182 [details]
Health Monitor HTTPS creation
Comment 2 Alexander Stafeyev 2018-05-02 13:20:40 UTC 
(In reply to Noam Manos from comment #0)
> Description of problem:
> Creating Octavia healthmonitor type HTTPS - Loadbalancer service is
> unavailable. 
> (When switching to healthmonitor of PING type, Loadbalancer works as
> expected).
> 
> 
> Version-Release number of selected component (if applicable):
> OSP: 13   
> Puddle: 2018-04-10.2
> 
> How reproducible:
> Always
> 
> 
> Steps to Reproduce:
> 
> (tester) [stack@undercloud-0 ~]$ openstack loadbalancer healthmonitor create
> --delay 5 --max-retries 4 --timeout 10 --type HTTPS pool1 --name
> https_monitor
> 
> 
> Actual results:
> [root@amphora-9b983ed7-5a3b-4197-8981-1695cc8a0897 ~]# tail -f
> /var/log/**/*.log
> 
> 
> Broadcast message from
> systemd-journald@amphora-9b983ed7-5a3b-4197-8981-1695cc8a0897 (Tue
> 2018-05-01 07:38:47 EDT):
> 
> haproxy[22916]: backend d001cc2c-a349-4495-a9b2-13b865676245 has no server
> available!
> 
> 
> Expected results:
> 
> 
> Additional info:
> Adding console output and amphora log.

Please share your member creation command.
Comment 3 Alexander Stafeyev 2018-05-02 13:25:51 UTC 
(In reply to Noam Manos from comment #0)
> Description of problem:
> Creating Octavia healthmonitor type HTTPS - Loadbalancer service is
> unavailable. 
> (When switching to healthmonitor of PING type, Loadbalancer works as
> expected).
> 
> 
> Version-Release number of selected component (if applicable):
> OSP: 13   
> Puddle: 2018-04-10.2
> 
> How reproducible:
> Always
> 
> 
> Steps to Reproduce:
> 
> (tester) [stack@undercloud-0 ~]$ openstack loadbalancer healthmonitor create
> --delay 5 --max-retries 4 --timeout 10 --type HTTPS pool1 --name
> https_monitor
> 
> 
> Actual results:
> [root@amphora-9b983ed7-5a3b-4197-8981-1695cc8a0897 ~]# tail -f
> /var/log/**/*.log
> 
> 
> Broadcast message from
> systemd-journald@amphora-9b983ed7-5a3b-4197-8981-1695cc8a0897 (Tue
> 2018-05-01 07:38:47 EDT):
> 
> haproxy[22916]: backend d001cc2c-a349-4495-a9b2-13b865676245 has no server
> available!
> 
> 
> Expected results:
> 
> 
> Additional info:
> Adding console output and amphora log.

Pls add this flag to the member create command : 
  --monitor-port <monitor_port>
                        An alternate protocol port used for health monitoring
                        a backend member.
Comment 5 Noam Manos 2018-05-02 15:46:31 UTC 
vm_name=vm-rht-1
vm_port=443
int_subnet_id=$(openstack subnet show int_subnet -c id -f value)
vm_ip=10.0.0.219


(tester) [stack@undercloud-0 ~]$ 
openstack loadbalancer member create --name ${vm_name}_$vm_port --subnet $int_subnet_id --address $vm_ip --protocol-port $vm_port pool1 --monitor-port $vm_port

+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| address             | 10.0.0.219                           |
| admin_state_up      | True                                 |
| created_at          | 2018-05-02T15:38:28                  |
| id                  | d8c8c27e-66a4-411f-b565-53927db29b97 |
| name                | vm-rht-1_443                         |
| operating_status    | OFFLINE                              |
| project_id          | f421dd896bcb47d28f692036f687fcd8     |
| protocol_port       | 443                                  |
| provisioning_status | PENDING_CREATE                       |
| subnet_id           | 34e5abbf-b084-40c0-8c62-846ae64968e0 |
| updated_at          | None                                 |
| weight              | 1                                    |
| monitor_port        | 443                                  |
| monitor_address     | None                                 |
+---------------------+--------------------------------------+

(tester) [stack@undercloud-0 ~]$ openstack loadbalancer member list pool1
+--------------------------------------+--------------+----------------------------------+---------------------+------------+---------------+------------------+--------+
| id                                   | name         | project_id                       | provisioning_status | address    | protocol_port | operating_status | weight |
+--------------------------------------+--------------+----------------------------------+---------------------+------------+---------------+------------------+--------+
| 730bd43d-949d-43d1-a436-57a8366904ed | vm-rht-1     | f421dd896bcb47d28f692036f687fcd8 | ACTIVE              | 10.0.0.219 |            80 | NO_MONITOR       |      1 |
| 66add177-5369-43d1-b7f3-0798d124eaf8 | vm-rht-2     | f421dd896bcb47d28f692036f687fcd8 | ACTIVE              | 10.0.0.214 |            80 | NO_MONITOR       |      1 |
| d8c8c27e-66a4-411f-b565-53927db29b97 | vm-rht-1_443 | f421dd896bcb47d28f692036f687fcd8 | ACTIVE              | 10.0.0.219 |           443 | OFFLINE          |      1 |
+--------------------------------------+--------------+----------------------------------+---------------------+------------+---------------+------------------+--------+

(tester) [stack@undercloud-0 ~]$ curl $lb_fip:80; curl $lb_fip:443

<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
Comment 6 Noam Manos 2018-05-02 15:49:31 UTC 
(In reply to Alexander Stafeyev from comment #3)
> 
> Pls add this flag to the member create command : 
>   --monitor-port <monitor_port>
>                         An alternate protocol port used for health monitoring
>                         a backend member.


Adding a new member with monitor-port=443 (and also protocol-port=443) did not resolve issue - still getting error 503 Service Unavailable.
Comment 7 Noam Manos 2018-05-07 08:02:00 UTC 
A workaround is to remove the HTTP pool, and create an HTTPS pool.
Then the Load Balancer (10.0.0.216) redirects traffic:

(tester) [stack@undercloud-0 ~]$ openstack loadbalancer pool create --name pool_https --loadbalancer LB --lb-algorithm ROUND_ROBIN --protocol HTTPS

(tester) [stack@undercloud-0 ~]$ openstack loadbalancer member create pool_https --name vm-rht-1_80 --subnet $int_subnet_id --address 10.0.0.219 --protocol-port 80



[stack@undercloud-0 ~]$ curl 10.0.0.216:80
Apache PHP Web Server:

 Connected client: 10.0.0.212
 Web server IP: 172.16.0.219
 Web server port: 80

 Host name: vm-rht-2
 Operating system: Linux
 Release name: 3.10.0-862.el7.x86_64
 Version information: #1 SMP Wed Mar 21 18:14:51 EDT 2018
 Machine type: x86_64


(tester) [stack@undercloud-0 ~]$ curl 10.0.0.216:443
Apache PHP Web Server:

 Connected client: 10.0.0.212
 Web server IP: 172.16.0.216
 Web server port: 443

 Host name: vm-rht-1
 Operating system: Linux
 Release name: 3.10.0-862.el7.x86_64
 Version information: #1 SMP Wed Mar 21 18:14:51 EDT 2018
 Machine type: x86_64


(Attaching full console output)
Comment 8 Noam Manos 2018-05-07 08:05:56 UTC 
Created attachment 1432520 [details]
workaround_creating_https_pool
Comment 10 Carlos Goncalves 2019-03-18 14:58:43 UTC 
HTTP pool associated to a HTTPS listener is an invalid configuration.

Validation of listener and pool protocols is being added at Octavia API level upstream and tracked in RHBZ #1668369.

*** This bug has been marked as a duplicate of bug 1668369 ***