Bug 2126730

Summary: Network API HttpException: 504, The server didn't respond in time.: 504 Gateway Time-out
Product: Red Hat OpenStack Reporter: Udi Shkalim <ushkalim>
Component: openstack-neutronAssignee: Yatin Karel <ykarel>
Status: CLOSED NEXTRELEASE QA Contact: Eran Kuris <ekuris>
Severity: high Docs Contact:
Priority: high    
Version: 17.0 (Wallaby)CC: chrisw, jlibosva, mschuppe, pkomarov, scohen, skaplons, twilson, ykarel
Target Milestone: ---Keywords: AutomationBlocker, TestBlocker, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2170352 (view as bug list) Environment:
Last Closed: 2023-02-03 20:11:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2170352    

Description Udi Shkalim 2022-09-14 11:07:00 UTC
Description of problem:
+ openstack router create internal_net_f7654d797a_router
+-------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                   | Value                                                                                                                                                                     |
+-------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up          | UP                                                                                                                                                                        |
| availability_zone_hints |                                                                                                                                                                           |
| availability_zones      |                                                                                                                                                                           |
| created_at              | 2022-09-13T20:38:06Z                                                                                                                                                      |
| description             |                                                                                                                                                                           |
| external_gateway_info   | null                                                                                                                                                                      |
| flavor_id               | None                                                                                                                                                                      |
| id                      | c28058cc-f6be-4781-a07f-e5cecc562183                                                                                                                                      |
| location                | cloud='overcloud', project.domain_id=, project.domain_name='Default', project.id='19d4f641681d4d479888389339af3b95', project.name='admin', region_name='regionOne', zone= |
| name                    | internal_net_f7654d797a_router                                                                                                                                            |
| project_id              | 19d4f641681d4d479888389339af3b95                                                                                                                                          |
| revision_number         | 1                                                                                                                                                                         |
| routes                  |                                                                                                                                                                           |
| status                  | ACTIVE                                                                                                                                                                    |
| tags                    |                                                                                                                                                                           |
| updated_at              | 2022-09-13T20:38:06Z                                                                                                                                                      |
+-------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+ echo 'Creating network internal_net_f7654d797a'
Creating network internal_net_f7654d797a
+ openstack network create internal_net_f7654d797a
Error while executing command: HttpException: 504, The server didn't respond in time.: 504 Gateway Time-out
+ echo 'Creating subnet internal_net_f7654d797a_subnet'
Creating subnet internal_net_f7654d797a_subnet
+ openstack subnet create --subnet-range 192.168.0.0/24 --allocation-pool start=192.168.0.10,end=192.168.0.100 --gateway 192.168.0.254 --dns-nameserver 172.22.0.1 --network internal_net_f7654d797a internal_net_f7654d797a_subnet
HttpException: 504: Server Error for url: https://overcloud.osptest.test.metalkube.org:13696/v2.0/subnets, 504 Gateway Time-out: The server didn't respond in time.
+ echo 'Add subnet internal_net_f7654d797a_subnet to router internal_net_f7654d797a_router'
Add subnet internal_net_f7654d797a_subnet to router internal_net_f7654d797a_router
+ openstack router add subnet internal_net_f7654d797a_router internal_net_f7654d797a_subnet
+ echo 'Set external-gateway for internal_net_f7654d797a_router'
Set external-gateway for internal_net_f7654d797a_router
+ openstack router set --external-gateway public internal_net_f7654d797a_router



Version-Release number of selected component (if applicable):
rhos-release 17.0 -p RHOS-17.0-RHEL-9-20220909.n.0 -r 9.0

How reproducible:
100%

Steps to Reproduce:
1. Run ansible sts-ha tests
2.
3.

Actual results:
Tests are failing

Expected results:
Tests are passing

Additional info:

Comment 2 Martin Schuppert 2022-09-15 07:14:47 UTC
might be the same as https://bugzilla.redhat.com/show_bug.cgi?id=2117182 , just for OSP17 ?

Comment 3 Martin Schuppert 2022-09-22 06:31:52 UTC
The logs in the sosreports on any of the controllers do only cover 2022-09-14, not 2022-09-13 where from comment 1 the test was done. Could you rerun the test and when creating the sosreport use --all-logs to collect all available logs regardless of size.

Comment 4 Udi Shkalim 2022-09-29 10:11:44 UTC
(In reply to Martin Schuppert from comment #2)
> might be the same as https://bugzilla.redhat.com/show_bug.cgi?id=2117182 ,
> just for OSP17 ?

Yes, it seems like it.

Comment 5 pkomarov 2022-12-29 08:08:24 UTC
reproduced also with me I'm adding my sos reports: 
sosreports,all overcloud_nodes /var/log, are at :  http://file.tlv.redhat.com/~pkomarov/sos_reports_2126730

Comment 6 pkomarov 2022-12-29 09:16:07 UTC
(In reply to pkomarov from comment #5)
> reproduced also with me I'm adding my sos reports: 
> sosreports,all overcloud_nodes /var/log, are at : 
> http://file.tlv.redhat.com/~pkomarov/sos_reports_2126730

Sorry sent too soon those point to a galera issue opened here : https://bugzilla.redhat.com/show_bug.cgi?id=2156886
I'll update when this issue reproduces

Comment 7 pkomarov 2022-12-29 14:28:33 UTC
issue reproduced here : 
During disruptive test : NETWORK_DISRUPTOR_CONTROLLER_MAIN_VIP 
context : which does ifdown to the br-osp inerface on the controller holding the main VIP pcs resource 
Then wait for the pacemaker cluster to recover the node or fence etc..
then try to boot a vm with network etc..
breaks on : 

+ openstack network create internal_net_90208ad795
Error while executing command: HttpException: 504, The server didn't respond in time.: 504 Gateway Time-out

Log date for the test was :(since) 12.29.22 10:48
sosreports,all overcloud_nodes /var/log, are at :  http://file.tlv.redhat.com/~pkomarov/sos_reports_21267302

Comment 14 pkomarov 2023-01-18 16:19:57 UTC
can you check again ? 
I got : 
+ openstack network create internal_net_9f92069461
Error while executing command: HttpException: 504, The server didn't respond in time.: 504 Gateway Time-out
after rebooting all controller in order oon the latest passed_phase2

Comment 22 Jakub Libosvar 2023-02-03 20:11:41 UTC
Will be fixed in 17.1.0