Bug 2117182
| Summary: | HA testing including hard reboot controllers and post OSP update - network resources creation can fail with Gateway Time-out | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | pkomarov |
| Component: | osp-director-operator-container | Assignee: | Jakub Libosvar <jlibosva> |
| Status: | CLOSED NOTABUG | QA Contact: | |
| Severity: | urgent | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 16.2 (Train) | CC: | egarciar, froyo, jlibosva, skaplons, ushkalim |
| Target Milestone: | z4 | Keywords: | Regression, Reopened, TestBlocker |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-11-16 14:14:19 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Comment 2
pkomarov
2022-08-15 20:22:25 UTC
Closing as this issue doesn't reproduce on a BM host with more power - 96 vs 60 cores and 250 G mem Re-opening this one as we see the same error in HA testing. Only after reboot on the OVN master node/controller does the cluster regain activity. IMO, not related to HW. Creating network internal_net_3244a4dee9 + openstack network create internal_net_3244a4dee9 Error while executing command: HttpException: 504, The server didn't respond in time.: 504 Gateway Time-out + echo 'Creating subnet internal_net_3244a4dee9_subnet' Creating subnet internal_net_3244a4dee9_subnet + openstack subnet create --subnet-range 192.168.0.0/24 --allocation-pool start=192.168.0.10,end=192.168.0.100 --gateway 192.168.0.254 --dns-nameserver 172.22.0.1 --network internal_net_3244a4dee9 internal_net_3244a4dee9_subnet HttpException: 504: Server Error for url: https://overcloud.osptest.test.metalkube.org:13696/v2.0/subnets, 504 Gateway Time-out: The server didn't respond in time. + echo 'Add subnet internal_net_3244a4dee9_subnet to router internal_net_3244a4dee9_router' Add subnet internal_net_3244a4dee9_subnet to router internal_net_3244a4dee9_router + openstack router add subnet internal_net_3244a4dee9_router internal_net_3244a4dee9_subnet No Router found for internal_net_3244a4dee9_router + echo 'Set external-gateway for internal_net_3244a4dee9_router' Set external-gateway for internal_net_3244a4dee9_router + openstack router set --external-gateway public internal_net_3244a4dee9_router No Router found for internal_net_3244a4dee9_router ## create security group openstack security group list | grep ${SECGROUP_NAME} + grep allow-icmp-ssh-3244a4dee9 + openstack security group list if [ $? -ne 0 ]; then echo "Creating security group ${SECGROUP_NAME}" openstack security group create ${SECGROUP_NAME} echo "Creating rules for ports 22,80,443 in security group ${SECGROUP_NAME}" openstack security group rule create --proto icmp ${SECGROUP_NAME} openstack security group rule create --proto tcp --dst-port 22 ${SECGROUP_NAME} openstack security group rule create --proto tcp --dst-port 80 ${SECGROUP_NAME} openstack security group rule create --proto tcp --dst-port 443 ${SECGROUP_NAME} fi + '[' 1 -ne 0 ']' + echo 'Creating security group allow-icmp-ssh-3244a4dee9' Creating security group allow-icmp-ssh-3244a4dee9 + openstack security group create allow-icmp-ssh-3244a4dee9 Error while executing command: HttpException: 504, 504 Gateway Time-out: The server didn't respond in time. + echo 'Creating rules for ports 22,80,443 in security group allow-icmp-ssh-3244a4dee9' Creating rules for ports 22,80,443 in security group allow-icmp-ssh-3244a4dee9 + openstack security group rule create --proto icmp allow-icmp-ssh-3244a4dee9 Error while executing command: HttpException: 504, 504 Gateway Time-out: The server didn't respond in time. + openstack security group rule create --proto tcp --dst-port 22 allow-icmp-ssh-3244a4dee9 Error while executing command: HttpException: 504, The server didn't respond in time.: 504 Gateway Time-out + openstack security group rule create --proto tcp --dst-port 80 allow-icmp-ssh-3244a4dee9 Error while executing command: HttpException: 504, The server didn't respond in time.: 504 Gateway Time-out + openstack security group rule create --proto tcp --dst-port 443 allow-icmp-ssh-3244a4dee9 While trying to have a reproducer environment it seems that the problem is fixed 5/5 HA runs are passing |