Bug 1956372

Summary: openshift-gcp-routes causes disruption during upgrade by stopping before all pods terminate
Product: OpenShift Container Platform Reporter: Clayton Coleman <ccoleman>
Component: Machine Config OperatorAssignee: Antonio Ojea <aojeagar>
Status: VERIFIED --- QA Contact: Michael Nguyen <mnguyen>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.8   
Target Milestone: ---   
Target Release: 4.8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: The Openshift script handling the Google Cloud Loadbalancer logic was exiting before the network was down. Consequence: The Openshift components that depend on loadbalancers were disrupted, so they can not exit gracefully Fix: Wait until the network is down before exiting Result: Graceful shutdown works correctly
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1966595    

Description Clayton Coleman 2021-05-03 14:41:27 UTC
openshift-gcp-routes is required to send traffic from GCP load balancers (because it actually connects the VIP to the host networking). Right now it stops *before* networking stops, which means while kube-apiserver is draining we kill the vip, which causes disruption.

We should terminate openshift-gcp-routes service when the network is shutting down, not before.

Comment 2 Michael Nguyen 2021-05-07 18:21:39 UTC
Verified on  4.8.0-0.nightly-2021-05-07-075528.  openshift-gcp-routes.service is stopped after network online target.

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-05-07-075528   True        False         32m     Cluster version is 4.8.0-0.nightly-2021-05-07-075528
$ oc get nodes
NAME                                       STATUS   ROLES    AGE   VERSION
ci-ln-9wr6012-f76d1-z7bjv-master-0         Ready    master   53m   v1.21.0-rc.0+291e731
ci-ln-9wr6012-f76d1-z7bjv-master-1         Ready    master   53m   v1.21.0-rc.0+291e731
ci-ln-9wr6012-f76d1-z7bjv-master-2         Ready    master   53m   v1.21.0-rc.0+291e731
ci-ln-9wr6012-f76d1-z7bjv-worker-b-gwn8c   Ready    worker   44m   v1.21.0-rc.0+291e731
ci-ln-9wr6012-f76d1-z7bjv-worker-c-c2ndb   Ready    worker   44m   v1.21.0-rc.0+291e731
ci-ln-9wr6012-f76d1-z7bjv-worker-d-2sc2x   Ready    worker   44m   v1.21.0-rc.0+291e731
$ oc debug node/ci-ln-9wr6012-f76d1-z7bjv-master-0
Starting pod/ci-ln-9wr6012-f76d1-z7bjv-master-0-debug ...
To use host binaries, run `chroot /host`
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# systemctl cat openshift-gcp-routes
# /etc/systemd/system/openshift-gcp-routes.service
[Unit]
Description=Update GCP routes for forwarded IPs.
ConditionKernelCommandLine=|ignition.platform.id=gce
ConditionKernelCommandLine=|ignition.platform.id=gcp
Before=network-online.target

[Service]
Type=simple
ExecStart=/bin/bash /opt/libexec/openshift-gcp-routes.sh start
ExecStopPost=/bin/bash /opt/libexec/openshift-gcp-routes.sh cleanup
User=root
RestartSec=30
Restart=always

[Install]
WantedBy=multi-user.target
# Ensure that network-online.target will not complete until the node has working external LBs.
RequiredBy=network-online.target
sh-4.4# journalctl
...snip...
May 07 17:15:11 ci-ln-9wr6012-f76d1-z7bjv-master-0.c.openshift-gce-devel-ci.inte systemd[1]: Stopped target Network is Online.
May 07 17:15:11 ci-ln-9wr6012-f76d1-z7bjv-master-0.c.openshift-gce-devel-ci.inte systemd[1]: node-valid-hostname.service: Succeeded.
May 07 17:15:11 ci-ln-9wr6012-f76d1-z7bjv-master-0.c.openshift-gce-devel-ci.inte systemd[1]: Stopped Ensure the node hostname is valid for the cluster.
May 07 17:15:11 ci-ln-9wr6012-f76d1-z7bjv-master-0.c.openshift-gce-devel-ci.inte systemd[1]: node-valid-hostname.service: Consumed 0 CPU time
May 07 17:15:11 ci-ln-9wr6012-f76d1-z7bjv-master-0.c.openshift-gce-devel-ci.inte systemd[1]: Stopping Update GCP routes for forwarded IPs....
May 07 17:15:11 ci-ln-9wr6012-f76d1-z7bjv-master-0.c.openshift-gce-devel-ci.inte systemd[1]: NetworkManager-wait-online.service: Succeeded.
May 07 17:15:11 ci-ln-9wr6012-f76d1-z7bjv-master-0.c.openshift-gce-devel-ci.inte systemd[1]: Stopped Network Manager Wait Online.
May 07 17:15:11 ci-ln-9wr6012-f76d1-z7bjv-master-0.c.openshift-gce-devel-ci.inte systemd[1]: NetworkManager-wait-online.service: Consumed 0 CPU time
May 07 17:15:11 ci-ln-9wr6012-f76d1-z7bjv-master-0.c.openshift-gce-devel-ci.inte systemd[1]: Stopped target sshd-keygen.target.
May 07 17:15:11 ci-ln-9wr6012-f76d1-z7bjv-master-0.c.openshift-gce-devel-ci.inte systemd[1]: systemd-user-sessions.service: Succeeded.
May 07 17:15:11 ci-ln-9wr6012-f76d1-z7bjv-master-0.c.openshift-gce-devel-ci.inte systemd[1]: Stopped Permit User Sessions.
May 07 17:15:11 ci-ln-9wr6012-f76d1-z7bjv-master-0.c.openshift-gce-devel-ci.inte systemd[1]: systemd-user-sessions.service: Consumed 13ms CPU time
May 07 17:15:11 ci-ln-9wr6012-f76d1-z7bjv-master-0.c.openshift-gce-devel-ci.inte systemd[1]: Stopped target Remote File Systems.
May 07 17:15:11 ci-ln-9wr6012-f76d1-z7bjv-master-0.c.openshift-gce-devel-ci.inte systemd[1]: Stopped target Network.
May 07 17:15:11 ci-ln-9wr6012-f76d1-z7bjv-master-0.c.openshift-gce-devel-ci.inte systemd[1]: Stopping Network Manager...