Bug 1877483
| Summary: | validate reliability of Azure booting in 4.6 | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Colin Walters <walters> | ||||
| Component: | RHCOS | Assignee: | Micah Abbott <miabbott> | ||||
| Status: | CLOSED NOTABUG | QA Contact: | Michael Nguyen <mnguyen> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | urgent | ||||||
| Version: | 4.6 | CC: | bbreard, imcleod, jligon, lucab, nstielau, sbatsche, sttts | ||||
| Target Milestone: | --- | ||||||
| Target Release: | 4.6.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | Type: | Bug | |||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Colin Walters
2020-09-09 17:49:41 UTC
Created attachment 1714336 [details]
azure serial console logs
I have reproduced this a few times, one or more master nodes will not be reachable. As you can see from log eth0 with not have an IP address assigned.
If OS boots without IP seems like it should reboot and retry? If I restart the node it comes up as expected. To compare with non failure case here is how node looks populated with IP. ``` Red Hat Enterprise Linux CoreOS 46.82.202009080940-0 (Ootpa) 4.6 SSH host key: SHA256:u7C6ANGUYM/wg0l7uCdRgjfSRc1880Yt5XqUxbaz1BA (ED25519) SSH host key: SHA256:isRZUSCtLxP9cZT7uS1HZZliJ8zec8Z58E2keN/vWr8 (ECDSA) SSH host key: SHA256:vuNAdjKZCcp7dUtU/csKReg/RcICf1ngOSpImYhjYH8 (RSA) eth0: 10.0.0.7 fe80::20d:3aff:fe11:c523 sbatsche-dhcp3-2020-0-p4dcp-master-0 login: [ 3237.856906] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue) [ 3268.002985] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue) ``` Isn't this an urgent issue? Medium does not fit. Since we see a direct relationship between this issue and CI failures boosting to urgent. example CI run showing missing nodes very possibly a CI repro. https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-azure-4.6/1303077792237228032 From Sam's initial logs, there are a few cycles in systemd dependency graph, all of them going through the Azure specific `openshift-azure-routes.path` unit. To the best of my systemd knowledge, cycle detection is deterministic while cycle breaking is not. Thus some boots will have the loop-cut taking out NetworkManager completely, while some other boots will miss some other service (with the breakage being less visible but still existing somewhere). Snippet for reference: ``` [ 324.981226] systemd[1]: network-online.target: Found ordering cycle on network.target/start [ 325.000115] systemd[1]: network-online.target: Found dependency on ovs-vswitchd.service/start [ 325.013025] systemd[1]: network-online.target: Found dependency on basic.target/start [ 325.027061] systemd[1]: network-online.target: Found dependency on paths.target/start [ 325.035592] systemd[1]: network-online.target: Found dependency on openshift-azure-routes.path/start [ 325.049473] systemd[1]: network-online.target: Found dependency on network-online.target/start [ 325.058497] systemd[1]: network-online.target: Job network.target/start deleted to break ordering cycle starting with network-online.target/start [ SKIP ] Ordering cycle found, skipping Network [ 325.090336] systemd[1]: NetworkManager-wait-online.service: Found ordering cycle on NetworkManager.service/start [ 325.102682] systemd[1]: NetworkManager-wait-online.service: Found dependency on openvswitch.service/start [ 325.123740] systemd[1]: NetworkManager-wait-online.service: Found dependency on ovs-vswitchd.service/start [ 325.133741] systemd[1]: NetworkManager-wait-online.service: Found dependency on basic.target/start [ 325.143301] systemd[1]: NetworkManager-wait-online.service: Found dependency on paths.target/start [ 325.155781] systemd[1]: NetworkManager-wait-online.service: Found dependency on openshift-azure-routes.path/start [ 325.168550] systemd[1]: NetworkManager-wait-online.service: Found dependency on network-online.target/start [ 325.187318] systemd[1]: NetworkManager-wait-online.service: Found dependency on NetworkManager-wait-online.service/start [ 325.206579] systemd[1]: NetworkManager-wait-online.service: Job NetworkManager.service/start deleted to break ordering cycle starting with NetworkManager-wait-online.service/start [ SKIP ] Ordering cycle found, skipping Network Manager [ 325.244759] systemd[1]: paths.target: Found ordering cycle on openshift-azure-routes.path/start [ 325.260943] systemd[1]: paths.target: Found dependency on network-online.target/start [ 325.274817] systemd[1]: paths.target: Found dependency on node-valid-hostname.service/start [ 325.294598] systemd[1]: paths.target: Found dependency on basic.target/start [ 325.311602] systemd[1]: paths.target: Found dependency on paths.target/start [ 325.321312] systemd[1]: paths.target: Job openshift-azure-routes.path/start deleted to break ordering cycle starting with paths.target/start [ SKIP ] Ordering cycle found, skipping Watch for downfile changes ``` Right, we have strong reason to believe that this bug was specific to that PR. I think we have a larger structural issue here around "basic coreos sanity testing across platforms without getting to the point of sending a PR to openshift/installer" but we've got an epic for that in 4.7. Right, we have strong reason to believe that this bug was specific to that PR. I think we have a larger structural issue here around "basic coreos sanity testing across platforms without getting to the point of sending a PR to openshift/installer" but we've got an epic for that in 4.7. |