Description of problem: After a 4.4 ipv6 single-stack installation on Azure failed(which is expected for now), use the installer to gather debugging logs. 1. Gather logs with ipv4 addresses $ ./openshift-install gather bootstrap --key ./x.pem --bootstrap 13.89.107.24 --master 10.0.0.5 INFO Pulling debug logs from the bootstrap machine INFO Bootstrap gather logs captured here "log-bundle-20200215055500.tar.gz" 2. Gather logs with ipv6 addresses $ ./openshift-install gather bootstrap --key ./x.pem --bootstrap 2603:1030:b:3::48 --master fd00::5 INFO Pulling debug logs from the bootstrap machine INFO Bootstrap gather logs captured here "log-bundle-20200215060021.tar.gz" 3. Check the logs collected from master after decompressing the log-bundle file. In the ipv4 one,container logs and system journals could be found. $ ls log_1/control-plane/10.0.0.5/ containers failed-units.txt journals unit-status In the ipv6 one, nothing there. $ ls log_2/control-plane/fd00\:\:5/ $ Version-Release number of the following components: Payload 4.4.0-0.nightly-2020-02-15-024625 $ ./openshift-install version ./openshift-install v4.4.0 built from commit a627412646a322daa6596bdb5bcefe8c64690439 release image registry.svc.ci.openshift.org/ocp/release@sha256:5de81fdc8361b00a6b46dfa822f52e7495f8555cbc88d99b4b634869c1f9ddb1 How reproducible: Always Steps to Reproduce: 1.[root@jialiu43az1-int-svc ~]# ./openshift-install gather bootstrap --key ./xxx.pem --bootstrap 2603:1030:b:3::48 --master fd00::5 --log-level debug DEBUG OpenShift Installer v4.4.0 DEBUG Built from commit a627412646a322daa6596bdb5bcefe8c64690439 INFO Pulling debug logs from the bootstrap machine DEBUG Added ./xxx.pem to installer's internal agent DEBUG Gathering bootstrap systemd summary ... DEBUG Gathering bootstrap failed systemd unit status ... DEBUG Gathering bootstrap journals ... DEBUG Gathering bootstrap containers ... DEBUG Gathering rendered assets... DEBUG Gathering cluster resources ... DEBUG Waiting for logs ... DEBUG Flag --config has been deprecated, use --kubeconfig instead DEBUG Flag --config has been deprecated, use --kubeconfig instead DEBUG Flag --config has been deprecated, use --kubeconfig instead DEBUG Flag --config has been deprecated, use --kubeconfig instead DEBUG Flag --config has been deprecated, use --kubeconfig instead DEBUG Flag --config has been deprecated, use --kubeconfig instead DEBUG Flag --config has been deprecated, use --kubeconfig instead DEBUG Flag --config has been deprecated, use --kubeconfig instead DEBUG Flag --config has been deprecated, use --kubeconfig instead DEBUG Flag --config has been deprecated, use --kubeconfig instead DEBUG Flag --config has been deprecated, use --kubeconfig instead DEBUG Flag --config has been deprecated, use --kubeconfig instead DEBUG Flag --config has been deprecated, use --kubeconfig instead DEBUG Flag --config has been deprecated, use --kubeconfig instead DEBUG Flag --config has been deprecated, use --kubeconfig instead DEBUG Flag --config has been deprecated, use --kubeconfig instead DEBUG Flag --config has been deprecated, use --kubeconfig instead DEBUG Flag --config has been deprecated, use --kubeconfig instead DEBUG Flag --config has been deprecated, use --kubeconfig instead DEBUG Flag --config has been deprecated, use --kubeconfig instead DEBUG Flag --config has been deprecated, use --kubeconfig instead DEBUG Flag --config has been deprecated, use --kubeconfig instead DEBUG Flag --config has been deprecated, use --kubeconfig instead DEBUG Flag --config has been deprecated, use --kubeconfig instead DEBUG Flag --config has been deprecated, use --kubeconfig instead DEBUG Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) DEBUG Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) DEBUG Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) DEBUG Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) DEBUG Unable to connect to the server: context deadline exceeded (Client.Timeout exceeded while awaiting headers) DEBUG Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) DEBUG Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) DEBUG Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) DEBUG Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) DEBUG Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) DEBUG Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) DEBUG Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) DEBUG Unable to connect to the server: context deadline exceeded (Client.Timeout exceeded while awaiting headers) DEBUG Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) DEBUG Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) DEBUG Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) DEBUG Unable to connect to the server: context deadline exceeded (Client.Timeout exceeded while awaiting headers) DEBUG Unable to connect to the server: context deadline exceeded (Client.Timeout exceeded while awaiting headers) DEBUG Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) DEBUG Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) DEBUG Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) DEBUG Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) DEBUG Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) DEBUG Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) DEBUG Gather remote logs DEBUG Collecting info from fd00::5 DEBUG Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) DEBUG lost connection EBUG Warning: Permanently added 'fd00::5' (ECDSA) to the list of known hosts. DEBUG Gathering master systemd summary ... DEBUG Gathering master failed systemd unit status ... DEBUG Gathering master journals ... DEBUG Gathering master containers ... DEBUG Waiting for logs ... DEBUG Log bundle written to /var/home/core/log-bundle-20200215075356.tar.gz INFO Bootstrap gather logs captured here "log-bundle-20200215075356.tar.gz" Actual results: Please include the entire output from the last TASK line through the end of output if an error is generated Expected results: Additional info: Please attach logs from ansible-playbook with the -vvv flag
Are you able to manuall SSH to the master node ??
More specifically, are you able to manually SSH from the bootstrap node to the master node?
It looks like the scp is failing with the syntax "core@fd00::5:", and so the installer-masters-gather.sh script is not being uploaded to the masters, and the resulting logs would not downloaded from the masters. Adding brackets appears to resolve this: "core@[fd00::5]:" and is backward compatible with ipv4. Will submit a PR shortly.
Verify this bug with openshift-install-linux-4.5.0-0.nightly-2020-03-17-232152, logs on master could be collected via private ipv6 address. $ ./openshift-install gather bootstrap --bootstrap 2603:1030:b:3::4d --master fd00::5 --key ./openshift-qe.pem --log-level debug DEBUG OpenShift Installer 4.5.0-0.nightly-2020-03-17-232152 DEBUG Built from commit 344e38f31fb65dfd27184bee420e6ec0043618b7 INFO Pulling debug logs from the bootstrap machine DEBUG Added ./openshift-qe.pem to installer's internal agent DEBUG Gathering bootstrap systemd summary ... DEBUG Gathering bootstrap failed systemd unit status ... DEBUG Gathering bootstrap journals ... DEBUG Gathering bootstrap containers ... DEBUG Gathering rendered assets... DEBUG Gathering cluster resources ... DEBUG Waiting for logs ... DEBUG Gather remote logs DEBUG Collecting info from fd00::5 EBUG Warning: Permanently added 'fd00::5' (ECDSA) to the list of known hosts. DEBUG Gathering master systemd summary ... DEBUG Gathering master failed systemd unit status ... DEBUG Gathering master journals ... DEBUG Gathering master containers ... DEBUG Waiting for logs ... DEBUG Log bundle written to /var/home/core/log-bundle-20200318073227.tar.gz INFO Bootstrap gather logs captured here "log-bundle-20200318073227.tar.gz"
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581