Bug 1858339
| Summary: | 4.5.1 installer fails with invalid memory address or nil pointer dereference | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Peter Larsen <plarsen> |
| Component: | Installer | Assignee: | Roberto Ciatti <rciatti> |
| Installer sub component: | OpenShift on RHV | QA Contact: | Lucie Leistnerova <lleistne> |
| Status: | CLOSED DUPLICATE | Docs Contact: | |
| Severity: | medium | ||
| Priority: | medium | CC: | dougsland |
| Version: | 4.5 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.6.0 | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-07-24 00:45:03 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Peter Larsen
2020-07-17 15:31:33 UTC
Version: [ocp@bastion ovirt]$ openshift-install version openshift-install 4.5.1 built from commit b714ff8ee8845b86da2dafce8fa5630ef1806f3b release image quay.io/openshift-release-dev/ocp-release@sha256:a656048696e79a30f0536f5acd5a1e8ec5ae331d4c7d21ca62bc8de412c79dc4 Running with debug: [ocp@bastion ovirt]$ openshift-install create cluster --dir=d1 --log-level=debug DEBUG OpenShift Installer 4.5.1 DEBUG Built from commit b714ff8ee8845b86da2dafce8fa5630ef1806f3b DEBUG Fetching Metadata... DEBUG Loading Metadata... DEBUG Loading Cluster ID... DEBUG Loading Install Config... DEBUG Loading SSH Key... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Cluster Name... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Pull Secret... DEBUG Loading Platform... DEBUG Loading Install Config... DEBUG Fetching Cluster ID... DEBUG Fetching Install Config... DEBUG Fetching SSH Key... DEBUG Generating SSH Key... ? SSH Public Key /home/ocp/.ssh/id_ose.pub DEBUG Fetching Base Domain... DEBUG Fetching Platform... DEBUG Generating Platform... ? Platform ovirt ? oVirt API endpoint URL rhev4m.peterlarsen.org ? Is the oVirt CA trusted locally? Yes ? oVirt certificate bundle -----BEGIN CERTIFICATE----- MIIDlzCCAn+gAwIBAgIBATANBgkqhkiG9w0BAQsFADA6MRgwFgYDVQQKDA9QRVRF UkxBUlNFTi5PUkcxHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0x ODA2MDkwMzEwNDJaFw0zODA2MDkwMzEwNDJaMDoxGDAWBgNVBAoMD1BFVEVSTEFS U0VOLk9SRzEeMBwGA1UEAwwVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MIIBIjANBgkq hkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArtyO18rbY4Yy4rzHMLzyNLVSNl0QVVim aRkTdLwIm4NLvJtJ2rLhcNyLnhN/3rWDuwxUwp6crm4Q1PZMK3qal4f2vV9xAtLl QGQs6MyMIJY/seu68TtHTCp35RYxYsH3aactZTj04FKF5n5rfAtFinKpsnYuRhcq u0xNmj4ktzrtqE86ukCYXTVE2/RsgqzsIr/M/ABD/c1NAj5F2EdpUuLc/WpXdPr/ 5duLfFVoWAKOLUtLlEo7b8s5GFmOgrCp88nVRS4YvZxHyCxtLzxJpk61Fcb+5zVU 3YA8hRQkzytpA0b1plKfxWAZjhFQxxF0VXgmqe5xegcpttO1qcyKWQIDAQABo4Gn MIGkMB8GA1UdIwQYMBaAFC4o1DKi0y7J1S5nxUCNH/LFOWqmMA8GA1UdEwEB/wQF MAMBAf8wDgYDVR0PAQH/BAQDAgHGMB0GA1UdDgQWBBQuKNQyotMuydUuZ8VAjR/y xTlqpjBBBggrBgEFBQcBAQQ1MDMwMQYIKwYBBQUHMAGGJWh0dHA6Ly9pcGEtY2Eu cGV0ZXJsYXJzZW4ub3JnL2NhL29jc3AwDQYJKoZIhvcNAQELBQADggEBAEfgVyYS JPhu7fWky/7Q9toUV4eh81UhpPhzd9a0qtZxGOkhkuB940UKYSYIDlkPtUfvCGh5 YnohG+VC3CkuV81Fiknsph3NfBntH/OWqz1+21/xC3pN24IvaXb6MFMsgPUi87eq JReq/mItPS0MTEuewnOzAl2BqfXM2EnnjkNt9eHo+q1T9ZXFyhFwLmeLFKYWIlQo MGi+vWWUjWDVHjwNK3NVoOzuSQ1YBoesnQW+dD8KDwyBDlTnLN7Za9XW0Tplir6u HivHpeyhphDvnjf7P/2BjJJru/qusEgWVdmdy8wF1BKDgpaxmZz+qrVZ8LRw0CK5 k4hlnMJ4zmYxhb8= -----END CERTIFICATE----- ? oVirt engine username ocp ? oVirt engine password ********* panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x1fc5cab] goroutine 1 [running]: github.com/ovirt/go-ovirt.(*Connection).testToken(0xc000582300, 0x0, 0x0, 0x0) (see description for full stack dump). Hi Peter,
I believe I have reproduced the report in this way:
$ trust list | grep -i engine
label: engine.medogz.home.52705 <------------ Engine CA is trusted
$ sudo su -
# mv /etc/pki/ca-trust/source/anchors/ca.pem ~/my-backup-dir
# update-ca-list
# exit
$ trust list | grep -i engine
$ <-------------- Cert is not trusted locally anymore
Run the installer again
--------------------------------------------
$ ./openshift-install create cluster --dir=ocp
? Platform ovirt
? oVirt API endpoint URL https://engine.medogz.home/ovirt-engine/api
? Is the oVirt CA trusted locally? Yes
? oVirt certificate bundle
-----BEGIN CERTIFICATE-----
MIICxDCCAoKgAwIBAgIETpxe9zALBgcqhkjOOAQDBQAwRDELMAkGA1UEBhMCVVMxDjAMBgNVBAoT
BWFwcDAxMQ0wCwYDVQQLEwRhcHBzMRYwFAYDVQQDDA1lYnNnb2xkX2FwcDAxMCAXDTExMTAxNzE2
NTkzNVoYDzIwNTExMDA3MTY1OTM1WjBEMQswCQYDVQQGEwJVUzEOMAwGA1UEChMFYXBwMDExDTAL
BgNVBAsTBGFwcHMxFjAUBgNVBAMMDWVic2dvbGRfYXBwMDEwqKYVDwT7g/bTxR7DAjVUE1oWkTL2
dfOuK2HXKu/yIgMZndFIAccCFQCXYFCPFSMLzLKSuYKi64QL8Fgc9QKBgQD34aCF1ps93su8q1w2
uFe5eZSvu/o66oL5V0wLPQeCZ1FZV4661FlP5nEHEIGAtEkWcSPoTCgWE7fPCTKMyKbhPBZ6i1R8
jSjgo64eK7OmdZFuo38L+iE1YvH7YnoBJDvMpPG+qFGQiaiD3+Fa5Z8GkotmXoB7VSVkAUw7/s9J
KgOBhQACgYEAz6MmM2ZIqg2KIta1LGq8rUJloM4h+j0rYyAe03Yc3prYrOtw1+e5ZhBgaKB3sgax
YXW4qS6nCi3Q0N6/mkPyy1Osr0ZbWuas/lJEsObVwHtGze9xY3dUKPA3vLcveo2xQgQKCuR474fq
QlHQRts5ij3sLKgDiKh1K7gT4CqWcKkwCwYHKoZIzjgEAwUAAy8AMCwCFHVj6UN0qT4hu2wHkiBs
mynaWQG3AhQ9JNpqp97p5+wSV3Bb/KeuraqI/g==
-----END CERTIFICATE-----
? oVirt engine username admin@internal
? oVirt engine password ************
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x1fc5cbb]
goroutine 1 [running]:
github.com/ovirt/go-ovirt.(*Connection).testToken(0xc00058a240, 0x0, 0x0, 0x0)
/go/src/github.com/openshift/installer/vendor/github.com/ovirt/go-ovirt/connection.go:97 +0x23b
github.com/ovirt/go-ovirt.(*Connection).Test(0xc00058a240, 0xc00058a240, 0x0)
/go/src/github.com/openshift/installer/vendor/github.com/ovirt/go-ovirt/connection.go:76 +0x2f
github.com/openshift/installer/pkg/asset/installconfig/ovirt.authenticated.func1(0xa7ab280, 0xc00080fd80, 0x0, 0x0)
/go/src/github.com/openshift/installer/pkg/asset/installconfig/ovirt/validaton.go:98 +0x274
gopkg.in/AlecAivazis/survey%2ev1.ComposeValidators.func1(0xa7ab280, 0xc00080fd80, 0xc00080fd80, 0x0)
/go/src/github.com/openshift/installer/vendor/gopkg.in/AlecAivazis/survey.v1/validate.go:68 +0x73
gopkg.in/AlecAivazis/survey%2ev1.Ask(0xc000b4ef60, 0x1, 0x1, 0xa57c620, 0xc000132260, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/openshift/installer/vendor/gopkg.in/AlecAivazis/survey.v1/survey.go:157 +0x2d2
github.com/openshift/installer/pkg/asset/installconfig/ovirt.askCredentials(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/openshift/installer/pkg/asset/installconfig/ovirt/credentials.go:78 +0x825
github.com/openshift/installer/pkg/asset/installconfig/ovirt.Platform(0x0, 0x0, 0x0)
/go/src/github.com/openshift/installer/pkg/asset/installconfig/ovirt/ovirt.go:16 +0xc07
github.com/openshift/installer/pkg/asset/installconfig.(*platform).Generate(0xc000606000, 0xc0007e6540, 0xc906b47, 0x12)
/go/src/github.com/openshift/installer/pkg/asset/installconfig/platform.go:83 +0x3a3
github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000a8a1e0, 0xde884e0, 0xc000606000, 0xc00076e470, 0x8, 0xc00076e470, 0x8)
/go/src/github.com/openshift/installer/pkg/asset/store/store.go:227 +0x7bf
github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000a8a1e0, 0xde88460, 0xc000aad080, 0xc00076e438, 0x6, 0xc00076e438, 0x6)
/go/src/github.com/openshift/installer/pkg/asset/store/store.go:221 +0x613
github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000a8a1e0, 0xde88360, 0xc000a72a20, 0xc000a9c3b0, 0x4, 0xc000a9c3b0, 0x4)
/go/src/github.com/openshift/installer/pkg/asset/store/store.go:221 +0x613
github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000a8a1e0, 0xde88320, 0xc000a72960, 0xc8b0257, 0x2, 0xc8b0257, 0x2)
/go/src/github.com/openshift/installer/pkg/asset/store/store.go:221 +0x613
github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000a8a1e0, 0x7f35952b3ac0, 0x1556c058, 0x0, 0x0, 0x40a0d5, 0xb488e00)
/go/src/github.com/openshift/installer/pkg/asset/store/store.go:221 +0x613
github.com/openshift/installer/pkg/asset/store.(*storeImpl).Fetch(0xc000a8a1e0, 0x7f35952b3ac0, 0x1556c058, 0x1553ebc0, 0x6, 0x6, 0xb600000000000000, 0xed6a56c79)
/go/src/github.com/openshift/installer/pkg/asset/store/store.go:77 +0x4b
main.runTargetCmd.func1(0x7ffe5fdfbace, 0x3, 0xc000a721c0, 0x0)
/go/src/github.com/openshift/installer/cmd/openshift-install/create.go:169 +0x133
main.runTargetCmd.func2(0x15546620, 0xc000b94100, 0x0, 0x2)
/go/src/github.com/openshift/installer/cmd/openshift-install/create.go:196 +0xb5
github.com/spf13/cobra.(*Command).execute(0x15546620, 0xc000b940c0, 0x2, 0x2, 0x15546620, 0xc000b940c0)
/go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:830 +0x2aa
github.com/spf13/cobra.(*Command).ExecuteC(0xc000b6e000, 0xc000b4fe30, 0x1, 0x1)
/go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
/go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:864
main.installerMain()
/go/src/github.com/openshift/installer/cmd/openshift-install/main.go:63 +0x301
main.main()
/go/src/github.com/openshift/installer/cmd/openshift-install/main.go:43 +0xc6
$
The error is gone if the CA is trusted locally. Copying the CA file to /etc/pki/ca-trust/source/anchors/ and run the command update-ca-trust.
Could you please confirm if that's the error you are seeing?
Just for the record: In 4.6 (master) we are in better shape: $ trust list | grep -i engine $ <-------------- Cert is not trusted locally anymore $ ./bin/openshift-install create cluster --log-level=debug --dir=ocp DEBUG OpenShift Installer unreleased-master-3336-g7c61e0c76bf7811f4f72ee9cdde7f3a8d8f7933b DEBUG Built from commit 7c61e0c76bf7811f4f72ee9cdde7f3a8d8f7933b DEBUG Fetching Metadata... DEBUG Loading Metadata... DEBUG Loading Cluster ID... DEBUG Loading Install Config... DEBUG Loading SSH Key... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Cluster Name... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Pull Secret... DEBUG Loading Platform... DEBUG Loading Install Config... DEBUG Fetching Cluster ID... DEBUG Fetching Install Config... DEBUG Fetching SSH Key... DEBUG Generating SSH Key... ? SSH Public Key /home/douglas/.ssh/id_rsa.pub DEBUG Fetching Base Domain... DEBUG Fetching Platform... DEBUG Generating Platform... ? Platform ovirt ? Engine FQDN[:PORT] engine.medogz.home DEBUG engine FQDN: engine.medogz.home DEBUG Engine URL: https://engine.medogz.home/ovirt-engine/api DEBUG checking URL response... urlAddr: https://engine.medogz.home/ovirt-engine/api skipVerify: true DEBUG PEM URL: https://engine.medogz.home/ovirt-engine/services/pki-resource?resource=ca-certificate&format=X509-PEM-CA DEBUG loaded /tmp/engine-412386034 into the system pool: DEBUG engine PEM temporary stored: /tmp/engine-412386034 DEBUG login attempts available: 3 ? Engine username admin@internal ? Engine password [Press Ctrl+C to switch username, ? for help] ****** X Sorry, your reply was invalid: failed to connect to Engine platform Post "https://engine.medogz.home/ovirt-engine/sso/oauth/token": x509: certificate signed by unknown authority (In reply to Douglas Schilling Landgraf from comment #3) > > The error is gone if the CA is trusted locally. Copying the CA file to > /etc/pki/ca-trust/source/anchors/ and run the command update-ca-trust. > Could you please confirm if that's the error you are seeing? The system I use is already trusting the CA locally. So I don't recognize this statement as true. I.e I can use "curl" against the management API and it will not complain about the cert. Process to reproduce: 1) Follow https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html/administration_guide/appe-red_hat_enterprise_virtualization_and_ssl#Replacing_the_Manager_CA_Certificate or add cert on install. 2) The CA used needs to be added to the node used running openshift-install (using freeipa just add the host to IPA). 3) Verify by using "curl https://<rhv-host.example.com/" and see that the cert is trusted 4) Run the openshift-install without a $HOME/.ovirt directory a) When asked, note it's a trusted cert b) When asked, paste the CA public cert in /etc/ipa/ca.crt 5) Watch the installer fail/crash Hi Peter, In comment: (In reply to Peter Larsen from comment #1) > Version: > [ocp@bastion ovirt]$ openshift-install version > openshift-install 4.5.1 > built from commit b714ff8ee8845b86da2dafce8fa5630ef1806f3b > release image > quay.io/openshift-release-dev/ocp-release@sha256: > a656048696e79a30f0536f5acd5a1e8ec5ae331d4c7d21ca62bc8de412c79dc4 > > > Running with debug: > [ocp@bastion ovirt]$ openshift-install create cluster --dir=d1 > --log-level=debug > DEBUG OpenShift Installer 4.5.1 > DEBUG Built from commit b714ff8ee8845b86da2dafce8fa5630ef1806f3b > DEBUG Fetching Metadata... > DEBUG Loading Metadata... > DEBUG Loading Cluster ID... > DEBUG Loading Install Config... > DEBUG Loading SSH Key... > DEBUG Loading Base Domain... > DEBUG Loading Platform... > DEBUG Loading Cluster Name... > DEBUG Loading Base Domain... > DEBUG Loading Platform... > DEBUG Loading Pull Secret... > DEBUG Loading Platform... > DEBUG Loading Install Config... > DEBUG Fetching Cluster ID... > DEBUG Fetching Install Config... > DEBUG Fetching SSH Key... > DEBUG Generating SSH Key... > ? SSH Public Key /home/ocp/.ssh/id_ose.pub > DEBUG Fetching Base Domain... > DEBUG Fetching Platform... > DEBUG Generating Platform... > ? Platform ovirt > ? oVirt API endpoint URL rhev4m.peterlarsen.org ^^^ here the problem. In 4.5 this should be https://rhev4m.peterlarsen.org/ovirt-engine/api Could you please confirm that's the issue you are facing? Moving to Roberto because he has been working in a similar bug about users using FQDN in the full url field. (In reply to Douglas Schilling Landgraf from comment #9) > > ? Platform ovirt > > ? oVirt API endpoint URL rhev4m.peterlarsen.org > > > ^^^ here the problem. In 4.5 this should be > https://rhev4m.peterlarsen.org/ovirt-engine/api > > Could you please confirm that's the issue you are facing? Entering a full URL indeed avoids the nullpointer failure. I checked the help message, and it does list it as you do. *** This bug has been marked as a duplicate of bug 1859874 *** |