Bug 1646584
| Summary: | Run puppet host configuration for step 1: Error: configuring ironic | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Filip Hubík <fhubik> | ||||||||
| Component: | puppet-ironic | Assignee: | Bob Fournier <bfournie> | ||||||||
| Status: | CLOSED ERRATA | QA Contact: | Filip Hubík <fhubik> | ||||||||
| Severity: | high | Docs Contact: | |||||||||
| Priority: | high | ||||||||||
| Version: | 14.0 (Rocky) | CC: | aschultz, asimonel, bfournie, dmacpher, fhubik, gcharot, hbrock, jjoyce, jschluet, jslagle, mburns, slinaber, tonyb, tvignaud, yselkowi | ||||||||
| Target Milestone: | beta | Keywords: | Automation, AutomationBlocker, Triaged, ZStream | ||||||||
| Target Release: | 14.0 (Rocky) | ||||||||||
| Hardware: | Unspecified | ||||||||||
| OS: | Unspecified | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | puppet-ironic-13.3.1-0.20181013115249.el7ost | Doc Type: | If docs needed, set a value | ||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2019-01-11 11:54:32 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Bug Depends On: | |||||||||||
| Bug Blocks: | 1588183 | ||||||||||
| Attachments: |
|
||||||||||
Created attachment 1502067 [details]
undercloud.conf
Created attachment 1502068 [details]
containers-prepare-parameter.yaml
Seeing this error in logs:
[\"/etc/puppet/modules/ironic/manifests/conductor.pp\", 223]:[\"/etc/puppet/modules/tripleo/manifests/profile/base/ironic/conductor.pp\", 41]",
"Error: /Stage[main]/Ironic::Pxe/Ironic::Pxe::Tftpboot_file[pxelinux.0]/File[/var/lib/ironic/tftpboot/pxelinux.0]: Could not evaluate: Could not retrieve information from environment production source(s) file:/usr/share/syslinux/pxelinux.0",
"Error: /Stage[main]/Ironic::Pxe/Ironic::Pxe::Tftpboot_file[chain.c32]/File[/var/lib/ironic/tftpboot/chain.c32]: Could not evaluate: Could not retrieve information from environment production source(s) file:/usr/share/syslinux/chain.c32",
Most likely related to fix here https://bugzilla.redhat.com/show_bug.cgi?id=1588183. Looking...
I can confirm that the failure is due to the puppet ironic::pxe error. In the logs, the following is the actual error message for the failure:
"2018-11-05 11:29:22,466 ERROR: 20741 -- ERROR configuring ironic",
From there, the items that Bob pointed out are what caused the configuring of ironic to fail.
Filip - can you provide more details on this topology? "1. Deploy OSP14 using InfraRed, topology 1:1:1:1" I'm able to install a UC using RHOS_RELEASE_ARGS="14 -r 7.6 -p 2018-11-04.1" ########################################################## The Undercloud has been successfully installed. Note in Comment 6, the puddle from 10-30 was actually being used because of a script issue which was why it passed at that time. This problem occurs when using puddle 2018-11-04.3 which uses openstack-ironic-pxe-container-14.0-70 and the fix for https://bugzilla.redhat.com/show_bug.cgi?id=1588183. *** Bug 1646117 has been marked as a duplicate of this bug. *** The cause is actually that puppet-ironic:manifests/params.pp hard-codes a requirement on syslinux-extlinux when $::osfamily is 'RedHat'. - $syslinux_package = 'syslinux-extlinux' - $syslinux_path = '/usr/share/syslinux' + $syslinux_package = 'syslinux-tftpboot' + $syslinux_path = '/var/lib/tftpboot' Note that: 1) In RHEL 7, the /usr/share/syslinux directory in question is actually in 'syslinux', not 'syslinux-extlinux'; 2) puppet-ironic does NOT have a Requires: syslinux or syslinux-extlinux. The only reason this worked in RHEL is that the ironic-pxe container pulled in syslinux as a requirement of extend_start.sh. The only reason this worked in RDO (which IIUC was using syslinux-tftpboot the whole time) is that syslinux-tftpboot used to Requires: syslinux, but that had to be dropped in order to make it noarch. Therefore, I believe the correct fix would be in puppet-ironic. Verified by CI, puppet-ironic-13.3.1-0.20181013115249 seems to be the fix we needed. *** Bug 1647001 has been marked as a duplicate of this bug. *** Greg,
Can we see which version of openstack-ironic is installed
> rpm -qa | grep ironic
That's the same error that this bug was originally created for but all testing has shown it be fixed.
Greg - note also that this puddle requires RHEL 7.6. I've verified the undercloud install worked fine using passed-phase2 sudo rhos-release -r 7.6 -p passed_phase2 Interesting I didn't use -r but i believe this should not be needed (i have rhel 7.6 repos enabled). I am indeed using RHEL 7.6: cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.6 (Maipo) RPMs: rpm -qa | grep ironic python2-ironicclient-2.5.0-0.20180810135843.fb94fb8.el7ost.noarch puppet-ironic-13.3.1-0.20181013115249.el7ost.noarch python2-ironic-inspector-client-3.3.0-0.20180810080932.53bf4e8.el7ost.noarch Images: sudo docker images | grep ironic 172.16.0.1:8787/rhosp14/openstack-ironic-conductor 14.0-76 490894bfaead 3 days ago 823 MB 172.16.0.1:8787/rhosp14/openstack-ironic-pxe 14.0-76 aa8f2a580eeb 3 days ago 766 MB 172.16.0.1:8787/rhosp14/openstack-nova-compute-ironic 14.0-81 96ec3273c957 3 days ago 1.36 GB 172.16.0.1:8787/rhosp14/openstack-ironic-api 14.0-76 f2273e22c27d 3 days ago 760 MB 172.16.0.1:8787/rhosp14/openstack-ironic-neutron-agent 14.0-46 34d7646ed508 3 days ago 794 MB 172.16.0.1:8787/rhosp14/openstack-ironic-inspector 14.0-82 964bdee25900 3 days ago 636 MB OK, not sure what's going on with your setup. Re-looking at the message in Comment 19 it does look the updated puppet-ironic is being used as its looking for file at /var/lib/ironic/tftpboot/pxelinux.0. Are you updating the undercloud after a previous install or is this a fresh installation? The docker images from 3 days ago do a look a little strange as passed_phase2 is the 2018-11-07.02 puddle (5 days ago). When I run that I get: stack@host06 ~]$ sudo docker images | grep ironic 192.168.0.1:8787/rhosp14/openstack-ironic-conductor 2018-11-06.1 f1bd143d3b8e 6 days ago 810 MB 192.168.0.1:8787/rhosp14/openstack-ironic-pxe 2018-11-06.1 d24535bd5d9c 6 days ago 752 MB 192.168.0.1:8787/rhosp14/openstack-nova-compute-ironic 2018-11-06.1 ed606589bdfa 6 days ago 1.35 GB 192.168.0.1:8787/rhosp14/openstack-ironic-api 2018-11-06.1 84a323f93a56 6 days ago 747 MB 192.168.0.1:8787/rhosp14/openstack-ironic-neutron-agent 2018-11-06.1 3921aad81be3 6 days ago 781 MB 192.168.0.1:8787/rhosp14/openstack-ironic-inspector 2018-11-06.1 4c23c1866456 6 days ago 623 MB It would be useful if we could get an sosreport. Thanks. I retried a deployment from scratch and got the same error. it looks like it is related to the container's image previous 14.0-75 docker image: docker run docker-registry.engineering.redhat.com/rhosp14/openstack-ironic-pxe:14.0-75 ls /var/lib/tftpboot/ cat.c32 chain.c32 cmd.c32 (...) 14.0-76 docker image: docker run docker-registry.engineering.redhat.com/rhosp14/openstack-ironic-pxe:14.0-76 ls /var/lib/tftpboot/ (nothing) I removed "tag: latest" from my containers-prepare-parameter.yaml and the deployment goes well, i end up with 1bf17999d89e 172.16.0.1:8787/rhosp14/openstack-ironic-pxe:14.0-77 "kolla_start" 9 minutes ago Up 9 minutes ironic_pxe_http 3fa2a5096691 172.16.0.1:8787/rhosp14/openstack-ironic-pxe:14.0-77 "/bin/bash -c 'BIN..." 9 minutes ago Up 9 minutes (healthy) ironic_pxe_tftp so version 14.0-77 works, 14.0-76 seems to be the issue, not sure how i ended up with this version. Thanks for the help Bob! Thanks for testing this Greg. It appears that a regression was introduced on an import in openstack-ironic-pxe:14.0-76. This was resolved in openstack-ironic-pxe:14.0-77. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:0045 |
Created attachment 1502054 [details] undercloud_install.log Description of problem: Undercloud deployment fails on errors: TASK [Debug output for task: Run puppet host configuration for step 1] ********* ok: [undercloud-0] => { "failed_when_result": false, ... config_image 192.168.24.1:8787/rhosp14/openstack-mariadb:2018-11-04.3", volumes []", check_mode 0", $ docker inspect --type container --format exists docker-puppet-nova_placement", $ docker inspect --type container --format exists docker-puppet-ironic_api", $ docker inspect --type container --format exists docker-puppet-mysql", ", ", Error: No such container: docker-puppet-nova_placement", Error: No such container: docker-puppet-mysql", Removing container: docker-puppet-mysql", Removing container: docker-puppet-nova_placement", ", Error: No such container: docker-puppet-ironic_api", Removing container: docker-puppet-ironic_api", Image already exists: 192.168.24.1:8787/rhosp14/openstack-ironic-api:2018-11-04.3", Image already exists: 192.168.24.1:8787/rhosp14/openstack-nova-placement-api:2018-11-04.3", ... # full logs attached Version-Release number of selected component (if applicable): OSP14, containerized UC, puddles 2018-11-04.3, 2018-11-02.1 How reproducible: always Steps to Reproduce: 1. Deploy OSP14 using InfraRed, topology 1:1:1:1 2. UC deployment fails