Bug 1515500
| Summary: | ironic containers are restarting. ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found Reason: No conductor service registered which supports driver pxe_ipmitool. | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | mlammon |
| Component: | openstack-containers | Assignee: | Dan Prince <dprince> |
| Status: | CLOSED ERRATA | QA Contact: | Artem Hrechanychenko <ahrechan> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 12.0 (Pike) | CC: | bfournie, derekh, dprince, jschluet, lruzicka, ltoscano, m.andre, mburns, mcornea, racedoro, rhallise, rhel-osp-director-maint, sasha, srevivo, tvignaud |
| Target Milestone: | rc | Keywords: | Triaged |
| Target Release: | 12.0 (Pike) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-ironic-pxe-12.0-20171122.1 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-12-13 19:16:19 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
The restarting containers are: ironic_pxe_http ironic_pxe_tftp ironic_conductor ########################################## ironic_pxe_http INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found ########################################## ironic_pxe_tftp INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found ########################################## ironic_conductor INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found Did some investigation and it looks like our "docker-puppet.py" config files fail to run cleanly for Ironic and Ironic PXE services. The Ironic API service however runs just fine and seems to startup correctly. Was able to debug this on the CLI by connecting to an OSP 12 overcloud controller and doing the following:
1) cd /var/lib/docker-puppet
2) create a temp file to configure just the Ironic containers in question. I did this by slicing up the /var/lib/docker-puppet/docker-puppet.json with jq. So this:
[overcloud-controller-0 docker-puppet]# cat ironic.json
[
{
"config_image": "172.19.0.3:8787/rhosp12/openstack-ironic-pxe-docker:20171116.1",
"step_config": "include ::tripleo::profile::base::ironic::conductor\n\ninclude ::tripleo::profile::base::database::mysql::client",
"config_volume": "ironic",
"puppet_tags": "ironic_config"
},
{
"config_image": "172.19.0.3:8787/rhosp12/openstack-ironic-pxe-docker:20171116.1",
"step_config": "",
"config_volume": "ironic",
"puppet_tags": "ironic_config"
}
]
3) finally I execute docker puppet manually like this:
NET_HOST=true CONFIG=ironic.json python ./docker-puppet.py
It outputs logs directly to the console and I can see that we appear to be missing a few files:
Error: /Stage[main]/Ironic::Pxe/Ironic::Pxe::Tftpboot_file[pxelinux.0]/File[/var/lib/ironic/tftpboot/pxelinux.0]: Could not evaluate: Could not retrieve information from environment production source(s) file:/usr/share/syslinux/pxelinux.0
Error: /Stage[main]/Ironic::Pxe/Ironic::Pxe::Tftpboot_file[chain.c32]/File[/var/lib/ironic/tftpboot/chain.c32]: Could not evaluate: Could not retrieve information from environment production source(s) file:/usr/share/syslinux/chain.c32
----
My guess is we need to bind mount in a couple of these directories into the config containers so that puppet can run correctly to generate the needed files.
The files in question are part of the syslinux package. I think the issue here is that we are simply missing these packages as part of the Ironic conductor/PXE containers. That would mean we are missing a Requires entry on an RPM or a package in kolla install list Okay. Built an ad-hoc docker container on to install two addition RPMs (mtools is a required dependency of syslinux). I manually grabbed the RPMs from the repo rather as it was quicker here. [stack@undercloud-0 ~]$ cat ironic-fix/Dockerfile FROM 172.19.0.3:8787/rhosp12/openstack-ironic-pxe-docker:20171116.1 USER root COPY syslinux-4.05-13.el7.x86_64.rpm /root COPY mtools-4.0.18-5.el7.x86_64.rpm /root RUN rpm -ivh /root/*.rpm && rm /root/*.rpm USER ironic ---- Did a docker build, pushed the container to the registry and retried the above test I performed with a new 172.19.0.3:8787/rhosp12/openstack-ironic-pxe-docker:20171116.2 (note the .2 here) image. The docker-puppet.py script ran successfully to generate the required config files. My recommendation is that we add the syslinux package to both the conductor and PXE containers as they are somewhat coupled as far as the puppet modules go. Perhaps we can tease these apart more cleanly in the future. Ended up just added the files to the Ironic PXE container as that is all we need I think. FYI, the puppet errors during the config generation should have been caught earlier and should have failed the deployment. This is being tracked in https://bugzilla.redhat.com/show_bug.cgi?id=1516204. requested changes made for container images and will be part of RC Image Respins I've manually tested the latest re-spin and successfully used ironic in the overcloud without any workarounds. This has been verified with ironic nodes deployed in overcloud. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3457 |
Description ironic containers are restarting. ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found Reason: No conductor service registered which supports driver pxe_ipmitool. Environment: openstack-ironic-conductor-9.1.3-0.20171106091155.fceb40f.el7ost.noarch puppet-ironic-11.3.1-0.20171030020204.66bf743.el7ost.noarch openstack-ironic-inspector-6.0.1-0.20171107185219.bad2e1c.el7ost.noarch python-ironicclient-1.17.0-1.el7ost.noarch python-ironic-lib-2.10.0-1.el7ost.noarch openstack-ironic-api-9.1.3-0.20171106091155.fceb40f.el7ost.noarch python-ironic-inspector-client-2.1.0-1.el7ost.noarch openstack-ironic-common-9.1.3-0.20171106091155.fceb40f.el7ost.noarch instack-undercloud-7.4.3-2.el7ost.noarch instack-7.0.1-1.el7ost.noarch Steps, 1. Installed OSP 12 latest with Ironic enabled in OC 2. Attempt to add baremetal nodes fails source /home/stack/overcloudrc openstack baremetal create overcloud-nodes.yaml Error from executing 'openstack baremetal create overcloud-nodes.yaml' 21:55:24 During resources creation, the following error(s) were encountered: 21:55:24 Unable to create the node with the specified attributes: {'ports': [{'address': '52:54:00:b8:9f:8f'}], 'driver': 'pxe_ipmitool', 'properties': {'memory_mb': '4096', 'cpu_arch': 'x86_64', 'local_gb': '40', 'cpus': '4'}, 'driver_info': {'ipmi_port': '6235', 'ipmi_address': '172.16.0.2', 'ipmi_username': 'admin', 'ipmi_password': 'password'}, 'name': 'ironic-0'}. The error is: No valid host was found. Reason: No conductor service registered which supports driver pxe_ipmitool. (HTTP 400) 21:55:24 Unable to create the node with the specified attributes: {'ports': [{'address': '52:54:00:c3:7e:3b'}], 'driver': 'pxe_ipmitool', 'properties': {'memory_mb': '4096', 'cpu_arch': 'x86_64', 'local_gb': '40', 'cpus': '4'}, 'driver_info': {'ipmi_port': '6234', 'ipmi_address': '172.16.0.2', 'ipmi_username': 'admin', 'ipmi_password': 'password'}, 'name': 'ironic-1'}. The error is: No valid host was found. Reason: No conductor service registered which supports driver pxe_ipmitool. (HTTP 400) Errors from controller node heat-admin@overcloud-controller-2 ~]$ sudo -i [root@overcloud-controller-2 ~]# docker ps|grep ironic c4963ae6c397 192.168.24.1:8787/rhosp12/openstack-nova-compute-ironic-docker:20171116.1 "kolla_start" 30 minutes ago Up 30 minutes (healthy) nova_compute fc5c160c2559 192.168.24.1:8787/rhosp12/openstack-ironic-pxe-docker:20171116.1 "kolla_start" 32 minutes ago Restarting (1) 5 minutes ago ironic_pxe_http 6f4ed69d8187 192.168.24.1:8787/rhosp12/openstack-ironic-pxe-docker:20171116.1 "kolla_start" 32 minutes ago Restarting (1) 5 minutes ago ironic_pxe_tftp 3d8593c0ecd4 192.168.24.1:8787/rhosp12/openstack-ironic-conductor-docker:20171116.1 "kolla_start" 32 minutes ago Restarting (1) 5 minutes ago ironic_conductor dea8bfa2eb0c 192.168.24.1:8787/rhosp12/openstack-ironic-api-docker:20171116.1 "kolla_start" 32 minutes ago Up 32 minutes ironic_api [root@overcloud-controller-2 ~]# docker exec -it ironic_conductor bash Error response from daemon: Container 3d8593c0ecd4d34b1556a4aaf7065b4260723cde83d3342fb22f7d12694e8a7b is restarting, wait until the container is running [root@overcloud-controller-2 ~]# docker exec -it iron^C_conductor bash [root@overcloud-controller-2 ~]# docker logs ironic_conductor INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/* file is not found