Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1807871

Summary: Hashed passwords for pxe_append_params not correctly read by oslo's config parser
Product: Red Hat OpenStack Reporter: Andreas Karis <akaris>
Component: openstack-ironicAssignee: RHOS Maint <rhos-maint>
Status: CLOSED DUPLICATE QA Contact: Alistair Tonner <atonner>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 13.0 (Queens)CC: bfournie, dtantsur, mburns
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-02-27 15:07:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Andreas Karis 2020-02-27 12:10:56 UTC
Description of problem:
Hashed passwords for pxe_append_params not correctly read by oslo's config parser

This one's likely for oslo's config parser but I'd like to send it through review by the ironic team first.

It's clearly related to the special characters in the password hash and to the way how oslo parses the string. But given that I'm single quoting, I expect no string parsing to happen.

Working:
~~~
/var/lib/config-data/puppet-generated/ironic/etc/ironic/ironic.conf:pxe_append_params = 'rootpwd="$1$pdkTvRVW$5hN5HLCjRnHXx2O3RDVXi/"'
[root@overcloud-controller-1 ~]# crudini --set /var/lib/config-data/puppet-generated/ironic/etc/ironic/ironic.conf pxe pxe_append_params  "'"'rootpwd="abc"'"'"
[root@overcloud-controller-1 ~]# crudini --get /var/lib/config-data/puppet-generated/ironic/etc/ironic/ironic.conf pxe pxe_append_params
'rootpwd="abc"'
[root@overcloud-controller-1 ~]# grep pxe_append_params /var/lib/config-data/puppet-generated/ironic/etc/ironic/ironic.conf
#pxe_append_params = nofb nomodeset vga=normal
pxe_append_params = 'rootpwd="abc"'
(reverse-i-search)`rs': tail -f /var/log/containe^C/ironic/ironic-conductor.log
[root@overcloud-controller-1 ~]# docker ps | awk '/ironic/ {print $NF}' | xargs docker restart
nova_compute
ironic_pxe_http
ironic_pxe_tftp
ironic_neutron_agent
ironic_conductor
ironic_api
[root@overcloud-controller-1 ~]# docker ps | grep ironic
7ea0afcb7039        registry.access.redhat.com/rhosp13/openstack-nova-compute-ironic:13.0-116.1580118163         "dumb-init --singl..."   36 hours ago        Up 14 seconds (health: starting)                       nova_compute
3278f382a9a9        registry.access.redhat.com/rhosp13/openstack-ironic-pxe:13.0-90.1580118205                   "dumb-init --singl..."   36 hours ago        Up 14 seconds                                          ironic_pxe_http
49696d427e74        registry.access.redhat.com/rhosp13/openstack-ironic-pxe:13.0-90.1580118205                   "dumb-init --singl..."   36 hours ago        Up 13 seconds (health: starting)                       ironic_pxe_tftp
a010eeb54d54        registry.access.redhat.com/rhosp13/openstack-ironic-neutron-agent:latest                     "dumb-init --singl..."   36 hours ago        Up 13 seconds                                          ironic_neutron_agent
0531578b3b57        registry.access.redhat.com/rhosp13/openstack-ironic-conductor:13.0-99.1580118204             "dumb-init --singl..."   36 hours ago        Up 13 seconds (health: starting)                       ironic_conductor
6bbf7eb02c7f        registry.access.redhat.com/rhosp13/openstack-ironic-api:13.0-98.1580118206                   "dumb-init --singl..."   36 hours ago        Up 11 seconds (health: starting)                       ironic_api
[root@overcloud-controller-1 ~]# tail -f /var/log/containers/ironic/ironic-conductor.log
2020-02-27 12:06:25.098 9 INFO ironic.common.driver_factory [req-eddcd604-d945-4eda-b63a-032d13021ea8 - - - - -] Loaded the following console interfaces: ['ipmitool-socat', 'no-console']
2020-02-27 12:06:25.099 9 INFO ironic.common.driver_factory [req-eddcd604-d945-4eda-b63a-032d13021ea8 - - - - -] Loaded the following power interfaces: ['ipmitool', 'redfish']
2020-02-27 12:06:25.101 9 INFO ironic.common.driver_factory [req-eddcd604-d945-4eda-b63a-032d13021ea8 - - - - -] Loaded the following deploy interfaces: ['iscsi', 'direct']
2020-02-27 12:06:25.104 9 INFO ironic.common.driver_factory [req-eddcd604-d945-4eda-b63a-032d13021ea8 - - - - -] Loaded the following raid interfaces: ['no-raid', 'agent']
2020-02-27 12:06:25.109 9 INFO ironic.common.driver_factory [req-eddcd604-d945-4eda-b63a-032d13021ea8 - - - - -] Loaded the following boot interfaces: ['pxe']
2020-02-27 12:06:25.110 9 INFO ironic.common.driver_factory [req-eddcd604-d945-4eda-b63a-032d13021ea8 - - - - -] Loaded the following inspect interfaces: ['no-inspect']
2020-02-27 12:06:25.112 9 INFO ironic.common.driver_factory [req-eddcd604-d945-4eda-b63a-032d13021ea8 - - - - -] Loaded the following vendor interfaces: ['ipmitool', 'no-vendor']
2020-02-27 12:06:25.113 9 INFO ironic.common.driver_factory [req-eddcd604-d945-4eda-b63a-032d13021ea8 - - - - -] Loaded the following rescue interfaces: ['no-rescue']
2020-02-27 12:06:25.543 9 INFO ironic.conductor.base_manager [req-eddcd604-d945-4eda-b63a-032d13021ea8 - - - - -] Successfully started conductor with hostname overcloud-controller-1.localdomain.
2020-02-27 12:06:25.544 9 INFO ironic.common.rpc_service [req-eddcd604-d945-4eda-b63a-032d13021ea8 - - - - -] Created RPC server for service ironic.conductor_manager on host overcloud-controller-1.localdomain.
^C
[root@overcloud-controller-1 ~]# docker ps | grep ironic
7ea0afcb7039        registry.access.redhat.com/rhosp13/openstack-nova-compute-ironic:13.0-116.1580118163         "dumb-init --singl..."   36 hours ago        Up 25 seconds (health: starting)                       nova_compute
3278f382a9a9        registry.access.redhat.com/rhosp13/openstack-ironic-pxe:13.0-90.1580118205                   "dumb-init --singl..."   36 hours ago        Up 25 seconds                                          ironic_pxe_http
49696d427e74        registry.access.redhat.com/rhosp13/openstack-ironic-pxe:13.0-90.1580118205                   "dumb-init --singl..."   36 hours ago        Up 24 seconds (health: starting)                       ironic_pxe_tftp
a010eeb54d54        registry.access.redhat.com/rhosp13/openstack-ironic-neutron-agent:latest                     "dumb-init --singl..."   36 hours ago        Up 24 seconds                                          ironic_neutron_agent
0531578b3b57        registry.access.redhat.com/rhosp13/openstack-ironic-conductor:13.0-99.1580118204             "dumb-init --singl..."   36 hours ago        Up 24 seconds (health: starting)                       ironic_conductor
6bbf7eb02c7f        registry.access.redhat.com/rhosp13/openstack-ironic-api:13.0-98.1580118206                   "dumb-init --singl..."   36 hours ago        Up 22 seconds (health: starting)                       ironic_api
[root@overcloud-controller-1 ~]# ^C
[root@overcloud-controller-1 ~]# tail -f /var/log/containers/ironic/ironic-conductor.log
2020-02-27 12:06:25.098 9 INFO ironic.common.driver_factory [req-eddcd604-d945-4eda-b63a-032d13021ea8 - - - - -] Loaded the following console interfaces: ['ipmitool-socat', 'no-console']
2020-02-27 12:06:25.099 9 INFO ironic.common.driver_factory [req-eddcd604-d945-4eda-b63a-032d13021ea8 - - - - -] Loaded the following power interfaces: ['ipmitool', 'redfish']
2020-02-27 12:06:25.101 9 INFO ironic.common.driver_factory [req-eddcd604-d945-4eda-b63a-032d13021ea8 - - - - -] Loaded the following deploy interfaces: ['iscsi', 'direct']
2020-02-27 12:06:25.104 9 INFO ironic.common.driver_factory [req-eddcd604-d945-4eda-b63a-032d13021ea8 - - - - -] Loaded the following raid interfaces: ['no-raid', 'agent']
2020-02-27 12:06:25.109 9 INFO ironic.common.driver_factory [req-eddcd604-d945-4eda-b63a-032d13021ea8 - - - - -] Loaded the following boot interfaces: ['pxe']
2020-02-27 12:06:25.110 9 INFO ironic.common.driver_factory [req-eddcd604-d945-4eda-b63a-032d13021ea8 - - - - -] Loaded the following inspect interfaces: ['no-inspect']
2020-02-27 12:06:25.112 9 INFO ironic.common.driver_factory [req-eddcd604-d945-4eda-b63a-032d13021ea8 - - - - -] Loaded the following vendor interfaces: ['ipmitool', 'no-vendor']
2020-02-27 12:06:25.113 9 INFO ironic.common.driver_factory [req-eddcd604-d945-4eda-b63a-032d13021ea8 - - - - -] Loaded the following rescue interfaces: ['no-rescue']
2020-02-27 12:06:25.543 9 INFO ironic.conductor.base_manager [req-eddcd604-d945-4eda-b63a-032d13021ea8 - - - - -] Successfully started conductor with hostname overcloud-controller-1.localdomain.
2020-02-27 12:06:25.544 9 INFO ironic.common.rpc_service [req-eddcd604-d945-4eda-b63a-032d13021ea8 - - - - -] Created RPC server for service ironic.conductor_manager on host overcloud-controller-1.localdomain.
^C
[root@overcloud-controller-1 ~]# docker ps | grep ironic
7ea0afcb7039        registry.access.redhat.com/rhosp13/openstack-nova-compute-ironic:13.0-116.1580118163         "dumb-init --singl..."   36 hours ago        Up 39 seconds (healthy)                       nova_compute
3278f382a9a9        registry.access.redhat.com/rhosp13/openstack-ironic-pxe:13.0-90.1580118205                   "dumb-init --singl..."   36 hours ago        Up 39 seconds                                 ironic_pxe_http
49696d427e74        registry.access.redhat.com/rhosp13/openstack-ironic-pxe:13.0-90.1580118205                   "dumb-init --singl..."   36 hours ago        Up 38 seconds (healthy)                       ironic_pxe_tftp
a010eeb54d54        registry.access.redhat.com/rhosp13/openstack-ironic-neutron-agent:latest                     "dumb-init --singl..."   36 hours ago        Up 38 seconds                                 ironic_neutron_agent
0531578b3b57        registry.access.redhat.com/rhosp13/openstack-ironic-conductor:13.0-99.1580118204             "dumb-init --singl..."   36 hours ago        Up 37 seconds (healthy)                       ironic_conductor
6bbf7eb02c7f        registry.access.redhat.com/rhosp13/openstack-ironic-api:13.0-98.1580118206                   "dumb-init --singl..."   36 hours ago        Up 35 seconds (healthy)                       ironic_api
[root@overcloud-controller-1 ~]# 
~~~

Not working:
~~~
[root@overcloud-controller-1 ~]# crudini --set /var/lib/config-data/puppet-generated/ironic/etc/ironic/ironic.conf pxe pxe_append_params  "'"'rootpwd="$1$pdkTvRVW$5hN5HLCjRnHXx2O3RDVXi/"'"'"
[root@overcloud-controller-1 ~]# crudini --get /var/lib/config-data/puppet-generated/ironic/etc/ironic/ironic.conf pxe pxe_append_params
'rootpwd="$1$pdkTvRVW$5hN5HLCjRnHXx2O3RDVXi/"'
[root@overcloud-controller-1 ~]# grep pxe_append_params /var/lib/config-data/puppet-generated/ironic/etc/ironic/ironic.conf
#pxe_append_params = nofb nomodeset vga=normal
pxe_append_params = 'rootpwd="$1$pdkTvRVW$5hN5HLCjRnHXx2O3RDVXi/"'
[root@overcloud-controller-1 ~]# docker ps | awk '/ironic/ {print $NF}' | xargs docker restart
nova_compute
ironic_pxe_http
ironic_pxe_tftp
ironic_neutron_agent
ironic_conductor
ironic_api
[root@overcloud-controller-1 ~]# tail -f /var/log/containers/ironic/ironic-conductor.log
2020-02-27 12:09:42.122 9 ERROR ironic   File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 3471, in __getitem__
2020-02-27 12:09:42.122 9 ERROR ironic     value = self.conf._get(key, namespace=self.namespace)
2020-02-27 12:09:42.122 9 ERROR ironic   File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2944, in _get
2020-02-27 12:09:42.122 9 ERROR ironic     value = self._do_get(name, group, namespace)
2020-02-27 12:09:42.122 9 ERROR ironic   File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2961, in _do_get
2020-02-27 12:09:42.122 9 ERROR ironic     info = self._get_opt_info(name, group)
2020-02-27 12:09:42.122 9 ERROR ironic   File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 3118, in _get_opt_info
2020-02-27 12:09:42.122 9 ERROR ironic     raise NoSuchOptError(opt_name, group)
2020-02-27 12:09:42.122 9 ERROR ironic NoSuchOptError: no such option pdkTvRVW in group [DEFAULT]
2020-02-27 12:09:42.122 9 ERROR ironic 
^C
[root@overcloud-controller-1 ~]# 
~~~




Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Bob Fournier 2020-02-27 12:35:24 UTC
I think we've come across this before, see Dmitry's comment here https://bugzilla.redhat.com/show_bug.cgi?id=1700085#c4.

Can you try doubling the $ as Dmitry recommends?

Comment 2 Andreas Karis 2020-02-27 14:34:41 UTC
I just commented the $ with \$  and it worked.

Comment 3 Andreas Karis 2020-02-27 14:35:02 UTC
s/commented/escaped/

Comment 4 Andreas Karis 2020-02-27 14:36:02 UTC
I'm directly quoting from a WIP KCS that I'm writing:
----------------------------------------------------------


    Generate a password hash with: openssl passwd -1 redhat

Raw

[root@overcloud-controller-1 ~]# openssl passwd -1 redhat
$1$pdkTvRVW$5hN5HLCjRnHXx2O3RDVXi/

Note, due to https://bugzilla.redhat.com/show_bug.cgi?id=1807871, the string will need to be escaped properly:
Raw

\$1\$pdkTvRVW$5hN5HLCjRnHXx2O3RDVXi/

    In the [pxe] section of ironic.conf of all controllers, set the hash as value to pxe_append_params like this:

Raw

pxe_append_params = 'rootpwd="\$1\$pdkTvRVW$5hN5HLCjRnHXx2O3RDVXi/"'

    Restart ironic containers on the controller

Bringing it all together in 2 commands, run the following on all 3 controller nodes:
Raw

crudini --set /var/lib/config-data/puppet-generated/ironic/etc/ironic/ironic.conf pxe pxe_append_params  "'"'rootpwd="\$1\$pdkTvRVW$5hN5HLCjRnHXx2O3RDVXi/"'"'"
docker ps | awk '/ironic/ {print $NF}' | xargs docker restart

Verification - make sure that this is set correctly on each controller:
Raw

[root@overcloud-controller-1 ~]# crudini --get /var/lib/config-data/puppet-generated/ironic/etc/ironic/ironic.conf pxe pxe_append_params
'rootpwd="\$1\$pdkTvRVW$5hN5HLCjRnHXx2O3RDVXi/"'

Verification - make sure that during node cleaning and/or deployment, file /var/lib/ironic/httpboot/pxelinux.cfg/{MAC_ADDRESS/:/-} exists on at least one controller and that it contains the root password hash - now without the additional '\':
Raw

[root@overcloud-controller-1 ~]#  cat /var/lib/ironic/httpboot/pxelinux.cfg/18-66-da-9f-bb-0f
#!ipxe

goto deploy

:deploy
imgfree
kernel http://192.168.29.11:8088/bfe415d9-d1c3-48ec-88cd-3b2f89305ee3/deploy_kernel selinux=0 troubleshoot=0 text rootpwd="$1$pdkTvRVW$5hN5HLCjRnHXx2O3RDVXi/" ip=${ip}:${next-server}:${gateway}:${netmask} BOOTIF=${mac} ipa-api-url=http://192.168.29.10:6385 initrd=deploy_ramdisk coreos.configdrive=0 || goto deploy

initrd http://192.168.29.11:8088/bfe415d9-d1c3-48ec-88cd-3b2f89305ee3/deploy_ramdisk || goto deploy
boot

:boot_partition
imgfree
kernel no_kernel root={{ ROOT }} ro text rootpwd="$1$pdkTvRVW$5hN5HLCjRnHXx2O3RDVXi/" initrd=ramdisk || goto boot_partition
initrd no_ramdisk || goto boot_partition
boot

:boot_whole_disk

Once the node comes up, one should be able to log in with the following credentials during the cleaning stage:
User: root
Password: redhat

Comment 5 Andreas Karis 2020-02-27 14:37:25 UTC
So we should IMO ship this BZ to oslo as they have to fix it. Single quoted strings shouldn't be expanded / evaluated by oslo especially not if they are part of a value to a parameter in a config file

Comment 6 Andreas Karis 2020-02-27 15:07:30 UTC
Meh ... it's a feature, not a bue: https://docs.openstack.org/oslo.config/latest/reference/configuration-files.html#option-value-interpolation

https://bugzilla.redhat.com/show_bug.cgi?id=1700085#c4

Thanks for pointing that out to me.

*** This bug has been marked as a duplicate of bug 1700085 ***