openstack-foreman: Failed to deploy 'controller' using foreman from grizzly on rhel6.4 (Invalid parameter token_driver at /opt/rh/ruby193/root/etc/puppet/environments/production/modules/openstack/manifests/keystone.pp:216) Environment: (RHEL 6.4 + Puddle 2013-11-01.1) --------------------------------------------- ruby193-rubygem-json-1.5.5-40.el6.x86_64 ruby193-ruby-1.9.3.448-40.el6.x86_64 ruby193-hiera-1.0.0-6.el6ost.noarch ruby-1.8.7.352-7.el6_2.x86_64 ruby-augeas-0.4.1-1.el6.x86_64 libselinux-ruby-2.0.94-5.3.el6.x86_64 ruby193-libyaml-0.1.4-4.el6.x86_64 ruby193-ruby-libs-1.9.3.448-40.el6.x86_64 ruby193-rubygem-bigdecimal-1.1.0-40.el6.x86_64 ruby193-rubygem-rdoc-3.12-12.el6ost.x86_64 ruby193-rubygems-1.8.24-9.el6ost.noarch ruby193-facter-1.6.18-5.el6ost.x86_64 ruby193-ruby-augeas-0.4.1-7.el6ost.x86_64 ruby193-puppet-3.1.1-11.1.el6ost.noarch ruby-libs-1.8.7.352-7.el6_2.x86_64 ruby-shadow-1.4.1-13.el6.x86_64 ruby193-runtime-1-10.el6.x86_64 ruby193-ruby-irb-1.9.3.448-40.el6.noarch ruby193-rubygem-io-console-0.3-40.el6.x86_64 ruby193-ruby-shadow-1.4.1-24.el6ost.x86_64 puppet-2.6.18-3.el6.noarch packstack-modules-puppet-2013.1.1-0.35.dev696.el6ost.noarch ruby193-puppet-3.1.1-11.1.el6ost.noarch ruby193-puppet-server-3.1.1-11.1.el6ost.noarch I'm having this problem when Installing controller using foreman - after puppet fails the host left with "E" Error (already reproduced few times) : Info: Loading facts in /opt/rh/ruby193/root/var/lib/puppet/lib/facter/puppet_vardir.rb Info: Loading facts in /opt/rh/ruby193/root/var/lib/puppet/lib/facter/pe_version.rb Info: Loading facts in /opt/rh/ruby193/root/var/lib/puppet/lib/facter/ip6tables_version.rb Info: Loading facts in /opt/rh/ruby193/root/var/lib/puppet/lib/facter/network.rb Info: Loading facts in /opt/rh/ruby193/root/var/lib/puppet/lib/facter/facter_dot_d.rb Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid parameter token_driver at /opt/rh/ruby193/root/etc/puppet/environments/production/modules/openstack/manifests/keystone.pp:216 on node puma01.scl.lab.tlv.redhat.com Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run From looking at this file on the server: vim +216 /opt/rh/ruby193/root/etc/puppet/environments/production/modules/openstack/manifests/keystone.pp class { '::keystone': verbose => $verbose, debug => $debug, bind_host => $bind_host, idle_timeout => $idle_timeout, catalog_type => 'sql', admin_token => $admin_token, token_driver => $token_driver, token_format => $token_format, enabled => $enabled, sql_connection => $sql_conn, use_syslog => $use_syslog, log_facility => $log_facility, } Note: the issue didn't occur on older puddle (2013-10-24.5)
I believe this error is coming from packstack-modules-puppet-2013.1.1-0.35.dev696.el6ost.noarch.rpm . The file in the stack trace above is copied by the foreman installer from /usr/share/packstack/modules/openstack/manifests/keystone.pp so it is really just an unaltered packstack-modules-puppet manifest. I'd be surprised if the packstack installer itself wasn't broken.
Looking at the grizzly puppet-keystone module there is no token_driver paramater (which is exactly what the error message saying): class keystone( $admin_token, $package_ensure = 'present', $bind_host = '0.0.0.0', $public_port = '5000', $admin_port = '35357', $compute_port = '8774', $verbose = false, $debug = false, $use_syslog = false, $catalog_type = 'sql', $token_format = 'PKI', $cache_dir = '/var/cache/keystone', $enabled = true, $sql_connection = 'sqlite:////var/lib/keystone/keystone.db', $idle_timeout = '200' ) ... So I guess you are using parameter which is not available in grizzly modules. BTW packstack is not broken since it is not using such parameter.
Also checked older modules (packstack-modules-puppet-2013.1.1-0.31.dev677.el6ost.noarch) and the parameter is not there, so I guess it wasn't available in grizzly ever.
Also checked packstack-modules-puppet-2013.1.1-0.33.dev695.el6ost.noarch from older puddle (2013-10-24.5) and token_driver isn't available.
The complaint from foreman is coming from the stanza beggining on line 202 in /usr/share/packstack/modules/openstack/manifests/keystone.pp (which I mentioned in previous comment, is copied to the location seen in the stack trace), which is owned by packstack-modules-puppet. This is theh stanza: class { '::keystone': verbose => $verbose, debug => $debug, bind_host => $bind_host, idle_timeout => $idle_timeout, catalog_type => 'sql', admin_token => $admin_token, token_driver => $token_driver, token_format => $token_format, enabled => $enabled, sql_connection => $sql_conn, use_syslog => $use_syslog, log_facility => $log_facility, } Here is a diff of /usr/share/packstack/modules/openstack/manifests/keystone.pp from packstack-modules-puppet-2013.1.1-0.35.dev696.el6ost.noarch.rpm and an older packstack-modules-puppet-2013.1.1-0.31.dev677.el6ost.noarch.rpm confirming that the parameter has changed *somewhere* in the 3.0 packstack-modules-rpm: --- usr/share/packstack/modules/openstack/manifests/keystone.pp 2013-11-05 12:24:00.340223927 -0500 +++ ../pmp/usr/share/packstack/modules/openstack/manifests/keystone.pp 2013-11-04 13:55:54.417237247 -0500 @@ -15,12 +15,16 @@ # [nova_user_password] Auth password for nova user. Required. # [public_address] Public address where keystone can be accessed. Required. # [public_protocol] Public protocol over which keystone can be accessed. Defaults to 'http' +# [token_format] Format keystone uses for tokens. Optional. Defaults to PKI. +# Supports PKI and UUID. # [db_type] Type of DB used. Currently only supports mysql. Optional. Defaults to 'mysql' # [db_user] Name of keystone db user. Optional. Defaults to 'keystone' # [db_name] Name of keystone DB. Optional. Defaults to 'keystone' # [admin_tenant] Name of keystone admin tenant. Optional. Defaults to 'admin' # [verbose] Log verbosely. Optional. Defaults to false. # [debug] Log at a debug-level. Optional. Defaults to false. +# [token_driver] Driver to use for managing tokens. +# Optional. Defaults to 'keystone.token.backends.kvs.Token' # [bind_host] Address that keystone binds to. Optional. Defaults to '0.0.0.0' # [internal_address] Internal address for keystone. Optional. Defaults to $public_address # [admin_address] Keystone admin address. Optional. Defaults to $internal_address @@ -30,6 +34,8 @@ # [swift_user_password] # Auth password for swift. # (Optional) Defaults to false. +# [use_syslog] Use syslog for logging. Defaults to false. +# [log_facility] Syslog facility to receive log lines. Defaults to LOG_USER. # [enabled] If the service is active (true) or passive (false). # Optional. Defaults to true # @@ -59,6 +65,7 @@ $quantum_user_password, $public_address, $public_protocol = 'http', + $token_format = 'PKI', $db_host = '127.0.0.1', $idle_timeout = '200', $swift_user_password = false, @@ -70,6 +77,7 @@ $debug = false, $bind_host = '0.0.0.0', $region = 'RegionOne', + $token_driver = 'keystone.token.backends.kvs.Token', $internal_address = false, $admin_address = false, $glance_public_address = false, @@ -92,6 +100,8 @@ $cinder = true, $quantum = true, $swift = false, + $use_syslog = false, + $log_facility = 'LOG_USER', $enabled = true ) { @@ -197,8 +207,12 @@ idle_timeout => $idle_timeout, catalog_type => 'sql', admin_token => $admin_token, + token_driver => $token_driver, + token_format => $token_format, enabled => $enabled, sql_connection => $sql_conn, + use_syslog => $use_syslog, + log_facility => $log_facility, } if ($enabled) {
Changing ownership to ichavero based on an email thread: 'It seems Ivan just found the problem. The git submodule that we were using was pointing to version that did not contain the commit adding the parameter that was missing. The submodule link is updated now in the Grizzly branch.'
Ah, now I see it. I was checking puppet-keystone module but the issue is in puppet-openstack module. Sorry for that. When [1] was backported to grizzly branch, this parameter was introduced. But then there's only one subsequent commit currently [2] and it is not modifying modules at all. So I guess that's one from RHOS specific backport patches downgrading the module back. I'm going to check that. [1] https://review.openstack.org/#/c/53376/ [2] https://review.openstack.org/#/q/status:merged+project:stackforge/packstack+branch:grizzly,n,z
Nah, puppet-openstack is ok, we have to update puppet-keystone to have puppet-openstack functional. Yep, I'm sometimes slow, sorry for that :).
OK, now I'm confused: [para@elysium packstack-build]$ cd grizzly/ [para@elysium grizzly]$ git submodule status | grep keystone 6f54428601c673baba4e70049ecbc798bd9bad64 packstack/puppet/modules/keystone (1.0.1-84-g6f54428) [para@elysium grizzly]$ cd ../havana [para@elysium havana]$ git submodule status | grep keystone 6f54428601c673baba4e70049ecbc798bd9bad64 packstack/puppet/modules/keystone (1.0.1-84-g6f54428) [para@elysium havana]$ And indeed there is no token_driver at this state: https://github.com/stackforge/puppet-keystone/blob/6f54428601c673baba4e70049ecbc798bd9bad64/manifests/init.pp So both Havana and Grizzly are broken by Maru's openstack::provision update. It's interesting that this does not showed up during packstack testing (both manual and automatic). We will have to update puppet-keystone in both branches and run tests. I will provide Foreman team with packstack-modules-puppet scratch build today.
I think the similar issue just occurred to me with Havana puddle (2013-11-08.1) on rhel6.5 - this issue might be not that consistent. Environment: ------------- rpm -qa | grep puppet packstack-modules-puppet-2013.2.1-0.9.dev840.el6ost.noarch puppet-server-3.2.4-1.el6_4.noarch puppet-3.2.4-1.el6_4.noarch Log: ----- Info: Retrieving plugin Info: Loading facts in /var/lib/puppet/lib/facter/iptables_version.rb Info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb Info: Loading facts in /var/lib/puppet/lib/facter/network.rb Info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb Info: Loading facts in /var/lib/puppet/lib/facter/ipa_client_configured.rb Info: Loading facts in /var/lib/puppet/lib/facter/hamysql_active_node.rb Info: Loading facts in /var/lib/puppet/lib/facter/ip6tables_version.rb Info: Loading facts in /var/lib/puppet/lib/facter/netns_support.rb Info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb Info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid parameter token_driver at /usr/share/packstack/modules/openstack/manifests/keystone.pp:301 on node puma01.scl.lab.tlv.redhat.com Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run vim /usr/share/packstack/modules/openstack/manifests/keystone.pp +301 --------------------------------------------------------------------- class { '::keystone': verbose => $verbose, debug => $debug, bind_host => $bind_host, idle_timeout => $idle_timeout, catalog_type => 'sql', admin_token => $admin_token, token_driver => $token_driver, token_format => $token_format, enabled => $enabled, sql_connection => $sql_conn, use_syslog => $use_syslog, log_facility => $log_facility, }
As I wrote already in comment #12 both grizzly and havana is broken. I've build s scratch grizzly build, so you can test foreman with patched puppet module. Scratch build is garbage collected now. We currently have updated puppet-keystone module in both branches, so this issue will be fixed in next builds.
Verified (Grizzly puddle 2013-11-13.1) packstack-modules-puppet-2013.1.1-0.35.dev699.el6ost.noarch. 'controller' host deployed successfully by foreman.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1510.html