Bug 1026279 - openstack-foreman: Failed to deploy 'controller' using foreman from grizzly on rhel6.4 (Invalid parameter token_driver at /opt/rh/ruby193/root/etc/puppet/environments/production/modules/openstack/manifests/keystone.pp:216)
Summary: openstack-foreman: Failed to deploy 'controller' using foreman from grizzly o...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-packstack
Version: 3.0
Hardware: x86_64
OS: Linux
high
high
Target Milestone: z3
: 3.0
Assignee: Martin Magr
QA Contact: Nir Magnezi
URL:
Whiteboard:
Depends On:
Blocks: 1029219
TreeView+ depends on / blocked
 
Reported: 2013-11-04 10:36 UTC by Omri Hochman
Modified: 2014-01-09 19:40 UTC (History)
13 users (show)

Fixed In Version: openstack-packstack-2013.1.1-0.35.dev699.el6ost
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1029219 (view as bug list)
Environment:
Last Closed: 2013-11-18 15:19:44 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 55412 0 None None None Never
OpenStack gerrit 55413 0 None None None Never
OpenStack gerrit 56038 0 None None None Never
OpenStack gerrit 56071 0 None None None Never
Red Hat Product Errata RHBA-2013:1510 0 normal SHIPPED_LIVE Red Hat OpenStack 3.0 bug fix and enhancement advisory 2013-11-18 20:11:18 UTC

Description Omri Hochman 2013-11-04 10:36:50 UTC
openstack-foreman: Failed to deploy 'controller' using foreman from grizzly on rhel6.4  (Invalid parameter token_driver at /opt/rh/ruby193/root/etc/puppet/environments/production/modules/openstack/manifests/keystone.pp:216)

Environment: (RHEL 6.4 + Puddle 2013-11-01.1)
---------------------------------------------
ruby193-rubygem-json-1.5.5-40.el6.x86_64
ruby193-ruby-1.9.3.448-40.el6.x86_64
ruby193-hiera-1.0.0-6.el6ost.noarch
ruby-1.8.7.352-7.el6_2.x86_64
ruby-augeas-0.4.1-1.el6.x86_64
libselinux-ruby-2.0.94-5.3.el6.x86_64
ruby193-libyaml-0.1.4-4.el6.x86_64
ruby193-ruby-libs-1.9.3.448-40.el6.x86_64
ruby193-rubygem-bigdecimal-1.1.0-40.el6.x86_64
ruby193-rubygem-rdoc-3.12-12.el6ost.x86_64
ruby193-rubygems-1.8.24-9.el6ost.noarch
ruby193-facter-1.6.18-5.el6ost.x86_64
ruby193-ruby-augeas-0.4.1-7.el6ost.x86_64
ruby193-puppet-3.1.1-11.1.el6ost.noarch
ruby-libs-1.8.7.352-7.el6_2.x86_64
ruby-shadow-1.4.1-13.el6.x86_64
ruby193-runtime-1-10.el6.x86_64
ruby193-ruby-irb-1.9.3.448-40.el6.noarch
ruby193-rubygem-io-console-0.3-40.el6.x86_64
ruby193-ruby-shadow-1.4.1-24.el6ost.x86_64
puppet-2.6.18-3.el6.noarch
packstack-modules-puppet-2013.1.1-0.35.dev696.el6ost.noarch
ruby193-puppet-3.1.1-11.1.el6ost.noarch
ruby193-puppet-server-3.1.1-11.1.el6ost.noarch

I'm having this problem when Installing controller using foreman - after puppet fails the host left with "E" Error 
(already reproduced few times) : 

Info: Loading facts in /opt/rh/ruby193/root/var/lib/puppet/lib/facter/puppet_vardir.rb
Info: Loading facts in /opt/rh/ruby193/root/var/lib/puppet/lib/facter/pe_version.rb
Info: Loading facts in /opt/rh/ruby193/root/var/lib/puppet/lib/facter/ip6tables_version.rb
Info: Loading facts in /opt/rh/ruby193/root/var/lib/puppet/lib/facter/network.rb
Info: Loading facts in /opt/rh/ruby193/root/var/lib/puppet/lib/facter/facter_dot_d.rb
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid parameter token_driver at /opt/rh/ruby193/root/etc/puppet/environments/production/modules/openstack/manifests/keystone.pp:216 on node puma01.scl.lab.tlv.redhat.com
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run


From looking at this file on the server: 
vim +216 /opt/rh/ruby193/root/etc/puppet/environments/production/modules/openstack/manifests/keystone.pp  

 
  class { '::keystone':
    verbose        => $verbose,
    debug          => $debug,
    bind_host      => $bind_host,
    idle_timeout   => $idle_timeout,
    catalog_type   => 'sql',
    admin_token    => $admin_token,
    token_driver   => $token_driver,
    token_format   => $token_format,
    enabled        => $enabled,
    sql_connection => $sql_conn,
    use_syslog     => $use_syslog,
    log_facility   => $log_facility,
  }

Note: the issue didn't occur on older puddle (2013-10-24.5)

Comment 3 Crag Wolfe 2013-11-04 21:26:40 UTC
I believe this error is coming from packstack-modules-puppet-2013.1.1-0.35.dev696.el6ost.noarch.rpm .  The file in the stack trace above is copied by the foreman installer from /usr/share/packstack/modules/openstack/manifests/keystone.pp so it is really just an unaltered packstack-modules-puppet manifest.  I'd be surprised if the packstack installer itself wasn't broken.

Comment 4 Martin Magr 2013-11-05 13:28:56 UTC
Looking at the grizzly puppet-keystone module there is no token_driver paramater (which is exactly what the error message saying):
class keystone(
  $admin_token,
  $package_ensure = 'present',
  $bind_host      = '0.0.0.0',
  $public_port    = '5000',
  $admin_port     = '35357',
  $compute_port   = '8774',
  $verbose        = false,
  $debug          = false,
  $use_syslog     = false,
  $catalog_type   = 'sql',
  $token_format   = 'PKI',
  $cache_dir      = '/var/cache/keystone',
  $enabled        = true,
  $sql_connection = 'sqlite:////var/lib/keystone/keystone.db',
  $idle_timeout   = '200'
)
...

So I guess you are using parameter which is not available in grizzly modules. BTW packstack is not broken since it is not using such parameter.

Comment 5 Martin Magr 2013-11-05 13:33:02 UTC
Also checked older modules (packstack-modules-puppet-2013.1.1-0.31.dev677.el6ost.noarch) and the parameter is not there, so I guess it wasn't available in grizzly ever.

Comment 6 Martin Magr 2013-11-05 13:50:58 UTC
Also checked packstack-modules-puppet-2013.1.1-0.33.dev695.el6ost.noarch from older puddle (2013-10-24.5) and token_driver isn't available.

Comment 8 Crag Wolfe 2013-11-05 17:34:18 UTC
The complaint from foreman is coming from the stanza beggining on line 202 in /usr/share/packstack/modules/openstack/manifests/keystone.pp (which I mentioned in previous comment, is copied to the location seen in the stack trace), which is owned by packstack-modules-puppet.    This is theh stanza:

 class { '::keystone':
    verbose        => $verbose,
    debug          => $debug,
    bind_host      => $bind_host,
    idle_timeout   => $idle_timeout,
    catalog_type   => 'sql',
    admin_token    => $admin_token,
    token_driver   => $token_driver,
    token_format   => $token_format,
    enabled        => $enabled,
    sql_connection => $sql_conn,
    use_syslog     => $use_syslog,
    log_facility   => $log_facility,
  }

Here is a diff of /usr/share/packstack/modules/openstack/manifests/keystone.pp from packstack-modules-puppet-2013.1.1-0.35.dev696.el6ost.noarch.rpm and an older 
packstack-modules-puppet-2013.1.1-0.31.dev677.el6ost.noarch.rpm confirming that the parameter has changed *somewhere* in the 3.0 packstack-modules-rpm:

--- usr/share/packstack/modules/openstack/manifests/keystone.pp	2013-11-05 12:24:00.340223927 -0500
+++ ../pmp/usr/share/packstack/modules/openstack/manifests/keystone.pp	2013-11-04 13:55:54.417237247 -0500
@@ -15,12 +15,16 @@
 # [nova_user_password] Auth password for nova user. Required.
 # [public_address] Public address where keystone can be accessed. Required.
 # [public_protocol] Public protocol over which keystone can be accessed. Defaults to 'http'
+# [token_format] Format keystone uses for tokens. Optional. Defaults to PKI.
+#   Supports PKI and UUID.
 # [db_type] Type of DB used. Currently only supports mysql. Optional. Defaults to  'mysql'
 # [db_user] Name of keystone db user. Optional. Defaults to  'keystone'
 # [db_name] Name of keystone DB. Optional. Defaults to  'keystone'
 # [admin_tenant] Name of keystone admin tenant. Optional. Defaults to  'admin'
 # [verbose] Log verbosely. Optional. Defaults to false.
 # [debug] Log at a debug-level. Optional. Defaults to false.
+# [token_driver] Driver to use for managing tokens.
+#   Optional.  Defaults to 'keystone.token.backends.kvs.Token'
 # [bind_host] Address that keystone binds to. Optional. Defaults to  '0.0.0.0'
 # [internal_address] Internal address for keystone. Optional. Defaults to  $public_address
 # [admin_address] Keystone admin address. Optional. Defaults to  $internal_address
@@ -30,6 +34,8 @@
 # [swift_user_password]
 #   Auth password for swift.
 #   (Optional) Defaults to false.
+# [use_syslog] Use syslog for logging. Defaults to false.
+# [log_facility] Syslog facility to receive log lines. Defaults to LOG_USER.
 # [enabled] If the service is active (true) or passive (false).
 #   Optional. Defaults to  true
 #
@@ -59,6 +65,7 @@
   $quantum_user_password,
   $public_address,
   $public_protocol          = 'http',
+  $token_format             = 'PKI',
   $db_host                  = '127.0.0.1',
   $idle_timeout             = '200',
   $swift_user_password      = false,
@@ -70,6 +77,7 @@
   $debug                    = false,
   $bind_host                = '0.0.0.0',
   $region                   = 'RegionOne',
+  $token_driver             = 'keystone.token.backends.kvs.Token',
   $internal_address         = false,
   $admin_address            = false,
   $glance_public_address    = false,
@@ -92,6 +100,8 @@
   $cinder                   = true,
   $quantum                  = true,
   $swift                    = false,
+  $use_syslog               = false,
+  $log_facility             = 'LOG_USER',
   $enabled                  = true
 ) {
 
@@ -197,8 +207,12 @@
     idle_timeout   => $idle_timeout,
     catalog_type   => 'sql',
     admin_token    => $admin_token,
+    token_driver   => $token_driver,
+    token_format   => $token_format,
     enabled        => $enabled,
     sql_connection => $sql_conn,
+    use_syslog     => $use_syslog,
+    log_facility   => $log_facility,
   }
 
   if ($enabled) {

Comment 9 Mike Orazi 2013-11-05 21:36:32 UTC
Changing ownership to ichavero based on an email thread:

'It seems Ivan just found the problem. The git submodule that we were using was pointing to version that did not contain the commit adding the parameter that was missing. The submodule link is updated now in the Grizzly branch.'

Comment 10 Martin Magr 2013-11-06 12:13:23 UTC
Ah, now I see it. I was checking puppet-keystone module but the issue is in puppet-openstack module. Sorry for that.

When [1] was backported to grizzly branch, this parameter was introduced. But then there's only one subsequent commit currently [2] and it is not modifying modules at all. So I guess that's one from RHOS specific backport patches downgrading the module back. I'm going to check that.

[1] https://review.openstack.org/#/c/53376/
[2] https://review.openstack.org/#/q/status:merged+project:stackforge/packstack+branch:grizzly,n,z

Comment 11 Martin Magr 2013-11-06 12:25:03 UTC
Nah, puppet-openstack is ok, we have to update puppet-keystone to have puppet-openstack functional. Yep, I'm sometimes slow, sorry for that :).

Comment 12 Martin Magr 2013-11-06 12:53:12 UTC
OK, now I'm confused:
[para@elysium packstack-build]$ cd grizzly/
[para@elysium grizzly]$ git submodule status | grep keystone
 6f54428601c673baba4e70049ecbc798bd9bad64 packstack/puppet/modules/keystone (1.0.1-84-g6f54428)
[para@elysium grizzly]$ cd ../havana
[para@elysium havana]$ git submodule status | grep keystone
 6f54428601c673baba4e70049ecbc798bd9bad64 packstack/puppet/modules/keystone (1.0.1-84-g6f54428)
[para@elysium havana]$ 

And indeed there is no token_driver at this state:
https://github.com/stackforge/puppet-keystone/blob/6f54428601c673baba4e70049ecbc798bd9bad64/manifests/init.pp

So both Havana and Grizzly are broken by Maru's openstack::provision update. It's interesting that this does not showed up during packstack testing (both manual and automatic). We will have to update puppet-keystone in both branches and run tests. I will provide Foreman team with packstack-modules-puppet scratch build today.

Comment 15 Omri Hochman 2013-11-11 17:36:40 UTC
I think the similar issue just occurred to me with Havana puddle (2013-11-08.1) on rhel6.5 - this issue might be not that consistent.  

Environment:
-------------
rpm -qa | grep puppet
packstack-modules-puppet-2013.2.1-0.9.dev840.el6ost.noarch
puppet-server-3.2.4-1.el6_4.noarch
puppet-3.2.4-1.el6_4.noarch


Log:
-----
Info: Retrieving plugin
Info: Loading facts in /var/lib/puppet/lib/facter/iptables_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/network.rb
Info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
Info: Loading facts in /var/lib/puppet/lib/facter/ipa_client_configured.rb
Info: Loading facts in /var/lib/puppet/lib/facter/hamysql_active_node.rb
Info: Loading facts in /var/lib/puppet/lib/facter/ip6tables_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/netns_support.rb
Info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
Info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid parameter token_driver at /usr/share/packstack/modules/openstack/manifests/keystone.pp:301 on node puma01.scl.lab.tlv.redhat.com
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run



vim /usr/share/packstack/modules/openstack/manifests/keystone.pp +301
---------------------------------------------------------------------

 class { '::keystone':
    verbose        => $verbose,
    debug          => $debug,
    bind_host      => $bind_host,
    idle_timeout   => $idle_timeout,
    catalog_type   => 'sql',
    admin_token    => $admin_token,
    token_driver   => $token_driver,
    token_format   => $token_format,
    enabled        => $enabled,
    sql_connection => $sql_conn,
    use_syslog     => $use_syslog,
    log_facility   => $log_facility,
  }

Comment 16 Martin Magr 2013-11-11 17:45:25 UTC
As I wrote already in comment #12 both grizzly and havana is broken.

I've build s scratch grizzly build, so you can test foreman with patched puppet module. Scratch build is garbage collected now. We currently have updated puppet-keystone module in both branches, so this issue will be fixed in next builds.

Comment 24 Omri Hochman 2013-11-13 21:08:09 UTC
Verified (Grizzly puddle 2013-11-13.1) packstack-modules-puppet-2013.1.1-0.35.dev699.el6ost.noarch. 

'controller' host deployed successfully by foreman.

Comment 27 errata-xmlrpc 2013-11-18 15:19:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1510.html


Note You need to log in before you can comment on or make changes to this bug.