Description of problem: Running the foreman_server.sh from openstack-foreman-installer-1.0.6-1.el6 on a clean RHEL or CentOS 6.5 with the EPEL and the rdo-release-havana-7, which installs the foreman 1.3 repos, the following error happens: rake aborted! undefined method `[]' for nil:NilClass Tasks: TOP => db:seed (See full trace by running task with --trace) This is the full output: http://fpaste.org/97739/ The variables used are the following: export PROVISIONING_INTERFACE=eth0 export FOREMAN_GATEWAY=192.168.5.100 export FOREMAN_PROVISIONING=true DHCP and Apache seem well configured but the Host Groups are not populated. Version-Release number of selected component (if applicable): foreman-mysql-1.3.2-1.el6.noarch foreman-mysql2-1.3.2-1.el6.noarch foreman-proxy-1.3.0-1.el6.noarch rubygem-foreman_api-0.1.9-1.el6.noarch openstack-foreman-installer-1.0.6-1.el6.noarch foreman-1.3.2-1.el6.noarch foreman-installer-1.3.1-1.el6.noarch openstack-puppet-modules-2013.2-9.el6.noarch puppet-server-3.5.1-1.el6.noarch puppet-3.5.1-1.el6.noarch How reproducible: Always Steps to Reproduce: 1. Install a clean RHEL 6.5 or CentOS 6.5 2. Enable EPEL 3. Enable the rdo-havana-7 repository 4. Install openstack-foreman-installer 5. Export the above mentioned variables 6. Run the script foreman_server.sh from /usr/share/openstack-foreman-installer/bin Actual results: rake aborted! undefined method `[]' for nil:NilClass Expected results: Script finishes with no error and foreman hostgroups are populated. Additional information: Might be similar to https://bugzilla.redhat.com/show_bug.cgi?id=1091619
I provisioned a new server and tried this puppet 3.2 (puppet-server-3.2.4-1.el6) and the result is very similar only that it took much longer: http://fpaste.org/97760/
As an update, I reproduced it in a new CentOS 6.5 server with the rdo-release-icehouse-3 which installs the foreman 1.5 repos and openstack-foreman-installer 2.0. Same error: rake aborted! undefined method `[]' for nil:NilClass Tasks: TOP => db:seed (See full trace by running task with --trace) Different output as openstack-foreman-installer version is different and apparently more tables have been filled. Full output: http://fpaste.org/97838 These are the versions in the latest installation: foreman-installer-1.5.0-0.2.RC2.el6.noarch foreman-1.5.0-0.2.RC2.el6.noarch openstack-foreman-installer-2.0.0.0-1.el6.noarch foreman-proxy-1.5.0-0.2.RC2.el6.noarch foreman-selinux-1.5.0-0.2.RC2.el6.noarch foreman-mysql2-1.5.0-0.2.RC2.el6.noarch puppet-3.5.1-1.el6.noarch openstack-puppet-modules-2014.1-8.el6.noarch puppet-server-3.5.1-1.el6.noarch
What ruby version do you use ? I think you need to enable scl repository for foreman 1.5 (requires ruby 1.9.3): http://theforeman.org/manuals/1.5/index.html The centos SCL is available here: http://wiki.centos.org/AdditionalResources/Repositories/SCL Just a guess. -- Thomas.
Also, as reported in the referenced BZ, when the user reverted to a slightly older puppet, the problem went away for him. Have you tried that version to compare behavior? I thought our repo was going to lock us down to 3.4.x, Padraig, am I remembering that wrong?
We've no repo wide pins on puppet version packstack seems fine with the new puppet version. Also note this new puppet version is in rawhide and el7 repos now. The suggestion was that foreman 1.5 is fine with the new puppet which is why we bumped that dependency in the first place. If foreman has large issues with the new puppet, then I suggest a (temp) pin in the astapor packages, however I'd try to fix it the right way and tweak things to be compat with the new puppet.
Reply to Comment 4: That's correct, if using Foreman 1.5 (installed with rdo-release-havana-8 or rdo-release-icehouse-3) SCL needs to be enabled. If not enabled, yum won't install openstack-foreman-installer due to ruby193 dependencies. Reply to Comment 5: I reproduced it with puppet-server-3.2.4-1.el6 instead of the default puppet-server-3.5.1-1.el as I mentioned in Comment 2. It is not locked to Puppet 3.4.x as puppet-server-3.5.1-1.el is in the latest one in the repo installed by rdo-release-havana-7 (http://yum.puppetlabs.com/el/6/products/x86_64/). I'm going to redeploy everything with 3.4 and will share here the results.
I just finished running the foreman_server.sh script with puppet-server-3.4.3-1.el6.noarch and the result is the same error. Full output: http://fpaste.org/98080/ In summary: The script will fail with Foreman 1.3 and Foreman 1.5 combined with either Puppet 3.2, Puppet 3.4 and Puppet 3.5. How it is supposed to work? i.e known working conditions such as other combinations of env variables as disabling Foreman provisioning. How was it QAed before the last release of Astapor which is packaged in openstack-foreman-installer-2.0.0.0-1.el6 available in the repos.
After setting FOREMAN_PROVISIONING=false and running foreman_server.sh with the default versions of Puppet and Foreman everything installed and the "Host groups" in the "Configure" section are now populated. It seems the script wasn't QAed with FOREMAN_PROVISIONING=true? I can help to further debugging as I would like it to work with bare metal provisioning.
Thanks for all the info, I will see if I can reproduce and debug this on my setup tomorrow. Apologies for being unclear in my earlier comment, I meant specifically 3.4 vs 3.5 for puppet, which you have now covered and shown to also have an issue. I'll report back my findings as soon as I have been able to do some debugging, just need to finish up some non-installer work we are in the middle of first.
OK, so I have done some testing here and have not yet been able to reproduce your issue on my setup, and one that QE let me borrow. I am highly suspicious right now of odd combinations of version of rpms, which I will list below, but let's also see if we can get some other test scenarios for you. So far you have verified behavior of: 1. Icehouse with provisioning=true and gateway=<ip> - FAIL 2. Havana with provisioning=true and gateway=<ip> - FAIL 3. <version?> with provisioning=false = SUCCESS The setup I tried for icehouse/rdo was: * Icehouse with provisioning=true and gateway=false - SUCCESS My next test will set gateway, but I am wondering if dependencies could have shifted under us between your tests and mine. If you were to get success in that setup too, that would narrow down potential issue areas. I have also run this successfully on rhos 4 using our 1.0.6/7 builds without issue. Some extra things you can do to help figure out where there may be a problem (if you are so inclined) are: 1. run foreman_server with 'bash -x' instead of just 'sh' 2. add a --trace to the 2 rake calls in foreman_server.sh (at the end is fine, inside the quotes) 3. find the 2 blocks in seeds.rb where we loop through the puppet classes to set things (around lines 414 and 429). Inside each of those block, add a puts, in the first one: puts "===== seeds hostgroup/puppet block: #{hg.inspect} =====" and in the second: puts "!!!!! seeds hostgroup block: #{hg.inspect} !!!!!!" These should go right after lines that look like: hostgroups.each do |hg| Doing that will show us which classes made it in successfully, assuming we have gotten that far. The repos my successful machine had for icehouse were: yum repolist Loaded plugins: product-id, subscription-manager This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. repo id repo name status epel Extra Packages for Enterprise Linux 6 - x86_64 10,739 foreman Foreman stable 158 foreman-plugins Foreman stable - plugins 65 openstack-icehouse OpenStack Icehouse Repository 1,012 puppetlabs-deps Puppet Labs Dependencies - x86_64 61 puppetlabs-products Puppet Labs Products - x86_64 363 rhel-6-server-optional-rpms Extra Packages for Enterprise Linux 6 - x86_64 2,921 rhel-6-server-rpms Extra Packages for Enterprise Linux 6 - x86_64 0 rhel-6-server-updates-rpms Extra Packages for Enterprise Linux 6 - x86_64 6,358 rhscl-ruby193-epel-6-x86_64 Ruby193 - epel-6-x86_64 504 RPM versions: rubygem-foreman_api-0.1.11-1.el6.noarch foreman-1.5.0-0.2.RC2.el6.noarch foreman-installer-1.5.0-0.2.RC2.el6.noarch ruby193-rubygem-foreman_simplify-0.0.5-1.el6.noarch openstack-foreman-installer-2.0.0.0-1.el6.noarch foreman-proxy-1.5.0-0.2.RC2.el6.noarch foreman-selinux-1.5.0-0.2.RC2.el6.noarch foreman-mysql2-1.5.0-0.2.RC2.el6.noarch puppet-3.5.1-1.el6.noarch openstack-puppet-modules-2014.1-9.el6.noarch puppet-server-3.5.1-1.el6.noarch ruby193-facter-1.6.18-4.el6.x86_64 facter-2.0.1-1.el6.x86_64
The setup I test had these packages: openstack-puppet-modules-2014.1-9.el6.noarch openstack-foreman-installer-2.0.0.0-1.el6.noarch puppet-3.5.1-1.el6.noarch rubygem-foreman_api-0.1.11-1.el6.noarch foreman-1.5.0-0.2.RC2.el6.noarch foreman-installer-1.5.0-0.2.RC2.el6.noarch ruby193-rubygem-foreman_simplify-0.0.5-1.el6.noarch openstack-foreman-installer-2.0.0.0-1.el6.noarch foreman-proxy-1.5.0-0.2.RC2.el6.noarch foreman-selinux-1.5.0-0.2.RC2.el6.noarch foreman-mysql2-1.5.0-0.2.RC2.el6.noarch And these settings: FOREMAN_PROVISIONING=true PROVISIONING_INTERFACE=eth0 FOREMAN_GATEWAY=<ip> This worked for me.
I'll have access again later this week to repeat the tests. Bear in mind the last tests were done with CentOS 6.5, EPEL and RDO as described above (not sure if that should make any difference). I understand that unsetting FOREMAN_GATEWAY seems to do the trick from Comment 11 and Comment 12, is that correct? In response to Comment 11: >> "3. <version?> with provisioning=false = SUCCESS" Version is Icehouse.
Jason, I followed your advice to get additional output from rake as you suggested in Comment 11: + sudo -u foreman scl enable ruby193 'cd /usr/share/foreman; rake --trace db:seed RAILS_ENV=production FOREMAN_PROVISIONING=true' ** Invoke db:seed (first_time) ** Execute db:seed ** Invoke db:abort_if_pending_migrations (first_time) ** Invoke environment (first_time) ** Execute environment ** Invoke db:load_config (first_time) ** Execute db:load_config ** Execute db:abort_if_pending_migrations Seeding /usr/share/foreman/db/seeds.d/05-architectures.rb Seeding /usr/share/foreman/db/seeds.d/07-config_templates.rb Seeding /usr/share/foreman/db/seeds.d/08-partition_tables.rb Seeding /usr/share/foreman/db/seeds.d/10-installation_media.rb Seeding /usr/share/foreman/db/seeds.d/11-permissions.rb Seeding /usr/share/foreman/db/seeds.d/11-roles.rb Seeding /usr/share/foreman/db/seeds.d/11-smart_proxy_features.rb Seeding /usr/share/foreman/db/seeds.d/12-auth_sources.rb Seeding /usr/share/foreman/db/seeds.d/13-compute_profiles.rb Seeding /usr/share/foreman/db/seeds.d/15-bookmarks.rb Seeding /usr/share/foreman/db/seeds.d/99-quickstack.rb rake aborted! undefined method `[]' for nil:NilClass Full log: http://fpaste.org/99890/ We have exactly the same versions as the ones you reported in Comment 11 and Brad reported in Comment 12: facter-2.0.1-1.el6.x86_64 foreman-1.5.0-0.2.RC2.el6.noarch foreman-installer-1.5.0-0.2.RC2.el6.noarch foreman-mysql2-1.5.0-0.2.RC2.el6.noarch foreman-proxy-1.5.0-0.2.RC2.el6.noarch foreman-selinux-1.5.0-0.2.RC2.el6.noarch openstack-foreman-installer-2.0.0.0-1.el6.noarch openstack-puppet-modules-2014.1-9.el6.noarch puppet-3.5.1-1.el6.noarch puppet-server-3.5.1-1.el6.noarch ruby193-facter-1.6.18-4.el6.x86_64 ruby193-rubygem-foreman_simplify-0.0.5-1.el6.noarch rubygem-foreman_api-0.1.11-1.el6.noarch As you can see, it failed again and the foreman host groups are not prepopulated. Jason, I believe we should not set FOREMAN_GATEWAY to false because that variable is used to set the gateway in the DHCP server settings for the Foreman host. Please, correct me if I'm wrong. A gateway setting should be an option. Brad, I'm not sure how you got it working as we have exactly the same versions, not sure if you could share the full output and if you checked if the database was populated and you didn't get the "undefined method `[]' for nil:NilClass". I should be able to reproduce exactly what you did and do further tests if needed.
Ramon, I am not suggesting that disabling the FOREMAN_GATEWAY is a solution, just trying to rule out possible error paths. If setting that to false got you further, that would give us something to work with for debugging. I looked at your paste, ad that is very disappointing that it fails before even any of the puts from seeds.rb were hit. I have to admit I am a bit stumped here right now, as there is not enough information in the error or debug output to suggest any changes, and I have been unable to reproduce the error myself. If I can get a setup where I can reproduce this, I have no doubt I can figure out what is going on here. Also, as you mentioned, you used cents, and we have been using rhea, which is an obvious difference (though of course if one works, they both should). I'll try another fresh setup in the next day or two once I finish up some things we are working on for Havana. We have a number of changes in master since the release at this point, and since Brad was unable to reproduce, if he tests master (icehouse) and has success, I may just tag a new release and get it pushed up to rod, at which point I'll update you here so you can give it a try to see if it helps. Sorry to not have a better answer, this a a tricky one to debug from afar.
I reproduced it again setting a value to FOREMAN_GATEWAY:http://fpaste.org/100468/ Now my only chance is using RHEL and the OSP channel which I'll try to do in the coming days but from what you say it won't happen with RHEL so I may be able to move forward after setting it up with RHEL OSP. Thanks.
I just noticed that I cannot install Icehouse from the RHEL subscription channels, the rhel-6-server-openstack-beta-rpms still has the old Havana packages. I believe there are no public repos for RHEL OSP 5 beta?
I deployed it with RHEL and openstack-foreman-installer-1.0.5-1.el6ost.noarch: [root@foreman bin]# yum repolist Loaded plugins: product-id, security, subscription-manager This system is receiving updates from Red Hat Subscription Management. rhel-6-server-openstack-4.0-rpms | 3.1 kB 00:00 rhel-6-server-optional-rpms | 3.5 kB 00:00 rhel-6-server-rpms | 3.7 kB 00:00 rhel-ha-for-rhel-6-server-rpms | 3.7 kB 00:00 rhel-lb-for-rhel-6-server-rpms | 3.7 kB 00:00 [...] Everything works in RHEL and rhel-6-server-openstack-4.0-rpms so I confirm this just happens with CentOS 6.5 and RDO. The difference I can tell is that RDO comes with openstack-foreman-installer-1.0.6-1.el6 whereas RHEL OSP 4 ships openstack-foreman-installer-1.0.5-1.el6ost.noarch Should the issue still be tracked in this bz? Thanks.
Moving to RDO.
I just had this same issue occur w/ RHEL 6.5 and RH OSP 4.
Does this help? + sudo -u foreman scl enable ruby193 'cd /usr/share/foreman; rake --trace db:seed RAILS_ENV=production FOREMAN_PROVISIONING=true' ** Invoke db:seed (first_time) ** Execute db:seed ** Invoke db:abort_if_pending_migrations (first_time) ** Invoke environment (first_time) ** Execute environment ** Invoke db:load_config (first_time) ** Execute db:load_config ** Execute db:abort_if_pending_migrations rake aborted! undefined method `[]' for nil:NilClass /usr/share/foreman/db/seeds.rb:242:in `<top (required)>' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/dependencies.rb:245:in `load' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/dependencies.rb:245:in `block in load' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/dependencies.rb:236:in `load_dependency' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/dependencies.rb:245:in `load' /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/engine.rb:520:in `load_seed' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/railties/databases.rake:309:in `block (2 levels) in <top (required)>' /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:in `call' /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:205:in `block in execute' /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:in `each' /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:200:in `execute' /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:158:in `block in invoke_with_call_chain' /opt/rh/ruby193/root/usr/share/ruby/monitor.rb:211:in `mon_synchronize' /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:151:in `invoke_with_call_chain' /opt/rh/ruby193/root/usr/share/ruby/rake/task.rb:144:in `invoke' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:116:in `invoke_task' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in `block (2 levels) in top_level' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in `each' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:94:in `block in top_level' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:in `standard_exception_handling' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:88:in `top_level' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:66:in `block in run' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:133:in `standard_exception_handling' /opt/rh/ruby193/root/usr/share/ruby/rake/application.rb:63:in `run' /opt/rh/ruby193/root/usr/bin/rake:32:in `<main>' Tasks: TOP => db:seed
I'm seeing this same problem with current (25 June 2014) RHEL 6.5 and RHOS 4 repositories. If I set FOREMAN_PROVISIONING=false the script succeeds and the Host Groups are created. repo id repo name status rhel-6-server-cf-tools-1-rpms Red Hat CloudForms Tools for RHEL 6 (RPMs) 31 rhel-6-server-mrg-messaging-2-rpms Red Hat Enterprise MRG Messaging 2 (for RHEL 6 Server) (RPMs) 184 rhel-6-server-openstack-4.0-rpms Red Hat OpenStack 4.0 (RPMs) 742 rhel-6-server-rhev-agent-rpms Red Hat Enterprise Virtualization Agents for RHEL 6 Server (RPMs) 50 rhel-6-server-rpms Red Hat Enterprise Linux 6 Server (RPMs) 12,642 rhel-ha-for-rhel-6-server-rpms Red Hat Enterprise Linux High Availability (for RHEL 6 Server) (RPMs) 376 rhel-lb-for-rhel-6-server-rpms Red Hat Enterprise Linux Load Balancer (for RHEL 6 Server) (RPMs) 15 repolist: 14,040 foreman-selinux-1.3.0-1.el6sat.noarch openstack-foreman-installer-1.0.12-1.el6ost.noarch foreman-1.3.0.4-1.el6sat.noarch foreman-mysql-1.3.0.4-1.el6sat.noarch foreman-proxy-1.3.0-5.el6sat.noarch foreman-installer-1.3.0-1.el6sat.noarch foreman-mysql2-1.3.0.4-1.el6sat.noarch rubygem-foreman_api-0.1.6-1.el6sat.noarch ruby193-puppet-3.1.1-11.el6ost.noarch puppet-3.2.4-3.el6_5.noarch openstack-puppet-modules-2013.2-9.1.el6ost.noarch puppet-server-3.2.4-3.el6_5.noarch /usr/share/openstack-foreman-installer/bin ** Invoke puppet:import:puppet_classes (first_time) ** Invoke environment (first_time) ** Execute environment ** Execute puppet:import:puppet_classes ** Invoke db:seed (first_time) ** Execute db:seed ** Invoke db:abort_if_pending_migrations (first_time) ** Invoke environment (first_time) ** Execute environment ** Invoke db:load_config (first_time) ** Execute db:load_config ** Execute db:abort_if_pending_migrations rake aborted! undefined method `[]' for nil:NilClass /usr/share/foreman/db/seeds.rb:242:in `<top (required)>'
Update. I created a clean install of RHEL 6.5 base and then followed the Openstack 4 Installation Guide to install Foreman. This time with provisioning enabled Foreman was successfully installed. I no longer get the error reported in this Bugzilla. I can also see all hostgroups in the web interface. I believe my problem was not starting from a fresh install of RHEL 6.5 for the Foreman server.
As I mentioned in Comment 18 it's only reproducible in CentOS 6.5, in RHEL 6.5 it works as you described.
I am experiencing this on RHEL6.5 and OSP4, on a fresh install on 6/24. I have not updated packages since then yet. Please let me know what additional information I can provide.
FWIW - I'm also triggering this issue with clean RHEL6.5 and OSP4. [root@foreman ~]# yum repolist Loaded plugins: product-id, security, subscription-manager This system is receiving updates from Red Hat Subscription Management. repo id repo name status rhel-6-server-mrg-messaging-2-rpms Red Hat Enterprise MRG Messaging 2 (for RHEL 6 Server) (RPMs) 205 rhel-6-server-openstack-4.0-rpms Red Hat OpenStack 4.0 (RPMs) 769 rhel-6-server-rpms Red Hat Enterprise Linux 6 Server (RPMs) 12,814 repolist: 13,788 [root@foreman ~]# rpm -q foreman foreman-1.3.0.4-1.el6sat.noarch [root@foreman ~]# rpm -q puppet-server puppet-server-3.2.4-3.el6_5.noarch [root@foreman ~]# yum update Loaded plugins: product-id, security, subscription-manager This system is receiving updates from Red Hat Subscription Management. Setting up Update Process No Packages marked for Update
openstack-foreman-installer has been deprecated and will no longer be receiving updates.