Description of problem: During Overcloud deploy, the rabbitmq_init_bundle container is launched but doesn't do anything after crm_node -n is launched by /usr/bin/ruby /usr/bin/puppet apply --verbose ... Version-Release number of selected component (if applicable): rabbitmq-server-3.6.15-3.el7ost.noarch How reproducible: Reproducible with every overcloud deploy Steps to Reproduce: 1. Run openstack overcloud deploy 2. Once we notice os-collect-config has stopped on the deployment of rabbitmq_init_bundle 3. If we verify which step it's up to using: $ hiera -c /etc/puppet/hiera.yaml step 2 We can see that its up to step 2, and verify using: $ cat /etc/puppet/hieradata/config_step.json {"step": 2} Then we try to manually re-run the puppet apply using: $ puppet apply --debug puppet_step_config.pp Actual results: We get the output in the attached file: puppet_apply_debug.txt Notable are the following warnings: $ egrep -i '^warning' puppet_apply_debug.txt Warning: Facter: Could not retrieve fact='rabbitmq_nodename', resolution='<anonymous>': undefined method `[]' for nil:NilClass Warning: Undefined variable '::deploy_config_name'; Warning: Undefined variable 'deploy_config_name'; Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Ip_address instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at ["/etc/puppet/modules/tripleo/manifests/profile/base/database/mysql/client.pp", 56]:["/var/lib/tripleo-config/puppet_step_config.pp", 35] Warning: This method is deprecated, please use the stdlib validate_legacy function, Warning: This method is deprecated, please use the stdlib validate_legacy function, Warning: This method is deprecated, please use the stdlib validate_legacy function, Warning: This method is deprecated, please use the stdlib validate_legacy function, Warning: This method is deprecated, please use the stdlib validate_legacy function, Warning: This method is deprecated, please use the stdlib validate_legacy function, Warning: This method is deprecated, please use the stdlib validate_legacy function, Within the container we see: ()[root@overcloud-controller-0 /]# ps -ef | more UID PID PPID C STIME TTY TIME CMD root 1 0 0 05:50 ? 00:00:00 /bin/bash /docker_puppet_apply.sh 2 file,file_line,concat,augeas,pacemaker::resource::bundle,pacemaker::property,pacemaker::resource::ocf,pacemak er::constraint::order,pacemaker::constraint::colocation,rabbitmq_policy,rabbitmq_user,rabbitmq_ready include ::tripleo::profile::base::pacemaker;include ::tripleo::profile::pacemaker::rabbitmq_bundle root 7 1 0 05:50 ? 00:00:02 /usr/bin/ruby /usr/bin/puppet apply --verbose --detailed-exitcodes --summarize --color=false --modulepath /etc/puppet/modules:/opt/stack/puppet-m odules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,pacemaker::resource::bundle,pacemaker::property,pacemaker::resource::ocf,pacemaker::constraint::order,pacemaker::const raint::colocation,rabbitmq_policy,rabbitmq_user,rabbitmq_ready -e include ::tripleo::profile::base::pacemaker;include ::tripleo::profile::pacemaker::rabbitmq_bundle root 399 7 0 05:50 ? 00:00:00 /usr/sbin/crm_node -n root 400 0 0 05:53 ? 00:00:00 /bin/bash root 424 400 0 05:54 ? 00:00:00 ps -ef root 425 400 0 05:54 ? 00:00:00 more Expected results: rabbitmq_init_bundle would complete and launch rabbitmq-bundle-docker Additional info:
@brendan - I think only the 1st line of the warning was captured. Pasting the Warnings with full lines as seen while running 'puppet apply --debug puppet_step_config.pp': Warning: Facter: Could not retrieve fact='rabbitmq_nodename', resolution='<anonymous>': undefined method `[]' for nil:NilClass Warning: Undefined variable '::deploy_config_name'; (file & line not available) Warning: Undefined variable 'deploy_config_name'; (file & line not available) Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Ip_address instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at ["/etc/puppet/modules/tripleo/manifests/profile/base/database/mysql/client.pp", 56]:["/var/lib/tripleo-config/puppet_step_config.pp", 35] (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in `deprecation') Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Bool. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/ntp/manifests/init.pp", 54]:["/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp", 29] (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in `deprecation') Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Absolute_Path. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/ntp/manifests/init.pp", 55]:["/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp", 29] (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in `deprecation') Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::String. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/ntp/manifests/init.pp", 56]:["/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp", 29] (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in `deprecation') Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Array. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/ntp/manifests/init.pp", 66]:["/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp", 29] (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in `deprecation') Warning: This method is deprecated, please use the stdlib validate_legacy function, with Pattern[]. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/ntp/manifests/init.pp", 68]:["/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp", 29] (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in `deprecation') Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Numeric. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/ntp/manifests/init.pp", 89]:["/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp", 29] (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in `deprecation') Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Hash. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/ssh/manifests/server.pp", 12]:["/var/lib/tripleo-config/puppet_step_config.pp", 41] (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in `deprecation')
upgrading to version 1.1.19 (previously 1.1.18) of pacemaker on the controller via the overcloud-full.qcow image that comes via the rhosp-director-images fixed this issue.