Rubygem-Staypuft: HA-neutron deployment gets paused with errors installing the third controller (2 have passed). Execution of '/usr/bin/systemctl start mariadb' returned 1: Job for mariadb.service failed Environment: rhel-osp-installer-0.1.9-1.el6ost.noarch openstack-foreman-installer-2.0.18-1.el6ost.noarch ruby193-rubygem-foreman_openstack_simplify-0.0.6-8.el6ost.noarch openstack-puppet-modules-2014.1-19.9.el6ost.noarch Steps to reproduce: 1. Install rhel-osp-installer 2. Configure/run an HA-neutron deployment (3 controllers + 2 computes) 3. in Advanced params, set the private_network to 192.168.0.0 (the address of the PXE network). Result: The deployment gets paused with error installing the third controller, while 2 other have passed. Note: Running the puppet agent manually on that controller finished with no error. Resuming the deployment afterwards results in successful deployment. Expected result: The deployment should complete successfully with no intervention.
Created attachment 925843 [details] /var/log/messages file from the controller that had the pupper errors.
Created attachment 925844 [details] /var/log/mariadb/mariadb.log file from the controller that had the pupper errors.
The yaml for the failed controller: --- classes: quickstack::openstack_common: quickstack::pacemaker::cinder: backend_eqlx: 'false' backend_eqlx_name: - eqlx_backend backend_glusterfs: false backend_glusterfs_name: glusterfs_backend backend_iscsi: 'false' backend_iscsi_name: iscsi_backend backend_nfs: 'true' backend_nfs_name: nfs_backend backend_rbd: 'false' backend_rbd_name: rbd_backend db_name: cinder db_ssl: false db_ssl_ca: '' db_user: cinder debug: false enabled: true eqlx_chap_login: - '' eqlx_chap_password: - '' eqlx_group_name: - group-0 eqlx_pool: - default eqlx_use_chap: - 'false' glusterfs_shares: [] log_facility: LOG_USER multiple_backends: false nfs_mount_options: nosharecache nfs_shares: - 192.168.0.1:/cinder qpid_heartbeat: '60' rbd_ceph_conf: /etc/ceph/ceph.conf rbd_flatten_volume_from_snapshot: 'false' rbd_max_clone_depth: '5' rbd_pool: volumes rbd_secret_uuid: 7fa6d270-5742-4bc1-ac68-0b17dd003040 rbd_user: volumes rpc_backend: cinder.openstack.common.rpc.impl_kombu san_ip: - '' san_login: - grpadmin san_password: - '' san_thin_provision: - 'false' use_syslog: false verbose: 'true' volume: true quickstack::pacemaker::common: fence_ipmilan_address: '' fence_ipmilan_expose_lanplus: 'true' fence_ipmilan_hostlist: '' fence_ipmilan_host_to_address: [] fence_ipmilan_interval: 60s fence_ipmilan_lanplus_options: '' fence_ipmilan_password: '' fence_ipmilan_username: '' fence_xvm_clu_iface: eth2 fence_xvm_clu_network: '' fence_xvm_key_file_password: '' fence_xvm_manage_key_file: 'false' fencing_type: disabled pacemaker_cluster_members: 192.168.0.8 192.168.0.10 192.168.0.11 pacemaker_cluster_name: openstack quickstack::pacemaker::galera: galera_monitor_password: monitor_pass galera_monitor_username: monitor_user mysql_root_password: 1ee3f43f37d5f76848595099681f68d9 wsrep_cluster_members: - 192.168.0.8 - 192.168.0.10 - 192.168.0.11 wsrep_cluster_name: galera_cluster wsrep_ssl: false wsrep_ssl_cert: /etc/pki/galera/galera.crt wsrep_ssl_key: /etc/pki/galera/galera.key wsrep_sst_method: rsync wsrep_sst_password: sst_pass wsrep_sst_username: sst_user quickstack::pacemaker::glance: backend: file db_name: glance db_ssl: false db_ssl_ca: '' db_user: glance debug: false enabled: true filesystem_store_datadir: /var/lib/glance/images/ log_facility: LOG_USER pcmk_fs_device: 192.168.0.1:/cinder pcmk_fs_dir: /var/lib/glance/images pcmk_fs_manage: 'true' pcmk_fs_options: nosharecache,context=\"system_u:object_r:glance_var_lib_t:s0\" pcmk_fs_type: nfs pcmk_swift_is_local: true rbd_store_pool: images rbd_store_user: images sql_idle_timeout: '3600' swift_store_auth_address: http://127.0.0.1:5000/v2.0/ swift_store_key: '' swift_store_user: '' use_syslog: false verbose: 'true' quickstack::pacemaker::heat: db_name: heat db_ssl: false db_ssl_ca: '' db_user: heat debug: false enabled: true log_facility: LOG_USER qpid_heartbeat: '60' use_syslog: false verbose: 'true' quickstack::pacemaker::horizon: horizon_ca: /etc/ipa/ca.crt horizon_cert: /etc/pki/tls/certs/PUB_HOST-horizon.crt horizon_key: /etc/pki/tls/private/PUB_HOST-horizon.key keystone_default_role: _member_ memcached_port: '11211' secret_key: 0702bc7e913a64258c190aa291077837 verbose: 'true' quickstack::pacemaker::keystone: admin_email: admin admin_password: 0164bce77dcb4ff7524e9cf43d9915d8 admin_tenant: admin admin_token: 884b4fe2d325d2941f5961435d0568d2 ceilometer: 'true' cinder: 'true' db_name: keystone db_ssl: 'false' db_ssl_ca: '' db_type: mysql db_user: keystone debug: 'false' enabled: 'true' glance: 'true' heat: 'true' heat_cfn: 'false' idle_timeout: '200' keystonerc: 'true' log_facility: LOG_USER nova: 'true' public_protocol: http region: RegionOne swift: 'false' token_driver: keystone.token.backends.sql.Token token_format: PKI use_syslog: 'false' verbose: 'true' quickstack::pacemaker::load_balancer: quickstack::pacemaker::memcached: quickstack::pacemaker::neutron: core_plugin: neutron.plugins.ml2.plugin.Ml2Plugin enabled: true enable_tunneling: 'true' external_network_bridge: br-ex ml2_flat_networks: - ! '*' ml2_mechanism_drivers: - openvswitch - l2population ml2_network_vlan_ranges: - physnet-external ml2_security_group: 'True' ml2_tenant_network_types: - vxlan ml2_tunnel_id_ranges: - 10:1000 ml2_type_drivers: - local - flat - vlan - gre - vxlan ml2_vxlan_group: 224.0.0.1 ovs_bridge_mappings: - physnet-external:br-ex ovs_bridge_uplinks: - br-ex:ens8 ovs_tunnel_iface: ens7 ovs_tunnel_network: '' ovs_tunnel_types: - vxlan - gre ovs_vlan_ranges: - physnet-external ovs_vxlan_udp_port: '4789' tenant_network_type: vlan tunnel_id_ranges: 1:1000 verbose: 'true' quickstack::pacemaker::nova: auto_assign_floating_ip: 'true' db_name: nova db_user: nova default_floating_pool: nova force_dhcp_release: 'false' image_service: nova.image.glance.GlanceImageService memcached_port: '11211' multi_host: 'true' neutron_metadata_proxy_secret: 8d89120a015781f3fa793cbe49c0b28b qpid_heartbeat: '60' rpc_backend: nova.openstack.common.rpc.impl_kombu scheduler_host_subset_size: '30' verbose: 'true' quickstack::pacemaker::params: amqp_group: amqp amqp_password: 27cec767f1845e46996143779c4e3cb9 amqp_port: '5672' amqp_provider: rabbitmq amqp_username: openstack amqp_vip: 192.168.0.99 ceilometer_admin_vip: 192.168.0.88 ceilometer_group: ceilometer ceilometer_private_vip: 192.168.0.88 ceilometer_public_vip: 192.168.0.88 ceilometer_user_password: 47c3f996059b9896aec51456c3c9646e cinder_admin_vip: 192.168.0.89 cinder_db_password: b2509f3bb880a3ce4bf54010c2b98b4e cinder_group: cinder cinder_private_vip: 192.168.0.89 cinder_public_vip: 192.168.0.89 cinder_user_password: 6654863e83a6ce9bd01b75a4f4ed3cb5 cluster_control_ip: 192.168.0.8 db_group: db db_vip: 192.168.0.90 glance_admin_vip: 192.168.0.91 glance_db_password: 7de2b3ba48b3bf9a287030c5602eb900 glance_group: glance glance_private_vip: 192.168.0.91 glance_public_vip: 192.168.0.91 glance_user_password: b7054bc8bf248aa06383c79d1e7ec6ee heat_admin_vip: 192.168.0.92 heat_auth_encryption_key: a32c809f960cea9b3e76309060532493 heat_cfn_admin_vip: 192.168.0.93 heat_cfn_enabled: 'true' heat_cfn_group: heat_cfn heat_cfn_private_vip: 192.168.0.93 heat_cfn_public_vip: 192.168.0.93 heat_cfn_user_password: 0579cc2f023286f5b39aacb8b034b1dc heat_cloudwatch_enabled: 'true' heat_db_password: 19e847f1c8a2f8b4fcc9ec399a9fe073 heat_group: heat heat_private_vip: 192.168.0.92 heat_public_vip: 192.168.0.92 heat_user_password: 6667b9a8ae5d0ae58f6d1ff99f7d43b8 horizon_admin_vip: 192.168.0.94 horizon_group: horizon horizon_private_vip: 192.168.0.94 horizon_public_vip: 192.168.0.94 include_amqp: 'true' include_cinder: 'true' include_glance: 'true' include_heat: 'true' include_horizon: 'true' include_keystone: 'true' include_mysql: 'true' include_neutron: 'true' include_nova: 'true' include_swift: 'false' keystone_admin_vip: 192.168.0.95 keystone_db_password: 7ea9ddade46825e50f2e6401fd8a5163 keystone_group: keystone keystone_private_vip: 192.168.0.95 keystone_public_vip: 192.168.0.95 keystone_user_password: 660d8620c76a70cdc104148f06463bcf lb_backend_server_addrs: - 192.168.0.8 - 192.168.0.10 - 192.168.0.11 lb_backend_server_names: - maca25400702876.example.com - maca25400702875.example.com - maca25400702877.example.com loadbalancer_group: loadbalancer loadbalancer_vip: 192.168.0.96 neutron: 'true' neutron_admin_vip: 192.168.0.98 neutron_db_password: f56c8fb8254a63cb10b15b013590dfe8 neutron_group: neutron neutron_metadata_proxy_secret: 8d89120a015781f3fa793cbe49c0b28b neutron_private_vip: 192.168.0.98 neutron_public_vip: 192.168.0.98 neutron_user_password: db83b3dbe314fb4d5e8a75ee1ada4955 nova_admin_vip: 192.168.0.97 nova_db_password: 554bdcba009e0978157ec4da8f06b13d nova_group: nova nova_private_vip: 192.168.0.97 nova_public_vip: 192.168.0.97 nova_user_password: a614ab98718f42fb9cfde6b676d5bc19 private_iface: '' private_ip: 10.8.29.147 private_network: 192.168.0.0 swift_group: swift swift_public_vip: 192.168.0.100 swift_user_password: '' quickstack::pacemaker::qpid: backend_port: '15672' config_file: /etc/qpidd.conf connection_backlog: '65535' haproxy_timeout: 120s log_to_file: UNSET manage_service: true max_connections: '65535' package_ensure: present package_name: qpid-cpp-server realm: QPID service_enable: true service_ensure: running service_name: qpidd worker_threads: '17' quickstack::pacemaker::swift: memcached_port: '11211' swift_internal_vip: 192.168.0.100 swift_shared_secret: faffe511acf79d72b6438920b8790863 swift_storage_device: '' swift_storage_ips: [] parameters: puppetmaster: staypuft.example.com domainname: Default domain used for provisioning hostgroup: base_RedHat_7/HA-neutron/HA Controller root_pw: $5$fm$EO4A.ybSB/ofUaZWkzNePd38XRpUQNXls8y1feWvIy3 puppet_ca: staypuft.example.com foreman_env: production owner_name: Admin User owner_email: root ntp-server: clock.redhat.com ui::cinder::driver_backend: nfs ui::cinder::eqlx_group_name: group-0 ui::cinder::eqlx_pool: default ui::cinder::nfs_uri: 192.168.0.1:/cinder ui::cinder::rbd_secret_uuid: 7fa6d270-5742-4bc1-ac68-0b17dd003040 ui::cinder::san_login: grpadmin ui::deployment::amqp_provider: rabbitmq ui::deployment::layout_name: High Availability Controllers / Compute ui::deployment::networking: neutron ui::deployment::platform: rhel7 ui::glance::driver_backend: nfs ui::glance::nfs_network_path: 192.168.0.1:/cinder ui::neutron::compute_tenant_interface: ens7 ui::neutron::external_interface_name: ens8 ui::neutron::networker_tenant_interface: ens7 ui::neutron::network_segmentation: vxlan ui::neutron::use_external_interface: 'true' ui::nova::network_manager: FlatDHCPManager ui::passwords::admin: 0164bce77dcb4ff7524e9cf43d9915d8 ui::passwords::amqp: 27cec767f1845e46996143779c4e3cb9 ui::passwords::amqp_nssdb: bb1755ade38caa4dd9a072787488f88d ui::passwords::ceilometer_metering_secret: a55ba849858684bd3c21b423c549ff92 ui::passwords::ceilometer_user: 47c3f996059b9896aec51456c3c9646e ui::passwords::cinder_db: b2509f3bb880a3ce4bf54010c2b98b4e ui::passwords::cinder_user: 6654863e83a6ce9bd01b75a4f4ed3cb5 ui::passwords::glance_db: 7de2b3ba48b3bf9a287030c5602eb900 ui::passwords::glance_user: b7054bc8bf248aa06383c79d1e7ec6ee ui::passwords::heat_auth_encrypt_key: a32c809f960cea9b3e76309060532493 ui::passwords::heat_cfn_user: 0579cc2f023286f5b39aacb8b034b1dc ui::passwords::heat_db: 19e847f1c8a2f8b4fcc9ec399a9fe073 ui::passwords::heat_user: 6667b9a8ae5d0ae58f6d1ff99f7d43b8 ui::passwords::horizon_secret_key: 0702bc7e913a64258c190aa291077837 ui::passwords::keystone_admin_token: 884b4fe2d325d2941f5961435d0568d2 ui::passwords::keystone_db: 7ea9ddade46825e50f2e6401fd8a5163 ui::passwords::keystone_user: 660d8620c76a70cdc104148f06463bcf ui::passwords::mode: random ui::passwords::mysql_root: 1ee3f43f37d5f76848595099681f68d9 ui::passwords::neutron_db: f56c8fb8254a63cb10b15b013590dfe8 ui::passwords::neutron_metadata_proxy_secret: 8d89120a015781f3fa793cbe49c0b28b ui::passwords::neutron_user: db83b3dbe314fb4d5e8a75ee1ada4955 ui::passwords::nova_db: 554bdcba009e0978157ec4da8f06b13d ui::passwords::nova_user: a614ab98718f42fb9cfde6b676d5bc19 ui::passwords::swift_admin: 3a7d0dcd78ba161dfbabca82cc77b636 ui::passwords::swift_shared_secret: faffe511acf79d72b6438920b8790863 ui::passwords::swift_user: b0535d312085ab6d60fff6bc843b9bd7 environment: production
Trying to piece together what happened here. The first puppet error messages is for heat: Aug 11 16:39:25 maca25400702876 start-puppet-agent: Error: unable to find group or resource: openstack-heat-api-cfn Aug 11 16:39:25 maca25400702876 puppet-agent[3364]: (/Stage[main]/Quickstack::Pacemaker::Heat/Quickstack::Pacemaker::Resource::Service[openstack-heat-api-cfn]/Pacemaker::Resource::Service[openstack-heat-api-cfn]/Pacemaker::Resource::Systemd[openstack-heat-api-cfn]/Pcmk_resource[openstack-heat-api-cfn]/ensure) created Aug 11 16:39:26 maca25400702876 puppet-agent[3364]: (/Stage[main]/Quickstack::Pacemaker::Heat/Quickstack::Pacemaker::Constraint::Base[heat-api-cfn-constr]/Pacemaker::Constraint::Base[heat-api-cfn-constr]/Exec[Creating order constraint heat-api-cfn-constr]/returns) Error: Resource 'openstack-heat-api-cfn-clone' does not exist Where did you see "Execution of '/usr/bin/systemctl start mariadb' returned 1: Job for mariadb.service failed" ? It is not in /var/log/messages. Is there a timestamp associated with the message?
I saw this message in the puppet report for the controller, where I had to manually re-run the puppet. Could not start Service[galera]: Execution of '/usr/bin/systemctl start mariadb' returned 1: Job for mariadb.service failed. See 'systemctl status mariadb.service' and 'journalctl -xn' for details. Wrapped exception: Execution of '/usr/bin/systemctl start mariadb' returned 1: Job for mariadb.service failed. See 'systemctl status mariadb.service' and 'journalctl -xn' for details.
Created attachment 925862 [details] /var/log/messages file from another controler that was managed to complete the deployment on its own.
Created attachment 925864 [details] /var/log/messages file from the third controler that was managed to complete the deployment on its own.
The first puppet error in the attachment 925862 [details] is: Aug 11 16:32:40 maca25400702876 Filesystem(fs-varlibglanceimages)[21759]: ERROR: Couldn't mount filesystem 192.168.0.1:/cinder on /var/lib/glance/images This implies the glance-related parameter, pcmk_fs_options, is incorrectly set to 192.168.0.1:/cinder . This error also occurs in the other attachment. I'm thinking the galera error reported in this BZ when re-running puppet by hand may be due to https://bugzilla.redhat.com/show_bug.cgi?id=1123312 , so I think we should wait until a fix is in for that to see if it addresses the issue. However, I'm concerned about the other errors that occurred so that none of the puppet runs completed successfully (cinder and heat).
openstack-foreman-installer-2.0.21-1.el6ost No galera errors. mariadb started on all nodes successfully.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2014-1090.html