Bug 1385678

Summary: rhel-osp-director: Minor update fails: exception: connect failed\n\u001b[1;31mWarning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: puppet-tripleoAssignee: Emilien Macchi <emacchi>
Status: CLOSED ERRATA QA Contact: Alexander Chuzhoy <sasha>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 10.0 (Newton)CC: cpaquin, dbecker, emacchi, jcoufal, jjoyce, jschluet, jslagle, mburns, morazi, rhallise, rhel-osp-director-maint, sclewis, slinaber, tvignaud
Target Milestone: rcKeywords: Triaged
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: puppet-tripleo-5.3.0-1.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-12-14 16:21:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alexander Chuzhoy 2016-10-17 13:29:26 UTC
rhel-osp-director: Minor update fails: exception: connect failed\n\u001b[1;31mWarning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.


Environment:
openstack-puppet-modules-9.0.0-0.20160915155755.8c758d6.el7ost.noarch
openstack-tripleo-heat-templates-5.0.0-0.20161003064637.d636e3a.1.1.el7ost.noarch
instack-undercloud-5.0.0-0.20160930175750.9d2a655.el7ost.noarch

Steps to reproduce:
1. Deploy overcloud with:
openstack overcloud deploy --debug --templates --libvirt-type kvm --ntp-server clock.redhat.com --neutron-network-type vxlan --neutron-tunnel-types vxlan --control-scale 3 --control-flavor controller-d75f3dec-c770-5f88-9d4c-3fea1bf9c484 --compute-scale 1 --compute-flavor compute-b634c10a-570f-59ba-bdbf-0c313d745a10 --ceph-storage-scale 2 --ceph-storage-flavor ceph-cf1f074b-dadb-5eb8-9eb0-55828273fab7 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e virt/ceph.yaml -e virt/hostnames.yml -e virt/network/network-environment.yaml --log-file overcloud_deployment_48.log

2. Update the undercloud.

3. Attempt to update the overcloud with:
yes ""| openstack overcloud update stack  -i overcloud



Result:
The update of overcloud fails:
+-------------------------------------------+----------------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+------------------------------------------------------------------------------------------------------------------------+                                                      
| resource_name                             | physical_resource_id                         | resource_type                                                                                                       | resource_status | updated_time         | stack_name                                                                                                             |                                                      
+-------------------------------------------+----------------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+------------------------------------------------------------------------------------------------------------------------+                                                      
| AllNodesDeploySteps                       | b6c3a711-e9bc-4480-9551-0e1fd2fc883a         | OS::TripleO::PostDeploySteps                                                                                        | UPDATE_FAILED   | 2016-10-17T02:22:16Z | overcloud                                                                                                              |                                                      
| ControllerDeployment_Step2                | efc5f026-a19c-4122-81e8-dfa1da75fe59         | OS::Heat::StructuredDeploymentGroup                                                                                 | UPDATE_FAILED   | 2016-10-17T02:31:04Z | overcloud-AllNodesDeploySteps-xrwscxo3z6w4                                                                             |                                                      
| CephStorageDeployment_Step2               | 11a7540c-8a9f-43ee-942c-107d092afe8d         | OS::Heat::StructuredDeploymentGroup                                                                                 | UPDATE_FAILED   | 2016-10-17T02:31:05Z | overcloud-AllNodesDeploySteps-xrwscxo3z6w4                                                                             |                                                      
| ComputeDeployment_Step2                   | 3f7503c6-85aa-4be7-a944-bcfb97ee336c         | OS::Heat::StructuredDeploymentGroup                                                                                 | UPDATE_FAILED   | 2016-10-17T02:31:05Z | overcloud-AllNodesDeploySteps-xrwscxo3z6w4                                                                             |                                                      
| 0                                         | 064323c6-cd18-4dde-9608-eda53971ac42         | OS::Heat::StructuredDeployment                                                                                      | UPDATE_FAILED   | 2016-10-17T02:31:06Z | overcloud-AllNodesDeploySteps-xrwscxo3z6w4-ComputeDeployment_Step2-iidvnfozw2ir                                        |                                                      
| 2                                         | 5d0dc3ab-424d-4c3c-9400-1fd94be10960         | OS::Heat::StructuredDeployment                                                                                      | UPDATE_FAILED   | 2016-10-17T02:31:06Z | overcloud-AllNodesDeploySteps-xrwscxo3z6w4-ControllerDeployment_Step2-n2wca4prwwsw                                     |                                                      
| 0                                         | 36b51bd3-f0f3-47fc-8d26-fdbfe693ec4b         | OS::Heat::StructuredDeployment                                                                                      | UPDATE_FAILED   | 2016-10-17T02:31:08Z | overcloud-AllNodesDeploySteps-xrwscxo3z6w4-ControllerDeployment_Step2-n2wca4prwwsw                                     |                                                      
| 1                                         | f94bd923-f416-4de2-b49b-5c140aca356d         | OS::Heat::StructuredDeployment                                                                                      | UPDATE_FAILED   | 2016-10-17T02:31:09Z | overcloud-AllNodesDeploySteps-xrwscxo3z6w4-ControllerDeployment_Step2-n2wca4prwwsw                                     |                                                      
+-------------------------------------------+----------------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+------------------------------------------------------------------------------------------------------------------------+        




[stack@undercloud-0 ~]$ heat deployment-show f94bd923-f416-4de2-b49b-5c140aca356d
WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead
{
  "status": "COMPLETE",
  "server_id": "d2280bfa-752c-44bd-b7fa-fb666c9830e0",
  "config_id": "ff21a2cf-d000-452f-a6da-d64d548cc3b0",
  "output_values": {
    "deploy_stdout": "Matching apachectl 'Server version: Apache/2.4.6 (Red Hat Enterprise Linux)\nServer built:   Aug  3 2016 08:33:27'\n\u001b[mNotice: Compiled catalog for controller-1.localdomain in environment production in 13.52 seconds\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceph/Package[ceph]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Swift/Package[swift]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Mongodb::Server::Config/File[/var/lib/mongodb/journal/j._0]/mode: mode changed '0600' to '0755'\u001b[0m\n\u001b[mNotice: /Stage[main]/Mongodb::Server::Config/File[/var/lib/mongodb/journal/lsn]/mode: mode changed '0600' to '0755'\u001b[0m\n\u001b[mNotice: /Stage[main]/Mongodb::Server::Service/Service[mongodb]: Triggered 'refresh' from 1 events\u001b[0m\n\u001b[mNotice: /Stage[main]/Swift::Deps/Anchor[swift::install::end]: Triggered 'refresh' from 1 events\u001b[0m\n\u001b[mNotice: /Stage[main]/Swift::Deps/Anchor[swift::service::begin]: Triggered 'refresh' from 1 events\u001b[0m\n\u001b[mNotice: Finished catalog run in 62.16 seconds\u001b[0m\n",
    "deploy_stderr": "exception: connect failed\n\u001b[1;31mWarning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.\u001b[0m\n\u001b[1;31mWarning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.\u001b[0m\n",
    "deploy_status_code": 0
  },

I see that yum update didn't run on any node.

Comment 2 Ryan Hallisey 2016-10-17 14:39:05 UTC
    "deploy_stdout": "Matching apachectl 'Server version: Apache/2.4.6 (CentOS)\nServer built:   Jul 18 2016 15:30:14'\n\u001b[mNotice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic i
s blocked.\u001b[0m\n\u001b[mNotice: Compiled catalog for overcloud-novacompute-0.localdomain in environment production in 2.06 seconds\u001b[0m\n", 
    "deploy_stderr": "exception: connect failed\n\u001b[1;31mError: Failed to apply catalog: Found 1 dependency cycle:\n(Exec[/usr/bin/systemctl daemon-reload] => Service[iptables] => Class[Firewall::Linux::Redh
at] => Stage[setup] => Stage[main] => Class[Tripleo::Packages] => Exec[package-upgrade] => Service[iptables])\nTry the '--graph' option and opening the resulting '.dot' file in OmniGraffle or GraphViz\u001b[0m\n
", 
    "deploy_status_code": 1

Comment 3 James Slagle 2016-10-17 18:24:48 UTC
i'm thinking this might be fixed by https://review.openstack.org/#/c/386637/ which is not yet downstream

Comment 4 James Slagle 2016-10-17 18:25:59 UTC
emilien, can you confirm comment 3?

Comment 5 Emilien Macchi 2016-10-17 19:44:23 UTC
After discussion with Ryan, we agreed this bug report hide the actual error, which is what Ryan copy pasted in comment #2.
If that's the case, the bug is fixed upstream and backported in stable/newton.

Comment 8 Alexander Chuzhoy 2016-11-08 23:12:35 UTC
Verified:

Environment:
puppet-tripleo-5.3.0-6.el7ost.noarch

The reported issue doesn't reproduce.

Comment 10 errata-xmlrpc 2016-12-14 16:21:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html