Bug 1297002 - a missing storage can cause cfme 5.5 to stop collecting data from openstack
a missing storage can cause cfme 5.5 to stop collecting data from openstack
Status: CLOSED ERRATA
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Providers (Show other bugs)
5.5.0
All All
high Severity high
: GA
: 5.6.0
Assigned To: Ladislav Smola
Ola Pavlenko
openstack
: ZStream
Depends On:
Blocks: 1310115 1348627
  Show dependency treegraph
 
Reported: 2016-01-08 11:56 EST by Felix Dewaleyne
Modified: 2017-08-29 23:46 EDT (History)
10 users (show)

See Also:
Fixed In Version: 5.6.0.4
Doc Type: Bug Fix
Doc Text:
A resource missing from a stack in an OpenStack environment caused CloudForms 5.5 to entirely stop collecting data from OpenStack, instead of issuing a warning about the problematic stack. This occurred when the `stack.resources` call issued an API call to OpenStack but bypassed the 404 handler in CloudForms. The code has been fixed to safely load orchestration stack relations, and this no longer occurs.
Story Points: ---
Clone Of:
: 1310115 1348627 (view as bug list)
Environment:
Last Closed: 2016-06-29 11:26:20 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Felix Dewaleyne 2016-01-08 11:56:15 EST
Description of problem:
a missing storage can cause cfme 5.5 to stop collecting data from openstack

Version-Release number of selected component (if applicable):
5.5.0.13

How reproducible:
too specific to reproduce entirely

Steps to Reproduce:
1. in openstack, fail to remove a stack (must end on an unknown error)
2. in cloudforms issue a refresh
3.

Actual results:
evm log :
[----] E, [2015-12-22T13:36:15.684101 #6625:80398c] ERROR -- : <Fog> excon.error     #<Excon::Errors::NotFound: Expected(200) <=> Actual(404 Not Found)
excon.error.response
  :body          => "404 Not Found\n\nThe resource could not be found.\n\n   "
  :headers       => {
    "Content-Length"         => "52"
    "Content-Type"           => "text/plain; charset=UTF-8"
    "Date"                   => "Tue, 22 Dec 2015 13:39:17 GMT"
    "X-Openstack-Request-Id" => "req-3a18cbfd-820b-4cf6-ad72-cd507ffa0afe"
  }
  :local_address => "172.22.32.103"
  :local_port    => 44299
  :reason_phrase => "Not Found"
  :remote_ip     => "172.22.224.17"
  :status        => 404
  :status_line   => "HTTP/1.1 404 Not Found\r\n"
>

[----] E, [2015-12-22T13:36:15.692418 #6625:80398c] ERROR -- : <Fog> excon.error     #<Excon::Errors::NotFound: Expected(200) <=> Actual(404 Not Found)
excon.error.response
  :body          => "404 Not Found\n\nThe resource could not be found.\n\n   "
  :headers       => {
    "Content-Length"         => "52"
    "Content-Type"           => "text/plain; charset=UTF-8"
    "Date"                   => "Tue, 22 Dec 2015 13:39:17 GMT"
    "X-Openstack-Request-Id" => "req-60dd9b18-599c-428f-b387-4062e2a545c0"
  }
  :local_address => "172.22.32.103"
  :local_port    => 44300
  :reason_phrase => "Not Found"
  :remote_ip     => "172.22.224.17"
  :status        => 404
  :status_line   => "HTTP/1.1 404 Not Found\r\n"
>

[----] E, [2015-12-22T13:36:15.700585 #6625:80398c] ERROR -- : <Fog> excon.error     #<Excon::Errors::NotFound: Expected(200) <=> Actual(404 Not Found)
excon.error.response
  :body          => "404 Not Found\n\nThe resource could not be found.\n\n   "
  :headers       => {
    "Content-Length"         => "52"
    "Content-Type"           => "text/plain; charset=UTF-8"
    "Date"                   => "Tue, 22 Dec 2015 13:39:17 GMT"
    "X-Openstack-Request-Id" => "req-a84d1b46-edfa-4dad-be8b-410b8acbdbd7"
  }
  :local_address => "172.22.32.103"
  :local_port    => 44301
  :reason_phrase => "Not Found"
  :remote_ip     => "172.22.224.17"
  :status        => 404
  :status_line   => "HTTP/1.1 404 Not Found\r\n"
>

[----] D, [2015-12-22T13:36:16.630988 #6625:80398c] DEBUG -- : MIQ(ManageIQ::Providers::Openstack::InfraManager::Refresher#refresh) EMS: [Openstack Beta undercloud], id: [99000000000005] Parsing Openstack inventory...Completed in 104.101
502848 seconds

no stats seem to be captured by openstack,  monitoring of the environment completely stopped


Expected results:
warnings issued but monitoring (capture ofdata for other stacks) is still performed

Additional info:
the customer reported that the removal of the stack that was causing the issue solved the problem but they are concerned that the presence of such an anomaly can stop entirely the capture of data from the openstack environment.
Comment 16 Greg Blomquist 2016-02-18 16:34:51 EST
@Ladas,

I tracked this down to here:

https://github.com/ManageIQ/manageiq/blob/e0ac3b6ee9c9aa1b8c22e79e6470faa89147de71/app/models/manageiq/providers/openstack/refresh_parser_common/orchestration_stacks.rb#L8

The problem with that line is that the `stack.resources` call issues an API call to OpenStack and it bypasses our 404 handler.

You can either update that line to catch the 404, or figure out how to get the `stack.resources` call to route through our OpenStack Handle code.  I'm sure the 404 catch is easier.
Comment 21 CFME Bot 2016-04-26 14:25:56 EDT
New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/575de9f95d9dde8d4ee63e585617550b7d9bee38

commit 575de9f95d9dde8d4ee63e585617550b7d9bee38
Author:     Ladislav Smola <lsmola@redhat.com>
AuthorDate: Mon Feb 22 14:25:28 2016 +0100
Commit:     Ladislav Smola <lsmola@redhat.com>
CommitDate: Thu Apr 21 12:34:20 2016 +0200

    Safely load orchestration stack relations
    
    Safely load orchestration stack relations
    
    Fixes BZ:
    https://bugzilla.redhat.com/show_bug.cgi?id=1297002

 .../openstack/cloud_manager/refresh_parser.rb      |  1 +
 .../refresh_parser_common/helper_methods.rb        | 24 +++++++++++++++++++++-
 .../refresh_parser_common/orchestration_stacks.rb  |  6 +++---
 .../openstack/openstack_handle/handled_list.rb     |  6 ++++++
 4 files changed, 33 insertions(+), 4 deletions(-)
Comment 23 Ronnie Rasouli 2016-06-16 09:09:17 EDT
Issue is not clear please provide steps to reproduce
Comment 32 errata-xmlrpc 2016-06-29 11:26:20 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1348

Note You need to log in before you can comment on or make changes to this bug.