Bug 1297002 - a missing storage can cause cfme 5.5 to stop collecting data from openstack
Summary: a missing storage can cause cfme 5.5 to stop collecting data from openstack
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Providers
Version: 5.5.0
Hardware: All
OS: All
high
high
Target Milestone: GA
: 5.6.0
Assignee: Ladislav Smola
QA Contact: Omri Hochman
URL:
Whiteboard: openstack
Depends On:
Blocks: 1310115 1348627
TreeView+ depends on / blocked
 
Reported: 2016-01-08 16:56 UTC by Felix Dewaleyne
Modified: 2019-12-16 05:15 UTC (History)
10 users (show)

Fixed In Version: 5.6.0.4
Doc Type: Bug Fix
Doc Text:
A resource missing from a stack in an OpenStack environment caused CloudForms 5.5 to entirely stop collecting data from OpenStack, instead of issuing a warning about the problematic stack. This occurred when the `stack.resources` call issued an API call to OpenStack but bypassed the 404 handler in CloudForms. The code has been fixed to safely load orchestration stack relations, and this no longer occurs.
Clone Of:
: 1310115 1348627 (view as bug list)
Environment:
Last Closed: 2016-06-29 15:26:20 UTC
Category: ---
Cloudforms Team: Openstack
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1348 0 normal SHIPPED_LIVE CFME 5.6.0 bug fixes and enhancement update 2016-06-29 18:50:04 UTC

Description Felix Dewaleyne 2016-01-08 16:56:15 UTC
Description of problem:
a missing storage can cause cfme 5.5 to stop collecting data from openstack

Version-Release number of selected component (if applicable):
5.5.0.13

How reproducible:
too specific to reproduce entirely

Steps to Reproduce:
1. in openstack, fail to remove a stack (must end on an unknown error)
2. in cloudforms issue a refresh
3.

Actual results:
evm log :
[----] E, [2015-12-22T13:36:15.684101 #6625:80398c] ERROR -- : <Fog> excon.error     #<Excon::Errors::NotFound: Expected(200) <=> Actual(404 Not Found)
excon.error.response
  :body          => "404 Not Found\n\nThe resource could not be found.\n\n   "
  :headers       => {
    "Content-Length"         => "52"
    "Content-Type"           => "text/plain; charset=UTF-8"
    "Date"                   => "Tue, 22 Dec 2015 13:39:17 GMT"
    "X-Openstack-Request-Id" => "req-3a18cbfd-820b-4cf6-ad72-cd507ffa0afe"
  }
  :local_address => "172.22.32.103"
  :local_port    => 44299
  :reason_phrase => "Not Found"
  :remote_ip     => "172.22.224.17"
  :status        => 404
  :status_line   => "HTTP/1.1 404 Not Found\r\n"
>

[----] E, [2015-12-22T13:36:15.692418 #6625:80398c] ERROR -- : <Fog> excon.error     #<Excon::Errors::NotFound: Expected(200) <=> Actual(404 Not Found)
excon.error.response
  :body          => "404 Not Found\n\nThe resource could not be found.\n\n   "
  :headers       => {
    "Content-Length"         => "52"
    "Content-Type"           => "text/plain; charset=UTF-8"
    "Date"                   => "Tue, 22 Dec 2015 13:39:17 GMT"
    "X-Openstack-Request-Id" => "req-60dd9b18-599c-428f-b387-4062e2a545c0"
  }
  :local_address => "172.22.32.103"
  :local_port    => 44300
  :reason_phrase => "Not Found"
  :remote_ip     => "172.22.224.17"
  :status        => 404
  :status_line   => "HTTP/1.1 404 Not Found\r\n"
>

[----] E, [2015-12-22T13:36:15.700585 #6625:80398c] ERROR -- : <Fog> excon.error     #<Excon::Errors::NotFound: Expected(200) <=> Actual(404 Not Found)
excon.error.response
  :body          => "404 Not Found\n\nThe resource could not be found.\n\n   "
  :headers       => {
    "Content-Length"         => "52"
    "Content-Type"           => "text/plain; charset=UTF-8"
    "Date"                   => "Tue, 22 Dec 2015 13:39:17 GMT"
    "X-Openstack-Request-Id" => "req-a84d1b46-edfa-4dad-be8b-410b8acbdbd7"
  }
  :local_address => "172.22.32.103"
  :local_port    => 44301
  :reason_phrase => "Not Found"
  :remote_ip     => "172.22.224.17"
  :status        => 404
  :status_line   => "HTTP/1.1 404 Not Found\r\n"
>

[----] D, [2015-12-22T13:36:16.630988 #6625:80398c] DEBUG -- : MIQ(ManageIQ::Providers::Openstack::InfraManager::Refresher#refresh) EMS: [Openstack Beta undercloud], id: [99000000000005] Parsing Openstack inventory...Completed in 104.101
502848 seconds

no stats seem to be captured by openstack,  monitoring of the environment completely stopped


Expected results:
warnings issued but monitoring (capture ofdata for other stacks) is still performed

Additional info:
the customer reported that the removal of the stack that was causing the issue solved the problem but they are concerned that the presence of such an anomaly can stop entirely the capture of data from the openstack environment.

Comment 16 Greg Blomquist 2016-02-18 21:34:51 UTC
@Ladas,

I tracked this down to here:

https://github.com/ManageIQ/manageiq/blob/e0ac3b6ee9c9aa1b8c22e79e6470faa89147de71/app/models/manageiq/providers/openstack/refresh_parser_common/orchestration_stacks.rb#L8

The problem with that line is that the `stack.resources` call issues an API call to OpenStack and it bypasses our 404 handler.

You can either update that line to catch the 404, or figure out how to get the `stack.resources` call to route through our OpenStack Handle code.  I'm sure the 404 catch is easier.

Comment 21 CFME Bot 2016-04-26 18:25:56 UTC
New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/575de9f95d9dde8d4ee63e585617550b7d9bee38

commit 575de9f95d9dde8d4ee63e585617550b7d9bee38
Author:     Ladislav Smola <lsmola>
AuthorDate: Mon Feb 22 14:25:28 2016 +0100
Commit:     Ladislav Smola <lsmola>
CommitDate: Thu Apr 21 12:34:20 2016 +0200

    Safely load orchestration stack relations
    
    Safely load orchestration stack relations
    
    Fixes BZ:
    https://bugzilla.redhat.com/show_bug.cgi?id=1297002

 .../openstack/cloud_manager/refresh_parser.rb      |  1 +
 .../refresh_parser_common/helper_methods.rb        | 24 +++++++++++++++++++++-
 .../refresh_parser_common/orchestration_stacks.rb  |  6 +++---
 .../openstack/openstack_handle/handled_list.rb     |  6 ++++++
 4 files changed, 33 insertions(+), 4 deletions(-)

Comment 23 Ronnie Rasouli 2016-06-16 13:09:17 UTC
Issue is not clear please provide steps to reproduce

Comment 32 errata-xmlrpc 2016-06-29 15:26:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1348


Note You need to log in before you can comment on or make changes to this bug.