Bug 1425947

Summary: 404 Not Found Errors in evm.log during Openstack CloudManager::Vm#perf_capture
Product: Red Hat CloudForms Management Engine Reporter: Dustin Scott <dscott>
Component: C&U Capacity and UtilizationAssignee: Marek Aufart <maufart>
Status: CLOSED WONTFIX QA Contact: Omri Hochman <ohochman>
Severity: medium Docs Contact:
Priority: medium    
Version: 5.7.0CC: abraverm, dscott, jhardy, maufart, obarenbo
Target Milestone: GA   
Target Release: cfme-future   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: openstack:c&u
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: Openstack Target Upstream Version:
Embargoed:

Description Dustin Scott 2017-02-22 20:31:30 UTC
Description of problem: Lots of error messages in /var/www/miq/vmdb/log/evm.log that look like this:

[----] I, [2017-02-22T15:05:27.671517 #7129:69913c]  INFO -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::Vm#perf_capture) [realtime] Capture for ManageIQ::Providers::Openstack::CloudManager::Vm name: [test], id: [11000000000002]...
[----] I, [2017-02-22T15:05:27.999618 #15587:879880]  INFO -- : Querying Openstack Ceilometer for events newer than 2017-02-22T11:15:21.574284...
[fog][WARNING] Unrecognized arguments: provider
[----] E, [2017-02-22T15:05:28.861821 #7129:69913c] ERROR -- : <Fog> excon.error     #<Excon::Error::NotFound: Expected(200) <=> Actual(404 Not Found)
excon.error.response
  :body          => "{\"code\": 404, \"description\": \"Resource 331efb38-2ca1-4330-99b7-d374baae5d5c does not exist\", \"title\": \"Not Found\"}"
  :cookies       => [
  ]
  :headers       => {
    "Connection"     => "close"
    "Content-Length" => "114"
    "Content-Type"   => "application/json; charset=UTF-8"
    "Date"           => "Wed, 22 Feb 2017 20:05:29 GMT"
    "Server"         => "Apache/2.4.6 (Red Hat Enterprise Linux)"
  }
  :host          => "myospems.exapmle.com"
  :local_address => "10.10.10.71"
  :local_port    => 46082
  :path          => "/v1/resource/generic/331efb38-2ca1-4330-99b7-d374baae5d5c"
  :port          => 13041
  :reason_phrase => "Not Found"
  :remote_ip     => "10.10.10.10"
  :status        => 404
  :status_line   => "HTTP/1.1 404 Not Found\r\n"
>

[----] W, [2017-02-22T15:05:28.862158 #7129:69913c]  WARN -- : OpenstackHandle::MetricDelegate list_meters filter:{"field"=>"resource_id", "value"=>"331efb38-2ca1-4330-99b7-d374baae5d5c"} Resource does not exist.


A curl confirms what we are seeing in the logs:

[root@cfme-01 vmdb]# curl -H 'X-Auth-Token:XXXXXXMYAUTHTOKEN' -X GET https://10.10.10.71:13041/v1/resource/generic/331efb38-2ca1-4330-99b7-d374baae5d5c -k
<html>
 <head>
  <title>404 Not Found</title>
 </head>
 <body>
  <h1>404 Not Found</h1>
  The resource could not be found.<br /><br />
Resource a187e624-c475-4613-b33f-4eea973c281b does not exist


 </body>



Version-Release number of selected component (if applicable):

[root@vlp-cldfrmwkr02 vmdb]# rpm -qa | grep cfme
cfme-5.7.0.17-1.el7cf.x86_64
cfme-gemset-5.7.0.17-1.el7cf.x86_64
cfme-appliance-5.7.0.17-1.el7cf.x86_64
rh-ruby23-rubygem-redhat_access_cfme-1.1.0-1.el7cf.noarch

Connecting to OSP 9

How reproducible:

100%


Steps to Reproduce:
1. Configure OSP Provider
2. Monitor evm.log for errors
3.

Actual results:

404 errors in evm.log


Expected results:

No 404 errors in evm.log when running ManageIQ::Providers::Openstack::CloudManager::Vm#perf_capture method

Additional info:

Comment 2 Dustin Scott 2017-02-22 20:35:12 UTC
Mixed up the :host, :local_address, and :remote_ip when sanitizing customer logs.

Comment 3 Tzu-Mainn Chen 2017-02-27 17:49:45 UTC
In order to investigate this properly, is it possible to get an idea of how this error surfaces outside of the log - failed refresh or events capture... ?

Comment 4 Dustin Scott 2017-03-07 14:03:37 UTC
The above logs seem to indicate that this happened during a performance capture (C&U collection possibly????), however, I am seeing much more of this in the logs now, which looks to be happening during an EMS Refresh:

[----] I, [2017-03-07T04:01:44.786279 #3080:c9b134]  INFO -- : MIQ(ManageIQ::Providers::Openstack::InfraManager::Refresher#refresh) Refreshing all targets...
[----] I, [2017-03-07T04:01:44.786398 #3080:c9b134]  INFO -- : MIQ(ManageIQ::Providers::Openstack::InfraManager::Refresher#refresh) EMS: [vlp-ospdirect02], id: [11000000000003] Refreshing targets for EMS...
[----] I, [2017-03-07T04:01:44.786502 #3080:c9b134]  INFO -- : MIQ(ManageIQ::Providers::Openstack::InfraManager::Refresher#refresh) EMS: [vlp-ospdirect02], id: [11000000000003]   ManageIQ::Providers::Openstack::InfraManager [direct02] id [11000000000003]
[----] I, [2017-03-07T04:01:44.786623 #3080:c9b134]  INFO -- : MIQ(ManageIQ::Providers::Openstack::InfraManager::Refresher#refresh_targets_for_ems) EMS: [direct02], id: [11000000000003] Refreshing target ManageIQ::Providers::Openstack::InfraManager [direct02] id [11000000000003]...
[----] E, [2017-03-07T04:01:47.274424 #3072:c9b134] ERROR -- : <Fog> excon.error     #<Excon::Error::NotFound: Expected(200) <=> Actual(404 Not Found)
excon.error.response
  :body          => "404 Not Found\n\nThe resource could not be found.\n\n   "
  :cookies       => [
  ]
  :headers       => {
    "Content-Length"         => "52"
    "Content-Type"           => "text/plain; charset=UTF-8"
    "Date"                   => "Tue, 07 Mar 2017 09:01:28 GMT"
    "X-Openstack-Request-Id" => "req-193ba00d-52b0-4c4f-9d4b-9ba5f486f87b"
  }
  :host          => "10.10.10.5"
  :local_address => "10.10.10.71"
  :local_port    => 52866
  :path          => "/v2.0/routers"
  :port          => 9696
  :reason_phrase => "Not Found"
  :remote_ip     => "10.10.10.5"
  :status        => 404
  :status_line   => "HTTP/1.1 404 Not Found\r\n"
>

[----] E, [2017-03-07T04:01:48.133105 #3072:c9b134] ERROR -- : <Fog> excon.error     #<Excon::Error::NotFound: Expected(200) <=> Actual(404 Not Found)
excon.error.response
  :body          => "404 Not Found\n\nThe resource could not be found.\n\n   "
  :cookies       => [
  ]
  :headers       => {
    "Content-Length"         => "52"
    "Content-Type"           => "text/plain; charset=UTF-8"
    "Date"                   => "Tue, 07 Mar 2017 09:01:29 GMT"
    "X-Openstack-Request-Id" => "req-173cb267-b2b6-4df2-8aca-6badd01b285d"
  }
  :host          => "10.10.10.5"
  :local_address => "10.10.10.71"
  :local_port    => 52894
  :path          => "/v2.0/floatingips"
  :port          => 9696
  :reason_phrase => "Not Found"
  :remote_ip     => "10.10.10.5"
  :status        => 404
  :status_line   => "HTTP/1.1 404 Not Found\r\n"
>

[----] I, [2017-03-07T04:01:48.134196 #3072:c9b134]  INFO -- : MIQ(EmsRefresh.save_ems_network_inventory) EMS: [direct02 Network Manager], id: [11000000000004] Saving EMS Network Inventory...

Comment 6 Josh Carter 2018-10-02 14:32:13 UTC
Dear customer, 

The CloudForms team is reviewing the current CloudForms Bug(defect) backlog in order to target engineering efforts. We are closing any bugs for versions that no longer have an active errata stream or that have hit their age limit. We are committing to better management of the backlog as we move forward. If you have an bug that you are still able to reproduce on a current version of CloudForms please open a new bug. 

If you have any concerns about this, please let us know.

Thanks and regards!

Comment 7 Josh Carter 2018-10-02 14:34:34 UTC
Dear customer, 

The CloudForms team is reviewing the current CloudForms Bug(defect) backlog in order to target engineering efforts. We are closing any bugs for versions that no longer have an active errata stream or that have hit their age limit. We are committing to better management of the backlog as we move forward. If you have an bug that you are still able to reproduce on a current version of CloudForms please open a new bug. 

If you have any concerns about this, please let us know.

Thanks and regards!