Bug 1365560

Summary: Connection to Ceilometer fails in fog/openstack
Product: Red Hat CloudForms Management Engine Reporter: Marek Aufart <maufart>
Component: ProvidersAssignee: Marek Aufart <maufart>
Status: CLOSED CURRENTRELEASE QA Contact: Ola Pavlenko <opavlenk>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 5.6.0CC: benglish, cpelland, ealcaniz, gblomqui, jdeubel, jfrey, jhardy, kmorey, ldomb, maufart, nachandr, nstephan, obarenbo, opavlenk, rrasouli, simaishi, vestival
Target Milestone: GAKeywords: TestOnly, ZStream
Target Release: 5.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: openstack:c&u
Fixed In Version: 5.7.0.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1365907 (view as bug list) Environment:
Last Closed: 2017-01-11 19:49:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: Openstack Target Upstream Version:
Embargoed:
Bug Depends On: 1384506    
Bug Blocks: 1337429, 1337676, 1364584, 1365907    

Description Marek Aufart 2016-08-09 14:30:20 UTC
Description of problem: Ceilometer related features do not work because connection to Ceilometer fails in fog. Details at [1]

Version-Release number of selected component (if applicable): 5.6.1 and current master

How reproducible: always

Steps to Reproduce:
1. Create Openstack cloud provider with Ceilometer events
2. Do some action like Instance provision
3. Check events on provider 

Actual results:
 - no EMS events are present
 - evm.log contains "Ceilometer ... no implicit conversion of String into Integer"
 - metrics are not loaded

Expected results:
 - EMS events should be displayed
 - evm.log should contain repeated Querying Ceilometer for events...
 - metric should be available after some time


Additional info: Issue looks to be caused by fog/openstack in method which choose endpoint version. The root cause is unknown (some hash or json lib change or patch IMO), but it can be fixed in fog/openstack by more robust data handling.  

[1] http://paste.openstack.org/show/552424/

Comment 2 Marek Aufart 2016-08-09 14:33:16 UTC
Fog/openstack fix https://github.com/fog/fog-openstack/pull/173

Comment 3 Greg Blomquist 2016-08-09 21:07:06 UTC
*** Bug 1365337 has been marked as a duplicate of this bug. ***

Comment 4 Greg Blomquist 2016-08-09 21:53:17 UTC
Moving back to ON-DEV because there's a MIQ piece that needs to be done here too.  Need to bump the version of the fog-openstack gem in the MIQ Gemfile.

Comment 5 Marek Aufart 2016-08-11 07:48:59 UTC
ManageIQ PR https://github.com/ManageIQ/manageiq/pull/10363 was merged.

Comment 6 Edu Alcaniz 2016-09-14 08:28:51 UTC
(In reply to Marek Aufart from comment #2)
> Fog/openstack fix https://github.com/fog/fog-openstack/pull/173

Hi, is it possible to this resolved?

Comment 9 Marek Aufart 2016-10-13 14:46:11 UTC
I don't understand how this BZ is blocked by 1384506. This BZ covers issue with connection to Ceilometer. Ceilometer events can be initiated by several actions, not only instance provision. Also issue from this BZ raised error in evm.log immediately after Ceilometer query attempt.

Comment 10 Colin Arnott 2016-10-13 19:25:05 UTC
My error looks like the following, however these loglines are from a dupe 1365337, not my appliance:

[----] E, [2016-08-09T13:41:02.553116 #17085:116b998] ERROR -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::MetricsCapture#perf_collect_metrics) [realtime] for: [ManageIQ::Providers::Openstack::CloudManager::Vm], [28], [nachandr-cfme56]   Timings at time of error: {:server_dequeue=>0.004283905029296875, :server_monitor=>0.17510271072387695, :capture_state=>0.3367500305175781, :connect=>0.8643429279327393}
[----] E, [2016-08-09T13:41:02.553481 #17085:116b998] ERROR -- : [Excon::Error::NotAcceptable]: Expected([200, 204, 300]) <=> Actual(406 Not Acceptable)
excon.error.response
  :body          => "{\"error_message\": \"406 Not Acceptable\\n\\nThe server could not comply with the request since it is either malformed or otherwise incorrect.\\n\\n   \"}"
  :cookies       => [
  ]
  :headers       => {
    "Content-Length" => "147"
    "Content-Type"   => "application/json"
    "Date"           => "Tue, 09 Aug 2016 17:37:55 GMT"
    "Server"         => "WSGIServer/0.1 Python/2.7.5"
  }
  :host          => "xx.yy.zz.aa"
  :local_address => "192.168.aa.bb"
  :local_port    => 36942
  :path          => "/"
  :port          => 8777
  :reason_phrase => "Not Acceptable"
  :remote_ip     => "10.8.58.50"
  :status        => 406
  :status_line   => "HTTP/1.0 406 Not Acceptable\r\n"
  Method:[rescue in perf_collect_metrics]
[----] E, [2016-08-09T13:41:02.553669 #17085:116b998] ERROR -- : /opt/rh/cfme-gemset/gems/excon-0.51.0/lib/excon/middlewares/expects.rb:6:in `response_call'
/opt/rh/cfme-gemset/gems/excon-0.51.0/lib/excon/middlewares/response_parser.rb:8:in `response_call'
/opt/rh/cfme-gemset/gems/excon-0.51.0/lib/excon/connection.rb:389:in `response'
/opt/rh/cfme-gemset/gems/excon-0.51.0/lib/excon/connection.rb:253:in `request'
/opt/rh/cfme-gemset/gems/fog-core-1.42.0/lib/fog/core/connection.rb:81:in `request'
/opt/rh/cfme-gemset/gems/fog-openstack-0.1.8/lib/fog/openstack.rb:489:in `get_supported_version'
/opt/rh/cfme-gemset/gems/fog-openstack-0.1.8/lib/fog/openstack/metering.rb:120:in `set_api_path'
/opt/rh/cfme-gemset/gems/fog-openstack-0.1.8/lib/fog/openstack/metering.rb:112:in `initialize'
/opt/rh/cfme-gemset/gems/fog-core-1.42.0/lib/fog/core/service.rb:115:in `new'
/opt/rh/cfme-gemset/gems/fog-core-1.42.0/lib/fog/core/service.rb:115:in `new'
/opt/rh/cfme-gemset/gems/fog-core-1.42.0/lib/fog/core/services_mixin.rb:16:in `new'
/var/www/miq/vmdb/gems/pending/openstack/openstack_handle/handle.rb:74:in `raw_connect'
/var/www/miq/vmdb/gems/pending/openstack/openstack_handle/handle.rb:49:in `block in raw_connect_try_ssl'
/var/www/miq/vmdb/gems/pending/openstack/openstack_handle/handle.rb:39:in `try_connection'
/var/www/miq/vmdb/gems/pending/openstack/openstack_handle/handle.rb:46:in `raw_connect_try_ssl'
/var/www/miq/vmdb/gems/pending/openstack/openstack_handle/handle.rb:170:in `connect'
/var/www/miq/vmdb/app/models/manageiq/providers/openstack/manager_mixin.rb:60:in `connect'
/var/www/miq/vmdb/app/models/manageiq/providers/openstack/base_metrics_capture.rb:29:in `block in perf_init_openstack'
/var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/var/www/miq/vmdb/app/models/manageiq/providers/openstack/base_metrics_capture.rb:28:in `perf_init_openstack'
/var/www/miq/vmdb/app/models/manageiq/providers/openstack/base_metrics_capture.rb:13:in `perf_collect_metrics'
/var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:6:in `perf_collect_metrics'
/var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:157:in `block in perf_capture'
/var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:154:in `perf_capture'
/var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:94:in `perf_capture_realtime'

Comment 11 Marek Aufart 2016-10-14 11:30:45 UTC
@Colin, the error unfortunately looks different to the issue resolved in this BZ. This BZ is looking not to be dupe of BZ1365337 most propably.

Comment 12 Ola Pavlenko 2016-10-27 09:35:49 UTC
I assume https://bugzilla.redhat.com/show_bug.cgi?id=1384506 was here by mistake.