Bug 1328087

I think that there is no relation to the fact that RHEV-M is running as a hosted engine.

Apparently what happens here is that RHEV-M is responding with a HTTP status code that CFME doesn't expect. When that happens the "ovirt" gem ends up calling the "return!" method of the rest-client gem with 2 parameters and a block:

  https://github.com/ManageIQ/ovirt/blob/v0.7.1/lib/ovirt/service.rb#L208

That works fine with the version of rest-client used upstream, 1.7.2:

  https://github.com/rest-client/rest-client/blob/v1.7.2/lib/restclient/abstract_response.rb#L34

But CFME seems to use 2.0.0.rc1 where the "return!" method has been changed to receive no parameter, only a block:

  https://github.com/rest-client/rest-client/blob/v2.0.0.rc1/lib/restclient/abstract_response.rb#L71

That needs to be fixed, either by updating the "ovirt" gem so that it calls correctly the "return!" method, or by downgrading the version of the "rest-client" gem.

In addition there is the question of why RHEV-M is returning an unexpected status code. Please ask the customer to try to access that URL in the browser:

  https://.../api/storagedomains?search=sortby%20name%20asc%20page%201

Ask them also to provide the log files of RHEV-M, located in /var/log/ovirt-engine/engine.log and /var/log/ovirt-engine/server.log. The web server access log is also helpful, it is located in /var/log/httpd/ssl_access_log.

The following pull request should fix the wrong call to the rest-client gem:

  https://github.com/ManageIQ/ovirt/pull/53

However, this won't make the refresh work correctly if RHEV-M is returning an error response.

I got the same result after fixing some errors with the NFS shared for export and ISO.

I deleted the NFS storage to change the configuraiton and I received the error that I couldn't add it again because it was already configured.

I deleted the storage manually in postgres and RHEV allowed me to create the NFS storage again, but then I started receiving this error in cloudforms and manageiq.

Looking at the RHEV-M logs, I was receiving a 404 error from RHEV-M when refressing the storage, even though the same session cookie was being used and the user and password was correct.

Connecting to the API directly, I was getting a message saying that b88c221e-0fce-4101-855c-26d8d18eede1 was not found.

Connecting again to the database, I found the starge configured both in storage_domain_static and storage_domain_dynamic.

I deleted the storage from both tables and everything started to work again

In Darga, the message in CFME interface was:
Ovirt::MissingResourceError

I just asked my customer - the same operation had been performed in his environment. From his description he probably did the same as you Sergio.

OK, looks like he have a reproducer. This will probably end up being a RHEV-M bug. I'm looking at it.

The only way I was able to reproduce this is the following:

1. Create a ISO storage domain.

2. Put it in maintenance.

3. Manually delete from the database the row of the "storage_server_connections" table, but keep the rows in the "storage_domain_static" and "storage_domain_dynamic":

  # delete from storage_server_connections where id='...';

After doing this the database is inconsistent and requests to the API fail with the following error returned to the caller

  HTTP/1.1 404 Not Found
  <fault>
    <reason>Operation Failed</reason>
    <detail>Entity not found: Storage server connection: id=f79e50f2-914b-49a1-8344-51d08f8d8109</detail>
  </fault>

There isn't much we can do if the database has been manipulated manually, but we could maybe have referential constraints (foreign keys) to avoid this kind of delete.

Anyhow, this isn't a CFME issue, but RHEV-M. I'm moving this bug back to POST, as we have a fix for the exception that this causes in CFME.

Felix, please open a new RHEV-M bug (or clone this one) if you think that we should do something to address this use case.

I'm also lowering severity, as this is something that should not happen if there aren't manual manipulations of the database.

The pull requests related to this bug are the following:

  Use version 2.0.0.rc1 of rest-client
  https://github.com/ManageIQ/ovirt/pull/53

  Release 0.9.1
  https://github.com/ManageIQ/ovirt/pull/54

  Update to version 0.9.1 of "ovirt" gem
  https://github.com/ManageIQ/manageiq/pull/8309

Verified on: 
cfme: 5.6.0.10-rc2.1.20160607103248_d06c141
rhevm 3.6 : 3.6.6.2-0.1.el6

Using reproduce steps from comment #17.

The refresh error, seen on CFME GUI:
Last Refresh: Error - 31 Minutes Ago
Ovirt::MissingResourceError
 
The refresh error, seen on evm.log:
[----] I, [2016-06-14T06:48:13.822004 #2925:10ad998]  INFO -- : MIQ(MiqServer#monitor_loop) Server Monitoring Complete - Timings: {:heartbeat=>0.009138822555541992, :server_dequeue=>0.004607439041137695, :server_monitor=>0.08786511421203613, :worker_monitor=>1.5567398071289062, :worker_dequeue=>0.009943723678588867, :total_time=>1.6685888767242432}
[----] E, [2016-06-14T06:48:14.194952 #31158:10ad998] ERROR -- : MIQ(ManageIQ::Providers::Redhat::InfraManager::Refresher#refresh) EMS: [RHEV36], id: [1000000000002] Refresh failed
[----] E, [2016-06-14T06:48:14.195349 #31158:10ad998] ERROR -- : [Ovirt::MissingResourceError]: Ovirt::MissingResourceError  Method:[rescue in block in refresh]
[----] E, [2016-06-14T06:48:14.195462 #31158:10ad998] ERROR -- : /opt/rh/cfme-gemset/gems/ovirt-0.10.0/lib/ovirt/service.rb:302:in `parse_error_response'
/opt/rh/cfme-gemset/gems/ovirt-0.10.0/lib/ovirt/service.rb:273:in `block in resource_verb'
/opt/rh/cfme-gemset/gems/rest-client-2.0.0.rc1/lib/restclient/request.rb:592:in `call'
/opt/rh/cfme-gemset/gems/rest-client-2.0.0.rc1/lib/restclient/request.rb:592:in `process_result'
/opt/rh/cfme-gemset/gems/rest-client-2.0.0.rc1/lib/restclient/request.rb:504:in `block in transmit'
/opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:853:in `start'
/opt/rh/cfme-gemset/gems/rest-client-2.0.0.rc1/lib/restclient/request.rb:494:in `transmit'
/opt/rh/cfme-gemset/gems/rest-client-2.0.0.rc1/lib/restclient/request.rb:202:in `execute'
/opt/rh/cfme-gemset/gems/rest-client-2.0.0.rc1/lib/restclient/request.rb:52:in `execute'
/opt/rh/cfme-gemset/gems/rest-client-2.0.0.rc1/lib/restclient/resource.rb:51:in `get'
/opt/rh/cfme-gemset/gems/ovirt-0.10.0/lib/ovirt/service.rb:268:in `resource_verb'
/opt/rh/cfme-gemset/gems/ovirt-0.10.0/lib/ovirt/service.rb:242:in `resource_get'
/opt/rh/cfme-gemset/gems/ovirt-0.10.0/lib/ovirt/service.rb:226:in `block in paginate_resource_get'
/opt/rh/cfme-gemset/gems/ovirt-0.10.0/lib/ovirt/service.rb:224:in `loop'
/opt/rh/cfme-gemset/gems/ovirt-0.10.0/lib/ovirt/service.rb:224:in `paginate_resource_get'
/opt/rh/cfme-gemset/gems/ovirt-0.10.0/lib/ovirt/service.rb:147:in `standard_collection'
/opt/rh/cfme-gemset/gems/ovirt-0.10.0/lib/ovirt/inventory.rb:113:in `standard_collection'
/opt/rh/cfme-gemset/gems/ovirt-0.10.0/lib/ovirt/inventory.rb:56:in `storagedomains'
/opt/rh/cfme-gemset/gems/ovirt-0.10.0/lib/ovirt/inventory.rb:154:in `block in collect_primary_jobs'
/opt/rh/cfme-gemset/gems/parallel-1.9.0/lib/parallel.rb:451:in `call'
/opt/rh/cfme-gemset/gems/parallel-1.9.0/lib/parallel.rb:451:in `call_with_index'
/opt/rh/cfme-gemset/gems/parallel-1.9.0/lib/parallel.rb:311:in `block (2 levels) in work_in_threads'
/opt/rh/cfme-gemset/gems/parallel-1.9.0/lib/parallel.rb:462:in `with_instrumentation'
/opt/rh/cfme-gemset/gems/parallel-1.9.0/lib/parallel.rb:310:in `block in work_in_threads'
/opt/rh/cfme-gemset/gems/parallel-1.9.0/lib/parallel.rb:192:in `block (2 levels) in in_threads'
/opt/rh/cfme-gemset/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `call'
/opt/rh/cfme-gemset/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
[----] E, [2016-06-14T06:48:14.195552 #31158:10ad998] ERROR -- : MIQ(ManageIQ::Providers::Redhat::InfraManager::Refresher#refresh) EMS: [RHEV36], id: [1000000000002] Unable to perform refresh for the following targets:
[----] E, [2016-06-14T06:48:14.195716 #31158:10ad998] ERROR -- : MIQ(ManageIQ::Providers::Redhat::InfraManager::Refresher#refresh)  --- ManageIQ::Providers::Redhat::InfraManager [RHEV36] id [1000000000002]
[----] I, [2016-06-14T06:48:14.222536 #31158:10ad998]  INFO -- : MIQ(ManageIQ::Providers::Redhat::InfraManager::Refresher#refresh) Refreshing all targets...Complete

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1348