happens on line 45 of

     41     def readline
     42       return legacy_readline if RUBY_VERSION.to_f <= 1.8_7
     43       buffer = ''
     44       begin
     45         buffer << @socket.read_nonblock(1) while buffer[-1] != "\n"
     46         buffer
     47       rescue Errno::EAGAIN, Errno::EWOULDBLOCK, IO::WaitReadable
     48         if timeout_reached('read')
     49           raise_timeout_error('read')
     50         else
     51           retry
     52         end


it could be that a read is issued when it should not.

Ladas,

this sorta sounds like the errors we were seeing when someone would add an OpenStack cloud to OpenStack infra...can you verify that?

Otherwise, can you take a look at how this would happen?

The error appears when user list Heat stacks for some tenant

/opt/rh/cfme-gemset/gems/fog-1.29.0/lib/fog/openstack/orchestration.rb:183:in `request'
/opt/rh/cfme-gemset/gems/fog-1.29.0/lib/fog/openstack/requests/orchestration/list_stack_data.rb:6:in `list_stack_data'

Could be that a user is missing role admin or heat_stack_owner on the tenant, but that should just throw Excon::Errors::Forbidden, which we can handle. Can you try to list stacks for each of your tenants in CLI to see if it's not broken?

the commands ran by the user time out when using v1, v2 or v3 of the api (cloudforms seems to use v2) - is it possible that a timeout would break the stream in an uncontrolled manner and this wouldn't be raised? 

the feedback from the customer regarding the errors raised while collecting the data is


Unable to establish connection to http://10.67.95.15:8004/v1/aaf0e4e740494f15bb3341efb58a945f/stacks

- or - 

[root@macb083fee7c1ff ~(openstack_admin)]# heat stack-list -g 2>&1 | tee heat_stack-list_global.out
Unable to establish connection to 
http://192.168.30.16:35357/v2.0/tokens

- or - 
Connection timeouts

I unfortunately do not have a better output to give but it looks plausible that the problem was that the stream was interrupted before its expected end somewhere upstream ?

I think this might be connected to usage of the admin endpoints looking at the urls. If the endpoints are inaccessible, e.g. on private network, it will timeout and can break like this.

@Felix could you investigate if that is the reason?

@Greg then we should connect to BZ that switches to public enpoints

Hi Felix,

Could you obtain the admin endpoints URLs from 'keystone endpoint-list' and then try to first do connectivity tests from the Cloudforms appliance to the IPs listed for the adminurl? If the ping tests are successful then try to curl the endpoints, you should be able to get a response from the Openstack side. If there's no response then it might indicate a connectivity issue.

Thanks

refresh failure Error - Less Than A Minute Ago
end of file reached (EOFError) 

[----] E, [2016-02-12T03:34:17.636520 #19433:347998] ERROR -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::Refresher#refresh) EMS: [rhos8_beta6], id: [13412000000000009] Refresh failed
[----] E, [2016-02-12T03:34:17.636740 #19433:347998] ERROR -- : [Excon::Errors::SocketError]: end of file reached (EOFError)  Method:[rescue in block in refresh]
[----] E, [2016-02-12T03:34:17.636883 #19433:347998] ERROR -- : /opt/rh/cfme-gemset/gems/excon-0.45.4/lib/excon/socket.rb:45:in `read_nonblock'
/opt/rh/cfme-gemset/gems/excon-0.45.4/lib/excon/socket.rb:45:in `readline'
/opt/rh/cfme-gemset/gems/excon-0.45.4/lib/excon/response.rb:53:in `parse'
/opt/rh/cfme-gemset/gems/excon-0.45.4/lib/excon/middlewares/response_parser.rb:6:in `response_call'
/opt/rh/cfme-gemset/gems/excon-0.45.4/lib/excon/connection.rb:372:in `response'
/opt/rh/cfme-gemset/gems/excon-0.45.4/lib/excon/connection.rb:236:in `request'
/opt/rh/cfme-gemset/gems/fog-core-1.35.0/lib/fog/core/connection.rb:81:in `request'
/opt/rh/cfme-gemset/gems/fog-2.0.0.pre.0/lib/fog/openstack/core.rb:550:in `get_supported_version'
/opt/rh/cfme-gemset/gems/fog-2.0.0.pre.0/lib/fog/openstack/image.rb:111:in `initialize'
/opt/rh/cfme-gemset/gems/fog-core-1.35.0/lib/fog/core/service.rb:115:in `new'
/opt/rh/cfme-gemset/gems/fog-core-1.35.0/lib/fog/core/service.rb:115:in `new'
/opt/rh/cfme-gemset/gems/fog-core-1.35.0/lib/fog/core/services_mixin.rb:16:in `new'
/var/www/miq/vmdb/gems/pending/openstack/openstack_handle/handle.rb:73:in `raw_connect'
/var/www/miq/vmdb/gems/pending/openstack/openstack_handle/handle.rb:48:in `block in raw_connect_try_ssl'
/var/www/miq/vmdb/gems/pending/openstack/openstack_handle/handle.rb:38:in `try_connection'
/var/www/miq/vmdb/gems/pending/openstack/openstack_handle/handle.rb:45:in `raw_connect_try_ssl'
/var/www/miq/vmdb/gems/pending/openstack/openstack_handle/handle.rb:161:in `connect'
/var/www/miq/vmdb/gems/pending/openstack/openstack_handle/handle.rb:245:in `detect_service'
/var/www/miq/vmdb/gems/pending/openstack/openstack_handle/handle.rb:223:in `detect_image_service'
/var/www/miq/vmdb/app/models/manageiq/providers/openstack/cloud_manager/refresh_parser.rb:27:in `initialize'
/var/www/miq/vmdb/app/models/manageiq/providers/openstack/cloud_manager/refresh_parser.rb:12:in `new'
/var/www/miq/vmdb/app/models/manageiq/providers/openstack/cloud_manager/refresh_parser.rb:12:in `ems_inv_to_hashes'
/var/www/miq/vmdb/app/models/manageiq/providers/openstack/cloud_manager/refresher.rb:6:in `parse_inventory'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:19:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:7:in `each'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:7:in `refresh'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:10:in `refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:77:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:76:in `each'
/var/www/miq/vmdb/app/models/ems_refresh.rb:76:in `refresh'
/var/www/miq/vmdb/app/models/miq_queue.rb:345:in `block in deliver'
/opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:89:in `block in timeout'
/opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `block in catch'
/opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `catch'
/opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `catch'
/opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:104:in `timeout'
/var/www/miq/vmdb/app/models/miq_queue.rb:341:in `deliver'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:106:in `deliver_queue_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:134:in `deliver_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:151:in `block in do_work'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:145:in `loop'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:145:in `do_work'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:334:in `block in do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:331:in `loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:331:in `do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:153:in `run'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:128:in `start'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:29:in `start_worker'
/var/www/miq/vmdb/lib/workers/bin/worker.rb:2:in `<top (required)>'
/opt/rh/cfme-gemset/gems/railties-4.2.5.1/lib/rails/commands/runner.rb:60:in `load'
/opt/rh/cfme-gemset/gems/railties-4.2.5.1/lib/rails/commands/runner.rb:60:in `<top (required)>'
/opt/rh/cfme-gemset/gems/railties-4.2.5.1/lib/rails/commands/commands_tasks.rb:123:in `require'
/opt/rh/cfme-gemset/gems/railties-4.2.5.1/lib/rails/commands/commands_tasks.rb:123:in `require_command!'
/opt/rh/cfme-gemset/gems/railties-4.2.5.1/lib/rails/commands/commands_tasks.rb:90:in `runner'
/opt/rh/cfme-gemset/gems/railties-4.2.5.1/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
/opt/rh/cfme-gemset/gems/railties-4.2.5.1/lib/rails/commands.rb:17:in `<top (required)>'
/var/www/miq/vmdb/bin/rails:4:in `require'
/var/www/mi

In compute node

Feb 12 03:33:58 overcloud-novacompute-1.localdomain libvirtd[15226]: End of file while reading data: Input/output error
Feb 12 03:33:58 overcloud-novacompute-1.localdomain libvirtd[15226]: libvirt version: 1.2.17, 
Feb 12 03:33:58 overcloud-novacompute-1.localdomain libvirtd[15226]: End of file while reading
Feb 12 03:33:58 overcloud-novacompute-1.localdomain systemd[1]: Starting OpenStack Nova Comput
Feb 12 03:34:00 overcloud-novacompute-1.localdomain nova-compute[19634]: No handlers could be 
Feb 12 03:34:12 overcloud-novacompute-1.localdomain os-collect-config[8276]: 2016-02-12 03:34:
Feb 12 03:34:12 overcloud-novacompute-1.localdomain os-collect-config[8276]: 2016-02-12 03:34:
Feb 12 03:34:12 overcloud-novacompute-1.localdomain os-collect-config[8276]: 2016-02-12 03:34:
Feb 12 03:34:12 overcloud-novacompute-1.localdomain os-collect-config[8276]: 2016-02-12 03:34:
Feb 12 03:34:12 overcloud-novacompute-1.localdomain nova-compute[19634]: 2016-02-12 03:34:12.4
Feb 12 03:34:22 overcloud-novacompute-1.localdomain nova-compute[19634]: 2016-02-12 03:34:22.4
Feb 12 03:34:32 overcloud-novacompute-1.localdomain nova-compute[19634]: 2016-02-12 03:34:32.4
Feb 12 03:34:40 overcloud-novacompute-1.localdomain polkitd[600]: Unregistered Authentication 

Nova compute service not running 
MainPID=13292 Id=openstack-nova-compute.service ActiveState=activating