Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
First Last Prev Next    This bug is not in your last search results.
Bug 1200075 - Broken Pipe in vsphere store due to inactive session [NEEDINFO]
Broken Pipe in vsphere store due to inactive session
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-glance-store (Show other bugs)
6.0 (Juno)
All Linux
high Severity medium
: z5
: 6.0 (Juno)
Assigned To: Flavio Percoco
Jaroslav Henner
: ZStream
Depends On:
Blocks: 1249164 1256371
  Show dependency tree / graph
 
Reported: 2015-03-09 12:01 EDT by Jack Waterworth
Modified: 2018-02-08 05:33 EST (History)
16 users (show)

See Also:
Fixed In Version: python-glance-store-0.1.10-4.el7ost
Doc Type: Bug Fix
Doc Text:
Prior to this update, Image Service's (glance) VMware vSphere store failed to handle 401 errors correctly. These errors were raised after the vSphere session had expired, causing long-standing operations, such as snapshots, to fail. This update resolves the issue, allowing these operations to succeed.
Story Points: ---
Clone Of:
: 1256371 (view as bug list)
Environment:
Last Closed: 2015-08-24 10:48:35 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
fpercoco: needinfo? (scohen)

Attachments (Terms of Use)
Add an attachment (proposed patch, testcase, etc.)


External Trackers
Tracker ID Priority Status Summary Last Updated
OpenStack gerrit 141665 None None None Never
Red Hat Product Errata RHBA-2015:1659 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform Bug Fix and Enhancement Advisory 2015-08-24 14:47:56 EDT

  None (edit)
Description Jack Waterworth 2015-03-09 12:01:17 EDT 
Description of problem:
Glance vSphere datastore image upload fails to complete with 104 error (connection reset by peer)

Steps to Reproduce:
1. configure glance for vmware vsphere
2. restart glance services
3. attempt to upload glance image

Actual results:
# glance image-create --progress --name 'cirros2' --disk-format 'vmdk' --container-format 'bare' --is-public 'true' --file '/root/CirrOS0.3.3-vmdk.raw';
[=============================>] 100%
<html>
 <head>
  <title>400 Bad Request</title>
 </head>
 <body>
  <h1>400 Bad Request</h1>
  Client disconnected before sending all data to backend<br /><br />

Expected results:
image should be successfully uploaded

Additional info:

# tail -f /var/log/glance/api.log /var/log/glance/registry.log
==> /var/log/glance/api.log <==
2015-03-06 09:24:42.081 14361 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.30.207.143

==> /var/log/glance/registry.log <==
2015-03-06 09:24:42.239 14434 INFO urllib3.connectionpool [-] Resetting dropped connection: 172.30.207.143
2015-03-06 09:24:42.393 14434 INFO glance.registry.api.v1.images [b42b72e1-479e-4204-b653-c6a2508f1941 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] Successfully created image c
80924f4-f412-4d4c-82fe-faf3179b9774
2015-03-06 09:24:42.394 14434 INFO glance.wsgi.server [b42b72e1-479e-4204-b653-c6a2508f1941 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] 172.30.207.100 - - [06/Mar/2015 09:24:4
2] "POST /images HTTP/1.1" 200 677 0.156951

==> /var/log/glance/api.log <==
2015-03-06 09:24:42.396 14361 INFO oslo.messaging._drivers.impl_rabbit [4fb8a6c0-9b48-4c8b-aa3f-32b72283f508 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] Connecting to AMQP ser
ver on 172.30.207.153:5672
2015-03-06 09:24:42.408 14361 INFO oslo.messaging._drivers.impl_rabbit [4fb8a6c0-9b48-4c8b-aa3f-32b72283f508 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] Connected to AMQP serv
er on 172.30.207.153:5672

==> /var/log/glance/registry.log <==
2015-03-06 09:24:42.415 14436 INFO urllib3.connectionpool [-] Resetting dropped connection: 172.30.207.143
2015-03-06 09:24:42.558 14436 INFO glance.wsgi.server [2352910c-e0d8-47c6-8a72-72cafe9cd46d 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] 172.30.207.100 - - [06/Mar/2015 09:24:4
2] "GET /images/c80924f4-f412-4d4c-82fe-faf3179b9774 HTTP/1.1" 200 677 0.143998
2015-03-06 09:24:42.561 14430 INFO urllib3.connectionpool [-] Resetting dropped connection: 172.30.207.143
2015-03-06 09:24:42.718 14430 INFO glance.registry.api.v1.images [216319bf-ef3f-4475-bac3-5ce928d6743d 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] Updating metadata for image
c80924f4-f412-4d4c-82fe-faf3179b9774
2015-03-06 09:24:42.719 14430 INFO glance.wsgi.server [216319bf-ef3f-4475-bac3-5ce928d6743d 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] 172.30.207.100 - - [06/Mar/2015 09:24:4
2] "PUT /images/c80924f4-f412-4d4c-82fe-faf3179b9774 HTTP/1.1" 200 677 0.158904

==> /var/log/glance/api.log <==
2015-03-06 09:24:42.740 14361 ERROR glance_store._drivers.vmware_datastore [4fb8a6c0-9b48-4c8b-aa3f-32b72283f508 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] Failed to upload c
ontent of image c80924f4-f412-4d4c-82fe-faf3179b9774
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore Traceback (most recent call last):
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore   File "/usr/lib/python2.7/site-packages/glance_store/_drivers/vmware_datastore.py", line 346, in add
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore     content=image_file)
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore   File "/usr/lib/python2.7/site-packages/glance_store/_drivers/vmware_datastore.py", line 489, in _get_http_conn
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore     conn.request(method, url, content, headers)
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore   File "/usr/lib64/python2.7/httplib.py", line 973, in request
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore     self._send_request(method, url, body, headers)
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore   File "/usr/lib64/python2.7/httplib.py", line 1007, in _send_request
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore     self.endheaders(body)
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore   File "/usr/lib64/python2.7/httplib.py", line 969, in endheaders
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore     self._send_output(message_body)
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore   File "/usr/lib64/python2.7/httplib.py", line 833, in _send_output
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore     self.send(message_body)
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore   File "/usr/lib64/python2.7/httplib.py", line 802, in send
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore     self.sock.sendall(datablock)
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore   File "/usr/lib/python2.7/site-packages/eventlet/green/ssl.py", line 137, in sendall
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore     v = self.send(data[count:])
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore   File "/usr/lib/python2.7/site-packages/eventlet/green/ssl.py", line 113, in send
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore     super(GreenSSLSocket, self).send, data, flags)
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore   File "/usr/lib/python2.7/site-packages/eventlet/green/ssl.py", line 80, in _call_trampolining
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore     return func(*a, **kw)
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore   File "/usr/lib64/python2.7/ssl.py", line 198, in send
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore     v = self._sslobj.write(data)
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore error: [Errno 104] Connection reset by peer
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore

==> /var/log/glance/registry.log <==
2015-03-06 09:24:42.768 14430 INFO glance.registry.api.v1.images [f3a2dcee-6d22-4a34-85bd-44ae5e013212 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] Updating metadata for image c80924f4-f412-4d4c-82fe-faf3179b9774
2015-03-06 09:24:42.769 14430 INFO glance.wsgi.server [f3a2dcee-6d22-4a34-85bd-44ae5e013212 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] 172.30.207.100 - - [06/Mar/2015 09:24:42] "PUT /images/c80924f4-f412-4d4c-82fe-faf3179b9774 HTTP/1.1" 200 677 0.026189

==> /var/log/glance/api.log <==
2015-03-06 09:24:43.123 14361 INFO glance.wsgi.server [4fb8a6c0-9b48-4c8b-aa3f-32b72283f508 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] 172.30.207.102 - - [06/Mar/2015 09:24:43] "POST /v1/images HTTP/1.1" 400 372 1.047594
Comment 3 Jack Waterworth 2015-03-09 12:02:32 EDT 
It appears this bug has been reported upstream and fixed for the kilo release found here https://bugs.launchpad.net/glance/+bug/1402354
Comment 16 Jaroslav Henner 2015-08-18 08:16:30 EDT 
I have got these errors so far when checking this on RHOS 6

python-oslo-vmware-0.6.0-1.el7ost.noarch
python-oslo-config-1.4.0-1.el7ost.noarch
python-oslo-vmware-0.6.0-1.el7ost.noarch


2015-08-18 08:10:48.693 25559 TRACE stevedore.extension   File "/usr/lib/python2.7/site-packages/glance_store/_drivers/vmware_datastore.py", line 27, in <module>
2015-08-18 08:10:48.693 25559 TRACE stevedore.extension     from oslo_utils import excutils
2015-08-18 08:10:48.693 25559 TRACE stevedore.extension ImportError: No module named oslo_utils

2015-08-18 08:09:00.087 25308 TRACE glance   File "/usr/lib/python2.7/site-packages/stevedore/driver.py", line 50, in _default_on_load_failure
2015-08-18 08:09:00.087 25308 TRACE glance     raise err
2015-08-18 08:09:00.087 25308 TRACE glance ImportError: No module named oslo_config

changing the imports to the oslo.config and oslo.utils seemed to help
Comment 17 Jaroslav Henner 2015-08-18 08:24:04 EDT 
Apart from that I wrote above, when trying to upload the image, I am getting

2015-08-18 08:23:01.265 27651 ERROR glance.api.v1.upload_utils [fb2db524-94b9-4ae5-befe-33cbf53193ef 24e9dcd37953452ea7429c1469609a40 6ee41f516a284148941732b3a664fed8 - - -] Failed to upload image 9e173d95-dbe2-4eca-bc51-f2505801fb0a
2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils Traceback (most recent call last):
2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils   File "/usr/lib/python2.7/site-packages/glance/api/v1/upload_utils.py", line 113, in upload_data_to_store
2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils     context=req.context)
2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils   File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 331, in store_add_to_backend
2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils     context=context)
2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils   File "/usr/lib/python2.7/site-packages/glance_store/_drivers/vmware_datastore.py", line 338, in add
2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils     cookie = self._build_vim_cookie_header(True)
2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils   File "/usr/lib/python2.7/site-packages/glance_store/_drivers/vmware_datastore.py", line 299, in _build_vim_cookie_header
2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils     if verify_session and not self.session.is_current_session_active():
2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils AttributeError: 'VMwareAPISession' object has no attribute 'is_current_session_active'
2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils 
2015-08-18 08:23:01.338 27651 INFO glance.wsgi.server [fb2db524-94b9-4ae5-befe-33cbf53193ef 24e9dcd37953452ea7429c1469609a40 6ee41f516a284148941732b3a664fed8 - - -] 10.34.69.130 - - [18/Aug/2015 08:23:01] "POST /v1/images HTTP/1.1" 500 431 0.546157
Comment 18 Jaroslav Henner 2015-08-18 08:27:44 EDT 
Adding underscore ( _is_current_sessi...) seems to help.
Comment 19 Jaroslav Henner 2015-08-18 09:25:32 EDT 
It seems image can be uploaded fine after doing the two manual fixes above, but BZ#1361197 prevents us to use it.
Comment 20 Flavio Percoco 2015-08-19 04:38:33 EDT 
In the Juno release, oslo packages used to use python namespaces. Hence the need for `.`

If backported to Juno, these imports need to be fixed. I'll propose a patch for the `_` issue upstream.
Comment 21 Flavio Percoco 2015-08-19 04:42:11 EDT 
Actually, the `_` issue is another backport specific change. In Juno, that method used to be private.
Comment 32 errata-xmlrpc 2015-08-24 10:48:35 EDT 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-1659.html
Note You need to log in before you can comment on or make changes to this bug.
First Last Prev Next    This bug is not in your last search results.