Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1200075

Summary: Broken Pipe in vsphere store due to inactive session
Product: Red Hat OpenStack Reporter: Jack Waterworth <jwaterwo>
Component: python-glance-storeAssignee: Flavio Percoco <fpercoco>
Status: CLOSED ERRATA QA Contact: Jaroslav Henner <jhenner>
Severity: medium Docs Contact:
Priority: high    
Version: 6.0 (Juno)CC: apevec, dmaley, ebarrera, eglynn, fpercoco, hguemar, jhenner, lhh, mlopes, nyechiel, sclewis, scohen, sgordon, sgotliv, yeylon
Target Milestone: z5Keywords: ZStream
Target Release: 6.0 (Juno)   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: python-glance-store-0.1.10-4.el7ost Doc Type: Bug Fix
Doc Text:
Prior to this update, Image Service's (glance) VMware vSphere store failed to handle 401 errors correctly. These errors were raised after the vSphere session had expired, causing long-standing operations, such as snapshots, to fail. This update resolves the issue, allowing these operations to succeed.
Story Points: ---
Clone Of:
: 1256371 (view as bug list) Environment:
Last Closed: 2015-08-24 14:48:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1249164, 1256371    

Description Jack Waterworth 2015-03-09 16:01:17 UTC
Description of problem:
Glance vSphere datastore image upload fails to complete with 104 error (connection reset by peer)

Steps to Reproduce:
1. configure glance for vmware vsphere
2. restart glance services
3. attempt to upload glance image

Actual results:
# glance image-create --progress --name 'cirros2' --disk-format 'vmdk' --container-format 'bare' --is-public 'true' --file '/root/CirrOS0.3.3-vmdk.raw';
[=============================>] 100%
<html>
 <head>
  <title>400 Bad Request</title>
 </head>
 <body>
  <h1>400 Bad Request</h1>
  Client disconnected before sending all data to backend<br /><br />

Expected results:
image should be successfully uploaded

Additional info:

# tail -f /var/log/glance/api.log /var/log/glance/registry.log
==> /var/log/glance/api.log <==
2015-03-06 09:24:42.081 14361 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.30.207.143

==> /var/log/glance/registry.log <==
2015-03-06 09:24:42.239 14434 INFO urllib3.connectionpool [-] Resetting dropped connection: 172.30.207.143
2015-03-06 09:24:42.393 14434 INFO glance.registry.api.v1.images [b42b72e1-479e-4204-b653-c6a2508f1941 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] Successfully created image c
80924f4-f412-4d4c-82fe-faf3179b9774
2015-03-06 09:24:42.394 14434 INFO glance.wsgi.server [b42b72e1-479e-4204-b653-c6a2508f1941 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] 172.30.207.100 - - [06/Mar/2015 09:24:4
2] "POST /images HTTP/1.1" 200 677 0.156951

==> /var/log/glance/api.log <==
2015-03-06 09:24:42.396 14361 INFO oslo.messaging._drivers.impl_rabbit [4fb8a6c0-9b48-4c8b-aa3f-32b72283f508 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] Connecting to AMQP ser
ver on 172.30.207.153:5672
2015-03-06 09:24:42.408 14361 INFO oslo.messaging._drivers.impl_rabbit [4fb8a6c0-9b48-4c8b-aa3f-32b72283f508 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] Connected to AMQP serv
er on 172.30.207.153:5672

==> /var/log/glance/registry.log <==
2015-03-06 09:24:42.415 14436 INFO urllib3.connectionpool [-] Resetting dropped connection: 172.30.207.143
2015-03-06 09:24:42.558 14436 INFO glance.wsgi.server [2352910c-e0d8-47c6-8a72-72cafe9cd46d 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] 172.30.207.100 - - [06/Mar/2015 09:24:4
2] "GET /images/c80924f4-f412-4d4c-82fe-faf3179b9774 HTTP/1.1" 200 677 0.143998
2015-03-06 09:24:42.561 14430 INFO urllib3.connectionpool [-] Resetting dropped connection: 172.30.207.143
2015-03-06 09:24:42.718 14430 INFO glance.registry.api.v1.images [216319bf-ef3f-4475-bac3-5ce928d6743d 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] Updating metadata for image
c80924f4-f412-4d4c-82fe-faf3179b9774
2015-03-06 09:24:42.719 14430 INFO glance.wsgi.server [216319bf-ef3f-4475-bac3-5ce928d6743d 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] 172.30.207.100 - - [06/Mar/2015 09:24:4
2] "PUT /images/c80924f4-f412-4d4c-82fe-faf3179b9774 HTTP/1.1" 200 677 0.158904

==> /var/log/glance/api.log <==
2015-03-06 09:24:42.740 14361 ERROR glance_store._drivers.vmware_datastore [4fb8a6c0-9b48-4c8b-aa3f-32b72283f508 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] Failed to upload c
ontent of image c80924f4-f412-4d4c-82fe-faf3179b9774
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore Traceback (most recent call last):
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore   File "/usr/lib/python2.7/site-packages/glance_store/_drivers/vmware_datastore.py", line 346, in add
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore     content=image_file)
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore   File "/usr/lib/python2.7/site-packages/glance_store/_drivers/vmware_datastore.py", line 489, in _get_http_conn
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore     conn.request(method, url, content, headers)
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore   File "/usr/lib64/python2.7/httplib.py", line 973, in request
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore     self._send_request(method, url, body, headers)
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore   File "/usr/lib64/python2.7/httplib.py", line 1007, in _send_request
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore     self.endheaders(body)
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore   File "/usr/lib64/python2.7/httplib.py", line 969, in endheaders
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore     self._send_output(message_body)
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore   File "/usr/lib64/python2.7/httplib.py", line 833, in _send_output
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore     self.send(message_body)
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore   File "/usr/lib64/python2.7/httplib.py", line 802, in send
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore     self.sock.sendall(datablock)
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore   File "/usr/lib/python2.7/site-packages/eventlet/green/ssl.py", line 137, in sendall
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore     v = self.send(data[count:])
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore   File "/usr/lib/python2.7/site-packages/eventlet/green/ssl.py", line 113, in send
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore     super(GreenSSLSocket, self).send, data, flags)
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore   File "/usr/lib/python2.7/site-packages/eventlet/green/ssl.py", line 80, in _call_trampolining
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore     return func(*a, **kw)
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore   File "/usr/lib64/python2.7/ssl.py", line 198, in send
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore     v = self._sslobj.write(data)
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore error: [Errno 104] Connection reset by peer
2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore

==> /var/log/glance/registry.log <==
2015-03-06 09:24:42.768 14430 INFO glance.registry.api.v1.images [f3a2dcee-6d22-4a34-85bd-44ae5e013212 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] Updating metadata for image c80924f4-f412-4d4c-82fe-faf3179b9774
2015-03-06 09:24:42.769 14430 INFO glance.wsgi.server [f3a2dcee-6d22-4a34-85bd-44ae5e013212 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] 172.30.207.100 - - [06/Mar/2015 09:24:42] "PUT /images/c80924f4-f412-4d4c-82fe-faf3179b9774 HTTP/1.1" 200 677 0.026189

==> /var/log/glance/api.log <==
2015-03-06 09:24:43.123 14361 INFO glance.wsgi.server [4fb8a6c0-9b48-4c8b-aa3f-32b72283f508 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] 172.30.207.102 - - [06/Mar/2015 09:24:43] "POST /v1/images HTTP/1.1" 400 372 1.047594

Comment 3 Jack Waterworth 2015-03-09 16:02:32 UTC
It appears this bug has been reported upstream and fixed for the kilo release found here https://bugs.launchpad.net/glance/+bug/1402354

Comment 16 Jaroslav Henner 2015-08-18 12:16:30 UTC
I have got these errors so far when checking this on RHOS 6

python-oslo-vmware-0.6.0-1.el7ost.noarch
python-oslo-config-1.4.0-1.el7ost.noarch
python-oslo-vmware-0.6.0-1.el7ost.noarch


2015-08-18 08:10:48.693 25559 TRACE stevedore.extension   File "/usr/lib/python2.7/site-packages/glance_store/_drivers/vmware_datastore.py", line 27, in <module>
2015-08-18 08:10:48.693 25559 TRACE stevedore.extension     from oslo_utils import excutils
2015-08-18 08:10:48.693 25559 TRACE stevedore.extension ImportError: No module named oslo_utils

2015-08-18 08:09:00.087 25308 TRACE glance   File "/usr/lib/python2.7/site-packages/stevedore/driver.py", line 50, in _default_on_load_failure
2015-08-18 08:09:00.087 25308 TRACE glance     raise err
2015-08-18 08:09:00.087 25308 TRACE glance ImportError: No module named oslo_config

changing the imports to the oslo.config and oslo.utils seemed to help

Comment 17 Jaroslav Henner 2015-08-18 12:24:04 UTC
Apart from that I wrote above, when trying to upload the image, I am getting

2015-08-18 08:23:01.265 27651 ERROR glance.api.v1.upload_utils [fb2db524-94b9-4ae5-befe-33cbf53193ef 24e9dcd37953452ea7429c1469609a40 6ee41f516a284148941732b3a664fed8 - - -] Failed to upload image 9e173d95-dbe2-4eca-bc51-f2505801fb0a
2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils Traceback (most recent call last):
2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils   File "/usr/lib/python2.7/site-packages/glance/api/v1/upload_utils.py", line 113, in upload_data_to_store
2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils     context=req.context)
2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils   File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 331, in store_add_to_backend
2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils     context=context)
2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils   File "/usr/lib/python2.7/site-packages/glance_store/_drivers/vmware_datastore.py", line 338, in add
2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils     cookie = self._build_vim_cookie_header(True)
2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils   File "/usr/lib/python2.7/site-packages/glance_store/_drivers/vmware_datastore.py", line 299, in _build_vim_cookie_header
2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils     if verify_session and not self.session.is_current_session_active():
2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils AttributeError: 'VMwareAPISession' object has no attribute 'is_current_session_active'
2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils 
2015-08-18 08:23:01.338 27651 INFO glance.wsgi.server [fb2db524-94b9-4ae5-befe-33cbf53193ef 24e9dcd37953452ea7429c1469609a40 6ee41f516a284148941732b3a664fed8 - - -] 10.34.69.130 - - [18/Aug/2015 08:23:01] "POST /v1/images HTTP/1.1" 500 431 0.546157

Comment 18 Jaroslav Henner 2015-08-18 12:27:44 UTC
Adding underscore ( _is_current_sessi...) seems to help.

Comment 19 Jaroslav Henner 2015-08-18 13:25:32 UTC
It seems image can be uploaded fine after doing the two manual fixes above, but BZ#1361197 prevents us to use it.

Comment 20 Flavio Percoco 2015-08-19 08:38:33 UTC
In the Juno release, oslo packages used to use python namespaces. Hence the need for `.`

If backported to Juno, these imports need to be fixed. I'll propose a patch for the `_` issue upstream.

Comment 21 Flavio Percoco 2015-08-19 08:42:11 UTC
Actually, the `_` issue is another backport specific change. In Juno, that method used to be private.

Comment 32 errata-xmlrpc 2015-08-24 14:48:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-1659.html

Comment 33 Red Hat Bugzilla 2023-09-18 00:11:17 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days