Description of problem: Glance vSphere datastore image upload fails to complete with 104 error (connection reset by peer) Steps to Reproduce: 1. configure glance for vmware vsphere 2. restart glance services 3. attempt to upload glance image Actual results: # glance image-create --progress --name 'cirros2' --disk-format 'vmdk' --container-format 'bare' --is-public 'true' --file '/root/CirrOS0.3.3-vmdk.raw'; [=============================>] 100% <html> <head> <title>400 Bad Request</title> </head> <body> <h1>400 Bad Request</h1> Client disconnected before sending all data to backend<br /><br /> Expected results: image should be successfully uploaded Additional info: # tail -f /var/log/glance/api.log /var/log/glance/registry.log ==> /var/log/glance/api.log <== 2015-03-06 09:24:42.081 14361 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.30.207.143 ==> /var/log/glance/registry.log <== 2015-03-06 09:24:42.239 14434 INFO urllib3.connectionpool [-] Resetting dropped connection: 172.30.207.143 2015-03-06 09:24:42.393 14434 INFO glance.registry.api.v1.images [b42b72e1-479e-4204-b653-c6a2508f1941 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] Successfully created image c 80924f4-f412-4d4c-82fe-faf3179b9774 2015-03-06 09:24:42.394 14434 INFO glance.wsgi.server [b42b72e1-479e-4204-b653-c6a2508f1941 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] 172.30.207.100 - - [06/Mar/2015 09:24:4 2] "POST /images HTTP/1.1" 200 677 0.156951 ==> /var/log/glance/api.log <== 2015-03-06 09:24:42.396 14361 INFO oslo.messaging._drivers.impl_rabbit [4fb8a6c0-9b48-4c8b-aa3f-32b72283f508 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] Connecting to AMQP ser ver on 172.30.207.153:5672 2015-03-06 09:24:42.408 14361 INFO oslo.messaging._drivers.impl_rabbit [4fb8a6c0-9b48-4c8b-aa3f-32b72283f508 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] Connected to AMQP serv er on 172.30.207.153:5672 ==> /var/log/glance/registry.log <== 2015-03-06 09:24:42.415 14436 INFO urllib3.connectionpool [-] Resetting dropped connection: 172.30.207.143 2015-03-06 09:24:42.558 14436 INFO glance.wsgi.server [2352910c-e0d8-47c6-8a72-72cafe9cd46d 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] 172.30.207.100 - - [06/Mar/2015 09:24:4 2] "GET /images/c80924f4-f412-4d4c-82fe-faf3179b9774 HTTP/1.1" 200 677 0.143998 2015-03-06 09:24:42.561 14430 INFO urllib3.connectionpool [-] Resetting dropped connection: 172.30.207.143 2015-03-06 09:24:42.718 14430 INFO glance.registry.api.v1.images [216319bf-ef3f-4475-bac3-5ce928d6743d 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] Updating metadata for image c80924f4-f412-4d4c-82fe-faf3179b9774 2015-03-06 09:24:42.719 14430 INFO glance.wsgi.server [216319bf-ef3f-4475-bac3-5ce928d6743d 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] 172.30.207.100 - - [06/Mar/2015 09:24:4 2] "PUT /images/c80924f4-f412-4d4c-82fe-faf3179b9774 HTTP/1.1" 200 677 0.158904 ==> /var/log/glance/api.log <== 2015-03-06 09:24:42.740 14361 ERROR glance_store._drivers.vmware_datastore [4fb8a6c0-9b48-4c8b-aa3f-32b72283f508 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] Failed to upload c ontent of image c80924f4-f412-4d4c-82fe-faf3179b9774 2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore Traceback (most recent call last): 2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore File "/usr/lib/python2.7/site-packages/glance_store/_drivers/vmware_datastore.py", line 346, in add 2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore content=image_file) 2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore File "/usr/lib/python2.7/site-packages/glance_store/_drivers/vmware_datastore.py", line 489, in _get_http_conn 2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore conn.request(method, url, content, headers) 2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore File "/usr/lib64/python2.7/httplib.py", line 973, in request 2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore self._send_request(method, url, body, headers) 2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore File "/usr/lib64/python2.7/httplib.py", line 1007, in _send_request 2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore self.endheaders(body) 2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore File "/usr/lib64/python2.7/httplib.py", line 969, in endheaders 2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore self._send_output(message_body) 2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore File "/usr/lib64/python2.7/httplib.py", line 833, in _send_output 2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore self.send(message_body) 2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore File "/usr/lib64/python2.7/httplib.py", line 802, in send 2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore self.sock.sendall(datablock) 2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore File "/usr/lib/python2.7/site-packages/eventlet/green/ssl.py", line 137, in sendall 2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore v = self.send(data[count:]) 2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore File "/usr/lib/python2.7/site-packages/eventlet/green/ssl.py", line 113, in send 2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore super(GreenSSLSocket, self).send, data, flags) 2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore File "/usr/lib/python2.7/site-packages/eventlet/green/ssl.py", line 80, in _call_trampolining 2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore return func(*a, **kw) 2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore File "/usr/lib64/python2.7/ssl.py", line 198, in send 2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore v = self._sslobj.write(data) 2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore error: [Errno 104] Connection reset by peer 2015-03-06 09:24:42.740 14361 TRACE glance_store._drivers.vmware_datastore ==> /var/log/glance/registry.log <== 2015-03-06 09:24:42.768 14430 INFO glance.registry.api.v1.images [f3a2dcee-6d22-4a34-85bd-44ae5e013212 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] Updating metadata for image c80924f4-f412-4d4c-82fe-faf3179b9774 2015-03-06 09:24:42.769 14430 INFO glance.wsgi.server [f3a2dcee-6d22-4a34-85bd-44ae5e013212 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] 172.30.207.100 - - [06/Mar/2015 09:24:42] "PUT /images/c80924f4-f412-4d4c-82fe-faf3179b9774 HTTP/1.1" 200 677 0.026189 ==> /var/log/glance/api.log <== 2015-03-06 09:24:43.123 14361 INFO glance.wsgi.server [4fb8a6c0-9b48-4c8b-aa3f-32b72283f508 47e2a8aefc024d45b767da4123304130 64b560aab329438c9b6a75dc51059150 - - -] 172.30.207.102 - - [06/Mar/2015 09:24:43] "POST /v1/images HTTP/1.1" 400 372 1.047594
It appears this bug has been reported upstream and fixed for the kilo release found here https://bugs.launchpad.net/glance/+bug/1402354
I have got these errors so far when checking this on RHOS 6 python-oslo-vmware-0.6.0-1.el7ost.noarch python-oslo-config-1.4.0-1.el7ost.noarch python-oslo-vmware-0.6.0-1.el7ost.noarch 2015-08-18 08:10:48.693 25559 TRACE stevedore.extension File "/usr/lib/python2.7/site-packages/glance_store/_drivers/vmware_datastore.py", line 27, in <module> 2015-08-18 08:10:48.693 25559 TRACE stevedore.extension from oslo_utils import excutils 2015-08-18 08:10:48.693 25559 TRACE stevedore.extension ImportError: No module named oslo_utils 2015-08-18 08:09:00.087 25308 TRACE glance File "/usr/lib/python2.7/site-packages/stevedore/driver.py", line 50, in _default_on_load_failure 2015-08-18 08:09:00.087 25308 TRACE glance raise err 2015-08-18 08:09:00.087 25308 TRACE glance ImportError: No module named oslo_config changing the imports to the oslo.config and oslo.utils seemed to help
Apart from that I wrote above, when trying to upload the image, I am getting 2015-08-18 08:23:01.265 27651 ERROR glance.api.v1.upload_utils [fb2db524-94b9-4ae5-befe-33cbf53193ef 24e9dcd37953452ea7429c1469609a40 6ee41f516a284148941732b3a664fed8 - - -] Failed to upload image 9e173d95-dbe2-4eca-bc51-f2505801fb0a 2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils Traceback (most recent call last): 2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/site-packages/glance/api/v1/upload_utils.py", line 113, in upload_data_to_store 2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils context=req.context) 2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 331, in store_add_to_backend 2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils context=context) 2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/site-packages/glance_store/_drivers/vmware_datastore.py", line 338, in add 2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils cookie = self._build_vim_cookie_header(True) 2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/site-packages/glance_store/_drivers/vmware_datastore.py", line 299, in _build_vim_cookie_header 2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils if verify_session and not self.session.is_current_session_active(): 2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils AttributeError: 'VMwareAPISession' object has no attribute 'is_current_session_active' 2015-08-18 08:23:01.265 27651 TRACE glance.api.v1.upload_utils 2015-08-18 08:23:01.338 27651 INFO glance.wsgi.server [fb2db524-94b9-4ae5-befe-33cbf53193ef 24e9dcd37953452ea7429c1469609a40 6ee41f516a284148941732b3a664fed8 - - -] 10.34.69.130 - - [18/Aug/2015 08:23:01] "POST /v1/images HTTP/1.1" 500 431 0.546157
Adding underscore ( _is_current_sessi...) seems to help.
It seems image can be uploaded fine after doing the two manual fixes above, but BZ#1361197 prevents us to use it.
In the Juno release, oslo packages used to use python namespaces. Hence the need for `.` If backported to Juno, these imports need to be fixed. I'll propose a patch for the `_` issue upstream.
Actually, the `_` issue is another backport specific change. In Juno, that method used to be private.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-1659.html
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days