Created attachment 1261584 [details] Patch for cinder/image/image_utils.py Description of problem: Backend: NetApp NFS (c-mode) Deploying a volume from image with an image larger than 20GB fails during conversion after image download with "ProcessExecutionError: Unexpected error while running command." Stack trace: 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 842, in trace_method _logging_wrapper 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume return f(*args, **kwargs) 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 842, in trace_method _logging_wrapper 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume return f(*args, **kwargs) 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/nfs_ cmode.py", line 482, in copy_image_to_volume 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume context, volume, image_service, image_id) 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 842, in trace_method _logging_wrapper 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume return f(*args, **kwargs) 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/nfs_ base.py", line 363, in copy_image_to_volume 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume context, volume, image_service, image_id) 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/remotefs.py", line 41 0, in copy_image_to_volume 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume run_as_root=run_as_root) 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/image/image_utils.py", line 252, in fetch_to_raw 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume run_as_root=run_as_root) 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/image/image_utils.py", line 350, in fetch_to_volume_format 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume data = qemu_img_info(dest, run_as_root=run_as_root) 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/image/image_utils.py", line 78, in q emu_img_info 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume prlimit=QEMU_IMG_LIMITS) 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 123, in execute 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume return processutils.execute(*cmd, **kwargs) 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 389 , in execute 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume cmd=sanitized_cmd) 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume ProcessExecutionError: Unexpected error while running command. 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume Command: /usr/bin/python2 -m oslo_concurrency.prlimit --as=1073741824 --cpu=2 -- sud o cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info /var/lib/cinder/mnt/a972c1ef86f3d8d0143da59338f98ace/volume-a266f203-95c5-4f51-bbd0-50243d b96a2a 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume Exit code: 137 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume Stdout: u'' 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume Stderr: u'' 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume Version-Release number of selected component (if applicable): RDO Newton python-cinder-9.1.2-1.el7.noarch openstack-cinder-9.1.2-1.el7.noarch How reproducible: Always Root cause analysis (and patch): After investigation, this problem is very similar to bug #2800111 The problem that the process reaches limits before it can finish. The solution is modifying QEMU_IMG_LIMITS increasing cpu time and address_space on cinder/image/image_utils.py. cpu_time=8, address_space=1 * units.Gi * 10 Patch in attach.
Can you give any more info about the image being used so I can try to reproduce this? (file format, any other interesting parameters)
Hi Eric, file format is qcow2, image size is ~20Gb. This is a Windows 2012 R2 image with customer's modifications. As it's an internal image, unfortunately I can't share. Below the details from 'openstack image show': | container_format | bare | disk_format | qcow2 | min_disk | 40 | name | Template_Win_2012_R2_64bit_auto_OpenStack | properties | description='Template Windows 2012', | protected | False | schema | /v2/schemas/image | size | 21050163200 | status | active | visibility | public | updated_at | 2017-03-01T17:06:00Z | virtual_size | None I can confirm that patching image_utils.py on all the compute nodes fixed the issue. I believe this is due to a slow network/response time from NetApp, although we have 10gb nic under the hood. Thanks, Giuseppe
Apologies, I referred to the wrong bugzilla id above ... a copy & paste mistake ... the real one is: #1402594 https://bugzilla.redhat.com/show_bug.cgi?id=1402594 Anyway, upstream is https://bugs.launchpad.net/nova/+bug/1646181
Any updates on this? Just want to know if this has been backported on the RPMs of openstack-newton Thanks
We are currently waiting for https://review.openstack.org/#/c/444996/ to merge into Cinder's stable/newton branch.
This will be in the openstack-cinder-9.1.4 builds.
Closing as change was merged.