Bug 1430756
| Summary: | NFS: problem deploying from image when image is large (more than 20GB), newton | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Community] RDO | Reporter: | Giuseppe Paterno' <gpaterno> | ||||
| Component: | openstack-cinder | Assignee: | Eric Harney <eharney> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Tzach Shefi <tshefi> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | urgent | ||||||
| Version: | unspecified | CC: | chris.brown, eharney, gpaterno, srevivo | ||||
| Target Milestone: | --- | ||||||
| Target Release: | trunk | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | openstack-cinder-9.1.4 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2017-08-25 15:36:30 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Can you give any more info about the image being used so I can try to reproduce this? (file format, any other interesting parameters) Hi Eric, file format is qcow2, image size is ~20Gb. This is a Windows 2012 R2 image with customer's modifications. As it's an internal image, unfortunately I can't share. Below the details from 'openstack image show': | container_format | bare | disk_format | qcow2 | min_disk | 40 | name | Template_Win_2012_R2_64bit_auto_OpenStack | properties | description='Template Windows 2012', | protected | False | schema | /v2/schemas/image | size | 21050163200 | status | active | visibility | public | updated_at | 2017-03-01T17:06:00Z | virtual_size | None I can confirm that patching image_utils.py on all the compute nodes fixed the issue. I believe this is due to a slow network/response time from NetApp, although we have 10gb nic under the hood. Thanks, Giuseppe Apologies, I referred to the wrong bugzilla id above ... a copy & paste mistake ... the real one is: #1402594 https://bugzilla.redhat.com/show_bug.cgi?id=1402594 Anyway, upstream is https://bugs.launchpad.net/nova/+bug/1646181 Any updates on this? Just want to know if this has been backported on the RPMs of openstack-newton Thanks We are currently waiting for https://review.openstack.org/#/c/444996/ to merge into Cinder's stable/newton branch. This will be in the openstack-cinder-9.1.4 builds. Closing as change was merged. |
Created attachment 1261584 [details] Patch for cinder/image/image_utils.py Description of problem: Backend: NetApp NFS (c-mode) Deploying a volume from image with an image larger than 20GB fails during conversion after image download with "ProcessExecutionError: Unexpected error while running command." Stack trace: 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 842, in trace_method _logging_wrapper 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume return f(*args, **kwargs) 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 842, in trace_method _logging_wrapper 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume return f(*args, **kwargs) 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/nfs_ cmode.py", line 482, in copy_image_to_volume 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume context, volume, image_service, image_id) 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 842, in trace_method _logging_wrapper 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume return f(*args, **kwargs) 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/nfs_ base.py", line 363, in copy_image_to_volume 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume context, volume, image_service, image_id) 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/remotefs.py", line 41 0, in copy_image_to_volume 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume run_as_root=run_as_root) 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/image/image_utils.py", line 252, in fetch_to_raw 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume run_as_root=run_as_root) 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/image/image_utils.py", line 350, in fetch_to_volume_format 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume data = qemu_img_info(dest, run_as_root=run_as_root) 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/image/image_utils.py", line 78, in q emu_img_info 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume prlimit=QEMU_IMG_LIMITS) 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 123, in execute 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume return processutils.execute(*cmd, **kwargs) 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 389 , in execute 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume cmd=sanitized_cmd) 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume ProcessExecutionError: Unexpected error while running command. 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume Command: /usr/bin/python2 -m oslo_concurrency.prlimit --as=1073741824 --cpu=2 -- sud o cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info /var/lib/cinder/mnt/a972c1ef86f3d8d0143da59338f98ace/volume-a266f203-95c5-4f51-bbd0-50243d b96a2a 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume Exit code: 137 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume Stdout: u'' 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume Stderr: u'' 2017-03-09 09:38:01.813 6366 ERROR cinder.volume.flows.manager.create_volume Version-Release number of selected component (if applicable): RDO Newton python-cinder-9.1.2-1.el7.noarch openstack-cinder-9.1.2-1.el7.noarch How reproducible: Always Root cause analysis (and patch): After investigation, this problem is very similar to bug #2800111 The problem that the process reaches limits before it can finish. The solution is modifying QEMU_IMG_LIMITS increasing cpu time and address_space on cinder/image/image_utils.py. cpu_time=8, address_space=1 * units.Gi * 10 Patch in attach.