Bug 1966634 - [OSP16.2]Failed to allocate memory: Cannot allocate memory , qemu-img coredump creating instances in overcloud
Summary: [OSP16.2]Failed to allocate memory: Cannot allocate memory , qemu-img coredum...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-ironic-lib
Version: 16.2 (Train)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: beta
: 16.2 (Train on RHEL 8.4)
Assignee: Julia Kreger
QA Contact: Paras Babbar
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-06-01 14:54 UTC by Paras Babbar
Modified: 2021-09-15 07:16 UTC (History)
5 users (show)

Fixed In Version: python-ironic-lib-2.21.2-1.20210603223309.cc50dab.el8ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-09-15 07:15:41 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 793351 0 None MERGED Limit the number of malloc arenas for qemu-img convert 2021-06-02 14:12:06 UTC
OpenStack gerrit 793931 0 None MERGED Increase the memory limit for qemu-img 2021-06-02 14:12:06 UTC
OpenStack gerrit 794034 0 None MERGED Limit qemu-img execution arenas 2021-06-02 14:12:06 UTC
Red Hat Product Errata RHEA-2021:3483 0 None None None 2021-09-15 07:16:07 UTC

Description Paras Babbar 2021-06-01 14:54:56 UTC 
Description of problem:

The issue is actually on the ironic conductor side which does the conversion. Following from the ironic conductor log:

2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils [req-f387e258-8ac7-4491-90bb-5169aacc48b0 fc2a283a64264b1682c1da2508ee007d 6f1ce8efc14f431b962691dc1c39eddc - default default] Node b19c943a-295e-4ca9-a13b-04e41b509c67 failed deploy step {'step': 'deploy', 'priority': 100, 'argsinfo': None, 'interface': 'deploy'} with unexpected error: Unexpected error while running command.
Command: /usr/bin/python3 -m oslo_concurrency.prlimit --as=1073741824 -- qemu-img convert -O raw /var/lib/ironic/master_images/tmpmx49mjnx/a35feba0-9a05-426e-b591-ade31665cd8a.part /var/lib/ironic/master_images/tmpmx49mjnx/a35feba0-9a05-426e-b591-ade31665cd8a.converted
Exit code: -6
Stdout: ''
Stderr: 'Failed to allocate memory: Cannot allocate memory\n': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Command: /usr/bin/python3 -m oslo_concurrency.prlimit --as=1073741824 -- qemu-img convert -O raw /var/lib/ironic/master_images/tmpmx49mjnx/a35feba0-9a05-426e-b591-ade31665cd8a.part /var/lib/ironic/master_images/tmpmx49mjnx/a35feba0-9a05-426e-b591-ade31665cd8a.converted
Exit code: -6
Stdout: ''
Stderr: 'Failed to allocate memory: Cannot allocate memory\n'
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils Traceback (most recent call last):
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils   File "/usr/lib/python3.6/site-packages/ironic/conductor/manager.py", line 3951, in _do_next_deploy_step
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils     result = interface.execute_deploy_step(task, step)
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils   File "/usr/lib/python3.6/site-packages/ironic/drivers/base.py", line 360, in execute_deploy_step
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils     return self._execute_step(task, step)
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils   File "/usr/lib/python3.6/site-packages/ironic/drivers/base.py", line 283, in _execute_step
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils     return getattr(self, step['step'])(task)
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils   File "/usr/lib/python3.6/site-packages/ironic_lib/metrics.py", line 60, in wrapped
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils     result = f(*args, **kwargs)
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils   File "/usr/lib/python3.6/site-packages/ironic/conductor/task_manager.py", line 148, in wrapper
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils     return f(*args, **kwargs)
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils   File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 422, in deploy
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils     deploy_utils.cache_instance_image(task.context, node)
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils   File "/usr/lib/python3.6/site-packages/ironic_lib/metrics.py", line 60, in wrapped
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils     result = f(*args, **kwargs)
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils   File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/deploy_utils.py", line 1197, in cache_instance_image
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils     force_raw)
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils   File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/deploy_utils.py", line 531, in fetch_images
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils     cache.fetch_image(href, path, ctx=ctx, force_raw=force_raw)
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils   File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/image_cache.py", line 140, in fetch_image
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils     href, master_path, dest_path, ctx=ctx, force_raw=force_raw)
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils   File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/image_cache.py", line 167, in _download_image
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils     _fetch(ctx, href, tmp_path, force_raw)
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils   File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/image_cache.py", line 324, in _fetch
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils     images.image_to_raw(image_href, path, path_tmp)
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils   File "/usr/lib/python3.6/site-packages/ironic/common/images.py", line 361, in image_to_raw
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils     disk_utils.convert_image(path_tmp, staged, 'raw')
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils   File "/usr/lib/python3.6/site-packages/ironic_lib/disk_utils.py", line 413, in convert_image
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils     use_standard_locale=True)
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils   File "/usr/lib/python3.6/site-packages/ironic_lib/utils.py", line 99, in execute
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils     result = processutils.execute(*cmd, **kwargs)
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils   File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 424, in execute
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils     cmd=sanitized_cmd)
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils Command: /usr/bin/python3 -m oslo_concurrency.prlimit --as=1073741824 -- qemu-img convert -O raw /var/lib/ironic/master_images/tmpmx49mjnx/a35feba0-9a05-426e-b591-ade31665cd8a.part /var/lib/ironic/master_images/tmpmx49mjnx/a35feba0-9a05-426e-b591-ade31665cd8a.converted
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils Exit code: -6
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils Stdout: ''
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils Stderr: 'Failed to allocate memory: Cannot allocate memory\n'
2020-12-02 16:11:58.756 8 ERROR ironic.conductor.utils 

Ironic's max memory limit for image conversion is 1024 MB: 

# grep image_convert_memory_limit /var/lib/config-data/puppet-generated/ironic/etc/ironic/ironic.conf
#image_convert_memory_limit = 1024

Version-Release number of selected component (if applicable):


How reproducible:
everytime(if we keep the memory to 1024)
Steps to Reproduce:
1. Deploy OSP with ironic in OC
2. run /usr/bin/python3 -m oslo_concurrency.prlimit --as=1073741824 -- qemu-img convert -O raw /var/lib/ironic/master_images/tmpmx49mjnx/a35feba0-9a05-426e-b591-ade31665cd8a.part /var/lib/ironic/master_images/tmpmx49mjnx/a35feba0-9a05-426e-b591-ade31665cd8a.converted
3. image conversion failed with OOM

Actual results:
failed image conversion

Expected results:

passed image conversion
Additional info:
Comment 7 errata-xmlrpc 2021-09-15 07:15:41 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform (RHOSP) 16.2 enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:3483
Note You need to log in before you can comment on or make changes to this bug.