Bug 1414900

Summary: python-ironic-lib: ironic in overcloud: Cannot allocate memory in /var/log/ironic/ironic-conductor.log on controller
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: python-ironic-libAssignee: RHOS Maint <rhos-maint>
Status: CLOSED ERRATA QA Contact: Alexander Chuzhoy <sasha>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 11.0 (Ocata)CC: apevec, dsneddon, dtantsur, dyasny, jjoyce, jschluet, lhh, lmartins, racedoro, sclewis
Target Milestone: Upstream M3Keywords: Triaged
Target Release: 11.0 (Ocata)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-ironic-lib-2.5.2-0.20170208212103.ace87b6.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-05-17 19:41:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Description Alexander Chuzhoy 2017-01-19 17:02:37 UTC
python-ironic-lib: ironic in overcloud: Cannot allocate memory in /var/log/ironic/ironic-conductor.log on controller 

Environment:
python-ironic-lib-2.1.2-1.el7ost.noarch
openstack-ironic-api-6.2.2-3.el7ost.noarch
python-ironic-inspector-client-1.9.0-2.el7ost.noarch
openstack-ironic-common-6.2.2-3.el7ost.noarch
openstack-ironic-conductor-6.2.2-3.el7ost.noarch
python-ironicclient-1.7.1-1.el7ost.noarch
puppet-ironic-9.4.1-2.el7ost.noarch
instack-undercloud-5.2.0-1.el7ost.noarch
openstack-tripleo-heat-templates-5.2.0-3.el7ost.noarch

Steps to reproduce:
1. Deployed a setup with controllers having 8G of ram (no swap) with ironic enabled:
openstack overcloud deploy --debug --templates --libvirt-type kvm --ntp-server clock.redhat.com --neutron-network-type vxlan --neutron-tunnel-types vxlan -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services/ironic.yaml -e virt/ceph.yaml -e virt/hostnames.yml -e virt/network/network-environment.yaml -e ironic.yaml -e flat_networks.yaml

2. Configured baremetal network, registered nodes.
3. Tried to clean:
The cleaning failed.
/var/log/ironic/ironic-conductor.log:2017-01-18 20:23:58.127 161689 ERROR ironic.conductor.manager [req-dd81e663-fc93-41b7-862f-4648c6b9bf43 - - - - -] Failed to prepare node 63a5184f-5327-4f97-a0fd-97709f0d5002 for cleaning: [Errno 12] Cannot allocate memory
/var/log/ironic/ironic-conductor.log:2017-01-18 20:23:58.127 161689 ERROR ironic.conductor.manager OSError: [Errno 12] Cannot allocate memory
/var/log/ironic/ironic-conductor.log:2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager [req-30479e22-d338-434a-a5d7-a539ffc07e95 - - - - -] Failed to prepare node 99d0b665-3343-4754-94a6-a63d66427134 for cleaning: [Errno 12] Cannot allocate memory
/var/log/ironic/ironic-conductor.log:2017-01-18 20:24:07.084 161689 ERROR ironic.conductor.manager OSError: [Errno 12] Cannot allocate memory

Comment 1 Alexander Chuzhoy 2017-01-19 17:04:58 UTC
According to the guide[1] it's enough to have 8GB on controllers:
"It is recommended to have at least 12 GiB of RAM on the undercloud and at least 8 GiB of RAM on the controllers. "

[1]http://docs.openstack.org/developer/tripleo-docs/advanced_deployment/baremetal_overcloud.html#bare-metal-instances-in-overcloud

Comment 2 Lucas Alvares Gomes 2017-01-20 10:00:08 UTC
So the issue is suppose to be fixed now. The patch upstream [0] is now merged and it has been backported to Ocata and Newton too [1]. I hope it solves the problem, if not, we will need to investigate more.

[0] https://review.openstack.org/422738
[1] https://review.openstack.org/#/q/topic:bug/1657808

Comment 3 Ramon Acedo 2017-02-24 10:43:44 UTC
Please, note that the minimum amount of RAM we support is 16 GB for the controllers [1]

[1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/html-single/director_installation_and_usage/#sect-Controller_Node_Requirements

Comment 6 Alexander Chuzhoy 2017-03-23 15:58:21 UTC
Verified:
Environment: 
python-ironic-lib-2.5.2-0.20170208212103.ace87b6.el7ost.noarch

The reported issue doesn't reproduce.

Comment 10 errata-xmlrpc 2017-05-17 19:41:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1245