Bug 1305419 - Failure to launch Hadoop HDP 2.0.6 Cluster via Sahara Wizards on RDO Liberty
Failure to launch Hadoop HDP 2.0.6 Cluster via Sahara Wizards on RDO Liberty
Status: CLOSED NOTABUG
Product: RDO
Classification: Community
Component: openstack-sahara (Show other bugs)
Liberty
Unspecified Unspecified
unspecified Severity unspecified
: ---
: Liberty
Assigned To: Elise Gafford
Luigi Toscano
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-08 01:57 EST by Boris Derzhavets
Modified: 2016-07-13 12:44 EDT (History)
0 users

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-07-13 12:44:16 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Boris Derzhavets 2016-02-08 01:57:22 EST
Description of problem:

I registered  Ubutu 14.04 image (also Ubntu 15.10 glance image was tested) with SAHARA on RDO Liberty with corresponding tags "hdp" and "2.0.6"

Then I created Master && Slave templates initializing daemons follow
samples :-
[root ServerCentOS7 version_2_0_6(keystone_admin)]# cat master.json
{
    "plugin_name": "hdp",
    "hadoop_version": "2.0.6",
    "node_processes": [
        "NAMENODE",
        "AMBARI_SERVER",
        "RESOURCEMANAGER",
        "HISTORYSERVER",
        "ZOOKEEPER_SERVER"
    ],
    "name": "hdp-206-default-master",
    "floating_ip_pool": "{floating_ip_pool}",
    "flavor_id": "{flavor_id}",
    "auto_security_group": "{auto_security_group}",
    "security_groups": "{security_groups}"
}


[root ServerCentOS7 version_2_0_6(keystone_admin)]# cat worker.json
{
    "plugin_name": "hdp",
    "hadoop_version": "2.0.6",
    "node_processes": [
        "DATANODE",
        "NODEMANAGER"
    ],
    "name": "hdp-206-default-worker",
    "floating_ip_pool": "{floating_ip_pool}",
    "flavor_id": "{flavor_id}",
    "auto_security_group": "{auto_security_group}",
    "security_groups": "{security_groups}

Creating Cluster's template I multiplied by 3 Slave's template
trying to fork 4 VMs at a time.

Finally spawning the HDP 2.0.6 Cluster results 
an error /val/log/sahara/sahara-engine.log :-


3eb98f8c6e7] HeatStackException: Heat stack failed with status Resource CREATE failed: resources.hdp-206-default-master: OverQuotaClient: resources[0].resources.hdp-206-cluster1-hdp-206-default-master-b33b997e: Quota exceeded for resources: ['security_group_rule']


Version-Release number of selected component (if applicable):

[root ServerCentOS7 ~]# rpm -qa \*sahara\*
openstack-sahara-engine-3.0.0-5.cc218ddgit.el7.noarch
python-saharaclient-0.11.1-1.el7.noarch
openstack-sahara-api-3.0.0-5.cc218ddgit.el7.noarch
openstack-sahara-common-3.0.0-5.cc218ddgit.el7.noarch

How reproducible:


Steps to Reproduce:
1. Create Master and Work templates
2. Create Cluster template 1xMaster + 3xWork 
3. Attempt to launch cluster via registered glance image Ubuntu 14.04 ( or
or Ubuntu 15.10 )

Actual results:

Finally spawning the HDP 2.0.6 Cluster results 
an error /val/log/sahara/sahara-engine.log :-


3eb98f8c6e7] HeatStackException: Heat stack failed with status Resource CREATE failed: resources.hdp-206-default-master: OverQuotaClient: resources[0].resources.hdp-206-cluster1-hdp-206-default-master-b33b997e: Quota exceeded for resources: ['security_group_rule']
26-02-07 21:21:01.359 1384 ERROR sahara.service.ops [instance: none, cluster: 902d7bc4-a0cb-45dc-9e81-33eb98f8c6e7] Error ID: 1826c6f4-bbc2-4c91-be0c-57bdcd02faf2
2016-02-07 21:21:01.359 1384 ERROR sahara.service.ops [instance: none, cluster: 902d7bc4-a0cb-45dc-9e81-33eb98f8c6e7]
2016-02-07 21:21:10.701 1384 WARNING sahara.service.heat.heat_engine [req-c0192e44-e84c-4f59-bf80-19c9584baa39 ] [instance: none, cluster: 902d7bc4-a0cb-45dc-9e81-33eb98f8c6e7] Cluster creation rollback (reason: Heat stack failed with status Resource CREATE failed: resources.hdp-206-default-master: OverQuotaClient: resources[0].resources.hdp-206-cluster1-hdp-206-default-master-b33b997e: Quota exceeded for resources: ['security_group_rule']
Error ID: 1826c6f4-bbc2-4c91-be0c-57bdcd02faf2)
2016-02-07 21:21:10.958 1384 INFO sahara.utils.cluster [req-c0192e44-e84c-4f59-bf80-19c9584baa39 ] [instance: none, cluster: 902d7bc4-a0cb-45dc-9e81-33eb98f8c6e7] Cluster status has been changed. New status=Error 


Expected results:

Cluster goes goes to Active. Hadoop Cluster HDP 2.0.6 gets deployed
across Master and 3 Work cloud VMs.

Additional info:
Comment 1 Luigi Toscano 2016-02-08 05:12:12 EST
Please raise the quota of your tenant. This is something that should be probably reported in the upstream documentation.

For example, the upstream Sahara CI increases the values of security group rules as follow:

https://github.com/openstack/sahara-ci-config/blob/master/config/devstack/local.sh#L63
Comment 2 Boris Derzhavets 2016-02-08 07:33:59 EST
I apologize for inconvenience. I am doing upstream image download for HDP 2.0.6
( 100 KB/sec ). Would I have problems with upstream image I will reopen  the
bug. For now status is closed. Upstream image for Vanilla 2.7.1 does work.
Comment 3 Luigi Toscano 2016-02-08 07:46:09 EST
I think you will still have to increase the quota for security group rules in your tenant, please consider this when performing your testing.
Comment 4 Elise Gafford 2016-02-08 08:34:31 EST
Luigi is correct; the default sec group rules can be insufficient for spawning Sahara clusters. Because the HDP cluster requires more rules (there are more services running), it is quite possible that the less rule-intense vanilla cluster could spawn using sec groups beneath a certain threshold, but that the HDP cluster would not. This is a stack operations issue, and the quota system is behaving appropriately (I increase the quotas for this setting regularly in my own envs, for this exact reason). If you find any other issues as you go, though, please do report them. Thanks!
Comment 5 Boris Derzhavets 2016-02-10 01:29:34 EST
Thanks to advise of Luigi Toscano [1] HDP 2.0.6 plug-in was successfully
tested. I was able to launch 5 VMs  Namenode, SecondaryNamenode , 3*(Worker Nodes) on top of RDO Liberty via templates generated in Sahara GUI environment.
Diagnostic obtained via http://master:50070 looks OK for me.
Quotas update via
https://github.com/openstack/sahara-ci-config/blob/master/config/devstack/local.sh#L63
seems to be important for both HDP 2.0.6 and CDH 5.0.4 Hadoop clusters launching
Comment 6 Elise Gafford 2016-07-13 12:44:16 EDT
Configuration fix r/t quotas successful per reporter.

Note You need to log in before you can comment on or make changes to this bug.