Bug 1298265

Summary: CDH 5.3 scenario tests fails because jobs stay in running state
Product: Red Hat OpenStack Reporter: Luigi Toscano <ltoscano>
Component: openstack-saharaAssignee: Elise Gafford <egafford>
Status: CLOSED WONTFIX QA Contact: Luigi Toscano <ltoscano>
Severity: high Docs Contact:
Priority: high    
Version: 8.0 (Liberty)CC: kbasil, matt, mimccune, pkshiras, yeylon
Target Milestone: ga   
Target Release: 8.0 (Liberty)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-01-19 16:34:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Scenario test log of CDH 5.3.0 RHEL 7.2 image, RHEL-OSP8 none

Description Luigi Toscano 2016-01-13 16:00:35 UTC
Description of problem:
Scenario tests (from master commit bce48970e81baf61a07d43cca27e1e4699ac62a0) fails with a CDH 5.3.0 RHEL 7.2 image. The cluster is created but the jobs stays in RUNNING state "forever" (raising the waiting time for jobs to 7200 seconds did not help).

Please note that the tests do *pass* for CDH 5.4.0 and HDP 2.0.6.

Version-Release number of selected component (if applicable):
openstack-sahara-common-3.0.0-5.el7ost.noarch
openstack-sahara-engine-3.0.0-5.el7ost.noarch
openstack-sahara-api-3.0.0-5.el7ost.noarch

The images have been generated with 
sahara-image-elements-3.0.0-3.el7ost.noarch

Apparently no relevant information in sahara-api and sahara-engine logs.

How reproducible:
$ python sahara/tests/scenario/runner.py --verbose -V scenario.ini etc/scenario/sahara-ci/credentials.yaml.mako etc/scenario/sahara-ci/edp.yaml.mako etc/scenario/sahara-ci/cdh-5.3.0.yaml.mako
[...]
Captured stderr:
~~~~~~~~~~~~~~~~
    /usr/lib/python2.7/site-packages/keystoneclient/service_catalog.py:196: UserWarning: Providing attr without filter_value to get_urls() is deprecated as of the 1.7.0 release and may be removed in the
 2.0.0 release. Either both should be provided or neither should be provided.
      'Providing attr without filter_value to get_urls() is '
    Traceback (most recent call last):

      File "sahara/tests/scenario/base.py", line 63, in wrapper
        return fct(self, *args, **kwargs)

      File "sahara/tests/scenario/base.py", line 198, in check_run_jobs
        self._job_batching(pre_exec)

      File "sahara/tests/scenario/base.py", line 207, in _job_batching
        self._poll_jobs_status(job_exec_ids)

      File "sahara/tests/scenario/base.py", line 314, in _poll_jobs_status
        self.fail("\n".join(report))

      File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 666, in fail
        raise self.failureException(msg)

    AssertionError: Job with id=9665bb83-b8fb-40ac-9adb-c082b8a5023b, name=test-0ba01c5f, type=Pig has status RUNNING
    Job with id=c480e198-2c19-4aa6-ae43-8d4178da6ed4, name=test-b1105ee4, type=MapReduce has status RUNNING
    Job with id=6847e9ec-f13b-4c7c-b24e-56e5748a36be, name=test-7802e9f5, type=MapReduce.Streaming has status RUNNING
    Job with id=31ebcb98-9f4a-4014-9ac7-e6d0929ce761, name=test-da2b82d1, type=Java has status RUNNING



Captured stdout:
~~~~~~~~~~~~~~~~
    Results of testing plugin cdh 5.3.0
    +-----------------------------+--------+-------------+
    | Check                       | Status | Duration, s |
    +-----------------------------+--------+-------------+
    | Create node group templates |   OK   |      0      |
    | Set flavor                  |   OK   |      0      |
    | Set flavor                  |   OK   |      0      |
    | Set flavor                  |   OK   |      0      |
    | Set flavor                  |   OK   |      0      |
    | Set flavor                  |   OK   |      0      |
    | Set flavor                  |   OK   |      0      |
    | Create cluster template     |   OK   |      0      |
    | Create cluster              |   OK   |      4      |
    | Check cluster state         |   OK   |     1561    |
    | Check cinder volumes        |   OK   |      2      |
    | Check event logs            |   OK   |      0      |
    | Check EDP jobs              | FAILED |     7203    |
    | Check sentry                |   OK   |      0      |
    +-----------------------------+--------+-------------+

Comment 1 Luigi Toscano 2016-01-13 16:08:41 UTC
Created attachment 1114445 [details]
Scenario test log of CDH 5.3.0 RHEL 7.2 image, RHEL-OSP8

Comment 2 Keith Basil 2016-01-19 16:35:11 UTC
We are moving to CDH5.4.