Bug 1298265 - CDH 5.3 scenario tests fails because jobs stay in running state
Summary: CDH 5.3 scenario tests fails because jobs stay in running state
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-sahara
Version: 8.0 (Liberty)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ga
: 8.0 (Liberty)
Assignee: Elise Gafford
QA Contact: Luigi Toscano
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-01-13 16:00 UTC by Luigi Toscano
Modified: 2016-01-19 16:35 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-01-19 16:34:46 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Scenario test log of CDH 5.3.0 RHEL 7.2 image, RHEL-OSP8 (13.48 KB, text/plain)
2016-01-13 16:08 UTC, Luigi Toscano
no flags Details

Description Luigi Toscano 2016-01-13 16:00:35 UTC
Description of problem:
Scenario tests (from master commit bce48970e81baf61a07d43cca27e1e4699ac62a0) fails with a CDH 5.3.0 RHEL 7.2 image. The cluster is created but the jobs stays in RUNNING state "forever" (raising the waiting time for jobs to 7200 seconds did not help).

Please note that the tests do *pass* for CDH 5.4.0 and HDP 2.0.6.

Version-Release number of selected component (if applicable):
openstack-sahara-common-3.0.0-5.el7ost.noarch
openstack-sahara-engine-3.0.0-5.el7ost.noarch
openstack-sahara-api-3.0.0-5.el7ost.noarch

The images have been generated with 
sahara-image-elements-3.0.0-3.el7ost.noarch

Apparently no relevant information in sahara-api and sahara-engine logs.

How reproducible:
$ python sahara/tests/scenario/runner.py --verbose -V scenario.ini etc/scenario/sahara-ci/credentials.yaml.mako etc/scenario/sahara-ci/edp.yaml.mako etc/scenario/sahara-ci/cdh-5.3.0.yaml.mako
[...]
Captured stderr:
~~~~~~~~~~~~~~~~
    /usr/lib/python2.7/site-packages/keystoneclient/service_catalog.py:196: UserWarning: Providing attr without filter_value to get_urls() is deprecated as of the 1.7.0 release and may be removed in the
 2.0.0 release. Either both should be provided or neither should be provided.
      'Providing attr without filter_value to get_urls() is '
    Traceback (most recent call last):

      File "sahara/tests/scenario/base.py", line 63, in wrapper
        return fct(self, *args, **kwargs)

      File "sahara/tests/scenario/base.py", line 198, in check_run_jobs
        self._job_batching(pre_exec)

      File "sahara/tests/scenario/base.py", line 207, in _job_batching
        self._poll_jobs_status(job_exec_ids)

      File "sahara/tests/scenario/base.py", line 314, in _poll_jobs_status
        self.fail("\n".join(report))

      File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 666, in fail
        raise self.failureException(msg)

    AssertionError: Job with id=9665bb83-b8fb-40ac-9adb-c082b8a5023b, name=test-0ba01c5f, type=Pig has status RUNNING
    Job with id=c480e198-2c19-4aa6-ae43-8d4178da6ed4, name=test-b1105ee4, type=MapReduce has status RUNNING
    Job with id=6847e9ec-f13b-4c7c-b24e-56e5748a36be, name=test-7802e9f5, type=MapReduce.Streaming has status RUNNING
    Job with id=31ebcb98-9f4a-4014-9ac7-e6d0929ce761, name=test-da2b82d1, type=Java has status RUNNING



Captured stdout:
~~~~~~~~~~~~~~~~
    Results of testing plugin cdh 5.3.0
    +-----------------------------+--------+-------------+
    | Check                       | Status | Duration, s |
    +-----------------------------+--------+-------------+
    | Create node group templates |   OK   |      0      |
    | Set flavor                  |   OK   |      0      |
    | Set flavor                  |   OK   |      0      |
    | Set flavor                  |   OK   |      0      |
    | Set flavor                  |   OK   |      0      |
    | Set flavor                  |   OK   |      0      |
    | Set flavor                  |   OK   |      0      |
    | Create cluster template     |   OK   |      0      |
    | Create cluster              |   OK   |      4      |
    | Check cluster state         |   OK   |     1561    |
    | Check cinder volumes        |   OK   |      2      |
    | Check event logs            |   OK   |      0      |
    | Check EDP jobs              | FAILED |     7203    |
    | Check sentry                |   OK   |      0      |
    +-----------------------------+--------+-------------+

Comment 1 Luigi Toscano 2016-01-13 16:08:41 UTC
Created attachment 1114445 [details]
Scenario test log of CDH 5.3.0 RHEL 7.2 image, RHEL-OSP8

Comment 2 Keith Basil 2016-01-19 16:35:11 UTC
We are moving to CDH5.4.


Note You need to log in before you can comment on or make changes to this bug.