Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1616320

Summary: [HA][CI] different test cases in CSIT's "Ha L3 Block Port" test suite fail with Check_Cluster_Is_In_Sync
Product: Red Hat OpenStack Reporter: Waldemar Znoinski <wznoinsk>
Component: opendaylightAssignee: Vishal Thapar <vthapar>
Status: CLOSED WONTFIX QA Contact: Noam Manos <nmanos>
Severity: medium Docs Contact:
Priority: medium    
Version: 13.0 (Queens)CC: aadam, mkolesni, oblaut, tjamrisk
Target Milestone: z6Keywords: Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: HA
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-03-06 16:16:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
controller-0 karaf logs
none
controller-1 karaf logs
none
controller-2 karaf logs
none
robot log.html part 1of2
none
robot log.html part 2of2 none

Description Waldemar Znoinski 2018-08-15 15:11:47 UTC
Description of problem:
recent d/s RH osp ci jobs running CSIT test suite started to show the error:

Keyword 'Check_Cluster_Is_In_Sync' failed after retrying for 1 minute. The last error was: Index 3 has incorrect status: False


Version-Release number of selected component (if applicable):
opendaylight-8.3.0-3
puddle: 2018-08-08.2

How reproducible:
75%

Steps to Reproduce:
1. deploy OSP13 with odl
2. run CSIT HA L3 block port suite
3.

Actual results:
below CSIT steps are failing:
- Allow Port On ODL1 Again
- Allow Port On ODL2 Again       
- Allow Port On ODL2 Finally
- Allow Port On ODL3 Again 
in different builds different combinations of ODLx Again/Finally fails 


Expected results:
test to pass

Additional info:
one of the d/s osp13 ci jobs showing these failures: https://url.corp.redhat.com/2350fe7

also, see attachments

Comment 1 Waldemar Znoinski 2018-08-16 08:51:38 UTC
Created attachment 1476377 [details]
controller-0 karaf logs

Comment 2 Waldemar Znoinski 2018-08-16 08:52:30 UTC
Created attachment 1476379 [details]
controller-1 karaf logs

Comment 3 Waldemar Znoinski 2018-08-16 08:53:17 UTC
Created attachment 1476380 [details]
controller-2 karaf logs

Comment 4 Waldemar Znoinski 2018-08-16 09:44:33 UTC
Created attachment 1476390 [details]
robot log.html part 1of2

Comment 5 Waldemar Znoinski 2018-08-16 09:44:59 UTC
Created attachment 1476391 [details]
robot log.html part 2of2

Comment 7 Vishal Thapar 2018-09-03 15:14:30 UTC
In the robot logs I noticed it is trying to pull EOS information but unable to because ODLtools is not installed. Would be good to have ODLtools installed and update the CSIt in tune with upstream where we added a new script to capture information about cluster status which will specifically help with issues like these.

U/S CSIT patch: https://git.opendaylight.org/gerrit/#/c/75259/

In absence of above, I am trying to figure out cluster changes from karaf log. Since test involves blocking akka ports, it is crucial to relate robot log timestamps to karaf timestamps.

Could you let me know the timezone difference between timestamps in karaf log and robot logs? They both seem to be in different timezones.

Comment 8 Waldemar Znoinski 2018-09-05 10:00:15 UTC
hey Vishal,

1. I've added installing odltools from pip to all d/s CSIT jobs, build 263 and any newer at https://url.corp.redhat.com/55dadb8 will have output of odltools as part of CSIT - you should catch the cluster sync problem in block port suite straight away

2. timezone in robot should be EDT and karaf.log UTC so timestamps in robot logs should be showing 4h behind timestamps in karaf.log

Comment 15 Franck Baudin 2019-03-06 16:16:53 UTC
As per depreciation notice [1], closing this bug. Please reopen if relevant for RHOSP13, as this is the only version shipping ODL.

[1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/14/html-single/release_notes/index#deprecated_functionality

Comment 16 Franck Baudin 2019-03-06 16:17:51 UTC
As per depreciation notice [1], closing this bug. Please reopen if relevant for RHOSP13, as this is the only version shipping ODL.

[1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/14/html-single/release_notes/index#deprecated_functionality