Bug 1640950

Created attachment 1495565 [details]
karaf log unhealthy

Description of problem:
after CSIT testing, part of which is bringing the opendaylight_api docker container down and up on controllers (HA_L2/3 test suite), as well as blocking ports (External Network suite) opendaylight_api on 1of3 overcloud controllers went into unhealthy state:

docker inspect opendaylight_api shows:
            "Health": {
                "Status": "unhealthy",
                "FailingStreak": 751,
                "Log": [
                    {
                        "Start": "2018-10-19T04:29:38.693654975-04:00",
                        "End": "2018-10-19T04:29:39.243704218-04:00",
                        "ExitCode": 1,
                        "Output": "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\n503 172.17.1.13:8081 0.005 seconds\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\ncurl: (22) The requested URL returned error: 503 Service Unavailable\n"
                    },
                    {
                        "Start": "2018-10-19T04:30:09.244109388-04:00",
                        "End": "2018-10-19T04:30:09.804145173-04:00",
                        "ExitCode": 1,
                        "Output": "\n503 172.17.1.13:8081 0.005 seconds\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\ncurl: (22) The requested URL returned error: 503 Service Unavailable\n"
                    },



Version-Release number of selected component (if applicable):
OSP14, puddle: 2018-10-10.3
opendaylight-8.3.0-4.el7ost.noarch
puppet-opendaylight-8.3.0-0.20180816081508.8c1f7e5.el7ost.noarch


How reproducible:
50%



Steps to Reproduce:
1. deploy osp14 + odl, optionally install a newer ovs2.10 as ovs2.10 in the above puddle is dying when working with ODL, more info in https://bugzilla.redhat.com/show_bug.cgi?id=1637926#c24
2. run ha_l2/3, external network csit test suites
3. observe opendaylight_api container on controller-1 goes into unhealthy state

Actual results:


Expected results:


Additional info: