Bug 1679060

Summary:

tendrl-monitoring-integration service fails to start when HTTPS is enabled

Product:

[Red Hat Storage] Red Hat Gluster Storage

Reporter:

Daniel Horák <dahorak>

Component:

web-admin-tendrl-monitoring-integration

Assignee:

gowtham <gshanmug>

Status:

CLOSED ERRATA

QA Contact:

Filip Balák <fbalak>

Severity:

high

Docs Contact:

Priority:

unspecified

Version:

rhgs-3.4

CC:

fbalak, mbukatov, nthomas, rcyriac, rhs-bugs, sankarshan

Target Milestone:

---

Keywords:

ZStream

Target Release:

RHGS 3.4.z Batch Update 4

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

tendrl-monitoring-integration-1.6.3-21.el7rhgs

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2019-03-27 03:49:38 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
Log from starting tendrl-monitoring-integration service	none

Description Daniel Horák 2019-02-20 09:43:43 UTC

Created attachment 1536623 [details]
Log from starting tendrl-monitoring-integration service

Description of problem:
  Service tendrl-monitoring-integration fails to start, if https is configured
  for Web Administration (accordingly to proposed steps in Bug 1634719).


Version-Release number of selected component (if applicable):
  Red Hat Enterprise Linux Server release 7.6 (Maipo)
  grafana-4.6.4-1.el7rhgs.x86_64
  graphite-web-1.1.4-1.el7rhgs.noarch
  httpd-2.4.6-88.el7.x86_64
  httpd-tools-2.4.6-88.el7.x86_64
  mod_ssl-2.4.6-88.el7.x86_64
  python-httplib2-0.9.2-1.el7.noarch
  tendrl-ansible-1.6.3-11.el7rhgs.noarch
  tendrl-api-1.6.3-12.el7rhgs.noarch
  tendrl-api-httpd-1.6.3-12.el7rhgs.noarch
  tendrl-commons-1.6.3-16.el7rhgs.noarch
  tendrl-grafana-plugins-1.6.3-20.el7rhgs.noarch
  tendrl-grafana-selinux-1.5.4-3.el7rhgs.noarch
  tendrl-monitoring-integration-1.6.3-20.el7rhgs.noarch
  tendrl-node-agent-1.6.3-17.el7rhgs.noarch
  tendrl-notifier-1.6.3-4.el7rhgs.noarch
  tendrl-selinux-1.5.4-3.el7rhgs.noarch
  tendrl-ui-1.6.3-15.el7rhgs.noarch


How reproducible:
  100%


Steps to Reproduce:
1. Install and configure Gluster Cluster and Web Administration.
2. Configure HTTPS for Web Administration as proposed in Bug 1634719.
3. Restart tendrl-monitoring-integration service.

Actual results:
  Service tendrl-monitoring-integration fails to start with following traceback
  in logs:
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  # systemctl status tendrl-monitoring-integration -l
  ● tendrl-monitoring-integration.service - Monitoring Integration
     Loaded: loaded (/usr/lib/systemd/system/tendrl-monitoring-integration.service; enabled; vendor preset: disabled)
     Active: failed (Result: start-limit) since Wed 2019-02-20 10:32:37 CET; 19s ago
    Process: 16594 ExecStart=/usr/bin/tendrl-monitoring-integration (code=exited, status=1/FAILURE)
   Main PID: 16594 (code=exited, status=1/FAILURE)

  Feb 20 10:32:36 tendrl-server systemd[1]: tendrl-monitoring-integration.service: main process exited, code=exited, status=1/FAILURE
  Feb 20 10:32:36 tendrl-server systemd[1]: Unit tendrl-monitoring-integration.service entered failed state.
  Feb 20 10:32:36 tendrl-server systemd[1]: tendrl-monitoring-integration.service failed.
  Feb 20 10:32:37 tendrl-server systemd[1]: tendrl-monitoring-integration.service holdoff time over, scheduling restart.
  Feb 20 10:32:37 tendrl-server systemd[1]: Stopped Monitoring Integration.
  Feb 20 10:32:37 tendrl-server systemd[1]: start request repeated too quickly for tendrl-monitoring-integration.service
  Feb 20 10:32:37 tendrl-server systemd[1]: Failed to start Monitoring Integration.
  Feb 20 10:32:37 tendrl-server systemd[1]: Unit tendrl-monitoring-integration.service entered failed state.
  Feb 20 10:32:37 tendrl-server systemd[1]: tendrl-monitoring-integration.service failed.
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  # tail -F /var/log/messages | grep tendrl 
  Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: Traceback (most recent call last):
  Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: File "/usr/bin/tendrl-monitoring-integration", line 11, in <module>
  Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: load_entry_point('tendrl-monitoring-integration==1.6.3', 'console_scripts', 'tendrl-monitoring-integration')()
  Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: File "/usr/lib/python2.7/site-packages/tendrl/monitoring_integration/manager/__init__.py", line 70, in main
  Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: monitoring_integration_manager.start()
  Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: File "/usr/lib/python2.7/site-packages/tendrl/monitoring_integration/manager/__init__.py", line 31, in start
  Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: dashboard.upload_default_dashboards()
  Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: File "/usr/lib/python2.7/site-packages/tendrl/monitoring_integration/grafana/dashboard.py", line 17, in upload_default_dashboards
  Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: main_org_id = grafana_org_utils.get_org_id(constants.MAIN_ORG)
  Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: File "/usr/lib/python2.7/site-packages/tendrl/monitoring_integration/grafana/grafana_org_utils.py", line 42, in get_org_id
  Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: auth=config.credentials
  Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: File "/usr/lib/python2.7/site-packages/requests/api.py", line 68, in get
  Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: return request('get', url, **kwargs)
  Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: File "/usr/lib/python2.7/site-packages/requests/api.py", line 50, in request
  Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: response = session.request(method=method, url=url, **kwargs)
  Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 464, in request
  Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: resp = self.send(prep, **send_kwargs)
  Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 602, in send
  Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: history = [resp for resp in gen] if allow_redirects else []
  Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 195, in resolve_redirects
  Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: allow_redirects=False,
  Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
  Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: r = adapter.send(request, **kwargs)
  Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 431, in send
  Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: raise SSLError(e, request=request)
  Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)
  Feb 20 10:32:36 tendrl-server systemd: tendrl-monitoring-integration.service: main process exited, code=exited, status=1/FAILURE
  Feb 20 10:32:36 tendrl-server systemd: Unit tendrl-monitoring-integration.service entered failed state.
  Feb 20 10:32:36 tendrl-server systemd: tendrl-monitoring-integration.service failed.
  Feb 20 10:32:37 tendrl-server systemd: tendrl-monitoring-integration.service holdoff time over, scheduling restart.
  Feb 20 10:32:37 tendrl-server systemd: start request repeated too quickly for tendrl-monitoring-integration.service
  Feb 20 10:32:37 tendrl-server systemd: Unit tendrl-monitoring-integration.service entered failed state.
  Feb 20 10:32:37 tendrl-server systemd: tendrl-monitoring-integration.service failed.
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  For the full log see the attached log.


Expected results:
  Service tendrl-monitoring-integration should start correctly.


Additional info:

Comment 1 gowtham 2019-02-20 17:55:14 UTC

Recently I have modified grafana API call from monitoring-integration via Vhost using a reverse proxy. When we enable the self-signed certificate for vhost 443 port then monitoring-integration unable to communicate with grafana. I have reverted back this change to direct 3000 port api call. 


Note:
  Except for an internal grafana-api call(monitoring-integration API call) all other calls happening to grafana happening via vhost only.

PR: https://github.com/Tendrl/monitoring-integration/pull/592

Comment 2 Martin Bukatovic 2019-02-27 13:28:45 UTC

This will be tested based on comment 1.

Comment 6 Filip Balák 2019-03-05 14:41:12 UTC

I was able to reproduce the issue with QE SSL CA and version:
tendrl-ansible-1.6.3-11.el7rhgs.noarch
tendrl-api-1.6.3-12.el7rhgs.noarch
tendrl-api-httpd-1.6.3-12.el7rhgs.noarch
tendrl-commons-1.6.3-17.el7rhgs.noarch
tendrl-grafana-plugins-1.6.3-20.el7rhgs.noarch
tendrl-grafana-selinux-1.5.4-3.el7rhgs.noarch
tendrl-monitoring-integration-1.6.3-20.el7rhgs.noarch
tendrl-node-agent-1.6.3-18.el7rhgs.noarch
tendrl-notifier-1.6.3-4.el7rhgs.noarch
tendrl-selinux-1.5.4-3.el7rhgs.noarch
tendrl-ui-1.6.3-15.el7rhgs.noarch

It is working with:
tendrl-ansible-1.6.3-11.el7rhgs.noarch
tendrl-api-1.6.3-13.el7rhgs.noarch
tendrl-api-httpd-1.6.3-13.el7rhgs.noarch
tendrl-commons-1.6.3-17.el7rhgs.noarch
tendrl-grafana-plugins-1.6.3-21.el7rhgs.noarch
tendrl-grafana-selinux-1.5.4-3.el7rhgs.noarch
tendrl-monitoring-integration-1.6.3-21.el7rhgs.noarch
tendrl-node-agent-1.6.3-18.el7rhgs.noarch
tendrl-notifier-1.6.3-4.el7rhgs.noarch
tendrl-selinux-1.5.4-3.el7rhgs.noarch
tendrl-ui-1.6.3-15.el7rhgs.noarch

--> VERIFIED

Comment 8 errata-xmlrpc 2019-03-27 03:49:38 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0660