Fedora Account System
Red Hat Associate
Red Hat Customer
Created attachment 1536623 [details] Log from starting tendrl-monitoring-integration service Description of problem: Service tendrl-monitoring-integration fails to start, if https is configured for Web Administration (accordingly to proposed steps in Bug 1634719). Version-Release number of selected component (if applicable): Red Hat Enterprise Linux Server release 7.6 (Maipo) grafana-4.6.4-1.el7rhgs.x86_64 graphite-web-1.1.4-1.el7rhgs.noarch httpd-2.4.6-88.el7.x86_64 httpd-tools-2.4.6-88.el7.x86_64 mod_ssl-2.4.6-88.el7.x86_64 python-httplib2-0.9.2-1.el7.noarch tendrl-ansible-1.6.3-11.el7rhgs.noarch tendrl-api-1.6.3-12.el7rhgs.noarch tendrl-api-httpd-1.6.3-12.el7rhgs.noarch tendrl-commons-1.6.3-16.el7rhgs.noarch tendrl-grafana-plugins-1.6.3-20.el7rhgs.noarch tendrl-grafana-selinux-1.5.4-3.el7rhgs.noarch tendrl-monitoring-integration-1.6.3-20.el7rhgs.noarch tendrl-node-agent-1.6.3-17.el7rhgs.noarch tendrl-notifier-1.6.3-4.el7rhgs.noarch tendrl-selinux-1.5.4-3.el7rhgs.noarch tendrl-ui-1.6.3-15.el7rhgs.noarch How reproducible: 100% Steps to Reproduce: 1. Install and configure Gluster Cluster and Web Administration. 2. Configure HTTPS for Web Administration as proposed in Bug 1634719. 3. Restart tendrl-monitoring-integration service. Actual results: Service tendrl-monitoring-integration fails to start with following traceback in logs: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ # systemctl status tendrl-monitoring-integration -l ● tendrl-monitoring-integration.service - Monitoring Integration Loaded: loaded (/usr/lib/systemd/system/tendrl-monitoring-integration.service; enabled; vendor preset: disabled) Active: failed (Result: start-limit) since Wed 2019-02-20 10:32:37 CET; 19s ago Process: 16594 ExecStart=/usr/bin/tendrl-monitoring-integration (code=exited, status=1/FAILURE) Main PID: 16594 (code=exited, status=1/FAILURE) Feb 20 10:32:36 tendrl-server systemd[1]: tendrl-monitoring-integration.service: main process exited, code=exited, status=1/FAILURE Feb 20 10:32:36 tendrl-server systemd[1]: Unit tendrl-monitoring-integration.service entered failed state. Feb 20 10:32:36 tendrl-server systemd[1]: tendrl-monitoring-integration.service failed. Feb 20 10:32:37 tendrl-server systemd[1]: tendrl-monitoring-integration.service holdoff time over, scheduling restart. Feb 20 10:32:37 tendrl-server systemd[1]: Stopped Monitoring Integration. Feb 20 10:32:37 tendrl-server systemd[1]: start request repeated too quickly for tendrl-monitoring-integration.service Feb 20 10:32:37 tendrl-server systemd[1]: Failed to start Monitoring Integration. Feb 20 10:32:37 tendrl-server systemd[1]: Unit tendrl-monitoring-integration.service entered failed state. Feb 20 10:32:37 tendrl-server systemd[1]: tendrl-monitoring-integration.service failed. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ # tail -F /var/log/messages | grep tendrl Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: Traceback (most recent call last): Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: File "/usr/bin/tendrl-monitoring-integration", line 11, in <module> Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: load_entry_point('tendrl-monitoring-integration==1.6.3', 'console_scripts', 'tendrl-monitoring-integration')() Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: File "/usr/lib/python2.7/site-packages/tendrl/monitoring_integration/manager/__init__.py", line 70, in main Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: monitoring_integration_manager.start() Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: File "/usr/lib/python2.7/site-packages/tendrl/monitoring_integration/manager/__init__.py", line 31, in start Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: dashboard.upload_default_dashboards() Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: File "/usr/lib/python2.7/site-packages/tendrl/monitoring_integration/grafana/dashboard.py", line 17, in upload_default_dashboards Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: main_org_id = grafana_org_utils.get_org_id(constants.MAIN_ORG) Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: File "/usr/lib/python2.7/site-packages/tendrl/monitoring_integration/grafana/grafana_org_utils.py", line 42, in get_org_id Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: auth=config.credentials Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: File "/usr/lib/python2.7/site-packages/requests/api.py", line 68, in get Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: return request('get', url, **kwargs) Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: File "/usr/lib/python2.7/site-packages/requests/api.py", line 50, in request Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: response = session.request(method=method, url=url, **kwargs) Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 464, in request Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: resp = self.send(prep, **send_kwargs) Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 602, in send Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: history = [resp for resp in gen] if allow_redirects else [] Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 195, in resolve_redirects Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: allow_redirects=False, Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: r = adapter.send(request, **kwargs) Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 431, in send Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: raise SSLError(e, request=request) Feb 20 10:32:36 tendrl-server tendrl-monitoring-integration: requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618) Feb 20 10:32:36 tendrl-server systemd: tendrl-monitoring-integration.service: main process exited, code=exited, status=1/FAILURE Feb 20 10:32:36 tendrl-server systemd: Unit tendrl-monitoring-integration.service entered failed state. Feb 20 10:32:36 tendrl-server systemd: tendrl-monitoring-integration.service failed. Feb 20 10:32:37 tendrl-server systemd: tendrl-monitoring-integration.service holdoff time over, scheduling restart. Feb 20 10:32:37 tendrl-server systemd: start request repeated too quickly for tendrl-monitoring-integration.service Feb 20 10:32:37 tendrl-server systemd: Unit tendrl-monitoring-integration.service entered failed state. Feb 20 10:32:37 tendrl-server systemd: tendrl-monitoring-integration.service failed. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For the full log see the attached log. Expected results: Service tendrl-monitoring-integration should start correctly. Additional info:
Recently I have modified grafana API call from monitoring-integration via Vhost using a reverse proxy. When we enable the self-signed certificate for vhost 443 port then monitoring-integration unable to communicate with grafana. I have reverted back this change to direct 3000 port api call. Note: Except for an internal grafana-api call(monitoring-integration API call) all other calls happening to grafana happening via vhost only. PR: https://github.com/Tendrl/monitoring-integration/pull/592
This will be tested based on comment 1.
I was able to reproduce the issue with QE SSL CA and version: tendrl-ansible-1.6.3-11.el7rhgs.noarch tendrl-api-1.6.3-12.el7rhgs.noarch tendrl-api-httpd-1.6.3-12.el7rhgs.noarch tendrl-commons-1.6.3-17.el7rhgs.noarch tendrl-grafana-plugins-1.6.3-20.el7rhgs.noarch tendrl-grafana-selinux-1.5.4-3.el7rhgs.noarch tendrl-monitoring-integration-1.6.3-20.el7rhgs.noarch tendrl-node-agent-1.6.3-18.el7rhgs.noarch tendrl-notifier-1.6.3-4.el7rhgs.noarch tendrl-selinux-1.5.4-3.el7rhgs.noarch tendrl-ui-1.6.3-15.el7rhgs.noarch It is working with: tendrl-ansible-1.6.3-11.el7rhgs.noarch tendrl-api-1.6.3-13.el7rhgs.noarch tendrl-api-httpd-1.6.3-13.el7rhgs.noarch tendrl-commons-1.6.3-17.el7rhgs.noarch tendrl-grafana-plugins-1.6.3-21.el7rhgs.noarch tendrl-grafana-selinux-1.5.4-3.el7rhgs.noarch tendrl-monitoring-integration-1.6.3-21.el7rhgs.noarch tendrl-node-agent-1.6.3-18.el7rhgs.noarch tendrl-notifier-1.6.3-4.el7rhgs.noarch tendrl-selinux-1.5.4-3.el7rhgs.noarch tendrl-ui-1.6.3-15.el7rhgs.noarch --> VERIFIED
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0660