Description of problem ====================== Service tendrl-monitoring-integration reports too long errors in systemd-journal log. It's not possible to identify individual log messages. This issue affects other python tendrl services as well, eg. tendrl-notifier (as can be seen in BZ 1513484 in log line shown in Actual Results section), which is the reason for tracking this bug under web admin tendrl commons component. Version-Release =============== tendrl-ansible-1.5.4-1.el7rhgs.noarch tendrl-api-1.5.4-2.el7rhgs.noarch tendrl-api-httpd-1.5.4-2.el7rhgs.noarch tendrl-commons-1.5.4-2.el7rhgs.noarch tendrl-grafana-plugins-1.5.4-3.el7rhgs.noarch tendrl-grafana-selinux-1.5.3-2.el7rhgs.noarch tendrl-monitoring-integration-1.5.4-3.el7rhgs.noarch tendrl-node-agent-1.5.4-2.el7rhgs.noarch tendrl-notifier-1.5.4-2.el7rhgs.noarch tendrl-selinux-1.5.3-2.el7rhgs.noarch tendrl-ui-1.5.4-2.el7rhgs.noarch How reproducible ================ 100% Steps to Reproduce ================== Reproducer from the original upstream issue: 1. Prepare machine for Tendrl Server 2. Install Tendrl Server there via tendrl-ansible, but make sure it doesn't configure etcd_connection in monitoring-integration.conf.yaml file (you can either use the commit linked above or remove line which configures this value from tendrl-ansible). 3. Open /etc/tendrl/monitoring-integration/monitoring-integration.conf.yaml and make sure that 0.0.0.0 is specified as etcd_connection. 4. See the logs of the service: journalctl -u tendrl-monitoring-integration Simplified reproducer: 1. Install RHGS WA via tendrl-ansible 2. Import gluster cluster with a volume 3. Check journald logs of various tendrl python components Note: original reproducer is more complicated, but provides results so that it's much easier to see the problem. Actual results ============== The error about the problem with connecting to etcd is not reported in a single line, so that multiple occurences of the problem are shown in the same log line, making the log hard to navigate: ``` Aug 22 17:51:29 mbukatov-usm1-server.example.com systemd[1]: Started Monitoring Integration. Aug 22 17:51:29 mbukatov-usm1-server.example.com systemd[1]: Starting Monitoring Integration... Aug 22 17:51:32 mbukatov-usm1-server.example.com systemd[1]: Stopping Monitoring Integration... Aug 22 17:51:32 mbukatov-usm1-server.example.com systemd[1]: Started Monitoring Integration. Aug 22 17:51:32 mbukatov-usm1-server.example.com systemd[1]: Starting Monitoring Integration... Aug 22 17:53:13 mbukatov-usm1-server.example.com tendrl-monitoring-integration[16421]: Creating namespace.monitoring from source tendrl.monitoring_integrationnamespace.monitoring created!Finding obje Aug 22 17:53:13 mbukatov-usm1-server.example.com tendrl-monitoring-integration[16421]: cting to central store (etcd), trying again...Error connecting to central store (etcd), trying again...Error con Aug 22 17:55:35 mbukatov-usm1-server.example.com tendrl-monitoring-integration[16421]: onnecting to central store (etcd), trying again...Error connecting to central store (etcd), trying again...Error Aug 22 17:55:35 mbukatov-usm1-server.example.com tendrl-monitoring-integration[16421]: or connecting to central store (etcd), trying again...Error connecting to central store (etcd), trying again...E Aug 22 17:57:59 mbukatov-usm1-server.example.com tendrl-monitoring-integration[16421]: .Error connecting to central store (etcd), trying again...Error connecting to central store (etcd), trying again Aug 22 17:57:59 mbukatov-usm1-server.example.com tendrl-monitoring-integration[16421]: in...Error connecting to central store (etcd), trying again...Error connecting to central store (etcd), trying a Aug 22 18:00:24 mbukatov-usm1-server.example.com tendrl-monitoring-integration[16421]: again...Error connecting to central store (etcd), trying again...Error connecting to central store (etcd), tryin Aug 22 18:00:24 mbukatov-usm1-server.example.com tendrl-monitoring-integration[16421]: ying again...Error connecting to central store (etcd), trying again...Error connecting to central store (etcd), Aug 22 18:02:48 mbukatov-usm1-server.example.com tendrl-monitoring-integration[16421]: , trying again...Error connecting to central store (etcd), trying again...Error connecting to central store (etc Aug 22 18:02:48 mbukatov-usm1-server.example.com tendrl-monitoring-integration[16421]: tcd), trying again...Error connecting to central store (etcd), trying again...Error connecting to central store Aug 22 18:05:13 mbukatov-usm1-server.example.com tendrl-monitoring-integration[16421]: e (etcd), trying again...Error connecting to central store (etcd), trying again...Error connecting to central st Aug 22 18:05:13 mbukatov-usm1-server.example.com tendrl-monitoring-integration[16421]: store (etcd), trying again...Error connecting to central store (etcd), trying again...Error connecting to centra Aug 22 18:07:37 mbukatov-usm1-server.example.com tendrl-monitoring-integration[16421]: ral store (etcd), trying again...Error connecting to central store (etcd), trying again...Error connecting to ce Aug 22 18:07:37 mbukatov-usm1-server.example.com tendrl-monitoring-integration[16421]: central store (etcd), trying again...Error connecting to central store (etcd), trying again...Error connecting t Aug 22 18:10:02 mbukatov-usm1-server.example.com tendrl-monitoring-integration[16421]: to central store (etcd), trying again...Error connecting to central store (etcd), trying again...Error connectin Aug 22 18:10:02 mbukatov-usm1-server.example.com tendrl-monitoring-integration[16421]: ting to central store (etcd), trying again...Error connecting to central store (etcd), trying again...Error conn Aug 22 18:12:26 mbukatov-usm1-server.example.com tendrl-monitoring-integration[16421]: nnecting to central store (etcd), trying again...Error connecting to central store (etcd), trying again...Error Aug 22 18:12:26 mbukatov-usm1-server.example.com tendrl-monitoring-integration[16421]: r connecting to central store (etcd), trying again...Error connecting to central store (etcd), trying again...Er ``` Logs of monitoring integration when I just install RHGS WA, import cluster and do nothing else. Log lines are too long, with individual messages not being reported on a single new log line: ``` -- Logs begin at Wed 2017-11-15 11:46:00 EST, end at Thu 2017-11-16 07:51:22 EST. -- Nov 15 12:13:13 mbukatov-usm1-server.example.com systemd[1]: Started Monitoring Integration. Nov 15 12:13:13 mbukatov-usm1-server.example.com systemd[1]: Starting Monitoring Integration... Nov 15 12:13:13 mbukatov-usm1-server.example.com tendrl-monitoring-integration[18460]: Creating namespace.monitoring from source tendrl.monitoring_integrationnamespace.monitoring created!Finding objects in namespace.monitoring.objectsRegistering object namespace.monitoring.objects.AlertTypesFinding atoms in namespace.monitoring.objects.AlertTypes.atomsFinding flows in namespace.monitoring.objects.AlertTypes.flowsRegistering object namespace.monitoring.objects.ConfigFinding atoms in namespace.monitoring.objects.Config.atomsFinding flows in namespace.monitoring.objects.Config.flowsRegistering object namespace.monitoring.objects.DefinitionFinding atoms in namespace.monitoring.objects.Definition.atomsFinding flows in namespace.monitoring.objects.Definition.flowsRegistering object namespace.monitoring.objects.TriggerFinding atoms in namespace.monitoring.objects.Trigger.atomsFinding flows in namespace.monitoring.objects.Trigger.flowsFinding flows in namespace.monitoring.flowsRegistering flow namespace.monitoring.flows.DeleteResourceFromGraphiteRegistering flow namespace.monitoring.flows.NewClusterDashboardRegistering flow namespace.monitoring.flows.UpdateDashboardRegistering flow namespace.monitoring.flows.UpdateDashboardSetup Tendrl definitions (.yml)for namespace.monitoringValidating registered (.py) flows in namespace.monitoring.flowsValidating defined (.yml) flows in namespace.monitoring.flowsValidating registered (.py) objects in namespace.monitoring.objectsValidating defined (.yml) objects in namespace.monitoring.objectsSetup Config for namespace.monitoringSetup central store clients for namespace.monitoringCreating namespace.tendrl from source tendrl.commonsnamespace.tendrl created!Finding objects in namespace.tendrl.objectsRegistering object namespace.tendrl.objects.AlertFinding atoms in namespace.tendrl.objects.Alert.atomsFinding flows in namespace.tendrl.objects.Alert.flowsRegistering object namespace.tendrl.objects.BlockDeviceFinding atoms in namespace.tendrl.objects.BlockDevice.atomsFinding flows in namespace.tendrl.objects.BlockDevice.flowsRegistering object namespace.tendrl.objects.ClusterF Nov 15 12:13:13 mbukatov-usm1-server.example.com tendrl-monitoring-integration[18460]: inding atoms in namespace.tendrl.objects.Cluster.atomsRegistering atom namespace.tendrl.objects.Cluster.atoms.CheckClusterAvailableRegistering atom namespace.tendrl.objects.Cluster.atoms.ConfigureMonitoringRegistering atom namespace.tendrl.objects.Cluster.atoms.CreateRegistering atom namespace.tendrl.objects.Cluster.atoms.ImportClusterRegistering atom namespace.tendrl.objects.Cluster.atoms.ImportCreatedClusterRegistering atom namespace.tendrl.objects.Cluster.atoms.PersistClusterDetailsAfterCreateRegistering atom namespace.tendrl.objects.Cluster.atoms.ValidCreateClusterParamsRegistering atom namespace.tendrl.objects.Cluster.atoms.ValidImportClusterParamsFinding flows in namespace.tendrl.objects.Cluster.flowsRegistering object namespace.tendrl.objects.AlertFinding atoms in namespace.tendrl.objects.Alert.atomsFinding flows in namespace.tendrl.objects.Alert.flowsRegistering object namespace.tendrl.objects.ClusterAlertFinding atoms in namespace.tendrl.objects.ClusterAlert.atomsFinding flows in namespace.tendrl.objects.ClusterAlert.flowsRegistering object namespace.tendrl.objects.ClusterAlertCountersFinding atoms in namespace.tendrl.objects.ClusterAlertCounters.atomsFinding flows in namespace.tendrl.objects.ClusterAlertCounters.flowsRegistering object namespace.tendrl.objects.ClusterNodeContextFinding atoms in namespace.tendrl.objects.ClusterNodeContext.atomsFinding flows in namespace.tendrl.objects.ClusterNodeContext.flowsRegistering object namespace.tendrl.objects.ClusterTendrlContextFinding atoms in namespace.tendrl.objects.ClusterTendrlContext.atomsFinding flows in namespace.tendrl.objects.ClusterTendrlContext.flowsRegistering object namespace.tendrl.objects.CpuFinding atoms in namespace.tendrl.objects.Cpu.atomsFinding flows in namespace.tendrl.objects.Cpu.flowsRegistering object namespace.tendrl.objects.DefinitionFinding atoms in namespace.tendrl.objects.Definition.atomsFinding flows in namespace.tendrl.objects.Definition.flowsRegistering object namespace.tendrl.objects.DetectedClusterFinding atoms in namespace.te Nov 15 12:13:14 mbukatov-usm1-server.example.com tendrl-monitoring-integration[18460]: ndrl.objects.DetectedCluster.atomsFinding flows in namespace.tendrl.objects.DetectedCluster.flowsRegistering object namespace.tendrl.objects.DiskFinding atoms in namespace.tendrl.objects.Disk.atomsFinding flows in namespace.tendrl.objects.Disk.flowsRegistering object namespace.tendrl.objects.JobFinding atoms in namespace.tendrl.objects.Job.atomsFinding flows in namespace.tendrl.objects.Job.flowsRegistering object namespace.tendrl.objects.MemoryFinding atoms in namespace.tendrl.objects.Memory.atomsFinding flows in namespace.tendrl.objects.Memory.flowsRegistering object namespace.tendrl.objects.NodeFinding atoms in namespace.tendrl.objects.Node.atomsRegistering atom namespace.tendrl.objects.Node.atoms.BaseAtomRegistering atom namespace.tendrl.objects.Node.atoms.CheckNodeUpRegistering atom namespace.tendrl.objects.Node.atoms.BaseAtomRegistering atom namespace.tendrl.objects.Node.atoms.CmdRegistering atom namespace.tendrl.objects.Node.atoms.IsNodeTendrlManagedFinding flows in namespace.tendrl.objects.Node.flowsRegistering object namespace.tendrl.objects.AlertFinding atoms in namespace.tendrl.objects.Alert.atomsFinding flows in namespace.tendrl.objects.Alert.flowsRegistering object namespace.tendrl.objects.NodeAlertFinding atoms in namespace.tendrl.objects.NodeAlert.atomsFinding flows in namespace.tendrl.objects.NodeAlert.flowsRegistering object namespace.tendrl.objects.NodeAlertCountersFinding atoms in namespace.tendrl.objects.NodeAlertCounters.atomsFinding flows in namespace.tendrl.objects.NodeAlertCounters.flowsRegistering object namespace.tendrl.objects.NodeContextFinding atoms in namespace.tendrl.objects.NodeContext.atomsFinding flows in namespace.tendrl.objects.NodeContext.flowsRegistering object namespace.tendrl.objects.NodeNetworkFinding atoms in namespace.tendrl.objects.NodeNetwork.atomsFinding flows in namespace.tendrl.objects.NodeNetwork.flowsRegistering object namespace.tendrl.objects.AlertFinding atoms in namespace.tendrl.objects.Alert.atomsFinding flows in namespace.tendrl.objects.Alert.flowsRegistering Nov 15 12:13:14 mbukatov-usm1-server.example.com tendrl-monitoring-integration[18460]: object namespace.tendrl.objects.NotificationOnlyAlertFinding atoms in namespace.tendrl.objects.NotificationOnlyAlert.atomsFinding flows in namespace.tendrl.objects.NotificationOnlyAlert.flowsRegistering object namespace.tendrl.objects.OsFinding atoms in namespace.tendrl.objects.Os.atomsFinding flows in namespace.tendrl.objects.Os.flowsRegistering object namespace.tendrl.objects.PlatformFinding atoms in namespace.tendrl.objects.Platform.atomsFinding flows in namespace.tendrl.objects.Platform.flowsRegistering object namespace.tendrl.objects.ServiceFinding atoms in namespace.tendrl.objects.Service.atomsRegistering atom namespace.tendrl.objects.Service.atoms.BaseAtomRegistering atom namespace.tendrl.objects.Service.atoms.CheckServiceStatusFinding flows in namespace.tendrl.objects.Service.flowsRegistering object namespace.tendrl.objects.TendrlContextFinding atoms in namespace.tendrl.objects.TendrlContext.atomsFinding flows in namespace.tendrl.objects.TendrlContext.flowsRegistering object namespace.tendrl.objects.DiskFinding atoms in namespace.tendrl.objects.Disk.atomsFinding flows in namespace.tendrl.objects.Disk.flowsRegistering object namespace.tendrl.objects.VirtualDiskFinding atoms in namespace.tendrl.objects.VirtualDisk.atomsFinding flows in namespace.tendrl.objects.VirtualDisk.flowsFinding flows in namespace.tendrl.flowsRegistering flow namespace.tendrl.flows.AuthorizeSshKeyRegistering flow namespace.tendrl.flows.ConfigureMonitoringRegistering flow namespace.tendrl.flows.CreateClusterRegistering flow namespace.tendrl.flows.ExpandClusterRegistering flow namespace.tendrl.flows.ImportClusterRegistering flow namespace.tendrl.flows.SetupSshSetup Tendrl definitions (.yml)for namespace.tendrlValidating registered (.py) flows in namespace.tendrl.flowsValidating defined (.yml) flows in namespace.tendrl.flowsValidating registered (.py) objects in namespace.tendrl.objectsValidating defined (.yml) objects in namespace.tendrl.objectsSetup NodeContext for namespace.tendrlLoad definitions (.yml) for namespace.tendrl.objects.No Nov 15 12:13:15 mbukatov-usm1-server.example.com tendrl-monitoring-integration[18460]: * Running on http://127.0.0.1:8789/ Nov 15 12:14:19 mbukatov-usm1-server.example.com systemd[1]: Stopping Monitoring Integration... Nov 15 12:14:19 mbukatov-usm1-server.example.com tendrl-monitoring-integration[18460]: deContextLoad definitions (.yml) for namespace.tendrl.objects.TendrlContext Nov 15 12:14:19 mbukatov-usm1-server.example.com systemd[1]: Started Monitoring Integration. Nov 15 12:14:19 mbukatov-usm1-server.example.com systemd[1]: Starting Monitoring Integration... Nov 15 12:14:20 mbukatov-usm1-server.example.com tendrl-monitoring-integration[19339]: Creating namespace.monitoring from source tendrl.monitoring_integrationnamespace.monitoring created!Finding objects in namespace.monitoring.objectsRegistering object namespace.monitoring.objects.AlertTypesFinding atoms in namespace.monitoring.objects.AlertTypes.atomsFinding flows in namespace.monitoring.objects.AlertTypes.flowsRegistering object namespace.monitoring.objects.ConfigFinding atoms in namespace.monitoring.objects.Config.atomsFinding flows in namespace.monitoring.objects.Config.flowsRegistering object namespace.monitoring.objects.DefinitionFinding atoms in namespace.monitoring.objects.Definition.atomsFinding flows in namespace.monitoring.objects.Definition.flowsRegistering object namespace.monitoring.objects.TriggerFinding atoms in namespace.monitoring.objects.Trigger.atomsFinding flows in namespace.monitoring.objects.Trigger.flowsFinding flows in namespace.monitoring.flowsRegistering flow namespace.monitoring.flows.DeleteResourceFromGraphiteRegistering flow namespace.monitoring.flows.NewClusterDashboardRegistering flow namespace.monitoring.flows.UpdateDashboardRegistering flow namespace.monitoring.flows.UpdateDashboardSetup Tendrl definitions (.yml)for namespace.monitoringValidating registered (.py) flows in namespace.monitoring.flowsValidating defined (.yml) flows in namespace.monitoring.flowsValidating registered (.py) objects in namespace.monitoring.objectsValidating defined (.yml) objects in namespace.monitoring.objectsSetup Config for namespace.monitoringSetup central store clients for namespace.monitoringCreating namespace.tendrl from source tendrl.commonsnamespace.tendrl created!Finding objects in namespace.tendrl.objectsRegistering object namespace.tendrl.objects.AlertFinding atoms in namespace.tendrl.objects.Alert.atomsFinding flows in namespace.tendrl.objects.Alert.flowsRegistering object namespace.tendrl.objects.BlockDeviceFinding atoms in namespace.tendrl.objects.BlockDevice.atomsFinding flows in namespace.tendrl.objects.BlockDevice.flowsRegistering object namespace.tendrl.objects.ClusterF Nov 15 12:14:20 mbukatov-usm1-server.example.com tendrl-monitoring-integration[19339]: inding atoms in namespace.tendrl.objects.Cluster.atomsRegistering atom namespace.tendrl.objects.Cluster.atoms.CheckClusterAvailableRegistering atom namespace.tendrl.objects.Cluster.atoms.ConfigureMonitoringRegistering atom namespace.tendrl.objects.Cluster.atoms.CreateRegistering atom namespace.tendrl.objects.Cluster.atoms.ImportClusterRegistering atom namespace.tendrl.objects.Cluster.atoms.ImportCreatedClusterRegistering atom namespace.tendrl.objects.Cluster.atoms.PersistClusterDetailsAfterCreateRegistering atom namespace.tendrl.objects.Cluster.atoms.ValidCreateClusterParamsRegistering atom namespace.tendrl.objects.Cluster.atoms.ValidImportClusterParamsFinding flows in namespace.tendrl.objects.Cluster.flowsRegistering object namespace.tendrl.objects.AlertFinding atoms in namespace.tendrl.objects.Alert.atomsFinding flows in namespace.tendrl.objects.Alert.flowsRegistering object namespace.tendrl.objects.ClusterAlertFinding atoms in namespace.tendrl.objects.ClusterAlert.atomsFinding flows in namespace.tendrl.objects.ClusterAlert.flowsRegistering object namespace.tendrl.objects.ClusterAlertCountersFinding atoms in namespace.tendrl.objects.ClusterAlertCounters.atomsFinding flows in namespace.tendrl.objects.ClusterAlertCounters.flowsRegistering object namespace.tendrl.objects.ClusterNodeContextFinding atoms in namespace.tendrl.objects.ClusterNodeContext.atomsFinding flows in namespace.tendrl.objects.ClusterNodeContext.flowsRegistering object namespace.tendrl.objects.ClusterTendrlContextFinding atoms in namespace.tendrl.objects.ClusterTendrlContext.atomsFinding flows in namespace.tendrl.objects.ClusterTendrlContext.flowsRegistering object namespace.tendrl.objects.CpuFinding atoms in namespace.tendrl.objects.Cpu.atomsFinding flows in namespace.tendrl.objects.Cpu.flowsRegistering object namespace.tendrl.objects.DefinitionFinding atoms in namespace.tendrl.objects.Definition.atomsFinding flows in namespace.tendrl.objects.Definition.flowsRegistering object namespace.tendrl.objects.DetectedClusterFinding atoms in namespace.te Nov 15 12:14:20 mbukatov-usm1-server.example.com systemd[1]: Stopping Monitoring Integration... Nov 15 12:14:20 mbukatov-usm1-server.example.com systemd[1]: Started Monitoring Integration. Nov 15 12:14:20 mbukatov-usm1-server.example.com systemd[1]: Starting Monitoring Integration... Nov 15 12:14:20 mbukatov-usm1-server.example.com systemd[1]: Stopping Monitoring Integration... Nov 15 12:14:21 mbukatov-usm1-server.example.com systemd[1]: Started Monitoring Integration. Nov 15 12:14:21 mbukatov-usm1-server.example.com systemd[1]: Starting Monitoring Integration... Nov 15 12:14:21 mbukatov-usm1-server.example.com tendrl-monitoring-integration[19507]: Creating namespace.monitoring from source tendrl.monitoring_integrationnamespace.monitoring created!Finding objects in namespace.monitoring.objectsRegistering object namespace.monitoring.objects.AlertTypesFinding atoms in namespace.monitoring.objects.AlertTypes.atomsFinding flows in namespace.monitoring.objects.AlertTypes.flowsRegistering object namespace.monitoring.objects.ConfigFinding atoms in namespace.monitoring.objects.Config.atomsFinding flows in namespace.monitoring.objects.Config.flowsRegistering object namespace.monitoring.objects.DefinitionFinding atoms in namespace.monitoring.objects.Definition.atomsFinding flows in namespace.monitoring.objects.Definition.flowsRegistering object namespace.monitoring.objects.TriggerFinding atoms in namespace.monitoring.objects.Trigger.atomsFinding flows in namespace.monitoring.objects.Trigger.flowsFinding flows in namespace.monitoring.flowsRegistering flow namespace.monitoring.flows.DeleteResourceFromGraphiteRegistering flow namespace.monitoring.flows.NewClusterDashboardRegistering flow namespace.monitoring.flows.UpdateDashboardRegistering flow namespace.monitoring.flows.UpdateDashboardSetup Tendrl definitions (.yml)for namespace.monitoringValidating registered (.py) flows in namespace.monitoring.flowsValidating defined (.yml) flows in namespace.monitoring.flowsValidating registered (.py) objects in namespace.monitoring.objectsValidating defined (.yml) objects in namespace.monitoring.objectsSetup Config for namespace.monitoringSetup central store clients for namespace.monitoringCreating namespace.tendrl from source tendrl.commonsnamespace.tendrl created!Finding objects in namespace.tendrl.objectsRegistering object namespace.tendrl.objects.AlertFinding atoms in namespace.tendrl.objects.Alert.atomsFinding flows in namespace.tendrl.objects.Alert.flowsRegistering object namespace.tendrl.objects.BlockDeviceFinding atoms in namespace.tendrl.objects.BlockDevice.atomsFinding flows in namespace.tendrl.objects.BlockDevice.flowsRegistering object namespace.tendrl.objects.ClusterF Nov 15 12:14:21 mbukatov-usm1-server.example.com tendrl-monitoring-integration[19507]: inding atoms in namespace.tendrl.objects.Cluster.atomsRegistering atom namespace.tendrl.objects.Cluster.atoms.CheckClusterAvailableRegistering atom namespace.tendrl.objects.Cluster.atoms.ConfigureMonitoringRegistering atom namespace.tendrl.objects.Cluster.atoms.CreateRegistering atom namespace.tendrl.objects.Cluster.atoms.ImportClusterRegistering atom namespace.tendrl.objects.Cluster.atoms.ImportCreatedClusterRegistering atom namespace.tendrl.objects.Cluster.atoms.PersistClusterDetailsAfterCreateRegistering atom namespace.tendrl.objects.Cluster.atoms.ValidCreateClusterParamsRegistering atom namespace.tendrl.objects.Cluster.atoms.ValidImportClusterParamsFinding flows in namespace.tendrl.objects.Cluster.flowsRegistering object namespace.tendrl.objects.AlertFinding atoms in namespace.tendrl.objects.Alert.atomsFinding flows in namespace.tendrl.objects.Alert.flowsRegistering object namespace.tendrl.objects.ClusterAlertFinding atoms in namespace.tendrl.objects.ClusterAlert.atomsFinding flows in namespace.tendrl.objects.ClusterAlert.flowsRegistering object namespace.tendrl.objects.ClusterAlertCountersFinding atoms in namespace.tendrl.objects.ClusterAlertCounters.atomsFinding flows in namespace.tendrl.objects.ClusterAlertCounters.flowsRegistering object namespace.tendrl.objects.ClusterNodeContextFinding atoms in namespace.tendrl.objects.ClusterNodeContext.atomsFinding flows in namespace.tendrl.objects.ClusterNodeContext.flowsRegistering object namespace.tendrl.objects.ClusterTendrlContextFinding atoms in namespace.tendrl.objects.ClusterTendrlContext.atomsFinding flows in namespace.tendrl.objects.ClusterTendrlContext.flowsRegistering object namespace.tendrl.objects.CpuFinding atoms in namespace.tendrl.objects.Cpu.atomsFinding flows in namespace.tendrl.objects.Cpu.flowsRegistering object namespace.tendrl.objects.DefinitionFinding atoms in namespace.tendrl.objects.Definition.atomsFinding flows in namespace.tendrl.objects.Definition.flowsRegistering object namespace.tendrl.objects.DetectedClusterFinding atoms in namespace.te Nov 15 12:14:22 mbukatov-usm1-server.example.com tendrl-monitoring-integration[19507]: ndrl.objects.DetectedCluster.atomsFinding flows in namespace.tendrl.objects.DetectedCluster.flowsRegistering object namespace.tendrl.objects.DiskFinding atoms in namespace.tendrl.objects.Disk.atomsFinding flows in namespace.tendrl.objects.Disk.flowsRegistering object namespace.tendrl.objects.JobFinding atoms in namespace.tendrl.objects.Job.atomsFinding flows in namespace.tendrl.objects.Job.flowsRegistering object namespace.tendrl.objects.MemoryFinding atoms in namespace.tendrl.objects.Memory.atomsFinding flows in namespace.tendrl.objects.Memory.flowsRegistering object namespace.tendrl.objects.NodeFinding atoms in namespace.tendrl.objects.Node.atomsRegistering atom namespace.tendrl.objects.Node.atoms.BaseAtomRegistering atom namespace.tendrl.objects.Node.atoms.CheckNodeUpRegistering atom namespace.tendrl.objects.Node.atoms.BaseAtomRegistering atom namespace.tendrl.objects.Node.atoms.CmdRegistering atom namespace.tendrl.objects.Node.atoms.IsNodeTendrlManagedFinding flows in namespace.tendrl.objects.Node.flowsRegistering object namespace.tendrl.objects.AlertFinding atoms in namespace.tendrl.objects.Alert.atomsFinding flows in namespace.tendrl.objects.Alert.flowsRegistering object namespace.tendrl.objects.NodeAlertFinding atoms in namespace.tendrl.objects.NodeAlert.atomsFinding flows in namespace.tendrl.objects.NodeAlert.flowsRegistering object namespace.tendrl.objects.NodeAlertCountersFinding atoms in namespace.tendrl.objects.NodeAlertCounters.atomsFinding flows in namespace.tendrl.objects.NodeAlertCounters.flowsRegistering object namespace.tendrl.objects.NodeContextFinding atoms in namespace.tendrl.objects.NodeContext.atomsFinding flows in namespace.tendrl.objects.NodeContext.flowsRegistering object namespace.tendrl.objects.NodeNetworkFinding atoms in namespace.tendrl.objects.NodeNetwork.atomsFinding flows in namespace.tendrl.objects.NodeNetwork.flowsRegistering object namespace.tendrl.objects.AlertFinding atoms in namespace.tendrl.objects.Alert.atomsFinding flows in namespace.tendrl.objects.Alert.flowsRegistering Nov 15 12:14:22 mbukatov-usm1-server.example.com tendrl-monitoring-integration[19507]: object namespace.tendrl.objects.NotificationOnlyAlertFinding atoms in namespace.tendrl.objects.NotificationOnlyAlert.atomsFinding flows in namespace.tendrl.objects.NotificationOnlyAlert.flowsRegistering object namespace.tendrl.objects.OsFinding atoms in namespace.tendrl.objects.Os.atomsFinding flows in namespace.tendrl.objects.Os.flowsRegistering object namespace.tendrl.objects.PlatformFinding atoms in namespace.tendrl.objects.Platform.atomsFinding flows in namespace.tendrl.objects.Platform.flowsRegistering object namespace.tendrl.objects.ServiceFinding atoms in namespace.tendrl.objects.Service.atomsRegistering atom namespace.tendrl.objects.Service.atoms.BaseAtomRegistering atom namespace.tendrl.objects.Service.atoms.CheckServiceStatusFinding flows in namespace.tendrl.objects.Service.flowsRegistering object namespace.tendrl.objects.TendrlContextFinding atoms in namespace.tendrl.objects.TendrlContext.atomsFinding flows in namespace.tendrl.objects.TendrlContext.flowsRegistering object namespace.tendrl.objects.DiskFinding atoms in namespace.tendrl.objects.Disk.atomsFinding flows in namespace.tendrl.objects.Disk.flowsRegistering object namespace.tendrl.objects.VirtualDiskFinding atoms in namespace.tendrl.objects.VirtualDisk.atomsFinding flows in namespace.tendrl.objects.VirtualDisk.flowsFinding flows in namespace.tendrl.flowsRegistering flow namespace.tendrl.flows.AuthorizeSshKeyRegistering flow namespace.tendrl.flows.ConfigureMonitoringRegistering flow namespace.tendrl.flows.CreateClusterRegistering flow namespace.tendrl.flows.ExpandClusterRegistering flow namespace.tendrl.flows.ImportClusterRegistering flow namespace.tendrl.flows.SetupSshSetup Tendrl definitions (.yml)for namespace.tendrlValidating registered (.py) flows in namespace.tendrl.flowsValidating defined (.yml) flows in namespace.tendrl.flowsValidating registered (.py) objects in namespace.tendrl.objectsValidating defined (.yml) objects in namespace.tendrl.objectsSetup NodeContext for namespace.tendrlLoad definitions (.yml) for namespace.tendrl.objects.No Nov 15 12:14:23 mbukatov-usm1-server.example.com tendrl-monitoring-integration[19507]: * Running on http://127.0.0.1:8789/ ``` Expected results ================ Each error should go into a single line: ``` tendrl-monitoring-integration[16421]: Creating namespace.monitoring from source tendrl.monitoring_integrationnamespace.monitoring created! tendrl-monitoring-integration[16421]: Finding object ... tendrl-monitoring-integration[16421]: Error connecting to central store (etcd), trying again... tendrl-monitoring-integration[16421]: Error connecting to central store (etcd), trying again... ``` (this example may not be 100% correct, but I hope we get the idea)
Fixed: https://github.com/Tendrl/monitoring-integration/issues/35
(In reply to Rohan Kanade from comment #3) > Fixed: https://github.com/Tendrl/monitoring-integration/issues/35 Could you either update FiV field or move this BZ into POST instead? Thank you.
Checking tendrl python components (which are using tendrl commons, which is a component this BZ is about): tendrl-commons-1.6.3-6.el7rhgs tendrl-gluster-integration-1.6.3-4.el7rhgs tendrl-monitoring-integration-1.6.3-4.el7rhgs tendrl-node-agent-1.6.3-6.el7rhgs tendrl-notifier-1.6.3-3.el7rhgs Which contains these systemd services: /usr/lib/systemd/system/tendrl-node-agent.service /usr/lib/systemd/system/tendrl-notifier.service /usr/lib/systemd/system/tendrl-monitoring-integration.service /usr/lib/systemd/system/tendrl-gluster-integration.service The journald log entries looks good at first sight, there are no obvious problems as noted in the BZ (2nd example from Actual Results), eg.: ``` # journalctl -u tendrl-monitoring-integration --output=cat | head -30 Started Monitoring Integration. Starting Monitoring Integration... Creating namespace.monitoring from source tendrl.monitoring_integration namespace.monitoring created! Finding objects in namespace.monitoring.objects Registering object namespace.monitoring.objects.AlertOrganization Finding atoms in namespace.monitoring.objects.AlertOrganization.atoms Finding flows in namespace.monitoring.objects.AlertOrganization.flows Registering object namespace.monitoring.objects.AlertTypes Finding atoms in namespace.monitoring.objects.AlertTypes.atoms Finding flows in namespace.monitoring.objects.AlertTypes.flows Registering object namespace.monitoring.objects.Config Finding atoms in namespace.monitoring.objects.Config.atoms Finding flows in namespace.monitoring.objects.Config.flows Registering object namespace.monitoring.objects.Definition Finding atoms in namespace.monitoring.objects.Definition.atoms Finding flows in namespace.monitoring.objects.Definition.flows Registering object namespace.monitoring.objects.Trigger Finding atoms in namespace.monitoring.objects.Trigger.atoms Finding flows in namespace.monitoring.objects.Trigger.flows Finding flows in namespace.monitoring.flows Registering flow namespace.monitoring.flows.DeleteMonitoringData Registering flow namespace.monitoring.flows.DeleteResourceFromGraphite Registering flow namespace.monitoring.flows.SetupClusterAlias Registering flow namespace.monitoring.flows.UpdateDashboard Setup Tendrl definitions (.yml)for namespace.monitoring Validating registered (.py) flows in namespace.monitoring.flows Validating defined (.yml) flows in namespace.monitoring.flows Validating registered (.py) objects in namespace.monitoring.objects Validating defined (.yml) objects in namespace.monitoring.objects ``` Besides that, I searched for log lines longer that 500 characters (based on the 2nd example from the Actual Results section, where lines are about 2000 characters long), and didn't find anything interesting: ``` # journalctl -u tendrl-monitoring-integration | awk 'length($0)>500' # ``` I used both ways to check all tendrl python services, as listed above. But when I retry the exact reproducer noted in this BZ (setting etcd_connection to 0.0.0.0), I see quite long line, which seems not right: ``` # journalctl -u tendrl-monitoring-integration -f --output cat | grep 'connecting to central store (etcd)' Finding flows in namespace.tendrl.objectError connecting to central store (etcd), trying again...Error connecting to central store (etcd), trying again...Error connecting to central store (etcd), trying again...Error connecting to central store (etcd), trying again...Error connecting to central store (etcd), trying again...s.Os.flows Finding flows in namespace.tendrl.objectError connecting to central store (etcd), trying again...Error connecting to central store (etcd), trying again...Error connecting to central store (etcd), trying again...Error connecting to central store (etcd), trying again...Error connecting to central store (etcd), trying again...s.Os.flows Finding flows in namespace.tendrl.objectError connecting to central store (etcd), trying again...Error connecting to central store (etcd), trying again...Error connecting to central store (etcd), trying again...Error connecting to central store (etcd), trying again...Error connecting to central store (etcd), trying again...s.Os.flows ``` Note that the problem is different compared to original report for this particular case, but it still differs from expected behavior as well.
Fixed via: https://github.com/Tendrl/commons/pull/987
Testing with tendrl-monitoring-integration-1.6.3-5.el7rhgs.noarch Based on comment 6, I retried the reproducer and now I see: ~~~ # journalctl -u tendrl-monitoring-integration -f --output cat | grep 'connecting to central store (etcd)' Error connecting to central store (etcd), trying again... Error connecting to central store (etcd), trying again... Error connecting to central store (etcd), trying again... Error connecting to central store (etcd), trying again... Error connecting to central store (etcd), trying again... ~~~
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2616