Bug 1174997

Summary: rubygem-staypuft: "Failed to record metering data: not master" errors in /var/log/ceilometer/collector.log on all controllers.
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: openstack-foreman-installerAssignee: Jason Guiditta <jguiditt>
Status: CLOSED ERRATA QA Contact: Leonid Natapov <lnatapov>
Severity: medium Docs Contact:
Priority: medium    
Version: unspecifiedCC: aberezin, cwolfe, jguiditt, mburns, mlopes, morazi, rhos-maint, yeylon
Target Milestone: ga   
Target Release: Installer   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-foreman-installer-3.0.8-1.el7ost Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-02-09 15:18:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1177026    
Attachments:
Description Flags
logs from controllers
none
log none

Description Alexander Chuzhoy 2014-12-16 22:53:32 UTC 
rubygem-staypuft: "Failed to record metering data: not master" errors in /var/log/ceilometer/collector.log on all controllers.


Environment:
openstack-foreman-installer-3.0.6-1.el7ost.noarch
ruby193-rubygem-staypuft-0.5.6-1.el7ost.noarch
ruby193-rubygem-foreman_openstack_simplify-0.0.6-8.el7ost.noarch
rhel-osp-installer-client-0.5.3-1.el7ost.noarch
openstack-puppet-modules-2014.2.7-2.el7ost.noarch
rhel-osp-installer-0.5.3-1.el7ost.noarch


Steps to reproduce:
1. install rhel-osp-installer.
2. Create/run haneutron deployment with 3 controllers+2 computes.

Result:
Checking the /var/log/ceilometer/collector.log file, found these errors on all controllers:

2014-12-16 17:36:15.067 7237 ERROR ceilometer.dispatcher.database [-] Failed to record metering data: not master
2014-12-16 17:36:15.067 7237 TRACE ceilometer.dispatcher.database     msg, allowable_errors)
2014-12-16 17:36:15.476 7237 ERROR ceilometer.dispatcher.database [-] Failed to record metering data: not master
2014-12-16 17:36:15.476 7237 TRACE ceilometer.dispatcher.database     msg, allowable_errors)

Expected result: no error in the log.
Comment 1 Alexander Chuzhoy 2014-12-16 22:57:49 UTC 
Created attachment 969786 [details]
logs from controllers
Comment 4 Crag Wolfe 2014-12-18 01:42:11 UTC 
I have not seen this issue yet with a fresh install which includes 3 controllers and a compute node with a launched VM.  I am seeing another (most likely unrelated) keystone error:

2014-12-17 14:04:08.454 10817 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2014-12-17 15:04:08.813 10817 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2014-12-17 16:04:09.184 10817 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2014-12-17 17:04:09.813 10817 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
Comment 5 Mike Burns 2014-12-18 13:57:36 UTC 
Fixed in latest build by other patches.
Comment 8 Leonid Natapov 2015-01-05 08:43:55 UTC 
1.tested on openstack-foreman-installer-3.0.8-1.el7ost.

2.Still see Failed to record messages in /var/log/ceilometer/collector.log

But this time those are different messages.

Instead of 2014-12-16 17:36:15.067 7237 ERROR ceilometer.dispatcher.database [-] Failed to record metering data: not master

I see

2015-01-04 11:07:48.977 57134 ERROR ceilometer.dispatcher.database [-] Failed to record metering data: could not connect to 10.35.173.181:27017: [Errno 113] EHOSTUNREACH

 
log attached
Comment 9 Leonid Natapov 2015-01-05 08:47:04 UTC 
Created attachment 976256 [details]
log
Comment 10 Mike Burns 2015-01-05 16:17:17 UTC 
(In reply to Leonid Natapov from comment #8)
> 1.tested on openstack-foreman-installer-3.0.8-1.el7ost.
> 
> 2.Still see Failed to record messages in /var/log/ceilometer/collector.log
> 
> But this time those are different messages.
> 
> Instead of 2014-12-16 17:36:15.067 7237 ERROR ceilometer.dispatcher.database
> [-] Failed to record metering data: not master
> 
> I see
> 
> 2015-01-04 11:07:48.977 57134 ERROR ceilometer.dispatcher.database [-]
> Failed to record metering data: could not connect to 10.35.173.181:27017:
> [Errno 113] EHOSTUNREACH
> 
>  
> log attached

This should be filed separately.  Different error messages generally mean different root causes.  

That being said, this appears to be an environmental issue.  If it couldn't connect to the ceilometer host, that appears environmental.
Comment 11 Leonid Natapov 2015-01-11 14:33:11 UTC 
The messages described in  the bug don't appear on openstack-foreman-installer-3.0.8-1.el7ost.
Verifying this bz.
Comment 14 errata-xmlrpc 2015-02-09 15:18:23 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0156.html