Bug 1125873

Summary: [upgrade rhel6]: the nova logs changed between rhos4 and rhos5 and after upgrade we have both logs and we keep on writing to old ones
Product: Red Hat OpenStack Reporter: Dafna Ron <dron>
Component: openstack-novaAssignee: Russell Bryant <rbryant>
Status: CLOSED ERRATA QA Contact: Dafna Ron <dron>
Severity: medium Docs Contact:
Priority: high    
Version: unspecifiedCC: apevec, dron, ndipanov, sclewis, sgordon, yeylon
Target Milestone: z1Keywords: Regression, ZStream
Target Release: 5.0 (RHEL 6)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-nova-2014.1.2-2.el6ost Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-09-30 17:52:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
nova.conf none

Description Dafna Ron 2014-08-01 09:37:59 UTC
Description of problem:

nova logs names changed between rhos4 and rhos5, for example, api.log was changed to nova-api.log 
I did the upgrade two days ago and looking at the logs dir, we have both logs but keep on writing to the old ones. 

Version-Release number of selected component (if applicable):


python-novaclient-2.17.0-2.el6ost.noarch
python-nova-2014.1.1-3.el6ost.noarch
openstack-nova-api-2014.1.1-3.el6ost.noarch
openstack-nova-common-2014.1.1-3.el6ost.noarch
openstack-nova-cert-2014.1.1-3.el6ost.noarch
openstack-nova-console-2014.1.1-3.el6ost.noarch
openstack-nova-novncproxy-2014.1.1-3.el6ost.noarch
openstack-nova-scheduler-2014.1.1-3.el6ost.noarch
openstack-nova-conductor-2014.1.1-3.el6ost.noarch
openstack-nova-compute-2014.1.1-3.el6ost.noarch


How reproducible:

100%

Steps to Reproduce:
setup is 1 controller + 3 computes installed with packstack

1. install rhos4 GA from this puddle: 2014-01-16.1
2. upgrade to rhoe4 A4 from this puddle: 2014-04-15.5
3. upgrade to rhos5 with service by service procedure to this puddle: 2014-07-23.2

Actual results:

we add the new logs but keep the old log format and continue to write to the old logs. 

Expected results:

we need to xz the old logs and start writing to the new ones

Additional info:

root@gizmo ~(keystone_admin)]# ls -ltr /var/log/nova/
total 294660
-rw-r--r--. 1 nova nova  2041534 Jul 30 03:43 api.log-20140730.gz
-rw-r--r--. 1 nova nova  4922357 Jul 30 03:43 compute.log-20140730.gz
-rw-r--r--. 1 nova nova 32370530 Jul 30 03:43 conductor.log-20140730.gz
-rw-r--r--. 1 nova nova  1769870 Jul 30 03:43 nova-api.log-20140730.gz
-rw-r--r--. 1 nova nova        0 Jul 30 03:43 nova-api-startup.log
-rw-r--r--. 1 nova nova  1735651 Jul 30 03:43 nova-api-startup.log-20140730.gz
-rw-r--r--. 1 nova nova  2121519 Jul 30 03:43 scheduler.log-20140730.gz
-rw-r--r--. 1 nova nova 38571929 Jul 30 13:23 nova-api.log
-rw-r--r--. 1 nova nova    86022 Jul 30 13:23 nova-manage.log
-rw-r--r--. 1 nova nova  3882330 Aug  1 12:36 scheduler.log
-rw-r--r--. 1 nova nova 80984189 Aug  1 12:36 api.log
-rw-r--r--. 1 nova nova 28552247 Aug  1 12:37 compute.log
-rw-r--r--. 1 nova nova  3083492 Aug  1 12:37 consoleauth.log
-rw-r--r--. 1 nova nova 99005843 Aug  1 12:37 conductor.log
-rw-r--r--. 1 nova nova  2552205 Aug  1 12:37 cert.log
[root@gizmo ~(keystone_admin)]# 
[root@gizmo ~(keystone_admin)]# date
Fri Aug  1 12:37:44 IDT 2014

Comment 1 Russell Bryant 2014-08-06 15:33:28 UTC
(In reply to Dafna Ron from comment #0)
> Actual results:
> 
> we add the new logs but keep the old log format and continue to write to the
> old logs. 
> 
> Expected results:
> 
> we need to xz the old logs and start writing to the new ones

Can you provide your nova.conf?  I'm curious if the log_file option is set.

If that option is *not* set, it sounds like the API service didn't get restarted.  Can you please verify that nova-api was restarted?

Comment 2 Dafna Ron 2014-08-06 16:56:19 UTC
# (Optional) Name of log file to output to. If no default is
# set, logging will go to stdout. (string value)
#log_file=<None>


part of the upgrade requires service restart: 

root@gizmo ~]# openstack-service status nova
openstack-nova-api (pid  14091) is running...
openstack-nova-cert (pid  14111) is running...
openstack-nova-compute (pid  14130) is running...
openstack-nova-conductor (pid  14149) is running...
openstack-nova-consoleauth (pid  14168) is running...
openstack-nova-novncproxy (pid  14197) is running...
openstack-nova-scheduler (pid  14212) is running...
[root@gizmo ~]# 

and it should start on reboot: 

[root@gizmo ~]# chkconfig  |grep nova-api
openstack-nova-api	0:off	1:off	2:on	3:on	4:on	5:on	6:off

Comment 3 Dafna Ron 2014-08-06 16:58:53 UTC
Created attachment 924535 [details]
nova.conf

Comment 4 Alan Pevec 2014-08-12 10:03:06 UTC
openstack-nova-$service.init in rhos5el6 have explicit--logfile parameter in service startup command line which sets it to /var/log/nova/$service.log

That was supposed to be removed as a fix for bug 966050 but looks it was fixed only for el7 and we have a regression in el6.

Comment 9 Dafna Ron 2014-09-11 16:27:21 UTC
we are writing to the new logs after upgrade. 
moving to verified on RHEL6.5 upgrade to puddle: 2014-09-10.1

http://pastebin.test.redhat.com/233492

Comment 11 errata-xmlrpc 2014-09-30 17:52:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1341.html