Bug 1320744

Summary:	Exec[tuned-virtual-host]/returns: NameError: global name 'retcode' is not defined
Product:	[Community] RDO	Reporter:	David Moreau Simard <dmsimard>
Component:	openstack-packstack	Assignee:	Ivan Chavero <ichavero>
Status:	CLOSED CURRENTRELEASE	QA Contact:	yeylon <yeylon>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	trunk	CC:	aortega, apevec, derekh, jpena, mmagr, srevivo, yeylon
Target Milestone:	---
Target Release:	trunk
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-03-30 23:06:31 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description David Moreau Simard 2016-03-23 22:12:32 UTC

There is a bugfix pending release for tuned, reference BZ:
- https://bugzilla.redhat.com/show_bug.cgi?id=1265660
- https://bugzilla.redhat.com/show_bug.cgi?id=1305903

Packstack installs, starts and sets up a tuned profile here:
https://github.com/openstack/packstack/blob/94eb2fdff99184b531603202260ca5b1383ec227/packstack/puppet/templates/nova_compute.pp#L47-L61

CI has been running into intermittent issues with the output being:
Debug: Executing '/usr/bin/systemctl is-active tuned'
Debug: Exec[tuned-virtual-host](provider=posix): Executing check '/usr/sbin/tuned-adm active | /bin/grep virtual-host'
Debug: Executing '/usr/sbin/tuned-adm active | /bin/grep virtual-host'
Debug: Exec[tuned-virtual-host](provider=posix): Executing '/usr/sbin/tuned-adm profile virtual-host'
Debug: Executing '/usr/sbin/tuned-adm profile virtual-host'
Notice: /Stage[main]/Main/Exec[tuned-virtual-host]/returns: DBus call to Tuned daemon failed
Notice: /Stage[main]/Main/Exec[tuned-virtual-host]/returns: Traceback (most recent call last):
Notice: /Stage[main]/Main/Exec[tuned-virtual-host]/returns:   File "/usr/sbin/tuned-adm", line 72, in <module>
Notice: /Stage[main]/Main/Exec[tuned-virtual-host]/returns:     result = action(**options)
Notice: /Stage[main]/Main/Exec[tuned-virtual-host]/returns:   File "/usr/lib/python2.7/site-packages/tuned/admin/admin.py", line 80, in profile
Notice: /Stage[main]/Main/Exec[tuned-virtual-host]/returns:     if retcode == 0:
Notice: /Stage[main]/Main/Exec[tuned-virtual-host]/returns: NameError: global name 'retcode' is not defined
Notice: /Stage[main]/Main/Exec[tuned-virtual-host]/returns: Trying to (re)start tuned...
Error: /usr/sbin/tuned-adm profile virtual-host returned 3 instead of one of [0]
Error: /Stage[main]/Main/Exec[tuned-virtual-host]/returns: change from notrun to 0 failed: /usr/sbin/tuned-adm profile virtual-host returned 3 instead of one of [0]

That particular error has been resolved upstream and will eventually land in the CentOS repositories.
In the meantime, the racy and intermittent nature of the issue seems to indicate that systemctl could report tuned as active (i.e, the process is running) but tuned would not actually be ready to receive commands just yet.

Comment 1 Alan Pevec 2016-03-23 23:04:53 UTC

> There is a bugfix pending release for tuned, reference BZ:

Even with that, there's race in tuned service, according to the trace it is reported as ready when it's not.
systemd assumes Type=dbus service is ready when its name is on dbus
which happens[1] before it is actually started[2]


[1] https://git.fedorahosted.org/cgit/tuned.git/tree/tuned.py#n66
[2] https://git.fedorahosted.org/cgit/tuned.git/tree/tuned.py#n76

Comment 2 Alan Pevec 2016-03-23 23:06:31 UTC

BTW in RDO bz we should be reporting only issues with RPM packaging, this could be reported in upstream bug tracker https://bugs.launchpad.net/packstack

Comment 3 David Moreau Simard 2016-03-23 23:47:44 UTC

(In reply to Alan Pevec from comment #1)
> > There is a bugfix pending release for tuned, reference BZ:
> 
> Even with that, there's race in tuned service, according to the trace it is
> reported as ready when it's not.
> systemd assumes Type=dbus service is ready when its name is on dbus
> which happens[1] before it is actually started[2]
> 
> 
> [1] https://git.fedorahosted.org/cgit/tuned.git/tree/tuned.py#n66
> [2] https://git.fedorahosted.org/cgit/tuned.git/tree/tuned.py#n76

Good catch on the actual race that we see!
Noted for packaging vs project bug.