Bug 1071453

Summary: tuned-adm not responding when executed near service start
Product: Red Hat Enterprise Linux 6 Reporter: Alon Bar-Lev <alonbl>
Component: tunedAssignee: Jaroslav Škarvada <jskarvad>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Jiri Belka <jbelka>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 6.5CC: aberezin, acanan, alonbl, bazulay, bblaskov, cpelland, eedri, iheim, jeder, jkurik, jskarvad, lbednar, nlevinki, psklenar, s.kieske, srevivo
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-09-26 09:33:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 871829, 1069119, 1078981    

Description Alon Bar-Lev 2014-02-28 20:05:43 UTC
(In reply to Jaroslav Škarvada from bug#1069119 comment #3)
> (In reply to Sandro Bonazzola from comment #2)
> > Jaroslav, can you take a look? Maybe it's a tuned bug.
> 
> I think this may be race. RHEL-6 tuned is not race free and it's something
> that cannot be correctly fixed without re-design (which happened in RHEL-7).
> 
> Could you try to add e.g. 5 seconds delay between '/sbin/service tuned
> start' and '/usr/bin/tuned-adm profile virtual-host'? In case it helps, I
> can try to workaround this specific problem.

(In reply to Meital Bourvine from bug#1069119 comment #8)
> This work around seems to be working.
> I run it 3 times.

Comment 2 Barak 2014-03-02 15:06:12 UTC
*** Bug 1069245 has been marked as a duplicate of this bug. ***

Comment 4 Jaroslav Škarvada 2014-03-17 15:51:36 UTC
Reproducer:

# tuned-adm profile virtual-host & service tuned start

This is not regression, the problem seems to be there since the beginning of the tuned project, but I am going to fix it.

Comment 5 Jaroslav Škarvada 2014-03-17 15:55:24 UTC
Also another thing I noticed during the overview of the ovirt-host-deploy log, you shouldn't (IMHO) mix upstart and sysvinit commands, i.e.: 

execute-output: ('/sbin/initctl', 'status', 'tuned')

This is asking about status of the upstart job, so for sysvinit service it will always return:
initctl: Unknown job: tuned

So next you are starting the service (now using the sysvinit syntax):
execute: ('/sbin/service', 'tuned', 'start')
even if the service is already started. This applies not only to tuned but also to other sysvinit services. Starting already started service should be harmless in most cases, but redundant.

Comment 6 Alon Bar-Lev 2014-03-17 15:58:08 UTC
(In reply to Jaroslav Škarvada from comment #4)
> Reproducer:
> 
> # tuned-adm profile virtual-host & service tuned start
> 
> This is not regression, the problem seems to be there since the beginning of
> the tuned project, but I am going to fix it.

Great. We are using same sequence since rhel-6.4 (2012-12-27), and have not experienced issue... up to after rhel-6.5... so it is strange... :)

Comment 7 Alon Bar-Lev 2014-03-17 16:00:33 UTC
(In reply to Jaroslav Škarvada from comment #5)
> Also another thing I noticed during the overview of the ovirt-host-deploy
> log, you shouldn't (IMHO) mix upstart and sysvinit commands, i.e.: 
> 
> execute-output: ('/sbin/initctl', 'status', 'tuned')

Right, unfortunately we have mixed mode services and should prefer upstart if available, so this failure is expected.

> So next you are starting the service (now using the sysvinit syntax):
> execute: ('/sbin/service', 'tuned', 'start')
> even if the service is already started. This applies not only to tuned but
> also to other sysvinit services. Starting already started service should be
> harmless in most cases, but redundant.

Correct... instead of asking for status and start, we start. Simpler code path.

Comment 8 Sven Kieske 2014-03-18 14:40:28 UTC
So is something blocking this getting fixed?

I fail to see where the problem is. Maybe someone could enlighten
me and everyone who wants to deploy RHEL 6.5 based Hosts :)

Comment 9 Jaroslav Škarvada 2014-03-18 16:24:46 UTC
(In reply to Sven Kieske from comment #8)
> So is something blocking this getting fixed?
> 
Unfortunately the reproducer from comment 4 is not reproducer for this bug (it reproduces similar problem). I noticed this later.

I think I know where the original problem is. I will provide scratch build for testing, because I am unable to reproduce the original problem.

Comment 10 Jaroslav Škarvada 2014-03-18 16:28:16 UTC
(In reply to Alon Bar-Lev from comment #6)
> (In reply to Jaroslav Škarvada from comment #4)
> > Reproducer:
> > 
> > # tuned-adm profile virtual-host & service tuned start
> > 
> > This is not regression, the problem seems to be there since the beginning of
> > the tuned project, but I am going to fix it.
> 
> Great. We are using same sequence since rhel-6.4 (2012-12-27), and have not
> experienced issue... up to after rhel-6.5... so it is strange... :)

Could you provide me version number of tuned which worked for you? I am afraid the problem was there all the time (but could manifest itself with much lower probability), and the probability was amplified by the latest update.

Comment 13 Jiri Belka 2014-04-01 13:45:47 UTC
Can't reproduce with tuned from RHN and even with tuned from brew.

I tried with (based on alonbl@ tip):

- while true; do service tuned stop ; service tuned start ; tuned-adm profile virtual-host; done
- while true; do service tuned stop; service messagebus stop ; service messagebus start; service tuned start ; tuned-adm profile virtual-host ; done

Comment 14 Jaroslav Škarvada 2014-04-01 13:51:52 UTC
(In reply to Jiri Belka from comment #13)
> Can't reproduce with tuned from RHN and even with tuned from brew.
> 
> I tried with (based on alonbl@ tip):
> 
> - while true; do service tuned stop ; service tuned start ; tuned-adm
> profile virtual-host; done
> - while true; do service tuned stop; service messagebus stop ; service
> messagebus start; service tuned start ; tuned-adm profile virtual-host ; done

Thanks, this is one of the reproducers that I have been trying, but I also haven't been successful in reproducing the problem. I think the only way is to try to re-provision the affected machine.

Comment 15 Eyal Edri 2014-04-06 10:14:10 UTC
Lukas - can you approve you tested the scratch build and it's reproducible on qe environment?

Comment 16 Lukas Bednar 2014-04-08 08:38:58 UTC
I reverted provided WA [1], and we were experiencing same problem during night.
Notice that it doesn't occur every run, I mean it is not 100% reproducible.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1069119#c8

Comment 17 Jaroslav Škarvada 2014-04-08 12:07:02 UTC
(In reply to Lukas Bednar from comment #16)
> I reverted provided WA [1], and we were experiencing same problem during
> night.
> Notice that it doesn't occur every run, I mean it is not 100% reproducible.
> 
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1069119#c8

Build from comment 12 adds locking to places that could cause deadlocks. No idea what else is wrong, could you provide simple reproducer?

You could also try running tuned with --debug option, just append --debug to line containing:
daemon $exec -d -c $config
i.e.:
daemon $exec -d -c $config --debug

in /etc/init.d/tuned
Maybe verbose logging would reveal something useful.