Bug 1443142

Summary: cpu-partitioning: systemctl restart returns before initrd is created
Product: Red Hat Enterprise Linux 7 Reporter: Luiz Capitulino <lcapitulino>
Component: tunedAssignee: Jaroslav Škarvada <jskarvad>
Status: CLOSED ERRATA QA Contact: Tereza Cerna <tcerna>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.4CC: jeder, jskarvad, olysonek, psklenar, tcerna, yturgema
Target Milestone: rcKeywords: Upstream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: tuned-2.9.0-0.1.rc1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-10 16:01:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1394932, 1467576, 1485946, 1516123    

Description Luiz Capitulino 2017-04-18 15:04:02 UTC
Description of problem:

The script below will say "tuned-initrd.img" does not exist. This happens because systemctl restart returns before tuned-initrd.img is really created. It will work if you uncomment the "sleep 5s" line.

#!/bin/bash

set -e 

TUNED_INITRD=/boot/tuned-initrd.img

echo "# Test comment ($RANDOM)" >> /etc/systemd/system.conf

systemctl restart tuned
# sleep 5s

lsinitrd /boot/tuned-initrd.img -f etc/systemd/system.conf > /tmp/system.conf
if ! cmp /tmp/system.conf /etc/systemd/system.conf; then
    echo system.conf differs
    exit 1
fi

Version-Release number of selected component (if applicable): tuned-2.8.0-2.el7


How reproducible:


Steps to Reproduce:
1. Active the cpu-partitioning profile
2. Run the script above

Comment 2 Jaroslav Škarvada 2017-04-18 15:08:44 UTC
Upstream report:
https://github.com/redhat-performance/tuned/issues/23

Comment 3 Luiz Capitulino 2017-04-18 15:10:29 UTC
Jaroslav,

I wasn't able to reproduce this issue at activation time with the script below:

#!/bin/bash

tuned-adm profile balanced
tuned-adm profile cpu-partitioning
if [ ! -f /boot/tuned-initrd.img ]; then
    echo ERROR: initrd image does not exist
    exit 1
else
    echo initrd image exists
fi

Is there anything in tuned that prevents this issue from happening at activation time? If yes and if this issue only happens with systemctl restart, then we can move it for 7.5 because it's not very serious.

Otherwise, if this happens at activation time then it's a very serious issue because people who activate the profile and then quickly reboot may get an unbootable system.

Comment 4 Jaroslav Škarvada 2017-04-18 18:41:20 UTC
Problem is that systemctl can return before the profile is applied. It always worked this way. We are trying to address this upstream (comment 2), but it is a big change in behaviour that needs to be tested, thus probably not ready and risky for 7.4.

The tuned-adm cannot return before the profile is applied (by default it's running in sync mode).

I think now the unbootable system could theoretically happen on load, but the probability of this 'bad state' is lowered by the current units ordering:

1. systemd patching
2. initrd generation
3. grub2 config patching

I.e. grub2 is patched after the initrd is successfully generated. If the machine is rebooted earlier it reboots without the initrd overlay - i.e. it's harmless. If it is rebooted after grub2 config is patched, the initrd is already there. There is a risk the quick reboot will trigger a race, but I think it's not very common scenario to reboot machines quickly after boot.

Comment 5 Luiz Capitulino 2017-04-18 18:54:31 UTC
OK, I'm moving to 7.5 because the scenario looks unlikely and it seems there isn't anything we can do for the 7.4 timeframe. But let's keep this on in the radar in case customers trig this.

Comment 6 Fedora Update System 2017-10-13 14:21:23 UTC
tuned-2.9.0-0.1.rc1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-d9c6b990df

Comment 8 Fedora Update System 2017-10-13 22:25:37 UTC
tuned-2.9.0-0.1.rc1.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-d9c6b990df

Comment 9 Fedora Update System 2017-10-13 23:25:30 UTC
tuned-2.9.0-0.1.rc1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-5f0849d207

Comment 10 Fedora Update System 2017-10-29 21:07:38 UTC
tuned-2.9.0-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-0e45ce4685

Comment 11 Fedora Update System 2017-10-29 21:13:43 UTC
tuned-2.9.0-1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-c30e9bd1ea

Comment 12 Luiz Capitulino 2018-01-23 13:55:46 UTC
I verified this BZ. With latest tuned the script from the description always work: meaning tuned-initrd.img is created before systemctl returns.

Comment 13 Tereza Cerna 2018-02-19 10:14:42 UTC
Thank you Luiz for your testing.
---------

Verified in tuned-2.9.0-1.el7.noarch
  * thanks to Luiz for his testing (see #12)
  * patch was applied in new version of tuned
-> VERIFIED

Comment 15 Ondřej Lysoněk 2018-03-26 11:48:47 UTC
*** Bug 1523194 has been marked as a duplicate of this bug. ***

Comment 17 errata-xmlrpc 2018-04-10 16:01:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0879