Bug 1443142 - cpu-partitioning: systemctl restart returns before initrd is created
cpu-partitioning: systemctl restart returns before initrd is created
Status: ON_QA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: tuned (Show other bugs)
7.4
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Jaroslav Škarvada
Luiz Capitulino
: Upstream
Depends On:
Blocks: 1394932 TUNED-7.5-REBASE 1485946
  Show dependency treegraph
 
Reported: 2017-04-18 11:04 EDT by Luiz Capitulino
Modified: 2017-10-29 17:13 EDT (History)
5 users (show)

See Also:
Fixed In Version: tuned-2.9.0-0.1.rc1.el7
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Luiz Capitulino 2017-04-18 11:04:02 EDT
Description of problem:

The script below will say "tuned-initrd.img" does not exist. This happens because systemctl restart returns before tuned-initrd.img is really created. It will work if you uncomment the "sleep 5s" line.

#!/bin/bash

set -e 

TUNED_INITRD=/boot/tuned-initrd.img

echo "# Test comment ($RANDOM)" >> /etc/systemd/system.conf

systemctl restart tuned
# sleep 5s

lsinitrd /boot/tuned-initrd.img -f etc/systemd/system.conf > /tmp/system.conf
if ! cmp /tmp/system.conf /etc/systemd/system.conf; then
    echo system.conf differs
    exit 1
fi

Version-Release number of selected component (if applicable): tuned-2.8.0-2.el7


How reproducible:


Steps to Reproduce:
1. Active the cpu-partitioning profile
2. Run the script above
Comment 2 Jaroslav Škarvada 2017-04-18 11:08:44 EDT
Upstream report:
https://github.com/redhat-performance/tuned/issues/23
Comment 3 Luiz Capitulino 2017-04-18 11:10:29 EDT
Jaroslav,

I wasn't able to reproduce this issue at activation time with the script below:

#!/bin/bash

tuned-adm profile balanced
tuned-adm profile cpu-partitioning
if [ ! -f /boot/tuned-initrd.img ]; then
    echo ERROR: initrd image does not exist
    exit 1
else
    echo initrd image exists
fi

Is there anything in tuned that prevents this issue from happening at activation time? If yes and if this issue only happens with systemctl restart, then we can move it for 7.5 because it's not very serious.

Otherwise, if this happens at activation time then it's a very serious issue because people who activate the profile and then quickly reboot may get an unbootable system.
Comment 4 Jaroslav Škarvada 2017-04-18 14:41:20 EDT
Problem is that systemctl can return before the profile is applied. It always worked this way. We are trying to address this upstream (comment 2), but it is a big change in behaviour that needs to be tested, thus probably not ready and risky for 7.4.

The tuned-adm cannot return before the profile is applied (by default it's running in sync mode).

I think now the unbootable system could theoretically happen on load, but the probability of this 'bad state' is lowered by the current units ordering:

1. systemd patching
2. initrd generation
3. grub2 config patching

I.e. grub2 is patched after the initrd is successfully generated. If the machine is rebooted earlier it reboots without the initrd overlay - i.e. it's harmless. If it is rebooted after grub2 config is patched, the initrd is already there. There is a risk the quick reboot will trigger a race, but I think it's not very common scenario to reboot machines quickly after boot.
Comment 5 Luiz Capitulino 2017-04-18 14:54:31 EDT
OK, I'm moving to 7.5 because the scenario looks unlikely and it seems there isn't anything we can do for the 7.4 timeframe. But let's keep this on in the radar in case customers trig this.
Comment 6 Fedora Update System 2017-10-13 10:21:23 EDT
tuned-2.9.0-0.1.rc1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-d9c6b990df
Comment 8 Fedora Update System 2017-10-13 18:25:37 EDT
tuned-2.9.0-0.1.rc1.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-d9c6b990df
Comment 9 Fedora Update System 2017-10-13 19:25:30 EDT
tuned-2.9.0-0.1.rc1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-5f0849d207
Comment 10 Fedora Update System 2017-10-29 17:07:38 EDT
tuned-2.9.0-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-0e45ce4685
Comment 11 Fedora Update System 2017-10-29 17:13:43 EDT
tuned-2.9.0-1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-c30e9bd1ea

Note You need to log in before you can comment on or make changes to this bug.