Description of problem:
We have a (significant) performance hit from enabling both PTI (Page Table Isolation) and disabling branch prediction (IBRS/IBPB) by default in order to mitigate "Meltdown" and "Spectre" exploits. While it's possible to add a kernel command line option to disable our mitigations, it would be ideal to have this be a very simple couple of options in the GUI that are by default checked, but easily unchecked by a user if they choose to do so.
Ideally we would want this to go into a spoke a user would intuitively choose to visit if they wanted to disable this functionality. Of the spokes which currently exist, however, none are good candidates for this; none seem like obvious choices if someone wanted to disable PTI and/or branch prediction.
But -- the oscap addon exists to allow users to select security profiles. So I think adding this option to the oscap addon screen might be viable. Watson, what do you think? My team would be happy to work with yours to add this feature.
We could also create a new spoke in anaconda (or as a new addon) to allow users to deselect this feature. That might have a higher impact on the translation team though. A new spoke for deselecting one thing also seems a little much.
I'll add any other options as we come up with them.
Okay, we have another option:
We can add this configuration option to the kdump addon screen. This is actually better than adding the option to the oscap screen since:
(a) the oscap addon screen is already quite full
(b) the kdump screen is already a place where people enable/disable/configure kdump, which they typically do for performance concerns. Allowing people to disable PTI/branch prediction for performance reasons fits.
We would have to rename the spoke to something a little more generic ('performance' maybe?), but hopefully that is not a significant concern.
I'm adding Dave Young to Cc, since I'd like to hear his opinion on this as the maintainer of the kdump addon. Dave, Martin Kolman from my team is happy to work with yours to add this feature.
Jon, let me know what you think as well.
For kickstart, you can disable the mitigations today with the bootloader command:
bootloader --append='nopti noibrs noibpb'
If we add a GUI option, then maybe we'll want a corresponding command for kickstart, but it's technically not necessary given the above method.
Samantha, I feel it is not good to add this in kdump addon which is sepecific for kdump only. But I'm not sure about oscap because I do not know what it is..
Or maybe it would be better to add another addon to tune bootloader appended parameters, and for the known bool switches, they can be set as radio buttons.
I've looked into adding a separate spoke screen for the mitigation controls. The relevant Anaconda hub screens currently look like this in RHEL 7.5:
Screen space wise TUI seem fine - adding another spoke should not be a problem.
The situation is less clear in the GUI, but it *should* be possible to add another spoke as long as it is added to the SYSTEM category without the need for scrolling on most screen sizes.
If we go this way - adding a new spoke for security mitigations - there is a couple of questions that need to be answered:
- how should this new screen be called ?
-> MITIGATIONS (as that's what will be configured there)
-> PERFORMANCE (as that's the actual reason for making it possible to turn the mitigations off)
-> KERNEL SETTINGS (as that's how you turn off the mitigations, but would not be very future proof)
-> SECURITY (might clash with the SECURITY POLICY screen provided by the Oscap Addon)
-> KERNEL & PERFORMANCE (combining some of the above)
-> something else ?
- what individual options should be there and what should they be called ?
-> as for representation I guess we want check boxes, checked by default that the user can uncheck, right ?
- do we want to provide some additional information about what these options do, where they are relevant and what's their expected impact ?
-> Anaconda has a built-in help system, which could be used to host this kind of information
For kickstart support - it has been noted in comment 4 there is a possible workaround via bootloader --append - is that good enough or do we want to have a separate kickstart command anyway ? Also I see the possibility of using bootloader --append for now and adding a dedicated kickstart command later.
We don't feel that oscap-anaconda-addon is a good place for this.
The addon is designed to select security profiles to apply, not for customization of profiles at installation time.
And OAA is related to security compliance and hardening, disabling security features that we might be checking for (depending on content loaded) feels strange.
Thanks for the input, Watson and Dave. You both have good points.
(In reply to Martin Kolman from comment #6)
> I've looked into adding a separate spoke screen for the mitigation controls.
> The relevant Anaconda hub screens currently look like this in RHEL 7.5:
> Screen space wise TUI seem fine - adding another spoke should not be a
> The situation is less clear in the GUI, but it *should* be possible to add
> another spoke as long as it is added to the SYSTEM category without the need
> for scrolling on most screen sizes.
> If we go this way - adding a new spoke for security mitigations - there is a
> couple of questions that need to be answered:
> - how should this new screen be called ?
> -> MITIGATIONS (as that's what will be configured there)
> -> PERFORMANCE (as that's the actual reason for making it possible to turn
> the mitigations off)
> -> KERNEL SETTINGS (as that's how you turn off the mitigations, but would
> not be very future proof)
> -> SECURITY (might clash with the SECURITY POLICY screen provided by the
> Oscap Addon)
> -> KERNEL & PERFORMANCE (combining some of the above)
> -> something else ?
Naming is always a bit tricky, but I think "performance" (which is nice and short) or "kernel & performance" would be good.
> - what individual options should be there and what should they be called ?
It sounds like "PTI (Page Table Isolation)" and "IBRS/IBPB (Indirect Branch Restricted Speculation/ Indirect Branch Prediction Barrier)" would be the options, going off the original comment.
> -> as for representation I guess we want check boxes, checked by default
> that the user can uncheck, right ?
Agreed, I think this sounds reasonable.
> - do we want to provide some additional information about what these options
> do, where they are relevant and what's their expected impact ?
> -> Anaconda has a built-in help system, which could be used to host this
> kind of information
I'm not sure about the help system. Maybe some short blurb could be displayed on the spoke itself. Honestly, whichever would be easier though.
> For kickstart support - it has been noted in comment 4 there is a possible
> workaround via bootloader --append - is that good enough or do we want to
> have a separate kickstart command anyway ? Also I see the possibility of
> using bootloader --append for now and adding a dedicated kickstart command
My opinion is that Jeff's suggestion in comment 4 is good enough. People who are aware enough to know they want these options disabled are most likely going to be savvy enough to add the boot parameters in their kickstart file. It also lessens the impact of this, since then we wouldn't need to make changes in pykickstart as well.
Jon, what is your opinion on all of this? I ask because I heard (outside of the BZ) that just requiring this in kickstart would be okay. In that case, we could just go with Jeff's suggestion in comment 4, and that would be super fast and easy. If an option for this is needed in the actual UI though, it would be great to have your thoughts on the suggestions outlined here.
Link to pull request: https://github.com/rhinstaller/anaconda/pull/1317
I have added a new spoke, called, "Kernel & performance", for both the Anaconda GUI and TUI. In both GUI and TUI the spoke has 3 switches, one each for toggling PTI, IBRS and IBPB.
Internally these switches add and remove the nopti/noibrs/noibpb options from the "bootaloder --append" line. If the user already has "bootloader --append='nopti'" in the kickstart, the PTI switch will be in the OFF state when the spoke is first visited. At that point it's possible to turn the mitigation back ON or leave it as is. Any changes in the spoke will also of course be reflected in "bootloader --append" in the resulting kickstart file Anaconda writes after the installation to /root.
How does the new spoke look like ?
Any feedback on what else to add or change welcome!
Martin, I think you did a great job. My one suggestion would be to remove the '.' in the spoke status ("All mitigations enabled") for consistency.
Any others on this bug, feel free to comment with your thoughts as well.
(In reply to Samantha N. Bueno from comment #10)
> Martin, I think you did a great job. My one suggestion would be to remove
> the '.' in the spoke status ("All mitigations enabled") for consistency.
Sure - I've updated the PR.
Help for the new spoke is not available and it's reported in bug 1545323
Is the spoke unique for each architecture?
The three parameters discussed above -- nopti, noibpb, noibrs -- are for x86. The other architectures supported by RHEL have different flags, e.g., no_rfi_flush on ppc64, or kpti=off on aarch64, or nobp=off on s390x.
(The ppc64 kernel will accept nopti as a synonym for no_rfi_flush, however, and I'm trying to add the same for aarch64, see bug 1536612)
(In reply to Jeff Bastian from comment #16)
> Is the spoke unique for each architecture?
> The three parameters discussed above -- nopti, noibpb, noibrs -- are for
> x86. The other architectures supported by RHEL have different flags, e.g.,
> no_rfi_flush on ppc64, or kpti=off on aarch64, or nobp=off on s390x.
> (The ppc64 kernel will accept nopti as a synonym for no_rfi_flush, however,
> and I'm trying to add the same for aarch64, see bug 1536612)
Unfortunately the spoke is the same across all arches, so it seems like this could be a problem.
I see a couple of options here:
(a) We can hide the spoke from the non-x86 arches. Since most graphical installs are probably done on x86 anyway, hopefully the impact of this would be minimal to other arches. They would also still have the option of disabling the options via kickstart or by entering parameters on the command line on boot.
(b) We can add the support in other arches for disabling the above options, but that is going to actually be a lot of work for us, it'll increase risk of this introducing a bug, and it's going to put a lot of burden on QE for testing.
My suggestion would be option (a) because this late in the release cycle, it's better to be risk-averse, especially with a change that has such high visibility to users.
I'll reach out to PM for input since I believe this decision is probably best left to them. Thanks for pointing this out though -- this would not have been good for ppc/s390/aarch64.
I agree with the option to hide the spoke on non-x86 systems; leaving the spoke enabled would just lead to confusion.
Moving this back to 'assigned' due to c#16.
I've created a PR that hides the "Kernel & performance" spoke on non-x86 architectures:
I've also created an updates image that can be used against the latest 7.5 compose (Anaconda 184.108.40.206-1) to test the change:
The updates.img works fine on s390x, the kernel & performance spoke is not visible anywhere, tried both vnc and text mode.
After consulting with CEE, I'm CLOSING the BZ as WONTFIX. Details below...
1. We should document the fix in kickstart installs. Maybe this is unnecessary,
given the boot option and well-documented ways to add these via kickstart.
2. For customers installing via GUI/TUI, it is too easy to make a bad
decision here, given the risk involved in making the wrong decision.
3. 95% or affected customers that need to make this change will not be
installing 7.5. They will be updating existing installations. We should
spend our energy on a solution to updating running systems.
4. This "simple" addition opens a huge set of future changes that we are not
ready to think about:
4.1 What other system/kernel options should be here too?
4.2 What other CVE mitigations do customer want to disable?
4.3 What more general system config settings should be presented at
Solving this problem fits better with cockpit or an ansible system role. I will open this conversation there to pursue a solution that helps customers at scale, where this problem is most impactful.
Removed the doc text from the 7.5 release notes.