Bug 1534833

Summary: Require kickstart/GUI spoke config option for certain security mitigations
Product: Red Hat Enterprise Linux 7 Reporter: Jon Masters <jcm>
Component: anacondaAssignee: Martin Kolman <mkolman>
Status: CLOSED WONTFIX QA Contact: Release Test Team <release-test-team-automation>
Severity: unspecified Docs Contact: Petr Bokoc <pbokoc>
Priority: unspecified    
Version: 7.5CC: bhsharma, dyoung, jbastian, jcm, jkurik, jstodola, jvavra, pbokoc, ruyang, rvykydal, salmy, sbueno, wsato
Target Milestone: rcKeywords: FutureFeature
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: anaconda-21.48.22.131-1 Doc Type: Enhancement
Doc Text:
The graphical and text mode installers now have a screen for disabling certain kernel vulnerability mitigations A new screen titled `Kernel & Performance` is now available in the graphical and text mode interactive installer. The screen allows you to switch off the kernel's mitigation mechanisms which were implemented to fix the Meltdown and Spectre vulnerabilities. Some of these mitigation mechanisms have a noticeable performance impact in certain scenarios, and at the same time may not be required in your environment. This new screen provides a way to disable them in these cases. The new screen contains an on/off switch for each mitigation: Page Table Isolation (PTI), Indirect Branch Restricted Speculation (IBRS), and Indirect Branch Prediction Barriers (IBPB). They can be turned on and off individually, and are all on by default. Red Hat strongly recommends keeping all three mitigation mechanisms enabled unless you or your organization performed an detailed analysis of potential risks and concluded that one or more are not necessary. There is no new Kickstart command to implement a way to switch these mechanisms on or off during a Kickstart installation. However, you can use the existing "bootloader --append" option to add options disabling one or more of these mechanisms to the kernel command line. To disable all three, use "bootloader --append='nopti noibrs noipbp'". This achieves the same result as switching all three mitigation techniques off in the installer, which also appends one or more of these kernel parameters to the boot command line. See the following article for additional information about the Meltdown and Spectre vulnerabilities: https://www.redhat.com/en/blog/what-are-meltdown-and-spectre-heres-what-you-need-know
Story Points: ---
Clone Of:
: 1545304 1546267 (view as bug list) Environment:
Last Closed: 2018-02-16 17:43:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1545304, 1546267    

Description Jon Masters 2018-01-16 03:12:29 UTC
Description of problem:

We have a (significant) performance hit from enabling both PTI (Page Table Isolation) and disabling branch prediction (IBRS/IBPB) by default in order to mitigate "Meltdown" and "Spectre" exploits. While it's possible to add a kernel command line option to disable our mitigations, it would be ideal to have this be a very simple couple of options in the GUI that are by default checked, but easily unchecked by a user if they choose to do so.

Comment 2 Samantha N. Bueno 2018-01-16 12:27:12 UTC
Ideally we would want this to go into a spoke a user would intuitively choose to visit if they wanted to disable this functionality. Of the spokes which currently exist, however, none are good candidates for this; none seem like obvious choices if someone wanted to disable PTI and/or branch prediction.

But -- the oscap addon exists to allow users to select security profiles. So I think adding this option to the oscap addon screen might be viable. Watson, what do you think? My team would be happy to work with yours to add this feature.

We could also create a new spoke in anaconda (or as a new addon) to allow users to deselect this feature. That might have a higher impact on the translation team though. A new spoke for deselecting one thing also seems a little much.

I'll add any other options as we come up with them.

Comment 3 Samantha N. Bueno 2018-01-16 13:46:17 UTC
Okay, we have another option:

We can add this configuration option to the kdump addon screen. This is actually better than adding the option to the oscap screen since:
(a) the oscap addon screen is already quite full
(b) the kdump screen is already a place where people enable/disable/configure kdump, which they typically do for performance concerns. Allowing people to disable PTI/branch prediction for performance reasons fits.

We would have to rename the spoke to something a little more generic ('performance' maybe?), but hopefully that is not a significant concern.

I'm adding Dave Young to Cc, since I'd like to hear his opinion on this as the maintainer of the kdump addon. Dave, Martin Kolman from my team is happy to work with yours to add this feature.

Jon, let me know what you think as well.

Comment 4 Jeff Bastian 2018-01-16 14:11:44 UTC
For kickstart, you can disable the mitigations today with the bootloader command:

bootloader --append='nopti noibrs noibpb'


If we add a GUI option, then maybe we'll want a corresponding command for kickstart, but it's technically not necessary given the above method.

Comment 5 Dave Young 2018-01-17 05:18:17 UTC
Samantha, I feel it is not good to add this in kdump addon which is sepecific for kdump only. But I'm not sure about oscap because I do not know what it is..

Or maybe it would be better to add another addon to tune bootloader appended parameters, and for the known bool switches, they can be set as radio buttons.

Comment 6 Martin Kolman 2018-01-17 17:54:56 UTC
I've looked into adding a separate spoke screen for the mitigation controls. The relevant Anaconda hub screens currently look like this in RHEL 7.5:

TUI
https://m4rtink.fedorapeople.org/screenshots/rhel/rhel_75_anaconda_tui_hub.png

GUI
https://m4rtink.fedorapeople.org/screenshots/rhel/rhel_75_anaconda_gui_summary_hub.png

Screen space wise TUI seem fine - adding another spoke should not be a problem. 

The situation is less clear in the GUI, but it *should* be possible to add another spoke as long as it is added to the SYSTEM category without the need for scrolling on most screen sizes.

If we go this way - adding a new spoke for security mitigations - there is a couple of questions that need to be answered:

- how should this new screen be called ?
-> MITIGATIONS (as that's what will be configured there)
-> PERFORMANCE (as that's the actual reason for making it possible to turn the mitigations off)
-> KERNEL SETTINGS (as that's how you turn off the mitigations, but would not be very future proof)
-> SECURITY (might clash with the SECURITY POLICY screen provided by the Oscap Addon)
-> KERNEL & PERFORMANCE (combining some of the above)
-> something else ?

- what individual options should be there and what should they be called ?
-> as for representation I guess we want check boxes, checked by default that the user can uncheck, right ?

- do we want to provide some additional information about what these options do, where they are relevant and what's their expected impact ?
-> Anaconda has a built-in help system[0], which could be used to host this kind of information


For kickstart support - it has been noted in comment 4 there is a possible workaround via bootloader --append - is that good enough or do we want to have a separate kickstart command anyway ? Also I see the possibility of using bootloader --append for now and adding a dedicated kickstart command later.


[0] https://m4rtink.fedorapeople.org/screenshots/rhel/rhel_75_anaconda_gui_hub_help.png

Comment 7 Watson Yuuma Sato 2018-01-18 09:29:52 UTC
Hello,

We don't feel that oscap-anaconda-addon is a good place for this.
The addon is designed to select security profiles to apply, not for customization of profiles at installation time.
And OAA is related to security compliance and hardening, disabling security features that we might be checking for (depending on content loaded) feels strange.

Comment 8 Samantha N. Bueno 2018-01-18 11:15:22 UTC
Thanks for the input, Watson and Dave. You both have good points.


(In reply to Martin Kolman from comment #6)
> I've looked into adding a separate spoke screen for the mitigation controls.
> The relevant Anaconda hub screens currently look like this in RHEL 7.5:
> 
> TUI
> https://m4rtink.fedorapeople.org/screenshots/rhel/rhel_75_anaconda_tui_hub.
> png
> 
> GUI
> https://m4rtink.fedorapeople.org/screenshots/rhel/
> rhel_75_anaconda_gui_summary_hub.png
> 
> Screen space wise TUI seem fine - adding another spoke should not be a
> problem. 
> 
> The situation is less clear in the GUI, but it *should* be possible to add
> another spoke as long as it is added to the SYSTEM category without the need
> for scrolling on most screen sizes.
> 
> If we go this way - adding a new spoke for security mitigations - there is a
> couple of questions that need to be answered:
> 
> - how should this new screen be called ?
> -> MITIGATIONS (as that's what will be configured there)
> -> PERFORMANCE (as that's the actual reason for making it possible to turn
> the mitigations off)
> -> KERNEL SETTINGS (as that's how you turn off the mitigations, but would
> not be very future proof)
> -> SECURITY (might clash with the SECURITY POLICY screen provided by the
> Oscap Addon)
> -> KERNEL & PERFORMANCE (combining some of the above)
> -> something else ?

Naming is always a bit tricky, but I think "performance" (which is nice and short) or "kernel & performance" would be good.


> 
> - what individual options should be there and what should they be called ?

It sounds like "PTI (Page Table Isolation)" and "IBRS/IBPB (Indirect Branch Restricted Speculation/ Indirect Branch Prediction Barrier)" would be the options, going off the original comment.


> -> as for representation I guess we want check boxes, checked by default
> that the user can uncheck, right ?

Agreed, I think this sounds reasonable.


> 
> - do we want to provide some additional information about what these options
> do, where they are relevant and what's their expected impact ?
> -> Anaconda has a built-in help system[0], which could be used to host this
> kind of information

I'm not sure about the help system. Maybe some short blurb could be displayed on the spoke itself. Honestly, whichever would be easier though.


> 
> 
> For kickstart support - it has been noted in comment 4 there is a possible
> workaround via bootloader --append - is that good enough or do we want to
> have a separate kickstart command anyway ? Also I see the possibility of
> using bootloader --append for now and adding a dedicated kickstart command
> later.

My opinion is that Jeff's suggestion in comment 4 is good enough. People who are aware enough to know they want these options disabled are most likely going to be savvy enough to add the boot parameters in their kickstart file. It also lessens the impact of this, since then we wouldn't need to make changes in pykickstart as well.

Jon, what is your opinion on all of this? I ask because I heard (outside of the BZ) that just requiring this in kickstart would be okay. In that case, we could just go with Jeff's suggestion in comment 4, and that would be super fast and easy. If an option for this is needed in the actual UI though, it would be great to have your thoughts on the suggestions outlined here.

Comment 9 Martin Kolman 2018-01-31 16:28:21 UTC
Link to pull request: https://github.com/rhinstaller/anaconda/pull/1317

I have added a new spoke, called, "Kernel & performance", for both the  Anaconda GUI and TUI. In both GUI and TUI the spoke has 3 switches, one each for toggling PTI, IBRS and IBPB.

Internally these switches add and remove the nopti/noibrs/noibpb options from the "bootaloder --append" line. If the user already has "bootloader --append='nopti'" in the kickstart, the PTI switch will be in the OFF state when the spoke is first visited. At that point it's possible to turn the mitigation back ON or leave it as is. Any changes in the spoke will also of course be reflected in "bootloader --append" in the resulting kickstart file Anaconda writes after the installation to /root.

How does the new spoke look like ?

TUI:
http://file.brq.redhat.com/~mkolman/screenshots/rhel7/rhel7.5/rhel_75_anaconda_tui_hub_with_kernel_and_performance_spoke.png
http://file.brq.redhat.com/~mkolman/screenshots/rhel7/rhel7.5/rhel_75_anaconda_tui_kernel_and_performance_spoke.png

GUI:
http://file.brq.redhat.com/~mkolman/screenshots/rhel7/rhel7.5/rhel_75_anaconda_gui_hub_with_kernel_and_performance_spoke.png
http://file.brq.redhat.com/~mkolman/screenshots/rhel7/rhel7.5/rhel_75_anaconda_gui_kernel_and_performance_spoke.png

Any feedback on what else to add or change welcome!

Comment 10 Samantha N. Bueno 2018-02-01 12:01:14 UTC
Martin, I think you did a great job. My one suggestion would be to remove the '.' in the spoke status ("All mitigations enabled") for consistency.

Any others on this bug, feel free to comment with your thoughts as well.

Comment 12 Martin Kolman 2018-02-05 17:34:57 UTC
(In reply to Samantha N. Bueno from comment #10)
> Martin, I think you did a great job. My one suggestion would be to remove
> the '.' in the spoke status ("All mitigations enabled") for consistency.
Sure - I've updated the PR.

Comment 15 Jan Stodola 2018-02-14 16:12:07 UTC
Help for the new spoke is not available and it's reported in bug 1545323

Comment 16 Jeff Bastian 2018-02-15 15:59:43 UTC
Is the spoke unique for each architecture?

The three parameters discussed above -- nopti, noibpb, noibrs -- are for x86.  The other architectures supported by RHEL have different flags, e.g., no_rfi_flush on ppc64, or kpti=off on aarch64, or nobp=off on s390x.

(The ppc64 kernel will accept nopti as a synonym for no_rfi_flush, however, and I'm trying to add the same for aarch64, see bug 1536612)

Comment 17 Samantha N. Bueno 2018-02-15 18:03:31 UTC
(In reply to Jeff Bastian from comment #16)
> Is the spoke unique for each architecture?
> 
> The three parameters discussed above -- nopti, noibpb, noibrs -- are for
> x86.  The other architectures supported by RHEL have different flags, e.g.,
> no_rfi_flush on ppc64, or kpti=off on aarch64, or nobp=off on s390x.
> 
> (The ppc64 kernel will accept nopti as a synonym for no_rfi_flush, however,
> and I'm trying to add the same for aarch64, see bug 1536612)

Unfortunately the spoke is the same across all arches, so it seems like this could be a problem.

I see a couple of options here:

(a) We can hide the spoke from the non-x86 arches. Since most graphical installs are probably done on x86 anyway, hopefully the impact of this would be minimal to other arches. They would also still have the option of disabling the options via kickstart or by entering parameters on the command line on boot.

(b) We can add the support in other arches for disabling the above options, but that is going to actually be a lot of work for us, it'll increase risk of this introducing a bug, and it's going to put a lot of burden on QE for testing.

My suggestion would be option (a) because this late in the release cycle, it's better to be risk-averse, especially with a change that has such high visibility to users.

I'll reach out to PM for input since I believe this decision is probably best left to them. Thanks for pointing this out though -- this would not have been good for ppc/s390/aarch64.

Comment 18 Jeff Bastian 2018-02-15 18:21:16 UTC
I agree with the option to hide the spoke on non-x86 systems; leaving the spoke enabled would just lead to confusion.

Comment 19 Samantha N. Bueno 2018-02-16 13:34:49 UTC
Moving this back to 'assigned' due to c#16.

Comment 21 Martin Kolman 2018-02-16 14:51:57 UTC
I've created a PR that hides the "Kernel & performance" spoke on non-x86 architectures: 
https://github.com/rhinstaller/anaconda/pull/1336

I've also created an updates image that can be used against the latest 7.5 compose (Anaconda 21.48.22.133-1) to test the change:
http://file.brq.redhat.com/~mkolman/update_images/rhel75_kp_spoke/rhel7_hide_kp_spoke_on_non_x86.img

Comment 22 Jan Stodola 2018-02-16 15:03:28 UTC
The updates.img works fine on s390x, the kernel & performance spoke is not visible anywhere, tried both vnc and text mode.

Comment 23 Steve Almy 2018-02-16 17:43:00 UTC
After consulting with CEE, I'm CLOSING the BZ as WONTFIX.  Details below...

1.  We should document the fix in kickstart installs.  Maybe this is unnecessary,
    given the boot option and well-documented ways to add these via kickstart.

2.  For customers installing via GUI/TUI, it is too easy to make a bad 
    decision here, given the risk involved in making the wrong decision.

3.  95% or affected customers that need to make this change will not be
    installing 7.5.  They will be updating existing installations.  We should
    spend our energy on a solution to updating running systems.

4.  This "simple" addition opens a huge set of future changes that we are not
    ready to think about:

   4.1 What other system/kernel options should be here too?
   4.2 What other CVE mitigations do customer want to disable?
   4.3 What more general system config settings should be presented at
       install time?

Solving this problem fits better with cockpit or an ansible system role.  I will open this conversation there to pursue a solution that helps customers at scale, where this problem is most impactful.

Comment 24 Petr Bokoc 2018-02-20 13:36:42 UTC
Removed the doc text from the 7.5 release notes.

Comment 26 Red Hat Bugzilla 2023-09-14 04:15:30 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days