Bug 1392051

Hi Evan,
I am not sure we want to support customized appliances, however we may want to request persisting customized after-deployment appliances, if it is not allowed yet.


In the case we are working together, I was thinking more about having a pre-made appliance that meets STIG standards. But this can be challenging, since there are multiple standards and one appliance may not work for all. 


In this case, we may want to allow also non-appliance deployments of HE VM, so that the customers would be able customizing their VMs the way they need from the very beginning. And it will be easier to support, rather then a customize appliance.


Let PM discuss this with the public sector team and weigh in on this.

DISA has Security Requirement Guides, or SRGs, that reflect DoD-mandated controls. They're organized by layers of the infrastructure -- e.g. network devices, operating systems, application servers, database servers, applications, so forth.

DoD Secure Technical Implementation Guides (DoD STIGS) reflect the vendor-specific configurations of a product to a DISA SRG. For example, the RHEL7 STIG reflects how to configure RHEL7 against the DISA Operating System SRG.

If we were to undertake building a secure appliance, we'd have to decompose RHEV into relevant subsystems (e.g. OS, app servers, app) and review the applicable DISA SRGs. We'd then have to interpret them for our components. For example, reading the DISA Database SRG and interpreting what components are applicable to embeded RHEV Postgres database servers. It's a big undertaking.

So, to make this consumable... ideas on how to start chunking this into smaller pieces:

(1) First we start with the operating system. For that, we'd want to apply the RHEL7 STIG and load the RHEV appliance. Thankfully we have published RHEL kickstarts for this. The RHEL7 STIG profile enables FIPS mode, sets various kernel and sysctl options, turns on verbose auditing. About 150-200 configuration checks in all.

The latest RHEL7 STIG can be reviewed here:
http://people.redhat.com/swells/scap-security-guide/RHEL/7/output/ssg-rhel7-guide-stig-rhel7-server-upstream.html

And a sample kickstart here:
https://github.com/OpenSCAP/scap-security-guide/blob/master/RHEL/7/kickstart/ssg-rhel7-ospp-ks.cfg#L123

On that sample kickstart, we'd want to change the profile on line 123 to "stig-rhel7-server-upstream". Once the RHEV appliance is loaded on the hardened RHEL image, general use testing should tell us what broke.

(2) RHEV embeds a Postgres database. I posted an HTML edition of the EnterpriseDB Postgres STIG, which can be used to harden the embedded RHEV database:
http://people.redhat.com/swells/edb-stig.html

(3) What JBoss backends does RHEV use? Posted a copy of the JBoss EAP6 STIG to give you an idea on what configurations need to be enabled:
http://people.redhat.com/swells/jboss-eap6-stig.html

(4) What other RHEV subsystems exist? SDN controllers? Getting the components listed out will help identify other applicable SRGs.

DISA STIG content can be found here:
http://iase.disa.mil/stigs/Pages/index.aspx

Hover over "STIGs" --> "STIGs Technologies", then select the area you're looking for. High-level requirements are called SRGs, e.g. "Database Server SRG"

Also, when evaluating what controls to select for a RHV STIG, here are the ones that VMWare ESX is being held to:

http://people.redhat.com/swells/vmare-esx-stig.html

First round of success would be documenting how RHV can be configured to the same requirements as VMWare.

So, the general idea here is that ansible playbook / module should be created (and possibly included in upstream ansible) and RHV-H cockpit / hosted-engine setup should consume it while deploying the RHV appliance. Have we got a BZ to track the ansible work needed on platform so we can depend on it?

(In reply to Sandro Bonazzola from comment #12)
> So, the general idea here is that ansible playbook / module should be
> created (and possibly included in upstream ansible) and RHV-H cockpit /
> hosted-engine setup should consume it while deploying the RHV appliance.
> Have we got a BZ to track the ansible work needed on platform so we can
> depend on it?

I believe that Satellite should manage the VM once it is up to make it complaint, we just want to make sure nothing break after the changes. 
Shawn, is this a correct approach?

I'm closing this as not a bug.
No change is needed within rhvm-appliance. Proposed solutions either uses satellite or ansible. Once a plan is set on how to customize the appliance after it's deployed, please open a documentation bug to cover it.

(In reply to Sandro Bonazzola from comment #14)
> I'm closing this as not a bug.
> No change is needed within rhvm-appliance. Proposed solutions either uses
> satellite or ansible. Once a plan is set on how to customize the appliance
> after it's deployed, please open a documentation bug to cover it.

Please do not close RFEs on RHV.
We would still need confirmation on this approach and then testing that nothing breaks.

I also need to customize an appliance.  One thing I would ask is can a way be developed where the user builds the OS the way they need it to meet their security and other requirements then a script/process adds RHV-M and.or anything else needed then packages it up into the same named RPM file that the hosted engine installer looks for?

This allows the flexibility of PXE booting while still using only image deployments for ease of install using hosted engine answer files.  Somehow though the host engine answer file needs to have a section added for Sat6 data so that the new system can be registered to a Sat6 server (and hopefully do that via the Sat6 server/capsule bootstrap.py system which would ideally also setup realm/IDM registration as part of the process).  Somehow IDM registration either via Sat or some other method (if someone does not have Sat6 but does have IDM) should be made available as well.

I solve the IDM issue for RHV-H Systems by getting Sat6 to build the OS then later either kick off help-hosted deployment via puppet OR if there is a cluster already just tell the existing RHV-M to go and get the host to install RHV-H on it.  This issue with security requirements and STIGS affects HRHEV-H hosts too not just RHV-M.

As I have CFME my total approach is for existing hosts that CFME based policies say need another -H system CFME tells Sat6 (Yes I have that automate code working 100% so Sat6 is the PXE system used by CFME) to build a RHEL7.  Then once built CFME API calls RHV to say to make this RH7 host a RHV-H system added to the cluster.  For new RHV SH setups (like in a new DC with only a capsule there and blank discovered hosts) I am hoping to be able to create my own RPM with a custom aoppliance image so that Sat6 builds the server, installs self-hosted engine RPMs, installs MY appliance image RPM, puts down the answer file(s) via puppet templating using hiera data, then uses puppet exec to fire off the self-hosted install.

Is this something I can help develop/test?

This should solve all security, stig, standards, etc. issues for -H and -M systems and users have a choice of using RH default image or use xxxx toolset to generate their own.

Also this should require no self-hosted SW changes as we are still doing images.

Long term I'd like to really see PXE added to self-hosted and I can help make that happen for a future release (which should really help smooth the Sat6 integration for errata checks and such from within RHV-M UI) but a new "build your own image" toolset should do the same end result with no changes and if possible very minor changes for adding Sat6 registration info to the answer file(s) or other minimal changes so the RHV-M is registered to Sat6  with REX, katello agent, puppet, etc. support.

As a side note I have had great success with 4.0 manual self-hosted installs where Sat6 builds the first hosts then discovers and then KS/builds the manager VM and installs via puppet the RHV-M SW.  It was 100% automatable except for the "press 1 then enter".  If the hosted engine setup could learn to look for a specific named empty file for each part and not need "1 + enter" then I can have it 100% automated in a few days.

See also:
Bug#1474519 - [RFE] Allow PXE boot for SHE VM appliance deployment

(In reply to Yaniv Lavi from comment #13)
> (In reply to Sandro Bonazzola from comment #12)
> > So, the general idea here is that ansible playbook / module should be
> > created (and possibly included in upstream ansible) and RHV-H cockpit /
> > hosted-engine setup should consume it while deploying the RHV appliance.
> > Have we got a BZ to track the ansible work needed on platform so we can
> > depend on it?
> 
> I believe that Satellite should manage the VM once it is up to make it
> complaint, we just want to make sure nothing break after the changes. 
> Shawn, is this a correct approach?

Missed this until Marina's update earlier today.

For the RHEL Anaconda plugins, because we cannot *guarantee* Satellite or Ansible will be present, we embed bash scripts to perform the hardening. Supplementary Ansible scripts are provided [0] but things default to bash as the lowest common denominator.

If RH-V has guaranteed access to Ansible during installation then sure, why not create remediation scripts in Ansible? 

There are generally two simultaneous work streams:
1) Translate the high-level US Gov requirements into SCAP for compliance checking;
2) Author remediation (e.g. Ansible playbook)

If BZs are needed as Sandro mentioned, I would suggest two be created.

[0] https://galaxy.ansible.com/RedHatGov/

Not sure why you would even want to try and get into any OS config at all.  I say that because it isn't just one stig or standard.  There are DOZENS and seemingly infinite combinations.  FedRaMP, FIPS 140-2, etc.  Some clients want more locked down systems compliand to mutiple standards/compliance regulations while others just need to check a box for the security team for best pratice(s).

My suggestion is a "Bring your OWN OS approach.  The idea is a toolset that takes AN OS configured the way the user needs it, has scripts that take data from RHV* answer file(s) and:

1. Performs sanity and basic checks on the provided OS
2. Prepares to register the System to IDM or not
3. Prepares to register the System to Satellite6 or not (including puppet/ancible install as needed)
4. Prepares to register the System to RH CDN or not
5. Installs and configures the RHV-M as needed so that cloud-init can take over when RHV-M gets deployed via self-hosted install using answer files
7. Creates the OVA file and packages that into the same name RPM as the image RPM from Red Hat.

The benefit is that now you no longer need to worry about ever changing security, compliance, corporate, etc. standards.  You are providing the flexibility the users need and do NOT need to complicate the RHV self-hosted or other code base.  There should be minimal (if any) code changes required at all.

This also can be done VERY quickly with a working system done in as little as 1 to 3 months.

To me this would be the quickest path to solve not only this specific STIG issue but ALL security, config, etc. issues and puts the user on the hook not RH for ongoing maintenance via puppet, ancible, etc. based on their needs and what that client has/does not have.

(In reply to Steven Mercurio from comment #21)
> Not sure why you would even want to try and get into any OS config at all. 
> I say that because it isn't just one stig or standard.  There are DOZENS and
> seemingly infinite combinations.  FedRaMP, FIPS 140-2, etc.  Some clients
> want more locked down systems compliand to mutiple standards/compliance
> regulations while others just need to check a box for the security team for
> best pratice(s).
> 
> My suggestion is a "Bring your OWN OS approach.  The idea is a toolset that
> takes AN OS configured the way the user needs it, has scripts that take data
> from RHV* answer file(s) and:
> 
> 1. Performs sanity and basic checks on the provided OS
> 2. Prepares to register the System to IDM or not
> 3. Prepares to register the System to Satellite6 or not (including
> puppet/ancible install as needed)
> 4. Prepares to register the System to RH CDN or not
> 5. Installs and configures the RHV-M as needed so that cloud-init can take
> over when RHV-M gets deployed via self-hosted install using answer files
> 7. Creates the OVA file and packages that into the same name RPM as the
> image RPM from Red Hat.
> 
> The benefit is that now you no longer need to worry about ever changing
> security, compliance, corporate, etc. standards.  You are providing the
> flexibility the users need and do NOT need to complicate the RHV self-hosted
> or other code base.  There should be minimal (if any) code changes required
> at all.
> 
> This also can be done VERY quickly with a working system done in as little
> as 1 to 3 months.
> 
> To me this would be the quickest path to solve not only this specific STIG
> issue but ALL security, config, etc. issues and puts the user on the hook
> not RH for ongoing maintenance via puppet, ancible, etc. based on their
> needs and what that client has/does not have.

On the configuration side, our OpenSCAP project was just described. It's a large collection of possible configuration checks logically grouped into profiles (e.g. DoD STIG, FISMA, PCI...). The project performs compliance scanning against the user-selected profile, and can remediate as well. It's neatly integrated into Anaconda to afford provisioning directly into compliant configurations.

In the scope of security configurations of RHV-H, we can make compliance profiles just like we do RHEL. For sustainment purposes, the OpenSCAP content is managed today by RHT Engineering and delivered via the scap-security-guide RPM.

The scope is too large for RHV 4.2. Moving to 4.3 for consideration once the needed changes are a bit more clear.

(In reply to Steven Mercurio from comment #21)
> 
> 1. Performs sanity and basic checks on the provided OS
> 2. Prepares to register the System to IDM or not
> 3. Prepares to register the System to Satellite6 or not (including
> puppet/ancible install as needed)
> 4. Prepares to register the System to RH CDN or not
> 5. Installs and configures the RHV-M as needed so that cloud-init can take
> over when RHV-M gets deployed via self-hosted install using answer files
> 7. Creates the OVA file and packages that into the same name RPM as the
> image RPM from Red Hat.

2 - 5 can't be done offline and can also happen after the HE is already deployed.
Why is that not good enough?

(In reply to Yaniv Lavi from comment #25)
> (In reply to Steven Mercurio from comment #21)
> > 
> > 1. Performs sanity and basic checks on the provided OS
> > 2. Prepares to register the System to IDM or not
> > 3. Prepares to register the System to Satellite6 or not (including
> > puppet/ancible install as needed)
> > 4. Prepares to register the System to RH CDN or not
> > 5. Installs and configures the RHV-M as needed so that cloud-init can take
> > over when RHV-M gets deployed via self-hosted install using answer files
> > 7. Creates the OVA file and packages that into the same name RPM as the
> > image RPM from Red Hat.
> 
> 2 - 5 can't be done offline and can also happen after the HE is already
> deployed.
> Why is that not good enough?

I can make 2-4 happen when HE deploys the VM and I can do #5 too with a bit of help answering questions as I am just going repeat what RH does to install RHV om the VM prior to packaging it up in an image then into an RPM.

The point is full hands off deplorment using self hosted install procedure which cant be done yet via pxe.

The ONLY reason it cant be done btw 100% hands off automated by SHE is because of the need to press 1 then enter.

If there was a switch that replaced asking you to press 1 then enter with a 2 min pause and retry the step again loop I'd have the SHE install kicked off by sat6 after building the server via PXE installing the RPMs, dropping the filled in answer file (from puppet template).

I have the entire thing hands off now but have to kick off the SHE specifying the answer file interactively only so I can press 1 then enter in RHV 4.0.

Trying to create an image myself that gets packaged into an RPM was just a fall back as I couldnt automate "pressing 1,enter then pause" in some kind of loop.

I can create scrips that takes the user supplied on and does needed steps to create the image packaged into rpm in less than 2 months with just a little help with answering a few questions.

The use case is simple.  A IAAS provider needs to be able to deploy RHVM via SHE but that RHV-M that meets their security and OS standards.  The security standards alone cary far too much and PXE is THE easiest solution.  If PXE isn't an option I can just create something that takes a user supplied OS, adds scripts that run on first boot, and does the RVM-M/ova pkg/RPM pkg same as RH does so user installs their RPM not one from RH.

 "STIG compliance" seems to mean something different to everyone you ask!.  Who has this exception, who has that one, etc.

The "bring your own OS" approach is the ONLY solution that covers ALL these bases and more.

For example FTRIB symlinks /var/tmp to /tmp and adds mount options like nosuid, etc.

Bring your own OS just takes a customer provider OS and

1. Does basic checks needed
2. Installs packages from Sat6 or CDN
3. Installs RHVm, and does basic config
4. Sanitizes OS
5. Creates OVA or instructs user to create OVA
6. Provides/runs script to create RPM from OVA ready to use in automated install.

The other option is just tell me how to install RHVm and I'll create a puppet module to do it.

I'll do ALL THE WORK!  I just need some basic info like the DOC the image RPM creaters use to do base install before certification of OVA/RPM."

can thios be made available so I can see it?

https://access.redhat.com/solutions/3275951

Any thoughts/opinions on the "Bring your own OS" approach?

(In reply to Steven Mercurio from comment #35)
> can thios be made available so I can see it?
> 
> https://access.redhat.com/solutions/3275951
> 
> Any thoughts/opinions on the "Bring your own OS" approach?

Tony, any thoughts on the approached being discussed comment 32? is it something that would be applicable for the group of users that you work with?

It'd be good to re-use https://docs.openstack.org/ansible-hardening/latest/index.html

(In reply to Steven Mercurio from comment #32)
>  "STIG compliance" seems to mean something different to everyone you ask!. 
> Who has this exception, who has that one, etc.
> 
> The "bring your own OS" approach is the ONLY solution that covers ALL these
> bases and more.
> 
> For example FTRIB symlinks /var/tmp to /tmp and adds mount options like
> nosuid, etc.
> 
> Bring your own OS just takes a customer provider OS and
> 
> 1. Does basic checks needed
> 2. Installs packages from Sat6 or CDN
> 3. Installs RHVm, and does basic config
> 4. Sanitizes OS
> 5. Creates OVA or instructs user to create OVA
> 6. Provides/runs script to create RPM from OVA ready to use in automated
> install.
> 
> The other option is just tell me how to install RHVm and I'll create a
> puppet module to do it.

That's doable today. Are you looking for a silent installation?

> 
> I'll do ALL THE WORK!  I just need some basic info like the DOC the image
> RPM creaters use to do base install before certification of OVA/RPM."

For example using Ansible, see https://github.com/rhevm-qe-automation/ovirt-ansible/blob/master/roles/ovirt-engine-setup/README.md

I want to create the OS myself using my standard Sat6 based KS.  I don't want to have dozens of different OD configs all over the place.  Also for clients what "STIGd" system is changes with every person and client you ask.

RH creating a "STIGd" RHVm image is UTTERLY USELESS as I have yet to see any 2 clients implement a "STIG" the same way.

Also I am doing a fully automated SHE install with only RH Sat6, puppet, Hiera 100% automated both in install AND ongoing maintenance like adding RHEL-RHV hosts with ZERO intervention.

My Sat6 also will auto-integrate products like IDM with RHV if IDM is present, etc.  My Sat6 already fully builds IDM servers, Sat6 servers, capsules, and SHE RHV is next.

The RHEL-RHVH systems are already built with Sat6 and managed with puppet for what RHV doesn't do AND puppet will use puppet to run needed CLI cmds on the manager to automate things like so imports, backups, etc.

And all this requires no code changes.  Just Hiera changes.

Ansible can't do anywhere near what I am doing on the level I'm working on.

It's like PCI 10.0.  basically skynet managed by a combination of CF with Sat6.

This and the whole security compliance mess is why I need the Bring your own OS approach.

(In reply to Yaniv Kaul from comment #38)
> (In reply to Steven Mercurio from comment #32)
> >  "STIG compliance" seems to mean something different to everyone you ask!. 
> > Who has this exception, who has that one, etc.
> > 
> > The "bring your own OS" approach is the ONLY solution that covers ALL these
> > bases and more.
> > 
> > For example FTRIB symlinks /var/tmp to /tmp and adds mount options like
> > nosuid, etc.
> > 
> > Bring your own OS just takes a customer provider OS and
> > 
> > 1. Does basic checks needed
> > 2. Installs packages from Sat6 or CDN
> > 3. Installs RHVm, and does basic config
> > 4. Sanitizes OS
> > 5. Creates OVA or instructs user to create OVA
> > 6. Provides/runs script to create RPM from OVA ready to use in automated
> > install.
> > 
> > The other option is just tell me how to install RHVm and I'll create a
> > puppet module to do it.
> 
> That's doable today. Are you looking for a silent installation?
> 


What I'd like to do is either be able to pke and install a SHE OR be able to build my own RHV-M image then turn that into an RPM so that I can use my RHEV-M image RPM instead of the one provided by RH.

Once I learn how to do it and get a iamge created based on my existing OS standard then I will automate everything from the kickstart/base OS setup forward so then anyone just need to build a base OS to their "standard" and run the setup to do everything else.  Once I have this working I'll upload it to upstream so someone can add any needed sanity checks to ensure the client provided OS is supportable before making an RHV-M image with it.

> > 
> > I'll do ALL THE WORK!  I just need some basic info like the DOC the image
> > RPM creaters use to do base install before certification of OVA/RPM."
> 
> For example using Ansible, see
> https://github.com/rhevm-qe-automation/ovirt-ansible/blob/master/roles/ovirt-
> engine-setup/README.md

The above link does not work and I have not heard of any ansible items that will take a user supplied OS and install RHV-M and do what is needed to create an image that takes an answer file just like the RH image does for an unattended SHE install.

Moran, why is this on external? which team is doing this?

not sure why it was addressed as external, for sorting this one out the expectation is it will be joint effort of node and infra teams.

moving it to node, since Ryan is leading this effort.

In partnership with the BU (Ryan Barry and team), US Public Sector has began work on this. 

First step is to evaluate which NIST 800-53 controls are applicable to RHV-H and RHV-M. Currently mappings are done through FISMA Low and available here:

RHV-H:
http://atopathways.redhatgov.io/product-documents/rhvh/

RHV-M:
http://atopathways.redhatgov.io/product-documents/rhvm/

OpenControl-formatted code behind those sites:
https://github.com/ComplianceAsCode/redhat

Will be updating through FISMA Moderate until November.

Upstream update:

Draft content for RHVH was released today:
https://github.com/ComplianceAsCode/content/releases/tag/v0.1.43

Hi Yuval, I am working on verification of this bug and ran into two potential issue:
1) This is probably a minor things, but 4.3 beta docs about hosted-engine deployment mentions all of the installation steps, but not the prompting for OpenSCAP profile: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3-beta/html-single/self-hosted_engine_guide/index
Should we create a doc bug about this?
2) This is maybe a bigger problem. While hosted-engine --deploy prompts the user for application of OpenSCAP profile, there is no such option in installation via cockpit, which is (according to the docs) the recommended way. I think both of these points should result in a new bug, what do you think?

(In reply to Jan Zmeskal from comment #61)
> Hi Yuval, I am working on verification of this bug and ran into two
> potential issue:
> 1) This is probably a minor things, but 4.3 beta docs about hosted-engine
> deployment mentions all of the installation steps, but not the prompting for
> OpenSCAP profile:
> https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3-
> beta/html-single/self-hosted_engine_guide/index
> Should we create a doc bug about this?

I think so, yes.

> 2) This is maybe a bigger problem. While hosted-engine --deploy prompts the
> user for application of OpenSCAP profile, there is no such option in
> installation via cockpit, which is (according to the docs) the recommended
> way. I think both of these points should result in a new bug, what do you
> think?

I think you're right here also :)

These are two bugs I created as a follow-up to comment #62:
https://bugzilla.redhat.com/show_bug.cgi?id=1695610
https://bugzilla.redhat.com/show_bug.cgi?id=1695613

Hello Yuval, this is what I did so far regarding verification:

1. yum install ovirt-hosted-engine-setup
2. yum install rhvm-appliance
3. hosted-engine --deploy
3.1. Verify that prompt "Do you want to apply a default OpenSCAP security profile" is displayed (with default being No)
3.2 Choose Yes
4. Finish hosted-engine installation
5. Check installation log and see if Ansible task [ovirt.hosted_engine_setup : Set default OpenSCAP profile] has been executed and its status is changed (see line 2138 in the attachment "hosted-engine setup log from RHEL based host")

After that, I copied https://github.com/ComplianceAsCode/content/releases/download/v0.1.43/scap-security-guide-0.1.43.zip to the hosted-engine VM and try to do the scan like this:
oscap xccdf eval --profile xccdf_org.ssgproject.content_profile_rhvh-vpp --report ~/scan-report.html scap-security-guide-0.1.43/ssg-rhv4-ds.xml

However, during the scan, I got "notapplicable" as a result for all the check. Only after that I noticed the last line in comment #53 and realized I was verifying on RHEL-based host.

So, I have a question: Is that feature supported only for RHVH-based host? If this is the case, I think we must at the very least inform the user about when the hosted-engine --deploy prompt is displayed. Also, does that mean that the Ansible task mentioned in step #5 took no action?

Jan

(In reply to Jan Zmeskal from comment #64)
> Hello Yuval, this is what I did so far regarding verification:
> 
> 1. yum install ovirt-hosted-engine-setup
> 2. yum install rhvm-appliance
> 3. hosted-engine --deploy
> 3.1. Verify that prompt "Do you want to apply a default OpenSCAP security
> profile" is displayed (with default being No)
> 3.2 Choose Yes
> 4. Finish hosted-engine installation
> 5. Check installation log and see if Ansible task [ovirt.hosted_engine_setup
> : Set default OpenSCAP profile] has been executed and its status is changed
> (see line 2138 in the attachment "hosted-engine setup log from RHEL based
> host")
> 
> After that, I copied
> https://github.com/ComplianceAsCode/content/releases/download/v0.1.43/scap-
> security-guide-0.1.43.zip to the hosted-engine VM and try to do the scan
> like this:
> oscap xccdf eval --profile xccdf_org.ssgproject.content_profile_rhvh-vpp
> --report ~/scan-report.html scap-security-guide-0.1.43/ssg-rhv4-ds.xml
> 
> However, during the scan, I got "notapplicable" as a result for all the
> check. Only after that I noticed the last line in comment #53 and realized I
> was verifying on RHEL-based host.
> 
> So, I have a question: Is that feature supported only for RHVH-based host?
> If this is the case, I think we must at the very least inform the user about
> when the hosted-engine --deploy prompt is displayed. Also, does that mean
> that the Ansible task mentioned in step #5 took no action?
> 
> Jan

For RHEL-based hosts the expectation is to use the RHEL data stream (e.g. ssg-rhel7-ds.xml)

e.g. the "VPP - Protection Profile for Virtualization v. 1.0 for Red Hat Enterprise Linux Hypervisor (RHELH)", or xccdf_org.ssgproject.content_profile_rhelh-vpp

oscap xccdf eval --profile xccdf_org.ssgproject.content_profile_rhelh-vpp
--report ~/scan-report.html scap-security-guide-0.1.43/ssg-rhel7-ds.xml

Jan, for the rhvm-appliance you should use the normal rhel disa stig profile as follows:

oscap xccdf eval --profile xccdf_org.ssgproject.content_profile_stig-rhel7-disa --report ~/scan-report.html /usr/share/xml/scap/ssg/content/ssg-rhel7-ds.xml

For RHVH you can use the profile that was mentioned in the bug, but keep in mind please that the support for RHVH was developed with the normal rhel disa stig as above.  If you would like to test the new RHVH profile, then you need to follow  Shawn's comment:

oscap xccdf eval --profile xccdf_org.ssgproject.content_profile_rhvh-vpp --report ~/scan-report.html [path-to]/scap-security-guide-0.1.43/ssg-rhv4-ds.xml

If you run a scan with this new profile please share the results, as I didn't try it out just yet :)

Hi Shawn and Yuval, after your comments, I realized I still have some questions:
1) The oscap command should be run in the *hosted-engine* VM and not on its underlying *host*, am I right?
2) You both reference different data stream. Shawn says I should run  data stream from scap-security-guide-0.1.43 in comment #66, while Yuval says I should run it from /usr/share/xml/scap/ssg/content/ (this is what is already present on the system and the version of that package for current rhvm-appliance is scap-security-guide-0.1.40-12. Which of those should I use? Or rather, what's the difference between them?

Anyway, what I did so far is that I ran this inside the hosted-engine VM:
oscap xccdf eval --profile xccdf_org.ssgproject.content_profile_rhelh-vpp --report /root/scan-report.html scap-security-guide-0.1.43/ssg-rhel7-ds.xml

The results can be found in attachment "Scan report 1". As you can see, almost half of the check failed.

PS: I was using RHEL-based host for hosted-engine VM deployment.

Jan

(In reply to Jan Zmeskal from comment #68)
> Hi Shawn and Yuval, after your comments, I realized I still have some
> questions:
> 1) The oscap command should be run in the *hosted-engine* VM and not on its
> underlying *host*, am I right?

For RHEL-based hypervisors, use the rhelh-vpp profile from the RHEL 7 data stream. E.g:

$ oscap xccdf eval --profile xccdf_org.ssgproject.content_profile_rhelh-vpp \
--report /root/scan-report.html \
scap-security-guide-0.1.43/ssg-rhel7-ds.xml

For RHVH-based hypervisors, use the rhvh-vpp profile from teh RHVH data stream. E.g.:

$ oscap xccdf eval --profile xccdf_org.ssgproject.content_profile_rhelh-vpp \
--report ~/scan-report.html scap-security-guide-0.1.43/ssg-rhel7-ds.xml

We haven't made content for RHVM (as an application) yet. Customers will be expected to follow the operating system STIG, and a yet-to-be-developed hardening guide for RHVM itself. In the mean time, suggest RHVM systems be hardened/scanned against the RHEL baseline, ideally the OSPP profile.



> 2) You both reference different data stream. Shawn says I should run  data
> stream from scap-security-guide-0.1.43 in comment #66, while Yuval says I
> should run it from /usr/share/xml/scap/ssg/content/ (this is what is already
> present on the system and the version of that package for current
> rhvm-appliance is scap-security-guide-0.1.40-12. Which of those should I
> use? Or rather, what's the difference between them?

Upstream vs downstream.

Content ships in RHEL via the scap-security-guide package. Currently shipping content does not have the virtualization profiles (they're scheduled for inclusion in RHEL 7.7).

So for now, to ensure what we *will* ship is good,

> 
> Anyway, what I did so far is that I ran this inside the hosted-engine VM:
> oscap xccdf eval --profile xccdf_org.ssgproject.content_profile_rhelh-vpp
> --report /root/scan-report.html scap-security-guide-0.1.43/ssg-rhel7-ds.xml
> 
> The results can be found in attachment "Scan report 1". As you can see,
> almost half of the check failed.
> 
> PS: I was using RHEL-based host for hosted-engine VM deployment.
> 
> Jan

Hello Yuval and Shawn,

I ran this command on RHEL-based host *after* running hosted-engine --deploy and applying default OpenSCAP security profile:
oscap xccdf eval --profile xccdf_org.ssgproject.content_profile_rhelh-vpp --report /root/scan-report.html scap-security-guide-0.1.43/ssg-rhel7-ds.xml

More info:
OS: RHEL7.6
openscap-1.2.17-2.el7.x86_64
ovirt-ansible-hosted-engine-setup-1.0.14-1.el7ev.noarch
ovirt-hosted-engine-setup-2.3.7-1.el7ev.noarch

Please check the attachment "Scan report 2". You can see there that 120 checks have failed. So it seems that the security profile was not actually applied. Do we move this to ASSIGNED?

Jan

(In reply to Jan Zmeskal from comment #71)
> Hello Yuval and Shawn,
> 
> I ran this command on RHEL-based host *after* running hosted-engine --deploy
> and applying default OpenSCAP security profile:
> oscap xccdf eval --profile xccdf_org.ssgproject.content_profile_rhelh-vpp
> --report /root/scan-report.html scap-security-guide-0.1.43/ssg-rhel7-ds.xml
> 
> More info:
> OS: RHEL7.6
> openscap-1.2.17-2.el7.x86_64
> ovirt-ansible-hosted-engine-setup-1.0.14-1.el7ev.noarch
> ovirt-hosted-engine-setup-2.3.7-1.el7ev.noarch
> 
> Please check the attachment "Scan report 2". You can see there that 120
> checks have failed. So it seems that the security profile was not actually
> applied. Do we move this to ASSIGNED?
> 

If remediation is desired, add the "--remediate" flag. Remember to use sudo since some remediations require root. For example:

sudo oscap xccdf eval --profile xccdf_org.ssgproject.content_profile_rhelh-vpp \
-- remediate \
--report /root/scan-report.html \
scap-security-guide-0.1.43/ssg-rhel7-ds.xml

Bugs with the content itself will/should be addressed by the Security Automation team (Marek Haicman, or "scap-security-guide" RPM in RHEL channel).

Believe this BZ is to ensure RHV still works once hardened (e.g. the operating system is placed into FIPS mode, audit is fully enabled, RHV installs with the restricted UMASK settings, etc).

p.s. Jan, since I believe we have not met before, I'm one of the SCAP content developers. Working to ensure the pass/fail content works, remediations actually remediate, overall workflow is sane. My colleague Gabe Alford and I are building out the RHV profiles.

Hi Shawn, the comment #73 by you sheds a way more light on the situation for me, thank you! I think I needed to read such explanation in the beginning. Excuse my uncanny questions, this is my first contact with OpenSCAP. 

> Believe this BZ is to ensure RHV still works once hardened

Does that mean that the proper verification of this BZ goes like this?
1) Scan host with OpenSCAP
2) Remedy the issue automatically using oscap with remediate flag
3) Make sure that the oscap scan passes 
4) Install hosted-engine on that host and make sure the installation succeeds
5) Verify basic functionality of the installed engine

Do I get it right now?

Jan

Hi Shawn,

I tried to follow verification steps outlined in comment #74 (please, confirm if they make sense to you). At step #2, I ran oscap with --remediate flag. Indeed most issues have been fixed, but two of them still failed with four reporting an error. Please take a look at "Scan report 3" attachment. Do you think this is a blocker for verification that the host has not been hardened completely? 

Jan

Hello Yuval,

your last comment is on the other hand quite clear and on the other it confuses the hell out out of me since it seems to be in contradiction with some of the thigh (I thought) we established.
Please, answer as specifically as you can to my questions:
1. When you are talking about RHV-H, are you talking about RHV-H operation system OR about RHV host (a machine) that can have either RHEL OS or RHV-H OS installed? From my previous conversation with you and Shawn I got an impression that it does not matter what OS the host is running.
2. What about scap-security-guide-0.1.43? In comment #68 I specifically asked if I should use this upstream version or the one released in upstream. In comment #70, Shawn replied that I should use data streams from scap-security-guide-0.1.43. However, you reference those that already exist on the system.
3. You posted steps how to verify on RHV-H and on RHV-M. Since this bug is about RHV-M, do we need to bother with the RHV-H part at all?

Jan

(In reply to Jan Zmeskal from comment #78)
> Hello Yuval,
> 
> your last comment is on the other hand quite clear and on the other it
> confuses the hell out out of me since it seems to be in contradiction with
> some of the thigh (I thought) we established.

Sorry :)

> Please, answer as specifically as you can to my questions:
> 1. When you are talking about RHV-H, are you talking about RHV-H operation
> system OR about RHV host (a machine) that can have either RHEL OS or RHV-H
> OS installed? From my previous conversation with you and Shawn I got an
> impression that it does not matter what OS the host is running.

I am talking about RHV-H only for now.

> 2. What about scap-security-guide-0.1.43? In comment #68 I specifically
> asked if I should use this upstream version or the one released in upstream.
> In comment #70, Shawn replied that I should use data streams from
> scap-security-guide-0.1.43. However, you reference those that already exist
> on the system.

Right, eventually we should use those profiles, but for now, since those profiles are new, they are not shipped yet in RHV-H nor RHV-M (not even in RHEL yet) so you can't test them.
Well, you can, but if you selected the regular rhel stig profile during installation, and then scanned with the new profile, you'll get wrong results.

> 3. You posted steps how to verify on RHV-H and on RHV-M. Since this bug is
> about RHV-M, do we need to bother with the RHV-H part at all?

We should, but I'm not sure if this bug covers RHV-H (bug 1654253 looks more relevant...)

Hi Yuval, so I followed your advice and did this for verification:

1. I had a host with RHV-H OS. I ran there:
1.1 yum install ovirt-hosted-engine-setup
1.2 yum install rhvm-appliance
2. Then I ran hosted-engine --deploy
2.1 I verified that prompt "Do you want to apply a default OpenSCAP security profile" is displayed (with default being No) and chose Yes
2.2 I waited for the installation to finish
3. I SSHed to the RHV-M appliance and ran the following:
oscap xccdf eval --profile xccdf_org.ssgproject.content_profile_stig-rhel7-disa --report /mnt/jzmeskal/scan-report-5.html /usr/share/xml/scap/ssg/content/ssg-rhel7-ds.xml

However, the result is not good. 119 tests passed but also 96 have failed. Please see the attachment "Scan report 5" for details.
I also attached logs from hosted-engine deployment, please see attachment "Hosted-engine deploy logs" for details.

It seems we must fail this verification, since this is quite far off from 90% pass result. 

Details of OS:
cat /etc/os-release 
NAME="Red Hat Enterprise Linux"
VERSION="7.6"
VERSION_ID="7.6"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Red Hat Virtualization Host"
VARIANT_ID="ovirt-node"
PRETTY_NAME="Red Hat Virtualization Host 4.3.0 (el7.6)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.6:beta:hypervisor"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

# FIXME
REDHAT_BUGZILLA_PRODUCT="Red Hat Virtualization"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.6
REDHAT_SUPPORT_PRODUCT="Red Hat Virtualization"
REDHAT_SUPPORT_PRODUCT_VERSION=7.6

Packages:
rpm -qa | egrep "hosted-engine|appliance"
ovirt-hosted-engine-ha-2.3.1-1.el7ev.noarch
ovirt-hosted-engine-setup-2.3.7-1.el7ev.noarch
rhvm-appliance-4.3-20190404.1.el7.x86_64
ovirt-ansible-hosted-engine-setup-1.0.15-1.el7ev.noarch

Hello Yuval,

> Jan can you please attach /var/log from the HE VM ?

I cannot attach the whole /var/log since it's too big. But I attached all the files from /var/log/ovirt* directories in attachment "Logs from RHEV-M appliance". Hope this helps.

Thanks Jan, can you please attach cloud-init.log and messages also ?

Moving back to ASSIGNED based on results of verification I posted in comment #80.

Yuval anything else to be merged in order to move this to modified?

Lets wait for the OST suite to make sure everything goes ok

System checked against profile xccdf_org.ssgproject.content_profile_stig-rhel7-disa in /usr/share/xml/scap/ssg/content/ssg-rhel7-ds.xml.

Result was:
196 passed
6 failed
26 other

6 failed tests are:
- Disable SSH Root Login
- Set Default firewalld Zone for Incoming Packets
- Set Boot Loader Password in grub2
- Install Smart Card Packages For Multifactor Authentication
- Ensure Users Re-Authenticate for Privilege Escalation - sudo NOPASSWD
- Install McAfee Virus Scanning Software

Creating remediation playbook from the result can fix (remediate system) only test "Disable SSH Root Login" which is correctly overrided during hosted-engine deploy (I wanted to not disable SSH access to engine) and I think that remediation content created from openscap tool is not in scope of this RFE.

But still I'm concerned from security perspective about the failed rules. Should we track these rules somewhere (as BZ or as known issue)?
Mainly for these two? 
- Set Default firewalld Zone for Incoming Packets
- Ensure Users Re-Authenticate for Privilege Escalation - sudo NOPASSWD

I will verify this bug because remediation was applied during hosted-engine-setup:
[ INFO ] TASK [ovirt.hosted_engine_setup : Initialize OpenSCAP variables]
[ INFO ] ok: [localhost -> <ENGINE.FQDN>]
[ INFO ] TASK [ovirt.hosted_engine_setup : Set OpenSCAP datastream path]
[ INFO ] ok: [localhost -> <ENGINE.FQDN>]
[ INFO ] TASK [ovirt.hosted_engine_setup : Verify OpenSCAP datastream]
[ INFO ] ok: [localhost -> <ENGINE.FQDN>]
[ INFO ] TASK [ovirt.hosted_engine_setup : Set default OpenSCAP profile]
[ INFO ] changed: [localhost -> <ENGINE.FQDN>]
[ INFO ] TASK [ovirt.hosted_engine_setup : Apply OpenSCAP profile]
[ INFO ] changed: [localhost -> <ENGINE.FQDN>]
[ INFO ] TASK [ovirt.hosted_engine_setup : Reset PermitRootLogin for sshd]
[ INFO ] changed: [localhost -> <ENGINE.FQDN>]
[ INFO ] TASK [ovirt.hosted_engine_setup : Reboot the engine VM to ensure that FIPS is enabled]
[ INFO ] changed: [localhost -> <ENGINE.FQDN>]
[ INFO ] TASK [ovirt.hosted_engine_setup : Check if FIPS is enabled]
[ INFO ] changed: [localhost -> <ENGINE.FQDN>]

Verified with 
scap-security-guide-0.1.43-13.el7.noarch
openscap-1.2.17-4.el7.x86_64

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:4232