Hi Evan,
I am not sure we want to support customized appliances, however we may want to request persisting customized after-deployment appliances, if it is not allowed yet.


In the case we are working together, I was thinking more about having a pre-made appliance that meets STIG standards. But this can be challenging, since there are multiple standards and one appliance may not work for all. 


In this case, we may want to allow also non-appliance deployments of HE VM, so that the customers would be able customizing their VMs the way they need from the very beginning. And it will be easier to support, rather then a customize appliance.


Let PM discuss this with the public sector team and weigh in on this.

DISA has Security Requirement Guides, or SRGs, that reflect DoD-mandated controls. They're organized by layers of the infrastructure -- e.g. network devices, operating systems, application servers, database servers, applications, so forth.

DoD Secure Technical Implementation Guides (DoD STIGS) reflect the vendor-specific configurations of a product to a DISA SRG. For example, the RHEL7 STIG reflects how to configure RHEL7 against the DISA Operating System SRG.

If we were to undertake building a secure appliance, we'd have to decompose RHEV into relevant subsystems (e.g. OS, app servers, app) and review the applicable DISA SRGs. We'd then have to interpret them for our components. For example, reading the DISA Database SRG and interpreting what components are applicable to embeded RHEV Postgres database servers. It's a big undertaking.

So, to make this consumable... ideas on how to start chunking this into smaller pieces:

(1) First we start with the operating system. For that, we'd want to apply the RHEL7 STIG and load the RHEV appliance. Thankfully we have published RHEL kickstarts for this. The RHEL7 STIG profile enables FIPS mode, sets various kernel and sysctl options, turns on verbose auditing. About 150-200 configuration checks in all.

The latest RHEL7 STIG can be reviewed here:
http://people.redhat.com/swells/scap-security-guide/RHEL/7/output/ssg-rhel7-guide-stig-rhel7-server-upstream.html

And a sample kickstart here:
https://github.com/OpenSCAP/scap-security-guide/blob/master/RHEL/7/kickstart/ssg-rhel7-ospp-ks.cfg#L123

On that sample kickstart, we'd want to change the profile on line 123 to "stig-rhel7-server-upstream". Once the RHEV appliance is loaded on the hardened RHEL image, general use testing should tell us what broke.

(2) RHEV embeds a Postgres database. I posted an HTML edition of the EnterpriseDB Postgres STIG, which can be used to harden the embedded RHEV database:
http://people.redhat.com/swells/edb-stig.html

(3) What JBoss backends does RHEV use? Posted a copy of the JBoss EAP6 STIG to give you an idea on what configurations need to be enabled:
http://people.redhat.com/swells/jboss-eap6-stig.html

(4) What other RHEV subsystems exist? SDN controllers? Getting the components listed out will help identify other applicable SRGs.

DISA STIG content can be found here:
http://iase.disa.mil/stigs/Pages/index.aspx

Hover over "STIGs" --> "STIGs Technologies", then select the area you're looking for. High-level requirements are called SRGs, e.g. "Database Server SRG"

Also, when evaluating what controls to select for a RHV STIG, here are the ones that VMWare ESX is being held to:

http://people.redhat.com/swells/vmare-esx-stig.html

First round of success would be documenting how RHV can be configured to the same requirements as VMWare.

So, the general idea here is that ansible playbook / module should be created (and possibly included in upstream ansible) and RHV-H cockpit / hosted-engine setup should consume it while deploying the RHV appliance. Have we got a BZ to track the ansible work needed on platform so we can depend on it?

(In reply to Sandro Bonazzola from comment #12)
> So, the general idea here is that ansible playbook / module should be
> created (and possibly included in upstream ansible) and RHV-H cockpit /
> hosted-engine setup should consume it while deploying the RHV appliance.
> Have we got a BZ to track the ansible work needed on platform so we can
> depend on it?

I believe that Satellite should manage the VM once it is up to make it complaint, we just want to make sure nothing break after the changes. 
Shawn, is this a correct approach?

I'm closing this as not a bug.
No change is needed within rhvm-appliance. Proposed solutions either uses satellite or ansible. Once a plan is set on how to customize the appliance after it's deployed, please open a documentation bug to cover it.

(In reply to Sandro Bonazzola from comment #14)
> I'm closing this as not a bug.
> No change is needed within rhvm-appliance. Proposed solutions either uses
> satellite or ansible. Once a plan is set on how to customize the appliance
> after it's deployed, please open a documentation bug to cover it.

Please do not close RFEs on RHV.
We would still need confirmation on this approach and then testing that nothing breaks.

I also need to customize an appliance.  One thing I would ask is can a way be developed where the user builds the OS the way they need it to meet their security and other requirements then a script/process adds RHV-M and.or anything else needed then packages it up into the same named RPM file that the hosted engine installer looks for?

This allows the flexibility of PXE booting while still using only image deployments for ease of install using hosted engine answer files.  Somehow though the host engine answer file needs to have a section added for Sat6 data so that the new system can be registered to a Sat6 server (and hopefully do that via the Sat6 server/capsule bootstrap.py system which would ideally also setup realm/IDM registration as part of the process).  Somehow IDM registration either via Sat or some other method (if someone does not have Sat6 but does have IDM) should be made available as well.

I solve the IDM issue for RHV-H Systems by getting Sat6 to build the OS then later either kick off help-hosted deployment via puppet OR if there is a cluster already just tell the existing RHV-M to go and get the host to install RHV-H on it.  This issue with security requirements and STIGS affects HRHEV-H hosts too not just RHV-M.

As I have CFME my total approach is for existing hosts that CFME based policies say need another -H system CFME tells Sat6 (Yes I have that automate code working 100% so Sat6 is the PXE system used by CFME) to build a RHEL7.  Then once built CFME API calls RHV to say to make this RH7 host a RHV-H system added to the cluster.  For new RHV SH setups (like in a new DC with only a capsule there and blank discovered hosts) I am hoping to be able to create my own RPM with a custom aoppliance image so that Sat6 builds the server, installs self-hosted engine RPMs, installs MY appliance image RPM, puts down the answer file(s) via puppet templating using hiera data, then uses puppet exec to fire off the self-hosted install.

Is this something I can help develop/test?

This should solve all security, stig, standards, etc. issues for -H and -M systems and users have a choice of using RH default image or use xxxx toolset to generate their own.

Also this should require no self-hosted SW changes as we are still doing images.

Long term I'd like to really see PXE added to self-hosted and I can help make that happen for a future release (which should really help smooth the Sat6 integration for errata checks and such from within RHV-M UI) but a new "build your own image" toolset should do the same end result with no changes and if possible very minor changes for adding Sat6 registration info to the answer file(s) or other minimal changes so the RHV-M is registered to Sat6  with REX, katello agent, puppet, etc. support.

As a side note I have had great success with 4.0 manual self-hosted installs where Sat6 builds the first hosts then discovers and then KS/builds the manager VM and installs via puppet the RHV-M SW.  It was 100% automatable except for the "press 1 then enter".  If the hosted engine setup could learn to look for a specific named empty file for each part and not need "1 + enter" then I can have it 100% automated in a few days.

See also:
Bug#1474519 - [RFE] Allow PXE boot for SHE VM appliance deployment

(In reply to Yaniv Lavi from comment #13)
> (In reply to Sandro Bonazzola from comment #12)
> > So, the general idea here is that ansible playbook / module should be
> > created (and possibly included in upstream ansible) and RHV-H cockpit /
> > hosted-engine setup should consume it while deploying the RHV appliance.
> > Have we got a BZ to track the ansible work needed on platform so we can
> > depend on it?
> 
> I believe that Satellite should manage the VM once it is up to make it
> complaint, we just want to make sure nothing break after the changes. 
> Shawn, is this a correct approach?

Missed this until Marina's update earlier today.

For the RHEL Anaconda plugins, because we cannot *guarantee* Satellite or Ansible will be present, we embed bash scripts to perform the hardening. Supplementary Ansible scripts are provided [0] but things default to bash as the lowest common denominator.

If RH-V has guaranteed access to Ansible during installation then sure, why not create remediation scripts in Ansible? 

There are generally two simultaneous work streams:
1) Translate the high-level US Gov requirements into SCAP for compliance checking;
2) Author remediation (e.g. Ansible playbook)

If BZs are needed as Sandro mentioned, I would suggest two be created.

[0] https://galaxy.ansible.com/RedHatGov/

Not sure why you would even want to try and get into any OS config at all.  I say that because it isn't just one stig or standard.  There are DOZENS and seemingly infinite combinations.  FedRaMP, FIPS 140-2, etc.  Some clients want more locked down systems compliand to mutiple standards/compliance regulations while others just need to check a box for the security team for best pratice(s).

My suggestion is a "Bring your OWN OS approach.  The idea is a toolset that takes AN OS configured the way the user needs it, has scripts that take data from RHV* answer file(s) and:

1. Performs sanity and basic checks on the provided OS
2. Prepares to register the System to IDM or not
3. Prepares to register the System to Satellite6 or not (including puppet/ancible install as needed)
4. Prepares to register the System to RH CDN or not
5. Installs and configures the RHV-M as needed so that cloud-init can take over when RHV-M gets deployed via self-hosted install using answer files
7. Creates the OVA file and packages that into the same name RPM as the image RPM from Red Hat.

The benefit is that now you no longer need to worry about ever changing security, compliance, corporate, etc. standards.  You are providing the flexibility the users need and do NOT need to complicate the RHV self-hosted or other code base.  There should be minimal (if any) code changes required at all.

This also can be done VERY quickly with a working system done in as little as 1 to 3 months.

To me this would be the quickest path to solve not only this specific STIG issue but ALL security, config, etc. issues and puts the user on the hook not RH for ongoing maintenance via puppet, ancible, etc. based on their needs and what that client has/does not have.

(In reply to Steven Mercurio from comment #21)
> Not sure why you would even want to try and get into any OS config at all. 
> I say that because it isn't just one stig or standard.  There are DOZENS and
> seemingly infinite combinations.  FedRaMP, FIPS 140-2, etc.  Some clients
> want more locked down systems compliand to mutiple standards/compliance
> regulations while others just need to check a box for the security team for
> best pratice(s).
> 
> My suggestion is a "Bring your OWN OS approach.  The idea is a toolset that
> takes AN OS configured the way the user needs it, has scripts that take data
> from RHV* answer file(s) and:
> 
> 1. Performs sanity and basic checks on the provided OS
> 2. Prepares to register the System to IDM or not
> 3. Prepares to register the System to Satellite6 or not (including
> puppet/ancible install as needed)
> 4. Prepares to register the System to RH CDN or not
> 5. Installs and configures the RHV-M as needed so that cloud-init can take
> over when RHV-M gets deployed via self-hosted install using answer files
> 7. Creates the OVA file and packages that into the same name RPM as the
> image RPM from Red Hat.
> 
> The benefit is that now you no longer need to worry about ever changing
> security, compliance, corporate, etc. standards.  You are providing the
> flexibility the users need and do NOT need to complicate the RHV self-hosted
> or other code base.  There should be minimal (if any) code changes required
> at all.
> 
> This also can be done VERY quickly with a working system done in as little
> as 1 to 3 months.
> 
> To me this would be the quickest path to solve not only this specific STIG
> issue but ALL security, config, etc. issues and puts the user on the hook
> not RH for ongoing maintenance via puppet, ancible, etc. based on their
> needs and what that client has/does not have.

On the configuration side, our OpenSCAP project was just described. It's a large collection of possible configuration checks logically grouped into profiles (e.g. DoD STIG, FISMA, PCI...). The project performs compliance scanning against the user-selected profile, and can remediate as well. It's neatly integrated into Anaconda to afford provisioning directly into compliant configurations.

In the scope of security configurations of RHV-H, we can make compliance profiles just like we do RHEL. For sustainment purposes, the OpenSCAP content is managed today by RHT Engineering and delivered via the scap-security-guide RPM.

The scope is too large for RHV 4.2. Moving to 4.3 for consideration once the needed changes are a bit more clear.

(In reply to Steven Mercurio from comment #21)
> 
> 1. Performs sanity and basic checks on the provided OS
> 2. Prepares to register the System to IDM or not
> 3. Prepares to register the System to Satellite6 or not (including
> puppet/ancible install as needed)
> 4. Prepares to register the System to RH CDN or not
> 5. Installs and configures the RHV-M as needed so that cloud-init can take
> over when RHV-M gets deployed via self-hosted install using answer files
> 7. Creates the OVA file and packages that into the same name RPM as the
> image RPM from Red Hat.

2 - 5 can't be done offline and can also happen after the HE is already deployed.
Why is that not good enough?

(In reply to Yaniv Lavi from comment #25)
> (In reply to Steven Mercurio from comment #21)
> > 
> > 1. Performs sanity and basic checks on the provided OS
> > 2. Prepares to register the System to IDM or not
> > 3. Prepares to register the System to Satellite6 or not (including
> > puppet/ancible install as needed)
> > 4. Prepares to register the System to RH CDN or not
> > 5. Installs and configures the RHV-M as needed so that cloud-init can take
> > over when RHV-M gets deployed via self-hosted install using answer files
> > 7. Creates the OVA file and packages that into the same name RPM as the
> > image RPM from Red Hat.
> 
> 2 - 5 can't be done offline and can also happen after the HE is already
> deployed.
> Why is that not good enough?

I can make 2-4 happen when HE deploys the VM and I can do #5 too with a bit of help answering questions as I am just going repeat what RH does to install RHV om the VM prior to packaging it up in an image then into an RPM.

The point is full hands off deplorment using self hosted install procedure which cant be done yet via pxe.

The ONLY reason it cant be done btw 100% hands off automated by SHE is because of the need to press 1 then enter.

If there was a switch that replaced asking you to press 1 then enter with a 2 min pause and retry the step again loop I'd have the SHE install kicked off by sat6 after building the server via PXE installing the RPMs, dropping the filled in answer file (from puppet template).

I have the entire thing hands off now but have to kick off the SHE specifying the answer file interactively only so I can press 1 then enter in RHV 4.0.

Trying to create an image myself that gets packaged into an RPM was just a fall back as I couldnt automate "pressing 1,enter then pause" in some kind of loop.

I can create scrips that takes the user supplied on and does needed steps to create the image packaged into rpm in less than 2 months with just a little help with answering a few questions.

The use case is simple.  A IAAS provider needs to be able to deploy RHVM via SHE but that RHV-M that meets their security and OS standards.  The security standards alone cary far too much and PXE is THE easiest solution.  If PXE isn't an option I can just create something that takes a user supplied OS, adds scripts that run on first boot, and does the RVM-M/ova pkg/RPM pkg same as RH does so user installs their RPM not one from RH.

 "STIG compliance" seems to mean something different to everyone you ask!.  Who has this exception, who has that one, etc.

The "bring your own OS" approach is the ONLY solution that covers ALL these bases and more.

For example FTRIB symlinks /var/tmp to /tmp and adds mount options like nosuid, etc.

Bring your own OS just takes a customer provider OS and

1. Does basic checks needed
2. Installs packages from Sat6 or CDN
3. Installs RHVm, and does basic config
4. Sanitizes OS
5. Creates OVA or instructs user to create OVA
6. Provides/runs script to create RPM from OVA ready to use in automated install.

The other option is just tell me how to install RHVm and I'll create a puppet module to do it.

I'll do ALL THE WORK!  I just need some basic info like the DOC the image RPM creaters use to do base install before certification of OVA/RPM."

can thios be made available so I can see it?

https://access.redhat.com/solutions/3275951

Any thoughts/opinions on the "Bring your own OS" approach?

(In reply to Steven Mercurio from comment #35)
> can thios be made available so I can see it?
> 
> https://access.redhat.com/solutions/3275951
> 
> Any thoughts/opinions on the "Bring your own OS" approach?

Tony, any thoughts on the approached being discussed comment 32? is it something that would be applicable for the group of users that you work with?

It'd be good to re-use https://docs.openstack.org/ansible-hardening/latest/index.html

(In reply to Steven Mercurio from comment #32)
>  "STIG compliance" seems to mean something different to everyone you ask!. 
> Who has this exception, who has that one, etc.
> 
> The "bring your own OS" approach is the ONLY solution that covers ALL these
> bases and more.
> 
> For example FTRIB symlinks /var/tmp to /tmp and adds mount options like
> nosuid, etc.
> 
> Bring your own OS just takes a customer provider OS and
> 
> 1. Does basic checks needed
> 2. Installs packages from Sat6 or CDN
> 3. Installs RHVm, and does basic config
> 4. Sanitizes OS
> 5. Creates OVA or instructs user to create OVA
> 6. Provides/runs script to create RPM from OVA ready to use in automated
> install.
> 
> The other option is just tell me how to install RHVm and I'll create a
> puppet module to do it.

That's doable today. Are you looking for a silent installation?

> 
> I'll do ALL THE WORK!  I just need some basic info like the DOC the image
> RPM creaters use to do base install before certification of OVA/RPM."

For example using Ansible, see https://github.com/rhevm-qe-automation/ovirt-ansible/blob/master/roles/ovirt-engine-setup/README.md

I want to create the OS myself using my standard Sat6 based KS.  I don't want to have dozens of different OD configs all over the place.  Also for clients what "STIGd" system is changes with every person and client you ask.

RH creating a "STIGd" RHVm image is UTTERLY USELESS as I have yet to see any 2 clients implement a "STIG" the same way.

Also I am doing a fully automated SHE install with only RH Sat6, puppet, Hiera 100% automated both in install AND ongoing maintenance like adding RHEL-RHV hosts with ZERO intervention.

My Sat6 also will auto-integrate products like IDM with RHV if IDM is present, etc.  My Sat6 already fully builds IDM servers, Sat6 servers, capsules, and SHE RHV is next.

The RHEL-RHVH systems are already built with Sat6 and managed with puppet for what RHV doesn't do AND puppet will use puppet to run needed CLI cmds on the manager to automate things like so imports, backups, etc.

And all this requires no code changes.  Just Hiera changes.

Ansible can't do anywhere near what I am doing on the level I'm working on.

It's like PCI 10.0.  basically skynet managed by a combination of CF with Sat6.

This and the whole security compliance mess is why I need the Bring your own OS approach.

(In reply to Yaniv Kaul from comment #38)
> (In reply to Steven Mercurio from comment #32)
> >  "STIG compliance" seems to mean something different to everyone you ask!. 
> > Who has this exception, who has that one, etc.
> > 
> > The "bring your own OS" approach is the ONLY solution that covers ALL these
> > bases and more.
> > 
> > For example FTRIB symlinks /var/tmp to /tmp and adds mount options like
> > nosuid, etc.
> > 
> > Bring your own OS just takes a customer provider OS and
> > 
> > 1. Does basic checks needed
> > 2. Installs packages from Sat6 or CDN
> > 3. Installs RHVm, and does basic config
> > 4. Sanitizes OS
> > 5. Creates OVA or instructs user to create OVA
> > 6. Provides/runs script to create RPM from OVA ready to use in automated
> > install.
> > 
> > The other option is just tell me how to install RHVm and I'll create a
> > puppet module to do it.
> 
> That's doable today. Are you looking for a silent installation?
> 


What I'd like to do is either be able to pke and install a SHE OR be able to build my own RHV-M image then turn that into an RPM so that I can use my RHEV-M image RPM instead of the one provided by RH.

Once I learn how to do it and get a iamge created based on my existing OS standard then I will automate everything from the kickstart/base OS setup forward so then anyone just need to build a base OS to their "standard" and run the setup to do everything else.  Once I have this working I'll upload it to upstream so someone can add any needed sanity checks to ensure the client provided OS is supportable before making an RHV-M image with it.

> > 
> > I'll do ALL THE WORK!  I just need some basic info like the DOC the image
> > RPM creaters use to do base install before certification of OVA/RPM."
> 
> For example using Ansible, see
> https://github.com/rhevm-qe-automation/ovirt-ansible/blob/master/roles/ovirt-
> engine-setup/README.md

The above link does not work and I have not heard of any ansible items that will take a user supplied OS and install RHV-M and do what is needed to create an image that takes an answer file just like the RH image does for an unattended SHE install.

Moran, why is this on external? which team is doing this?

not sure why it was addressed as external, for sorting this one out the expectation is it will be joint effort of node and infra teams.

moving it to node, since Ryan is leading this effort.

In partnership with the BU (Ryan Barry and team), US Public Sector has began work on this. 

First step is to evaluate which NIST 800-53 controls are applicable to RHV-H and RHV-M. Currently mappings are done through FISMA Low and available here:

RHV-H:
http://atopathways.redhatgov.io/product-documents/rhvh/

RHV-M:
http://atopathways.redhatgov.io/product-documents/rhvm/

OpenControl-formatted code behind those sites:
https://github.com/ComplianceAsCode/redhat

Will be updating through FISMA Moderate until November.