Bug 1712960

Summary: vSphere OVA is of an old virtual HW revision and a generic OS type
Product: OpenShift Container Platform Reporter: Ian McLeod <imcleod>
Component: RHCOSAssignee: Steve Milner <smilner>
Status: CLOSED ERRATA QA Contact: Micah Abbott <miabbott>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.1.zCC: anli, bbreard, behoward, dahernan, dustymabe, imcleod, jligon, jsavanyo, lmeyer, nstielau, sponnaga, tbielawa, vmware-gos-qa, walters
Target Milestone: ---Keywords: OSE41z_next
Target Release: 4.1.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: 4.1.4
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-07-04 09:01:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ian McLeod 2019-05-22 15:16:07 UTC
Our contacts at VMWare have suggested the following metadata-only changes to the OVA we generate for RHCOS.

1) Set "HWV = 13" ("vmx-13" in the actual OVA XML) - This will enforce ESXi >= 6.5 when deploying the OVA template

2) Set the OS type to rhel7-64 ("rhel7_64Guest" in the actual OVA XML) - It's unclear what concrete effect this has when deploying.

We currently set the HWV to 7,8 and the OS type to "other26xlinux-64"

Comment 12 Micah Abbott 2019-06-21 15:31:58 UTC
Upgrade to 4.1.0-0.nightly-2019-06-20-015058 and inspected nodes for updated, signed package:

```
$ oc get clusterversion/version
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.1.0-0.nightly-2019-06-20-015058   True        False         2m36s   Cluster version is 4.1.0-0.nightly-2019-06-20-015058

$ oc get nodes
NAME                                         STATUS                     ROLES    AGE   VERSION
ip-10-0-131-47.us-west-2.compute.internal    Ready                      worker   46h   v1.13.4+9b19d73a0
ip-10-0-141-41.us-west-2.compute.internal    Ready                      master   46h   v1.13.4+9252851b0
ip-10-0-146-21.us-west-2.compute.internal    Ready                      master   46h   v1.13.4+9252851b0                                                                                                             ip-10-0-154-37.us-west-2.compute.internal    Ready                      worker   46h   v1.13.4+9252851b0                                                                                                            
ip-10-0-168-93.us-west-2.compute.internal    Ready                      worker   46h   v1.13.4+9252851b0
ip-10-0-174-123.us-west-2.compute.internal   Ready,SchedulingDisabled   master   46h   v1.13.4+9252851b0

$ oc debug node/ip-10-0-131-47.us-west-2.compute.internal                                                         
Starting pod/ip-10-0-131-47us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
* pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dfb630dadbc16f15507c01b6d7783c8cb5140ac280ff518296c5ec00484afc
              CustomOrigin: Managed by pivot tool
                   Version: 410.8.20190619.1 (2019-06-19T21:27:54Z)

  pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:067bba989d1c6c570fc2b17424207c8020cdcef528513eeafa48faf3fa5d496e
              CustomOrigin: Managed by pivot tool
                   Version: 410.8.20190619.0 (2019-06-19T00:47:13Z)
sh-4.4# rpm -qi open-vm-tools
Name        : open-vm-tools
Version     : 10.3.10
Release     : 2.el8
Architecture: x86_64
Install Date: Wed Jun 19 21:21:34 2019
Group       : Applications/System
Size        : 2820502
License     : GPLv2
Signature   : RSA/SHA256, Tue Jun  4 16:07:37 2019, Key ID 199e2f91fd431d51
Source RPM  : open-vm-tools-10.3.10-2.el8.src.rpm
Build Date  : Tue Jun  4 13:07:20 2019
Build Host  : x86-vm-05.build.eng.bos.redhat.com
Relocations : (not relocatable)
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Vendor      : Red Hat, Inc.
URL         : https://github.com/vmware/open-vm-tools
Summary     : Open Virtual Machine Tools for virtual machines hosted on VMware
Description :
The open-vm-tools project is an open source implementation of VMware Tools. It
is a suite of open source virtualization utilities and drivers to improve the
functionality, user experience and administration of VMware virtual machines.
This package contains only the core user-space programs and libraries of
open-vm-tools.
# cat /etc/vmware-tools/tools.conf 
[guestosinfo]
short-name = rhel8-64

$ oc debug node/ip-10-0-141-41.us-west-2.compute.internal
Starting pod/ip-10-0-141-41us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
* pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dfb630dadbc16f15507c01b6d7783c8cb5140ac280ff518296c5ec00484afc
              CustomOrigin: Managed by pivot tool
                   Version: 410.8.20190619.1 (2019-06-19T21:27:54Z)

  pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:067bba989d1c6c570fc2b17424207c8020cdcef528513eeafa48faf3fa5d496e
              CustomOrigin: Managed by pivot tool
                   Version: 410.8.20190619.0 (2019-06-19T00:47:13Z)
sh-4.4# rpm -qi open-vm-tools
Name        : open-vm-tools
Version     : 10.3.10
Release     : 2.el8
Architecture: x86_64
Install Date: Wed Jun 19 21:21:34 2019
Group       : Applications/System
Size        : 2820502
License     : GPLv2
Signature   : RSA/SHA256, Tue Jun  4 16:07:37 2019, Key ID 199e2f91fd431d51
Source RPM  : open-vm-tools-10.3.10-2.el8.src.rpm
Build Date  : Tue Jun  4 13:07:20 2019
Build Host  : x86-vm-05.build.eng.bos.redhat.com
Relocations : (not relocatable)
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Vendor      : Red Hat, Inc.
URL         : https://github.com/vmware/open-vm-tools
Summary     : Open Virtual Machine Tools for virtual machines hosted on VMware
Description :
The open-vm-tools project is an open source implementation of VMware Tools. It
is a suite of open source virtualization utilities and drivers to improve the
functionality, user experience and administration of VMware virtual machines.
This package contains only the core user-space programs and libraries of
open-vm-tools.
# cat /etc/vmware-tools/tools.conf 
[guestosinfo]
short-name = rhel8-64
```

Comment 13 Micah Abbott 2019-06-24 17:19:52 UTC
This was mistakenly moved to VERIFIED, as I thought the provided package contained all the fixes necessary.

In addition to this package, the RHCOS team needs to make additional changes to how the RHCOS compose is made in order to fully fix this BZ.

Moving back to ASSIGNED until the remaining changes are provided.

Comment 14 Steve Milner 2019-06-25 14:33:27 UTC
The only issue that is still present is that the 4.1 ART pipeline is using a cosa that does not have Ian's changes. All other work for this BZ is finished.

Comment 15 Micah Abbott 2019-06-25 14:33:48 UTC
The last bit of the fix lives in https://github.com/coreos/coreos-assembler/pull/528

We will need ART to produce builds of RHCOS using a version of `coreos-assembler` that includes the commits in that PR.

Comment 19 Micah Abbott 2019-07-01 13:50:22 UTC
Using the 4.1.4 release candidate from https://openshift-release.svc.ci.openshift.org/releasestream/4-stable/release/4.1.4...

```
$ oc image info -a ./all-the-pull-secrets.json $(oc adm release info --image-for=machine-os-content -a ./all-the-pull-secrets.json quay.io/openshift-release-dev/ocp-release:4.1.4)
Name:       quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2dd133df40fa0fe6393e92ea41c3ffbfda47d86380d68e6402d3e81361557168
Media Type: application/vnd.docker.distribution.manifest.v2+json
Created:    3d ago
Image Size: 595.2MB
OS:         linux
Arch:       amd64
Entrypoint: /noentry
Labels:     com.coreos.ostree-commit=b608c00b0da5d5edbdc7d453d980f6cebff53fdba0081bb49a13867b6ffaf437
            version=410.8.20190627.0

$ curl -J -L -O https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.1/410.8.20190627.0/rhcos-410.8.20190627.0-vmware.ova

$ tar -xvf rhcos-410.8.20190627.0-vmware.ova
desc.ovf
disk.vmdk

$ grep vmw:osType desc.ovf
    <OperatingSystemSection ovf:id="80" ovf:version="6" vmw:osType="rhel7_64Guest">

$ grep vssd:VirtualSystemType desc.ovf
        <vssd:VirtualSystemType>vmx-13</vssd:VirtualSystemType>
```

The RPM changes were previously verified in 4.1 builds; marking this as VERIFIED.

Comment 25 Colin Walters 2019-07-03 16:35:14 UTC
> Our contacts at VMWare have suggested the following metadata-only changes to the OVA we generate for RHCOS.

If this requires respinning the OVA we would need to have a procedure to respin just *one* bootimage type, or respin *all* bootimages.

This also most notably requires a PR to bump the bootimages used by the installer.

Also, on this topic I would like to suggest that the bootimages provided on the portal exactly match that pinned in the installer.  In other words, let us never again do what we did for 4.1.0 where the installer (IPI) uses one set of bootimages and the version we uploaded manually used another.

Note that 4.1.0 RHCOS bootimages are vulnerable to https://access.redhat.com/security/vulnerabilities/tcpsack too.

Comment 26 errata-xmlrpc 2019-07-04 09:01:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1635

Comment 27 Micah Abbott 2019-08-12 13:35:47 UTC
*** Bug 1740199 has been marked as a duplicate of this bug. ***