Bug 1889694

Summary: usbguard fails to install as RHCOS extension due to missing libprotobuf.so.15
Product: OpenShift Container Platform Reporter: Sinny Kumari <skumari>
Component: RHCOSAssignee: Jonathan Lebon <jlebon>
Status: CLOSED ERRATA QA Contact: Michael Nguyen <mnguyen>
Severity: high Docs Contact:
Priority: medium    
Version: 4.7CC: bbreard, imcleod, jlebon, jligon, miabbott, nstielau, smilner
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-24 15:26:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sinny Kumari 2020-10-20 11:46:45 UTC
Description of problem:

usbguard fails to install due to missing libprotobuf.so on RHCOS system. This could be related to package dependency changed in base RHCOS packages or usbguard started depending on this new lib.

This issue was cought my MCO e2e-gcp-op ci run

Example e2e-gcp-op job - https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_machine-config-operator/2025/pull-ci-openshift-machine-config-operator-master-e2e-gcp-op/1318182107079512064

MCD Log: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_machine-config-operator/2025/pull-ci-openshift-machine-config-operator-master-e2e-gcp-op/1318182107079512064/artifacts/e2e-gcp-op/pods/openshift-machine-config-operator_machine-config-daemon-llwtn_machine-config-daemon.log
...
E1019 14:47:38.790081    2229 writer.go:135] Marking Degraded due to: error running rpm-ostree update --install usbguard: Checking out tree a527c24...done
Enabled rpm-md repositories: coreos-extensions
Resolving dependencies...done
error: Could not depsolve transaction; 1 problem detected:
 Problem: conflicting requests
  - nothing provides libprotobuf.so.15()(64bit) needed by usbguard-0.7.4-4.el8.x86_64
: exit status 1


Expected results: usbguard should successfully install as RHCOS extensions

Comment 1 Sinny Kumari 2020-10-20 11:47:53 UTC
Setting priority to high since this blocks MCO PR getting merged

Comment 2 Jonathan Lebon 2020-10-20 14:54:38 UTC
The proper fix for this is https://github.com/coreos/rpm-ostree/issues/2055. For now, I'll just add the missing RPMs to https://github.com/openshift/os/blob/master/scripts/download-extensions. Though we really should also have a basic test to confirm that extensions overlay cleanly.

Comment 3 Steve Milner 2020-10-20 15:14:22 UTC
Targeting 4.7 since this was found in the 4.7 images.

Sinny: Are you seeing this in previous builds as well?

Comment 5 Sinny Kumari 2020-10-20 15:41:54 UTC
Not sure, there is no e2e-gcp-op test run in ci since 17 October for 4.6 branch. I will check this on a local cluster and update here.

Comment 6 Sinny Kumari 2020-10-20 18:20:43 UTC
don't see this issue in 4.6, tested with latest nightly 4.6.0-0.nightly-2020-10-20-101225

Comment 8 Micah Abbott 2020-10-21 13:21:29 UTC
The proper version of protobuf landed in 47.82.202010201950-0 and that version of RHCOS was promoted into a nightly 4.7 payload.

Comment 10 Michael Nguyen 2020-10-27 01:45:08 UTC
Verified on 4.7.0-0.nightly-2020-10-26-124513.  usbguard installs successfully.

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.nightly-2020-10-26-124513   True        False         5h23m   Cluster version is 4.7.0-0.nightly-2020-10-26-124513
$ oc get nodes
NAME                                         STATUS   ROLES    AGE     VERSION
ip-10-0-136-111.us-west-2.compute.internal   Ready    worker   5h37m   v1.19.0+e67f5dc
ip-10-0-138-139.us-west-2.compute.internal   Ready    master   5h43m   v1.19.0+e67f5dc
ip-10-0-166-15.us-west-2.compute.internal    Ready    worker   5h37m   v1.19.0+e67f5dc
ip-10-0-187-209.us-west-2.compute.internal   Ready    master   5h43m   v1.19.0+e67f5dc
ip-10-0-218-114.us-west-2.compute.internal   Ready    worker   5h32m   v1.19.0+e67f5dc
ip-10-0-219-131.us-west-2.compute.internal   Ready    master   5h44m   v1.19.0+e67f5dc
$ oc debug node/ip-10-0-218-114.us-west-2.compute.internal -- chroot /host rpm-ostree status
Starting pod/ip-10-0-218-114us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
State: idle
Deployments:
* pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dc9cdce1901a5f5ca6cbab5e435c48d924eb813a01f5c48e49dd91c1d2e340c
              CustomOrigin: Managed by machine-config-operator
                   Version: 47.82.202010241242-0 (2020-10-24T12:46:24Z)

  ostree://6dff002defa66e3f9f53b57da7dc3eef37db39762655c00c13b977d72516649c
                   Version: 47.82.202010211043-0 (2020-10-21T10:47:13Z)

Removing debug pod ...
$ oc debug node/ip-10-0-218-114.us-west-2.compute.internal -- chroot /host rpm -q protobuf
Starting pod/ip-10-0-218-114us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
package protobuf is not installed

Removing debug pod ...
$ cat << EOF > 80-extensions.yaml
> apiVersion: machineconfiguration.openshift.io/v1
> kind: MachineConfig
> metadata:
>   labels:
>     machineconfiguration.openshift.io/role: worker
>   name: 80-worker-extensions
> spec:
>   config:
>     ignition:
>       version: 3.1.0
>   extensions:
>     - usbguard
> EOF
$ oc create -f 80-extensions.yaml 
machineconfig.machineconfiguration.openshift.io/80-worker-extensions created
$ oc get machineconfig 80-worker-extensions
NAME                   GENERATEDBYCONTROLLER   IGNITIONVERSION   AGE
80-worker-extensions                           3.1.0             15s
$ oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-107f3aa4ffa04a053e0b320fe1d89688   True      False      False      3              3                   3                     0                      5h49m
worker   rendered-worker-c4d33fa1d6014ab23f18932769e390c2   False     True       False      3              0                   0                     0                      5h49m
$ watch oc get mcp
$ oc debug node/ip-10-0-218-114.us-west-2.compute.internal -- chroot /host rpm -q usbguard
Starting pod/ip-10-0-218-114us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
usbguard-0.7.4-4.el8.x86_64

Removing debug pod ...

Comment 12 Jonathan Lebon 2021-01-07 18:47:37 UTC
(Marking as No Doc Update because this was a transient thing that happened during 4.7 development which never leaked out IIRC.)

Comment 14 errata-xmlrpc 2021-02-24 15:26:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633