Bug 1814834

Summary: Unattended installation of repo GPG keys is failing in RHEL 8.1
Product: Red Hat Enterprise Linux 8 Reporter: Jaroslav Mracek <jmracek>
Component: dnfAssignee: Packaging Maintenance Team <packaging-team-maint>
Status: CLOSED DUPLICATE QA Contact: Radek Bíba <rbiba>
Severity: unspecified Docs Contact:
Priority: medium    
Version: 8.1CC: ericedens, james.antill, jecoulte, swm-qe
Target Milestone: rcKeywords: Triaged
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1792506 Environment:
Last Closed: 2020-04-28 10:44:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1781601, 1792506    
Bug Blocks:    

Description Jaroslav Mracek 2020-03-18 18:55:25 UTC 
+++ This bug was initially created as a clone of Bug #1792506 +++

## Description of problem:

Unattended installation of repo GPG keys is failing in RHEL 8.1, however it was working in RHEL 8.

## Version-Release number of selected component (if applicable):

RHEL 8.1 with DNF 4.2.7-7. Full details:

### /etc/os-release:
  NAME="Red Hat Enterprise Linux"
  VERSION="8.1 (Ootpa)"
  ID="rhel"
  ID_LIKE="fedora"
  VERSION_ID="8.1"
  PLATFORM_ID="platform:el8"
  PRETTY_NAME="Red Hat Enterprise Linux 8.1 (Ootpa)"
  ANSI_COLOR="0;31"
  CPE_NAME="cpe:/o:redhat:enterprise_linux:8.1:GA"
  HOME_URL="https://www.redhat.com/"
  BUG_REPORT_URL="https://bugzilla.redhat.com/"

  REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
  REDHAT_BUGZILLA_PRODUCT_VERSION=8.1
  REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
  REDHAT_SUPPORT_PRODUCT_VERSION="8.1"

### dnf --version
  Installed: dnf-0:4.2.7-7.el8_1.noarch at Wed 11 Dec 2019 12:11:03 AM GMT
  Built    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla> at Mon 21 Oct 2019 12:28:04 PM GMT

  Installed: rpm-0:4.14.2-25.el8.x86_64 at Wed 11 Dec 2019 12:09:26 AM GMT
  Built    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla> at Wed 07 Aug 2019 01:32:37 PM GMT

## How reproducible:

Always reproducible in a non-interactive environment. Successful if run in a terminal.

## Steps to Reproduce:

To reproduce the issue, run [setup.sh] in a non-interactive environment on RHEL 8.1. Two concrete examples are shown below:

### Example 1:

Launch an instance of GCP using a RHEL 8.1 image from  https://console.cloud.google.com/marketplace/details/rhel-cloud/rhel-8 (for example rhel-8-v20191210) using [setup.sh] as a startup script.

### Example 2:

This can be quickly reproduced using CentOS 8 and CentOS 8.1 in Docker. Run the following examples with "docker build ." in a directory with setup.sh from below.

Passing example:

  Dockerfile:
    FROM centos:8
    COPY setup.sh /
    RUN /setup.sh


Failing example:

  Dockerfile:
    FROM centos:8.1.1911
    COPY setup.sh /
    RUN /setup.sh


## Actual results:

The call to `dnf -y install google-cloud-sdk` fails with: 

Importing GPG key 0xA7317B0F:
 Userid     : "Google Cloud Packages Automatic Signing Key <gc-team>"
 Fingerprint: D0BC 747F D8CA F711 7500 D6FA 3746 C208 A731 7B0F
 From       : https://packages.cloud.google.com/yum/doc/yum-key.gpg
INFO startup-script: Return code 141.


## Expected results:

The call to `dnf -y install google-cloud-sdk` succeeds, with the side effect that the repo's GPG keys are accepted.

## Additional info:

### setup.sh
dnf -y remove google-cloud-sdk

cat << EOM > /etc/yum.repos.d/google-cloud.repo
[google-compute-engine]
name=Google Compute Engine
baseurl=https://packages.cloud.google.com/yum/repos/google-compute-engine-el8-x86_64-stable
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
       https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
[google-cloud-sdk]
name=Google Cloud SDK
baseurl=https://packages.cloud.google.com/yum/repos/cloud-sdk-el8-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
       https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOM

curl -O https://packages.cloud.google.com/yum/doc/yum-key.gpg
curl -O https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
rpm --import yum-key.gpg
rpm --import rpm-package-key.gpg
rm -f yum-key.gpg rpm-package-key.gpg

dnf -y install google-cloud-sdk

--- Additional comment from Jaroslav Mracek on 2020-02-18 15:52:07 UTC ---

I am sorry but I was unable to reproduce the issue. I tried on fedora 31. 

Dockerfile:
    FROM centos:8.1.1911
    COPY setup.sh /
    RUN /setup.sh

$ podman build .

gpg keys were successfully imported (see bellow).

Please could you provide an alternative reproducer?



STEP 1: FROM centos:8.1.1911
STEP 2: COPY setup.sh /
80ef5d8a252716d37d5a5b8b983c84bd57ba8a1fb6a477d22d8c2f8cbb32c6ea
STEP 3: RUN /setup.sh
No match for argument: google-cloud-sdk
No packages marked for removal.
Dependencies resolved.
Nothing to do.
Complete!
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1885  100  1885    0     0  11493      0 --:--:-- --:--:-- --:--:-- 11493
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   975  100   975    0     0   8405      0 --:--:-- --:--:-- --:--:--  8405
CentOS-8 - AppStream                            4.7 MB/s | 6.4 MB     00:01    
CentOS-8 - Base                                 5.5 MB/s | 5.0 MB     00:00    
CentOS-8 - Extras                               6.0 kB/s | 2.1 kB     00:00    
Google Compute Engine                           328  B/s | 454  B     00:01    
Google Compute Engine                            16 kB/s | 1.8 kB     00:00    
Importing GPG key 0xA7317B0F:
 Userid     : "Google Cloud Packages Automatic Signing Key <gc-team>"
 Fingerprint: D0BC 747F D8CA F711 7500 D6FA 3746 C208 A731 7B0F
 From       : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Importing GPG key 0xBA07F4FB:
 Userid     : "Google Cloud Packages Automatic Signing Key <gc-team>"
 Fingerprint: 54A6 47F9 048D 5688 D7DA 2ABE 6A03 0B21 BA07 F4FB
 From       : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Google Compute Engine                           7.7 kB/s | 975  B     00:00    
Importing GPG key 0x3E1BA8D5:
 Userid     : "Google Cloud Packages RPM Signing Key <gc-team>"
 Fingerprint: 3749 E1BA 95A8 6CE0 5454 6ED2 F09C 394C 3E1B A8D5
 From       : https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
Google Compute Engine                           3.0 kB/s | 5.8 kB     00:01    
Google Cloud SDK                                426  B/s | 454  B     00:01    
Google Cloud SDK                                 16 kB/s | 1.8 kB     00:00    
Importing GPG key 0xA7317B0F:
 Userid     : "Google Cloud Packages Automatic Signing Key <gc-team>"
 Fingerprint: D0BC 747F D8CA F711 7500 D6FA 3746 C208 A731 7B0F
 From       : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Importing GPG key 0xBA07F4FB:
 Userid     : "Google Cloud Packages Automatic Signing Key <gc-team>"
 Fingerprint: 54A6 47F9 048D 5688 D7DA 2ABE 6A03 0B21 BA07 F4FB
 From       : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Google Cloud SDK                                8.7 kB/s | 975  B     00:00    
Importing GPG key 0x3E1BA8D5:
 Userid     : "Google Cloud Packages RPM Signing Key <gc-team>"
 Fingerprint: 3749 E1BA 95A8 6CE0 5454 6ED2 F09C 394C 3E1B A8D5
 From       : https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
Google Cloud SDK                                6.1 MB/s |  31 MB     00:05    
Dependencies resolved.
================================================================================
 Package           Arch   Version                               Repo       Size
================================================================================
Installing:
 google-cloud-sdk  noarch 280.0.0-1                             google-cloud-sdk
                                                                           52 M
Installing dependencies:
 python2           x86_64 2.7.16-12.module_el8.1.0+219+cf9e6ac9 AppStream 109 k
 python2-libs      x86_64 2.7.16-12.module_el8.1.0+219+cf9e6ac9 AppStream 6.0 M
 python2-pip-wheel noarch 9.0.3-14.module_el8.1.0+219+cf9e6ac9  AppStream 1.2 M
 python2-setuptools-wheel
                   noarch 39.0.1-11.module_el8.1.0+219+cf9e6ac9 AppStream 289 k
Installing weak dependencies:
 python2-pip       noarch 9.0.3-14.module_el8.1.0+219+cf9e6ac9  AppStream 2.0 M
 python2-setuptools
                   noarch 39.0.1-11.module_el8.1.0+219+cf9e6ac9 AppStream 643 k
Enabling module streams:
 python27                 2.7                                                  

Transaction Summary
================================================================================
Install  7 Packages

Total download size: 62 M
Installed size: 275 M
Downloading Packages:
(1/7): python2-2.7.16-12.module_el8.1.0+219+cf9 744 kB/s | 109 kB     00:00    
(2/7): python2-pip-9.0.3-14.module_el8.1.0+219+ 2.7 MB/s | 2.0 MB     00:00    
(3/7): python2-pip-wheel-9.0.3-14.module_el8.1. 1.8 MB/s | 1.2 MB     00:00    
(4/7): python2-libs-2.7.16-12.module_el8.1.0+21 5.0 MB/s | 6.0 MB     00:01    
(5/7): python2-setuptools-39.0.1-11.module_el8. 1.4 MB/s | 643 kB     00:00    
(6/7): python2-setuptools-wheel-39.0.1-11.modul 735 kB/s | 289 kB     00:00    
(7/7): eae1d38e8895c3e067a21bba76e809fcdec3a8ae 8.4 MB/s |  52 MB     00:06    
--------------------------------------------------------------------------------
Total                                           8.1 MB/s |  62 MB     00:07     
warning: /var/cache/dnf/AppStream-02e86d1c976ab532/packages/python2-2.7.16-12.module_el8.1.0+219+cf9e6ac9.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 8483c65d: NOKEY
CentOS-8 - AppStream                            227 kB/s | 1.6 kB     00:00    
Importing GPG key 0x8483C65D:
 Userid     : "CentOS (CentOS Official Signing Key) <security>"
 Fingerprint: 99DB 70FA E1D7 CE22 7FB6 4882 05B5 55B3 8483 C65D
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1 
  Installing       : python2-setuptools-wheel-39.0.1-11.module_el8.1.0+21   1/7 
  Installing       : python2-pip-wheel-9.0.3-14.module_el8.1.0+219+cf9e6a   2/7 
  Installing       : python2-libs-2.7.16-12.module_el8.1.0+219+cf9e6ac9.x   3/7 
  Installing       : python2-pip-9.0.3-14.module_el8.1.0+219+cf9e6ac9.noa   4/7 
  Installing       : python2-setuptools-39.0.1-11.module_el8.1.0+219+cf9e   5/7 
  Installing       : python2-2.7.16-12.module_el8.1.0+219+cf9e6ac9.x86_64   6/7 
  Running scriptlet: python2-2.7.16-12.module_el8.1.0+219+cf9e6ac9.x86_64   6/7 
  Installing       : google-cloud-sdk-280.0.0-1.noarch                      7/7 
  Running scriptlet: google-cloud-sdk-280.0.0-1.noarch                      7/7 
  Verifying        : python2-2.7.16-12.module_el8.1.0+219+cf9e6ac9.x86_64   1/7 
  Verifying        : python2-libs-2.7.16-12.module_el8.1.0+219+cf9e6ac9.x   2/7 
  Verifying        : python2-pip-9.0.3-14.module_el8.1.0+219+cf9e6ac9.noa   3/7 
  Verifying        : python2-pip-wheel-9.0.3-14.module_el8.1.0+219+cf9e6a   4/7 
  Verifying        : python2-setuptools-39.0.1-11.module_el8.1.0+219+cf9e   5/7 
  Verifying        : python2-setuptools-wheel-39.0.1-11.module_el8.1.0+21   6/7 
  Verifying        : google-cloud-sdk-280.0.0-1.noarch                      7/7 

Installed:
  google-cloud-sdk-280.0.0-1.noarch                                             
  python2-pip-9.0.3-14.module_el8.1.0+219+cf9e6ac9.noarch                       
  python2-setuptools-39.0.1-11.module_el8.1.0+219+cf9e6ac9.noarch               
  python2-2.7.16-12.module_el8.1.0+219+cf9e6ac9.x86_64                          
  python2-libs-2.7.16-12.module_el8.1.0+219+cf9e6ac9.x86_64                     
  python2-pip-wheel-9.0.3-14.module_el8.1.0+219+cf9e6ac9.noarch                 
  python2-setuptools-wheel-39.0.1-11.module_el8.1.0+219+cf9e6ac9.noarch         

Complete!
STEP 4: COMMIT

--- Additional comment from ericedens on 2020-02-19 16:27:01 UTC ---

Hi Jaroslav Mracek,

The script needs to be run from a non-interactive environment to trigger the bug. I have a limited understanding of podman, but it looks like it doesn't use a daemon process as docker does, so perhaps when you run `podman build` the podman process is interactive via your terminal.

Docker will be easiest repro. Alternatively, you can execute `setup.sh` on a RHEL 8.1 or CentOS 8.1 machine directly. The bug won't occur if you run the script from a terminal interactively; something non-interactive like a startup script or a cron will repro it.

Thanks,
Eric

--- Additional comment from Eric Coulter on 2020-03-11 15:47:07 UTC ---

I'm able to reproduce easily - ran into this on a Centos 8 instance on an Openstack cloud. 
Happens every time I try something similar - my script was slightly different, but tested with ericedens script over ssh. 

See output below; can provide additional info if needed. 

Cheers,
Eric C.

#### /etc/os-release
$ ssh centos@$test_vm_ip "cat /etc/os-release"
Warning: Permanently added '$test_vm_ip' (ECDSA) to the list of known hosts.
NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="8"

#### dnf --version
$ ssh centos@$test_vm_ip "dnf --version"
Warning: Permanently added '$test_vm_ip' (ECDSA) to the list of known hosts.
4.2.7
  Installed: dnf-0:4.2.7-7.el8_1.noarch at Mon 13 Jan 2020 09:50:35 PM GMT 
  Built    : CentOS Buildsys <bugs> at Thu 19 Dec 2019 03:44:23 PM GMT 

  Installed: rpm-0:4.14.2-25.el8.x86_64 at Mon 13 Jan 2020 09:49:51 PM GMT 
  Built    : CentOS Buildsys <bugs> at Fri 08 Nov 2019 10:56:14 PM GMT 

### Example - running the above setup.sh over ssh

$ scp dnf_bug_setup.sh centos@$test_vm_ip:
Warning: Permanently added '$test_vm_ip' (ECDSA) to the list of known hosts.
dnf_bug_setup.sh                                                                                                                                                       100%  937    26.7KB/s   0.9KB/s   00:00     $ ssh centos@$test_vm_ip "sudo ./dnf_bug_setup.sh"
Warning: Permanently added '$test_vm_ip' (ECDSA) to the list of known hosts.
Waiting for process with pid 4172 to finish. 
No match for argument: google-cloud-sdk
No packages marked for removal.
Dependencies resolved. 
Nothing to do.
Complete!
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1885  100  1885    0     0  12401      0 --:--:-- --:--:-- --:--:-- 12401
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   975  100   975    0     0   7558      0 --:--:-- --:--:-- --:--:--  7558
Google Compute Engine                           512  B/s | 454  B     00:00    
Google Compute Engine                            14 kB/s | 1.8 kB     00:00    
Importing GPG key 0xA7317B0F:                        
 Userid     : "Google Cloud Packages Automatic Signing Key <gc-team>"
 Fingerprint: D0BC 747F D8CA F711 7500 D6FA 3746 C208 A731 7B0F
 From       : https://packages.cloud.google.com/yum/doc/yum-key.gpg

## confirmation of error code
$ ssh centos@$test_vm_ip "echo $?"
Warning: Permanently added '$test_vm_ip' (ECDSA) to the list of known hosts.
141

--- Additional comment from Jaroslav Mracek on 2020-03-18 18:53:48 UTC ---

With docker I am able to reproduce it. When I moved `dnf -y install google-cloud-sdk` from shell script and I run it as another RUN in Docker file everything worked fine. The process was somehow terminated outside of DNF.

Also the issue is not reproducible with podman therefore I would like to ask developers of docker if they could identify the source issue. I will create a clone on docker to keep track.
Comment 1 Tomas Tomecek 2020-03-19 14:34:45 UTC 
Two things:

1. there is no python-docker in RHEL 8

2. there is even no docker in RHEL 8

Therefore unassigning it from python-docker - please pick a more suitable component, I just want to make it super-clear that python-docker has nothing to do with this bug.
Comment 3 Daniel Mach 2020-04-28 10:44:23 UTC 

*** This bug has been marked as a duplicate of bug 1792506 ***