Bug 1884001 - 3.11.z installer should required docker 1.13.1-104 [NEEDINFO]
Summary: 3.11.z installer should required docker 1.13.1-104
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.11.z
Assignee: Russell Teague
QA Contact: Gaoyun Pei
: 1890328 (view as bug list)
Depends On:
Blocks: 1883226
TreeView+ depends on / blocked
Reported: 2020-09-30 17:57 UTC by Adam Kaplan
Modified: 2020-11-13 20:28 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2020-11-13 20:28:27 UTC
Target Upstream Version:
rteague: needinfo? (adam.kaplan)

Attachments (Terms of Use)

Description Adam Kaplan 2020-09-30 17:57:04 UTC
Version: 3.11.z

Platform: aws

Please specify: ansible-install (3.11 CI)

What happened?

Ansible installer allows docker v1.13.1-103. This includes a bug in how symlinks are treated, which breaks OpenShift builds.

What did you expect to happen?

Docker v1.13.1-104 or higher should be enforced.

How to reproduce it (as minimally and precisely as possible)?

Run 3.11 extended_builds suite (via Jenkins)

Anything else we need to know?

See https://prow.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/25561/test_pull_request_origin_extended_builds_311/172/

Comment 1 Lukasz Szaszkiewicz 2020-10-06 12:42:25 UTC
Is someone working on this issue? At the moment it is blocking 3.11 PRs.

For example, we can't merge https://github.com/openshift/origin/pull/25009 because some unit tests are failing.
We can't fix unit tests https://github.com/openshift/origin/pull/25561 because "extended_builds" job fails.

Comment 2 Russell Teague 2020-10-06 13:42:11 UTC
I'm seeing docker.x86_64 2:1.13.1-162.git64e9980.el7_8 being installed in https://storage.googleapis.com/origin-ci-test/pr-logs/pull/25561/pull-ci-openshift-origin-release-3.11-e2e-gcp/1311358965027704832/build-log.txt

Can you link to where docker v1.13.1-103 is being installed?

Comment 3 Venkata Siva Teja Areti 2020-10-06 14:40:49 UTC
This is one run in which I see docker 1.13.1-103 being installed.


These are the messages in installation logs that show 103 is installed.

TASK [container_runtime : Install Docker] **************************************
task path: /usr/share/ansible/openshift-ansible/roles/container_runtime/tasks/package_docker.yml:23
changed: [localhost] => {
    "attempts": 1, 
    "changed": true, 
    "generated_timestamp": "2020-09-30 18:06:38.368471", 
    "msg": "", 
    "rc": 0, 
    "results": [
        "2:docker-1.13.1-103.git7f2769b.el7.x86_64 providing docker is already installed",

Comment 4 Russell Teague 2020-10-06 17:53:39 UTC
The AMI which is being used for this job has not been updated in nearly a year.  It has an old version of docker installed and openshift-ansible does not force upgrades if docker is already installed.  The image base image the job is using should be updated.

I do not know who supports the oct/playbooks/provision/aws-up.yml playbook being used to provision the AWS instances.

    "item": {
        "ami_id": "ami-01a9f71561742665f", 
        "architecture": "x86_64", 
        "block_device_mapping": {
            "/dev/sda1": {
                "delete_on_termination": true, 
                "encrypted": false, 
                "size": 75, 
                "snapshot_id": "snap-0bace4ea1d7a4225e", 
                "volume_type": "gp2"
            "/dev/sdb": {
                "delete_on_termination": true, 
                "encrypted": false, 
                "size": 50, 
                "snapshot_id": "snap-0138bfa2d206e5160", 
                "volume_type": "gp2"
        "creationDate": "2019-10-18T15:43:05.000Z", 
        "description": "ami_build_origin_int_rhel_base_2162", 
        "hypervisor": "xen", 
        "is_public": false, 
        "location": "531415883065/ami_build_origin_int_rhel_base_2162", 
        "name": "ami_build_origin_int_rhel_base_2162", 
        "owner_id": "531415883065", 
        "platform": null, 
        "root_device_name": "/dev/sda1", 
        "root_device_type": "ebs", 
        "state": "available", 
        "tags": {
            "Name": "ami_build_origin_int_rhel_base_2162", 
            "image_stage": "base", 
            "operating_system": "rhel", 
            "ready": "yes"

Comment 7 Maru Newby 2020-10-22 00:00:50 UTC
Any update on this? I'm initiating an effort to get 3.11 built with golang 1.15 and without this being fixed we can't merge anything to origin's release-3.11 branch without overriding the unit test job.

Comment 8 Scott Dodson 2020-10-22 00:47:15 UTC
Interesting, well the whole reason it's failing is because the repo in which we're installing golang 1.9.2 info the AMI went away. If we're no longer going to be installing golang 1.9.2 and you'll presumably have to find another repo to retrieve 1.15 from then at least that problem goes away. Are you going to be updating the AMI build?

Comment 9 Maru Newby 2020-10-22 03:38:39 UTC
(In reply to Scott Dodson from comment #8)
> Interesting, well the whole reason it's failing is because the repo in which
> we're installing golang 1.9.2 info the AMI went away. If we're no longer
> going to be installing golang 1.9.2 and you'll presumably have to find
> another repo to retrieve 1.15 from then at least that problem goes away. Are
> you going to be updating the AMI build?

I'm still coming up to speed with what is involved in building and testing 3.11. The goal is to clarify what risks (if any) are involved in switching to golang 1.15. That involves getting CI clusters with golang 1.15-built artifacts built and tested in CI. As far as I know we'll be building with ci containers that are built by downloading upstream golang binaries, e.g.


Only once we'd successfully vetted 1.15-built 3.11 in CI would we be have to worry about what's involved in releasing 1.15-built 3.11 artifacts. Would that be when AMIs become relevant, or am I missing something?

Comment 10 Scott Dodson 2020-10-22 14:09:59 UTC
*** Bug 1890328 has been marked as a duplicate of this bug. ***

Comment 11 Brenton Leanhardt 2020-10-22 19:49:11 UTC
Maru, is this bug primarily about fixing the AMI.  Should you own this?

Comment 12 Maru Newby 2020-10-23 19:52:59 UTC
(In reply to Brenton Leanhardt from comment #11)
> Maru, is this bug primarily about fixing the AMI.  Should you own this?

I should definitely not own this. I'm tasked with enabling testing of 3.11 built with golang 1.15, and lack the background necessary to understand why AMIs are even required for 3.11 testing.

Comment 13 Russell Teague 2020-11-11 19:11:18 UTC
This may help unblock the AMI build.

Comment 14 Russell Teague 2020-11-11 22:03:02 UTC
Looks like if we can get the AMI build job working it should have working docker version installed.

From the logs:
Package docker.x86_64 2:1.13.1-203.git0be3e21.el7_9 will be installed

Comment 15 Russell Teague 2020-11-12 13:24:25 UTC
It looks like the extended_builds job is passing, and was passing before we updated the AMI.

Can someone verify https://github.com/openshift/origin/pull/25561 from comment 1?

Comment 16 Adam Kaplan 2020-11-12 16:28:14 UTC
In the interim several updates were made to get the extended_build tests on 3.11 passing. We are currently skipping the secrets test [1] - as a part of fixing this bug the secrets test should be re-enabled.

Looking at the latest run on CI, we are still using the old version of docker [2]:

TASK [container_runtime : Install Docker] **************************************
task path: /usr/share/ansible/openshift-ansible/roles/container_runtime/tasks/package_docker.yml:23
changed: [localhost] => {
    "attempts": 1, 
    "changed": true, 
    "generated_timestamp": "2020-11-09 14:02:53.517859", 
    "msg": "", 
    "rc": 0, 
    "results": [
        "2:docker-1.13.1-103.git7f2769b.el7.x86_64 providing docker is already installed", 
        "Loaded plugins: amazon-id, rhui-lb, search-disabled-repos\nResolving Dependencies\n--> Running transaction check\n---> Package atomic.x86_64 1:1.22.1-33.gitb507039.el7_8 will be installed\n--> Processing Dependency: runc for package: 1:atomic-1.22.1-33.gitb507039.el7_8.x86_64\n--> Processing Dependency: python-slip-dbus for package: 1:atomic-1.22.1-33.gitb507039.el7_8.x86_64\n--> Processing Dependency: ostree for package: 1:atomic-1.22.1-33.gitb507039.el7_8.x86_64\n--> Processing Dependency: gomtree for package: 1:atomic-1.22.1-33.gitb507039.el7_8.x86_64\n---> Package skopeo.x86_64 1:0.1.40-11.el7_8 will be installed\n--> Processing Dependency: containers-common = 1:0.1.40-11.el7_8 for package: 1:skopeo-0.1.40-11.el7_8.x86_64\n--> Running transaction check\n---> Package containers-common.x86_64 1:0.1.37-3.el7 will be updated\n---> Package containers-common.x86_64 1:0.1.40-11.el7_8 will be an update\n--> Processing Dependency: slirp4netns for package: 1:containers-common-0.1.40-11.el7_8.x86_64\n--> Processing Dependency: fuse-overlayfs for package: 1:containers-common-0.1.40-11.el7_8.x86_64\n---> Package gomtree.x86_64 0:0.5.0-0.2.git16da0f8.el7 will be installed\n---> Package ostree.x86_64 0:2019.1-2.el7 will be installed\n---> Package python-slip-dbus.noarch 0:0.4.0-4.el7 will be installed\n--> Processing Dependency: python-slip = 0.4.0-4.el7 for package: python-slip-dbus-0.4.0-4.el7.noarch\n---> Package runc.x86_64 0:1.0.0-67.rc10.el7_8 will be installed\n--> Processing Dependency: criu for package: runc-1.0.0-67.rc10.el7_8.x86_64\n--> Running transaction check\n---> Package criu.x86_64 0:3.12-2.el7 will be installed\n--> Processing Dependency: libprotobuf-c.so.1(LIBPROTOBUF_C_1.0.0)(64bit) for package: criu-3.12-2.el7.x86_64\n--> Processing Dependency: libprotobuf-c.so.1()(64bit) for package: criu-3.12-2.el7.x86_64\n--> Processing Dependency: libnet.so.1()(64bit) for package: criu-3.12-2.el7.x86_64\n---> Package fuse-overlayfs.x86_64 0:0.7.2-6.el7_8 will be installed\n--> Processing Dependency: libfuse3.so.3(FUSE_3.2)(64bit) for package: fuse-overlayfs-0.7.2-6.el7_8.x86_64\n--> Processing Dependency: libfuse3.so.3(FUSE_3.0)(64bit) for package: fuse-overlayfs-0.7.2-6.el7_8.x86_64\n--> Processing Dependency: libfuse3.so.3()(64bit) for package: fuse-overlayfs-0.7.2-6.el7_8.x86_64\n---> Package python-slip.noarch 0:0.4.0-4.el7 will be installed\n---> Package slirp4netns.x86_64 0:0.4.3-4.el7_8 will be installed\n--> Running transaction check\n---> Package fuse3-libs.x86_64 0:3.6.1-4.el7 will be installed\n---> Package libnet.x86_64 0:1.1.6-7.el7 will be installed\n---> Package protobuf-c.x86_64 0:1.0.2-3.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package           Arch   Version           Repository                     Size\n================================================================================\nInstalling:\n atomic            x86_64 1:1.22.1-33.gitb507039.el7_8\n                                            oso-rhui-rhel-server-extras   917 k\n skopeo            x86_64 1:0.1.40-11.el7_8 oso-rhui-rhel-server-extras   5.8 M\nInstalling for dependencies:\n criu              x86_64 3.12-2.el7        oso-rhui-rhel-server-releases 453 k\n fuse-overlayfs    x86_64 0.7.2-6.el7_8     oso-rhui-rhel-server-extras    55 k\n fuse3-libs        x86_64 3.6.1-4.el7       oso-rhui-rhel-server-extras    82 k\n gomtree           x86_64 0.5.0-0.2.git16da0f8.el7\n                                            oso-rhui-rhel-server-extras   667 k\n libnet            x86_64 1.1.6-7.el7       oso-rhui-rhel-server-releases  59 k\n ostree            x86_64 2019.1-2.el7      oso-rhui-rhel-server-extras   513 k\n protobuf-c        x86_64 1.0.2-3.el7       oso-rhui-rhel-server-releases  28 k\n python-slip       noarch 0.4.0-4.el7       oso-rhui-rhel-server-releases  31 k\n python-slip-dbus  noarch 0.4.0-4.el7       oso-rhui-rhel-server-releases  32 k\n runc              x86_64 1.0.0-67.rc10.el7_8\n                                            oso-rhui-rhel-server-extras   2.7 M\n slirp4netns       x86_64 0.4.3-4.el7_8     oso-rhui-rhel-server-extras    82 k\nUpdating for dependencies:\n containers-common x86_64 1:0.1.40-11.el7_8 oso-rhui-rhel-server-extras    43 k\n\nTransaction Summary\n================================================================================\nInstall  2 Packages (+11 Dependent packages)\nUpgrade             (  1 Dependent package)\n\nTotal download size: 11 M\nDownloading packages:\nNo Presto metadata available for oso-rhui-rhel-server-extras\n--------------------------------------------------------------------------------\nTotal                                               17 MB/s |  11 MB  00:00     \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n  Installing : ostree-2019.1-2.el7.x86_64                                  1/15 \n  Installing : libnet-1.1.6-7.el7.x86_64                                   2/15 \n  Installing : python-slip-0.4.0-4.el7.noarch                              3/15 \n  Installing : python-slip-dbus-0.4.0-4.el7.noarch                         4/15 \n  Installing : slirp4netns-0.4.3-4.el7_8.x86_64                            5/15 \n  Installing : fuse3-libs-3.6.1-4.el7.x86_64                               6/15 \n  Installing : fuse-overlayfs-0.7.2-6.el7_8.x86_64                         7/15 \n  Updating   : 1:containers-common-0.1.40-11.el7_8.x86_64                  8/15 \n  Installing : 1:skopeo-0.1.40-11.el7_8.x86_64                             9/15 \n  Installing : gomtree-0.5.0-0.2.git16da0f8.el7.x86_64                    10/15 \n  Installing : protobuf-c-1.0.2-3.el7.x86_64                              11/15 \n  Installing : criu-3.12-2.el7.x86_64                                     12/15 \n  Installing : runc-1.0.0-67.rc10.el7_8.x86_64                            13/15 \n  Installing : 1:atomic-1.22.1-33.gitb507039.el7_8.x86_64                 14/15 \n  Cleanup    : 1:containers-common-0.1.37-3.el7.x86_64                    15/15 \n  Verifying  : protobuf-c-1.0.2-3.el7.x86_64                               1/15 \n  Verifying  : gomtree-0.5.0-0.2.git16da0f8.el7.x86_64                     2/15 \n  Verifying  : runc-1.0.0-67.rc10.el7_8.x86_64                             3/15 \n  Verifying  : 1:skopeo-0.1.40-11.el7_8.x86_64                             4/15 \n  Verifying  : fuse3-libs-3.6.1-4.el7.x86_64                               5/15 \n  Verifying  : fuse-overlayfs-0.7.2-6.el7_8.x86_64                         6/15 \n  Verifying  : slirp4netns-0.4.3-4.el7_8.x86_64                            7/15 \n  Verifying  : python-slip-0.4.0-4.el7.noarch                              8/15 \n  Verifying  : criu-3.12-2.el7.x86_64                                      9/15 \n  Verifying  : 1:atomic-1.22.1-33.gitb507039.el7_8.x86_64                 10/15 \n  Verifying  : ostree-2019.1-2.el7.x86_64                                 11/15 \n  Verifying  : libnet-1.1.6-7.el7.x86_64                                  12/15 \n  Verifying  : 1:containers-common-0.1.40-11.el7_8.x86_64                 13/15 \n  Verifying  : python-slip-dbus-0.4.0-4.el7.noarch                        14/15 \n  Verifying  : 1:containers-common-0.1.37-3.el7.x86_64                    15/15 \n\nInstalled:\n  atomic.x86_64 1:1.22.1-33.gitb507039.el7_8   skopeo.x86_64 1:0.1.40-11.el7_8  \n\nDependency Installed:\n  criu.x86_64 0:3.12-2.el7                                                      \n  fuse-overlayfs.x86_64 0:0.7.2-6.el7_8                                         \n  fuse3-libs.x86_64 0:3.6.1-4.el7                                               \n  gomtree.x86_64 0:0.5.0-0.2.git16da0f8.el7                                     \n  libnet.x86_64 0:1.1.6-7.el7                                                   \n  ostree.x86_64 0:2019.1-2.el7                                                  \n  protobuf-c.x86_64 0:1.0.2-3.el7                                               \n  python-slip.noarch 0:0.4.0-4.el7                                              \n  python-slip-dbus.noarch 0:0.4.0-4.el7                                         \n  runc.x86_64 0:1.0.0-67.rc10.el7_8                                             \n  slirp4netns.x86_64 0:0.4.3-4.el7_8                                            \n\nDependency Updated:\n  containers-common.x86_64 1:0.1.40-11.el7_8                                    \n\nComplete!\n"

[1] https://github.com/openshift/origin/pull/25616
[2] https://storage.googleapis.com/origin-ci-test/pr-logs/pull/25661/test_pull_request_origin_extended_builds_311/202/build-log.txt

Comment 17 Russell Teague 2020-11-12 20:47:38 UTC
As of earlier today, a new AMI has been built (ami-085e3add674833626) that contains docker-1.13.1-203.git0be3e21.el7_9.x86_64.  I kicked off the extended_builds test on the PR[1] and confirmed it is using this AMI.  The job is currently failing because the PR has merge conflicts.  Please resolve the merge conflicts and test again.

[1] https://github.com/openshift/origin/pull/25561

Comment 18 Russell Teague 2020-11-13 20:28:27 UTC
Confirmed the extended builds job now has newer docker installed.

"2:docker-1.13.1-203.git0be3e21.el7_9.x86_64 providing docker is already installed"

Verified in job https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_extended_builds_311/207

Since this is a CI issue, closing.

Note You need to log in before you can comment on or make changes to this bug.