Version: 3.11.z Platform: aws Please specify: ansible-install (3.11 CI) What happened? Ansible installer allows docker v1.13.1-103. This includes a bug in how symlinks are treated, which breaks OpenShift builds. What did you expect to happen? Docker v1.13.1-104 or higher should be enforced. How to reproduce it (as minimally and precisely as possible)? Run 3.11 extended_builds suite (via Jenkins) Anything else we need to know? See https://prow.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/25561/test_pull_request_origin_extended_builds_311/172/
Is someone working on this issue? At the moment it is blocking 3.11 PRs. For example, we can't merge https://github.com/openshift/origin/pull/25009 because some unit tests are failing. We can't fix unit tests https://github.com/openshift/origin/pull/25561 because "extended_builds" job fails.
I'm seeing docker.x86_64 2:1.13.1-162.git64e9980.el7_8 being installed in https://storage.googleapis.com/origin-ci-test/pr-logs/pull/25561/pull-ci-openshift-origin-release-3.11-e2e-gcp/1311358965027704832/build-log.txt Can you link to where docker v1.13.1-103 is being installed?
This is one run in which I see docker 1.13.1-103 being installed. https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/25561/test_pull_request_origin_extended_builds_311/174/build-log.txt These are the messages in installation logs that show 103 is installed. TASK [container_runtime : Install Docker] ************************************** task path: /usr/share/ansible/openshift-ansible/roles/container_runtime/tasks/package_docker.yml:23 changed: [localhost] => { "attempts": 1, "changed": true, "generated_timestamp": "2020-09-30 18:06:38.368471", "msg": "", "rc": 0, "results": [ "2:docker-1.13.1-103.git7f2769b.el7.x86_64 providing docker is already installed",
The AMI which is being used for this job has not been updated in nearly a year. It has an old version of docker installed and openshift-ansible does not force upgrades if docker is already installed. The image base image the job is using should be updated. I do not know who supports the oct/playbooks/provision/aws-up.yml playbook being used to provision the AWS instances. "item": { "ami_id": "ami-01a9f71561742665f", "architecture": "x86_64", "block_device_mapping": { "/dev/sda1": { "delete_on_termination": true, "encrypted": false, "size": 75, "snapshot_id": "snap-0bace4ea1d7a4225e", "volume_type": "gp2" }, "/dev/sdb": { "delete_on_termination": true, "encrypted": false, "size": 50, "snapshot_id": "snap-0138bfa2d206e5160", "volume_type": "gp2" } }, "creationDate": "2019-10-18T15:43:05.000Z", "description": "ami_build_origin_int_rhel_base_2162", "hypervisor": "xen", "is_public": false, "location": "531415883065/ami_build_origin_int_rhel_base_2162", "name": "ami_build_origin_int_rhel_base_2162", "owner_id": "531415883065", "platform": null, "root_device_name": "/dev/sda1", "root_device_type": "ebs", "state": "available", "tags": { "Name": "ami_build_origin_int_rhel_base_2162", "image_stage": "base", "operating_system": "rhel", "ready": "yes" },
Any update on this? I'm initiating an effort to get 3.11 built with golang 1.15 and without this being fixed we can't merge anything to origin's release-3.11 branch without overriding the unit test job.
Interesting, well the whole reason it's failing is because the repo in which we're installing golang 1.9.2 info the AMI went away. If we're no longer going to be installing golang 1.9.2 and you'll presumably have to find another repo to retrieve 1.15 from then at least that problem goes away. Are you going to be updating the AMI build?
(In reply to Scott Dodson from comment #8) > Interesting, well the whole reason it's failing is because the repo in which > we're installing golang 1.9.2 info the AMI went away. If we're no longer > going to be installing golang 1.9.2 and you'll presumably have to find > another repo to retrieve 1.15 from then at least that problem goes away. Are > you going to be updating the AMI build? I'm still coming up to speed with what is involved in building and testing 3.11. The goal is to clarify what risks (if any) are involved in switching to golang 1.15. That involves getting CI clusters with golang 1.15-built artifacts built and tested in CI. As far as I know we'll be building with ci containers that are built by downloading upstream golang binaries, e.g. https://github.com/openshift/release/blob/master/projects/origin-release/golang-1.15/Dockerfile Only once we'd successfully vetted 1.15-built 3.11 in CI would we be have to worry about what's involved in releasing 1.15-built 3.11 artifacts. Would that be when AMIs become relevant, or am I missing something?
*** Bug 1890328 has been marked as a duplicate of this bug. ***
Maru, is this bug primarily about fixing the AMI. Should you own this?
(In reply to Brenton Leanhardt from comment #11) > Maru, is this bug primarily about fixing the AMI. Should you own this? I should definitely not own this. I'm tasked with enabling testing of 3.11 built with golang 1.15, and lack the background necessary to understand why AMIs are even required for 3.11 testing.
This may help unblock the AMI build. https://github.com/openshift/aos-cd-jobs/pull/2413
Looks like if we can get the AMI build job working it should have working docker version installed. From the logs: Package docker.x86_64 2:1.13.1-203.git0be3e21.el7_9 will be installed
It looks like the extended_builds job is passing, and was passing before we updated the AMI. https://prow.ci.openshift.org/job-history/gs/origin-ci-test/pr-logs/directory/test_pull_request_origin_extended_builds_311 Can someone verify https://github.com/openshift/origin/pull/25561 from comment 1?
In the interim several updates were made to get the extended_build tests on 3.11 passing. We are currently skipping the secrets test [1] - as a part of fixing this bug the secrets test should be re-enabled. Looking at the latest run on CI, we are still using the old version of docker [2]: ``` TASK [container_runtime : Install Docker] ************************************** task path: /usr/share/ansible/openshift-ansible/roles/container_runtime/tasks/package_docker.yml:23 changed: [localhost] => { "attempts": 1, "changed": true, "generated_timestamp": "2020-11-09 14:02:53.517859", "msg": "", "rc": 0, "results": [ "2:docker-1.13.1-103.git7f2769b.el7.x86_64 providing docker is already installed", "Loaded plugins: amazon-id, rhui-lb, search-disabled-repos\nResolving Dependencies\n--> Running transaction check\n---> Package atomic.x86_64 1:1.22.1-33.gitb507039.el7_8 will be installed\n--> Processing Dependency: runc for package: 1:atomic-1.22.1-33.gitb507039.el7_8.x86_64\n--> Processing Dependency: python-slip-dbus for package: 1:atomic-1.22.1-33.gitb507039.el7_8.x86_64\n--> Processing Dependency: ostree for package: 1:atomic-1.22.1-33.gitb507039.el7_8.x86_64\n--> Processing Dependency: gomtree for package: 1:atomic-1.22.1-33.gitb507039.el7_8.x86_64\n---> Package skopeo.x86_64 1:0.1.40-11.el7_8 will be installed\n--> Processing Dependency: containers-common = 1:0.1.40-11.el7_8 for package: 1:skopeo-0.1.40-11.el7_8.x86_64\n--> Running transaction check\n---> Package containers-common.x86_64 1:0.1.37-3.el7 will be updated\n---> Package containers-common.x86_64 1:0.1.40-11.el7_8 will be an update\n--> Processing Dependency: slirp4netns for package: 1:containers-common-0.1.40-11.el7_8.x86_64\n--> Processing Dependency: fuse-overlayfs for package: 1:containers-common-0.1.40-11.el7_8.x86_64\n---> Package gomtree.x86_64 0:0.5.0-0.2.git16da0f8.el7 will be installed\n---> Package ostree.x86_64 0:2019.1-2.el7 will be installed\n---> Package python-slip-dbus.noarch 0:0.4.0-4.el7 will be installed\n--> Processing Dependency: python-slip = 0.4.0-4.el7 for package: python-slip-dbus-0.4.0-4.el7.noarch\n---> Package runc.x86_64 0:1.0.0-67.rc10.el7_8 will be installed\n--> Processing Dependency: criu for package: runc-1.0.0-67.rc10.el7_8.x86_64\n--> Running transaction check\n---> Package criu.x86_64 0:3.12-2.el7 will be installed\n--> Processing Dependency: libprotobuf-c.so.1(LIBPROTOBUF_C_1.0.0)(64bit) for package: criu-3.12-2.el7.x86_64\n--> Processing Dependency: libprotobuf-c.so.1()(64bit) for package: criu-3.12-2.el7.x86_64\n--> Processing Dependency: libnet.so.1()(64bit) for package: criu-3.12-2.el7.x86_64\n---> Package fuse-overlayfs.x86_64 0:0.7.2-6.el7_8 will be installed\n--> Processing Dependency: libfuse3.so.3(FUSE_3.2)(64bit) for package: fuse-overlayfs-0.7.2-6.el7_8.x86_64\n--> Processing Dependency: libfuse3.so.3(FUSE_3.0)(64bit) for package: fuse-overlayfs-0.7.2-6.el7_8.x86_64\n--> Processing Dependency: libfuse3.so.3()(64bit) for package: fuse-overlayfs-0.7.2-6.el7_8.x86_64\n---> Package python-slip.noarch 0:0.4.0-4.el7 will be installed\n---> Package slirp4netns.x86_64 0:0.4.3-4.el7_8 will be installed\n--> Running transaction check\n---> Package fuse3-libs.x86_64 0:3.6.1-4.el7 will be installed\n---> Package libnet.x86_64 0:1.1.6-7.el7 will be installed\n---> Package protobuf-c.x86_64 0:1.0.2-3.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n atomic x86_64 1:1.22.1-33.gitb507039.el7_8\n oso-rhui-rhel-server-extras 917 k\n skopeo x86_64 1:0.1.40-11.el7_8 oso-rhui-rhel-server-extras 5.8 M\nInstalling for dependencies:\n criu x86_64 3.12-2.el7 oso-rhui-rhel-server-releases 453 k\n fuse-overlayfs x86_64 0.7.2-6.el7_8 oso-rhui-rhel-server-extras 55 k\n fuse3-libs x86_64 3.6.1-4.el7 oso-rhui-rhel-server-extras 82 k\n gomtree x86_64 0.5.0-0.2.git16da0f8.el7\n oso-rhui-rhel-server-extras 667 k\n libnet x86_64 1.1.6-7.el7 oso-rhui-rhel-server-releases 59 k\n ostree x86_64 2019.1-2.el7 oso-rhui-rhel-server-extras 513 k\n protobuf-c x86_64 1.0.2-3.el7 oso-rhui-rhel-server-releases 28 k\n python-slip noarch 0.4.0-4.el7 oso-rhui-rhel-server-releases 31 k\n python-slip-dbus noarch 0.4.0-4.el7 oso-rhui-rhel-server-releases 32 k\n runc x86_64 1.0.0-67.rc10.el7_8\n oso-rhui-rhel-server-extras 2.7 M\n slirp4netns x86_64 0.4.3-4.el7_8 oso-rhui-rhel-server-extras 82 k\nUpdating for dependencies:\n containers-common x86_64 1:0.1.40-11.el7_8 oso-rhui-rhel-server-extras 43 k\n\nTransaction Summary\n================================================================================\nInstall 2 Packages (+11 Dependent packages)\nUpgrade ( 1 Dependent package)\n\nTotal download size: 11 M\nDownloading packages:\nNo Presto metadata available for oso-rhui-rhel-server-extras\n--------------------------------------------------------------------------------\nTotal 17 MB/s | 11 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : ostree-2019.1-2.el7.x86_64 1/15 \n Installing : libnet-1.1.6-7.el7.x86_64 2/15 \n Installing : python-slip-0.4.0-4.el7.noarch 3/15 \n Installing : python-slip-dbus-0.4.0-4.el7.noarch 4/15 \n Installing : slirp4netns-0.4.3-4.el7_8.x86_64 5/15 \n Installing : fuse3-libs-3.6.1-4.el7.x86_64 6/15 \n Installing : fuse-overlayfs-0.7.2-6.el7_8.x86_64 7/15 \n Updating : 1:containers-common-0.1.40-11.el7_8.x86_64 8/15 \n Installing : 1:skopeo-0.1.40-11.el7_8.x86_64 9/15 \n Installing : gomtree-0.5.0-0.2.git16da0f8.el7.x86_64 10/15 \n Installing : protobuf-c-1.0.2-3.el7.x86_64 11/15 \n Installing : criu-3.12-2.el7.x86_64 12/15 \n Installing : runc-1.0.0-67.rc10.el7_8.x86_64 13/15 \n Installing : 1:atomic-1.22.1-33.gitb507039.el7_8.x86_64 14/15 \n Cleanup : 1:containers-common-0.1.37-3.el7.x86_64 15/15 \n Verifying : protobuf-c-1.0.2-3.el7.x86_64 1/15 \n Verifying : gomtree-0.5.0-0.2.git16da0f8.el7.x86_64 2/15 \n Verifying : runc-1.0.0-67.rc10.el7_8.x86_64 3/15 \n Verifying : 1:skopeo-0.1.40-11.el7_8.x86_64 4/15 \n Verifying : fuse3-libs-3.6.1-4.el7.x86_64 5/15 \n Verifying : fuse-overlayfs-0.7.2-6.el7_8.x86_64 6/15 \n Verifying : slirp4netns-0.4.3-4.el7_8.x86_64 7/15 \n Verifying : python-slip-0.4.0-4.el7.noarch 8/15 \n Verifying : criu-3.12-2.el7.x86_64 9/15 \n Verifying : 1:atomic-1.22.1-33.gitb507039.el7_8.x86_64 10/15 \n Verifying : ostree-2019.1-2.el7.x86_64 11/15 \n Verifying : libnet-1.1.6-7.el7.x86_64 12/15 \n Verifying : 1:containers-common-0.1.40-11.el7_8.x86_64 13/15 \n Verifying : python-slip-dbus-0.4.0-4.el7.noarch 14/15 \n Verifying : 1:containers-common-0.1.37-3.el7.x86_64 15/15 \n\nInstalled:\n atomic.x86_64 1:1.22.1-33.gitb507039.el7_8 skopeo.x86_64 1:0.1.40-11.el7_8 \n\nDependency Installed:\n criu.x86_64 0:3.12-2.el7 \n fuse-overlayfs.x86_64 0:0.7.2-6.el7_8 \n fuse3-libs.x86_64 0:3.6.1-4.el7 \n gomtree.x86_64 0:0.5.0-0.2.git16da0f8.el7 \n libnet.x86_64 0:1.1.6-7.el7 \n ostree.x86_64 0:2019.1-2.el7 \n protobuf-c.x86_64 0:1.0.2-3.el7 \n python-slip.noarch 0:0.4.0-4.el7 \n python-slip-dbus.noarch 0:0.4.0-4.el7 \n runc.x86_64 0:1.0.0-67.rc10.el7_8 \n slirp4netns.x86_64 0:0.4.3-4.el7_8 \n\nDependency Updated:\n containers-common.x86_64 1:0.1.40-11.el7_8 \n\nComplete!\n" ] } ``` [1] https://github.com/openshift/origin/pull/25616 [2] https://storage.googleapis.com/origin-ci-test/pr-logs/pull/25661/test_pull_request_origin_extended_builds_311/202/build-log.txt
As of earlier today, a new AMI has been built (ami-085e3add674833626) that contains docker-1.13.1-203.git0be3e21.el7_9.x86_64. I kicked off the extended_builds test on the PR[1] and confirmed it is using this AMI. The job is currently failing because the PR has merge conflicts. Please resolve the merge conflicts and test again. [1] https://github.com/openshift/origin/pull/25561
Confirmed the extended builds job now has newer docker installed. "2:docker-1.13.1-203.git0be3e21.el7_9.x86_64 providing docker is already installed" Verified in job https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_extended_builds_311/207 Since this is a CI issue, closing.
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days