Description of problem: Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Description of problem: minor update of openstack 15 deployment fails in overcloud update when updating Controller attempts to pull ovn-northd Version-Release number of selected component (if applicable): RHOS_TRUNK-15.0-RHEL-8-20190813.n.0 openstack-heat-agents.noarch 1.8.1-0.20190523210450.1e15344.el8ost @rhelosp-15.0 openstack-heat-api.noarch 1:12.0.1-0.20190805120452.3476f1d.el8ost @rhelosp-15.0 openstack-heat-common.noarch 1:12.0.1-0.20190805120452.3476f1d.el8ost @rhelosp-15.0 openstack-heat-engine.noarch 1:12.0.1-0.20190805120452.3476f1d.el8ost @rhelosp-15.0 openstack-heat-monolith.noarch 1:12.0.1-0.20190805120452.3476f1d.el8ost @rhelosp-15.0 openstack-selinux.noarch 0.8.19-0.20190606150404.06faac7.el8ost @rhelosp-15.0 openstack-tripleo-common.noarch 10.8.1-0.20190808130448.b942458.el8ost @rhelosp-15.0 openstack-tripleo-common-containers.noarch 10.8.1-0.20190808130448.b942458.el8ost @rhelosp-15.0 openstack-tripleo-heat-templates.noarch 10.6.1-0.20190812140519.2a684c0.el8ost @rhelosp-15.0 openstack-tripleo-image-elements.noarch 10.4.1-0.20190705161217.2c8a6a5.el8ost @rhelosp-15.0 openstack-tripleo-puppet-elements.noarch 10.3.2-0.20190710165331.c89fe3c.el8ost @rhelosp-15.0 openstack-tripleo-validations.noarch 10.5.1-0.20190812110443.80a9f7b.el8ost @rhelosp-15.0 puppet-openstack_extras.noarch 14.4.1-0.20190420090934.6b1b687.el8ost @rhelosp-15.0 puppet-openstacklib.noarch 14.4.1-0.20190420125152.3719ca1.el8ost @rhelosp-15.0 python-openstackclient-lang.noarch 3.18.0-0.20190312140834.6868499.el8ost @rhelosp-15.0 python3-openstackclient.noarch 3.18.0-0.20190312140834.6868499.el8ost @rhelosp-15.0 python3-openstacksdk.noarch How reproducible: Steps to Reproduce: 1. Deploy OSP 15 and run minor update process 2. 3. Actual results: 2019-08-14 18:10:30 | fatal: [controller-0]: FAILED! => {"changed": true, "cmd": ["podman", "pull", "192.168.24.1:8787/rhosp15/openstack-ovn-northd:20190812.1"], "delta": "0:00:00.070167", "end": "2019-08-14 18:10:29.867197", "msg": "non-zero return code", "rc": 125, "start": "2019-08-14 18:10:29.797030", "stderr": "Trying to pull 192.168.24.1:8787/rhosp15/openstack-ovn-northd:20190812.1...Failed\nerror pulling image \"192.168.24.1:8787/rhosp15/openstack-ovn-northd:20190812.1\": unable to pull 192.168.24.1:8787/rhosp15/openstack-ovn-northd:20190812.1: unable to pull image: Error determining manifest MIME type for docker://192.168.24.1:8787/rhosp15/openstack-ovn-northd:20190812.1: Error reading manifest 20190812.1 in 192.168.24.1:8787/rhosp15/openstack-ovn-northd: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: \"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\">\\n<html><head>\\n<title>404 Not Found</title>\\n</head><body>\\n<h1>Not Found</h1>\\n<p>The requested URL /v2/rhosp15/openstack-ovn-northd/manifests/20190812.1 was not found on this server.</p>\\n</body></html>\\n\"", "stderr_lines": ["Trying to pull 192.168.24.1:8787/rhosp15/openstack-ovn-northd:20190812.1...Faile -> overall update fails some time later. Expected results: Openstack overcloud update command completes successfully. Additional info: https://rhos-ci-staging-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/hardware_provisioning/view/rqci/job/DFG-hardware_provisioning-rqci-15_director-8.0-virthost-3cont_3ceph_1comp_2ironic-ipv4-geneve-minor_update-RHELOSP-48859/16/
Created attachment 1603896 [details] controller update log
Created attachment 1603897 [details] overcloud minor update stage log
I am also hitting this issue 019-08-15 21:13:14.383786", "msg": "non-zero return code", "rc": 125, "start": "2019-08-15 21:13:13.921758", "stderr": "Trying to pull registry.access.redhat.com/rhosp15/openstack-ovn-northd:15.0...Failed\nerror pulling image \"registry.access.redhat.com/rhosp15/openstack-ovn-northd:15.0\": unable to pull registry.access.redhat.com/rhosp15/openstack-ovn-northd:15.0: unable to pull image: Error determining manifest MIME type for docker://registry.access.redhat.com/rhosp15/openstack-ovn-northd:15.0: Error reading manifest 15.0 in registry.access.redhat.com/rhosp15/openstack-ovn-northd: name unknown: Repo not found", "stderr_lines": ["Trying to pull registry.access.redhat.com/rhosp15/openstack-ovn-northd:15.0...Failed", "error pulling image \"registry.access.redhat.com/rhosp15/openstack-ovn-northd:15.0\": unable to pull registry.access.redhat.com/rhosp15/openstack-ovn-northd:15.0: unable to pull image: Error determining manifest MIME type for docker://registry.access.redhat.com/rhosp15/openstack-ovn-northd:15.0: Error reading manifest 15.0 in registry.access.redhat.com/rhosp15/openstack-ovn-northd: name unknown: Repo not found"], "stdout": "", "stdout_lines": []}
From comment #5 it would look like the job might not be picking up the right parameters in the container image prepare step or whatever it has become now. It is defaulting to general pullspec for the container images registry.access.redhat.com/rhosp15/XXXX which is not available yet as OSP 15 is not GA, so you would either need to be pointing to beta content or internal registry for the content. By providing the right parameters to the prepare stage of things for registry to fetch images from.
From the provided logs it appears the issue is with the registry URL. I'm closing this bug as there is nothing wrong with OVN and sounds more like a job misconfiguration issue as pointed out in the comment 7.