Description of problem: When enabling metrics deployment, and using an local docker registry for installation base, the IMAGE_PREFIX isn't set to reflect the additional (new) registry. As a result, the pull is attempted from registry.access.redhat.com regardless of parameter. Error: ======== ERROR ========= validate_deployment_artifacts: Pod hawkular-cassandra-1-z720w from ReplicationController hawkular-cassandra-1 is in a Pending state. This is most often due to waiting for the container image to pull and should eventually resolve. * * * * Pod hawkular-metrics-3o7p6 from ReplicationController hawkular-metrics is in a Pending state. This is most often due to waiting for the container image to pull and should eventually resolve. * * * * Pod heapster-rhs8v from ReplicationController heapster specified an image that cannot be pulled. ERROR: This is most often due to the image name being wrong or the docker registry being unavailable. Ensure that you used the correct IMAGE_PREFIX and IMAGE_VERSION with the deployment. There was an event for this pod with the following message: Failed to pull image "registry.access.redhat.com/openshift3/metrics-heapster:3.4.0": image pull failed for registry.access.redhat.com/openshift3/metrics-heapster:3.4.0, this may be because there are no credentials on this request. details: (net/http: request canceled) Version-Release number of selected component (if applicable): 3.4 How reproducible: Every time Steps to Reproduce: 1. Install with openshift_hosted_metrics_deploy=true, and with a custom registry 2. Deployment of metrics fails - ansible does not fail, but deployment ends with failure (see above). Expected results: Metrics installation run should set IMAGE_PREFIX to reflect the additional_registry setting. Or there needs to be a parameter to set this value (registry/logging should follow the same setup). Additional info:
The templates are updated when a combination of oreg_url and openshift_examples_modify_imagestreams are set. Can you please provide your full inventory?
Created attachment 1245918 [details] Anisble inventory used for install The requested ansible inventory used to install from.
(In reply to Scott Dodson from comment #1) > The templates are updated when a combination of oreg_url and > openshift_examples_modify_imagestreams are set. Can you please provide your > full inventory? I've uploaded the inventory file, and here are the settings relevant for this issue: openshift_docker_additional_registries=satellite.peterlarsen.org:5000 openshift_docker_insecure=satellite.peterlarsen.org openshift_docker_disable_push_dockerhub=True oreg_url=peterlarsen_org-ocp_docker_registry-ose-${component}:${version}
Forgot this line: openshift_examples_modify_imagestreams=true
(In reply to Peter Larsen from comment #3) > (In reply to Scott Dodson from comment #1) > > openshift_docker_additional_registries=satellite.peterlarsen.org:5000 > openshift_docker_insecure=satellite.peterlarsen.org > openshift_docker_disable_push_dockerhub=True > oreg_url=peterlarsen_org-ocp_docker_registry-ose-${component}:${version} These work for me, I raised an upstream ticket (https://github.com/openshift/openshift-ansible/issues/3263) about the missing openshift_hosted_metrics_deployer_version example from the reference file: openshift_docker_additional_registries=registry.example.com:5000 openshift_docker_insecure_registries=registry.example.com:5000 openshift_docker_disable_push_dockerhub=True oreg_url=registry.example.com:5000/openshift3/ose-${component}:${version} openshift_examples_modify_imagestreams=true openshift_hosted_metrics_deployer_version=latest openshift_hosted_logging_deployer_version=latest But please also note https://bugzilla.redhat.com/show_bug.cgi?id=1419493 and https://bugzilla.redhat.com/show_bug.cgi?id=1419497. Thanks.
How is this a metrics issue exactly?
"Metrics installation run should set IMAGE_PREFIX to reflect the additional_registry setting. Or there needs to be a parameter to set this value (registry/logging should follow the same setup)." Can you not just set the "openshift_hosted_metrics_deployer_prefix" value to point to your own registry that you want to use? The default value for this is "registry.access.redhat.com/openshift3/" so you will probably just want to change the 'registry.access.redhat.com' part.
(In reply to Matt Wringe from comment #8) > "Metrics installation run should set IMAGE_PREFIX to reflect the > additional_registry setting. Or there needs to be a parameter to set this > value (registry/logging should follow the same setup)." > > Can you not just set the "openshift_hosted_metrics_deployer_prefix" value to > point to your own registry that you want to use? > > The default value for this is "registry.access.redhat.com/openshift3/" so > you will probably just want to change the 'registry.access.redhat.com' part. I'm not seeing such an option in the inventory file? I see "openshift_hosted_logging_deployer_prefix" nothing for metrics.
(In reply to Peter Larsen from comment #9) > I'm not seeing such an option in the inventory file? I see > "openshift_hosted_logging_deployer_prefix" nothing for metrics. If its not set it will use a default. What happens if you set this value? Is being able to set this value all you are needing here? Or are you expecting that something else should be happening?
(In reply to Matt Wringe from comment #10) > (In reply to Peter Larsen from comment #9) > > I'm not seeing such an option in the inventory file? I see > > "openshift_hosted_logging_deployer_prefix" nothing for metrics. > > If its not set it will use a default. What happens if you set this value? > > Is being able to set this value all you are needing here? Or are you > expecting that something else should be happening? The default is what value? Right now, when I run the setup, logging and everything else is pulled fine from the local repository. Except metrics; This bug is about the missing parameter/variable for metrics, just like we have for logging, so we can do an "offline" install that includes metrics. Note - I can pass the IMAGE_PREFIX manually to the deployer template of metrics. I just cannot get that done through the ansible installer. Or at least it's not documented how to make it work.
See https://bugzilla.redhat.com/show_bug.cgi?id=1417526#c8 for the default Unless specifying that value does not work for you, this issue needs to be closed and another opened against the ansible documentation.
"Right now, when I run the setup, logging and everything else is pulled fine from the local repository. Except metrics;" Are you doing anything special with logging? or are you also setting a prefix type value for it?
(In reply to Matt Wringe from comment #13) > "Right now, when I run the setup, logging and everything else is pulled fine > from the local repository. Except metrics;" > > Are you doing anything special with logging? or are you also setting a > prefix type value for it? I think we have a "slight" miscommunication here. I'm trying to install metrics. From the ansible installer. From an offline repository. Because I don't have a metrics "prefix" parameter, the installer does NOT set IMAGE_PREFIX when it starts the metrics deployer, and hence the installation fails as it cannot find the required images. That's as short and precise I can describe the issue. I'm not sure if this constitute "something special" but it sure is functionality that's not there.
I am not sure how much more clear I can be here. I have asked multiple times if you have set the "openshift_hosted_metrics_deployer_prefix" like what you have to do with logging (https://bugzilla.redhat.com/show_bug.cgi?id=1417526#c8 https://bugzilla.redhat.com/show_bug.cgi?id=1417526#c12) In 3.5 there are a lot of changes to the ansible install, and the documentation does outline a value 'openshift_metrics_image_prefix' which can be used to specify the prefix (see https://github.com/openshift/openshift-docs/pull/3765)
Set to verified since in 3.5.0 user can set the image prefix in the inventory file like openshift_metrics_image_prefix= and run the playbook and it works.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:0884