Attachment 1470025 Details for Bug 1579675 – scalegroup nodes are unable to pull image

scalegroup nodes are unable to pull image

scalegroup_nodes_pull_failed.txt (text/plain), 17.05 KB, created by Stefanie Forrester on 2018-07-23 18:29:04 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Stefanie Forrester

Created: 2018-07-23 18:29:04 UTC

Size: 17.05 KB

patch

obsolete

>
>
>[root@free-int-master-3c664 ~]# oc adm new-project dakinitest20180723
>Created project dakinitest20180723
>[root@free-int-master-3c664 ~]# oc new-app --template=laravel-mysql-persistent  -n dakinitest20180723
>--> Deploying template "openshift/laravel-mysql-persistent" to project dakinitest20180723
>
>     Laravel + MySQL (Persistent)
>     ---------
>     An example Laravel application with a MySQL database. For more information about using this template, including OpenShift considerations, see https://github.com/luciddreamz/laravel-ex/blob/master/readme.md.
>
>     The following service(s) have been created in your project: laravel-mysql-persistent, mysql.
>
>     For more information about using this template, including OpenShift considerations, see https://github.com/luciddreamz/laravel-ex/blob/master/readme.md.
>
>     * With parameters:
>        * Name=laravel-mysql-persistent
>        * Namespace=openshift
>        * Git Repository URL=https://github.com/luciddreamz/laravel-ex.git
>        * Git Reference=
>        * Laravel Application Environment=development
>        * Laravel Application Debug Mode=true
>        * OPcache Revalidation Frequency=0
>        * Custom Composer Mirror URL=
>        * Cluster Name=preview
>        * Router Shard=44fs
>        * Memory Limit=512Mi
>        * Memory Limit (MySQL)=512Mi
>        * Volume Capacity=1Gi
>        * Context Directory=
>        * Application Hostname=
>        * GitHub Webhook Secret=QIVspFh0GxJmjKoDMhasjhqm1Sma4NjWPSMEYp2v # generated
>        * Database Service Name=mysql
>        * Database Engine=mysql
>        * Database Name=default
>        * Database User=laravel
>        * Database Password=iQ25J14EEXncCqga # generated
>        * Laravel Encryption Key=J7EPFR7LOpZZbMrm5oYAQy7uhIFnpyfX # generated
>
>--> Creating resources ...
>    secret "laravel-mysql-persistent" created
>    service "laravel-mysql-persistent" created
>    route "laravel-mysql-persistent" created
>    imagestream "laravel-mysql-persistent" created
>    buildconfig "laravel-mysql-persistent" created
>    persistentvolumeclaim "mysql" created
>    deploymentconfig "laravel-mysql-persistent" created
>    service "mysql" created
>    deploymentconfig "mysql" created
>--> Success
>    Access your application via route 'laravel-mysql-persistent-dakinitest20180723.d800.free-int.openshiftapps.com'
>    Build scheduled, use 'oc logs -f bc/laravel-mysql-persistent' to track its progress.
>    Run 'oc status' to view your app.
>[root@free-int-master-3c664 ~]# oc logs -f laravel-mysql-persistent-1-build -n dakinitest20180723
>Error from server (BadRequest): container "sti-build" in pod "laravel-mysql-persistent-1-build" is waiting to start: PodInitializing
>[root@free-int-master-3c664 ~]# oc get pods -n dakinitest20180723
>NAME                               READY     STATUS              RESTARTS   AGE
>laravel-mysql-persistent-1-build   0/1       Init:ErrImagePull   0          18s
>mysql-1-deploy                     0/1       ErrImagePull        0          18s
>[root@free-int-master-3c664 ~]# oc get events -n dakinitest20180723
>LAST SEEN   FIRST SEEN   COUNT     NAME                                                KIND                    SUBOBJECT                        TYPE      REASON                  SOURCE                                   MESSAGE
>28s         28s          1         laravel-mysql-persistent-1-build.1544127193b71cc3   Pod                                                      Normal    Scheduled               default-scheduler                        Successfully assigned laravel-mysql-persistent-1-build to ip-172-31-59-112.ec2.internal
>27s         27s          1         laravel-mysql-persistent-1-build.15441271fbb7ead1   Pod                     spec.initContainers{git-clone}   Normal    Pulling                 kubelet, ip-172-31-59-112.ec2.internal   pulling image "registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder:v3.10.15"
>26s         26s          1         laravel-mysql-persistent-1-build.154412720262cd0c   Pod                     spec.initContainers{git-clone}   Warning   Failed                  kubelet, ip-172-31-59-112.ec2.internal   Failed to pull image "registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder:v3.10.15": rpc error: code = Unknown desc = unable to retrieve auth token: invalid username/password
>26s         26s          1         laravel-mysql-persistent-1-build.15441272026334ab   Pod                     spec.initContainers{git-clone}   Warning   Failed                  kubelet, ip-172-31-59-112.ec2.internal   Error: ErrImagePull
>2s          25s          8         laravel-mysql-persistent-1-build.15441272500fe3a1   Pod                                                      Normal    SandboxChanged          kubelet, ip-172-31-59-112.ec2.internal   Pod sandbox changed, it will be killed and re-created.
>28s         28s          1         mysql-1-deploy.154412719449b451                     Pod                                                      Normal    Scheduled               default-scheduler                        Successfully assigned mysql-1-deploy to ip-172-31-58-173.ec2.internal
>26s         26s          1         mysql-1-deploy.154412722cf3998e                     Pod                     spec.containers{deployment}      Normal    Pulling                 kubelet, ip-172-31-58-173.ec2.internal   pulling image "registry.reg-aws.openshift.com:443/openshift3/ose-deployer:v3.10.15"
>26s         26s          1         mysql-1-deploy.154412722fa2966f                     Pod                     spec.containers{deployment}      Warning   Failed                  kubelet, ip-172-31-58-173.ec2.internal   Failed to pull image "registry.reg-aws.openshift.com:443/openshift3/ose-deployer:v3.10.15": rpc error: code = Unknown desc = unable to retrieve auth token: invalid username/password
>26s         26s          1         mysql-1-deploy.154412722fa2f4b0                     Pod                     spec.containers{deployment}      Warning   Failed                  kubelet, ip-172-31-58-173.ec2.internal   Error: ErrImagePull
>1s          25s          8         mysql-1-deploy.154412726601d0e7                     Pod                                                      Normal    SandboxChanged          kubelet, ip-172-31-58-173.ec2.internal   Pod sandbox changed, it will be killed and re-created.
>28s         28s          1         mysql.1544127189834c2b                              DeploymentConfig                                         Normal    DeploymentCreated       deploymentconfig-controller              Created new replication controller "mysql-1" for version 1
>28s         28s          1         mysql.154412719ab00c19                              PersistentVolumeClaim                                    Normal    ProvisioningSucceeded   persistentvolume-controller              Successfully provisioned volume pvc-21665951-8ea5-11e8-b23c-0a0ddbb52ad4 using kubernetes.io/aws-ebs
>[root@free-int-master-3c664 ~]#
>
>
>
>
>[sedgar@bastion-nasa-2 openshift-ansible-ops (prod $%>)]$ opssh -i -c free-int "docker --config=/var/lib/origin/.docker pull registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder:v3.10.15"
>[1] 18:23:14 [FAILURE] free-int-compute-ip-172-31-53-77.ec2.internal Exited with error code 255
>Stderr: Warning: Permanently added 'free-int-compute-ip-172-31-53-77.ec2.internal,34.228.157.90' (ECDSA) to the list of known hosts.
>Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
>[2] 18:23:14 [FAILURE] free-int-compute-ip-172-31-55-28.ec2.internal Exited with error code 255
>Stderr: Warning: Permanently added 'free-int-compute-ip-172-31-55-28.ec2.internal,54.160.29.6' (ECDSA) to the list of known hosts.
>Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
>[3] 18:23:14 [FAILURE] free-int-compute-ip-172-31-56-28.ec2.internal Exited with error code 255
>Stderr: Warning: Permanently added 'free-int-compute-ip-172-31-56-28.ec2.internal,34.204.13.3' (ECDSA) to the list of known hosts.
>Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
>[4] 18:23:14 [FAILURE] free-int-compute-ip-172-31-59-112.ec2.internal Exited with error code 1
>Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder ...
>Stderr: Warning: Permanently added 'free-int-compute-ip-172-31-59-112.ec2.internal,34.228.168.106' (ECDSA) to the list of known hosts.
>Get https://registry.reg-aws.openshift.com:443/v2/openshift3/ose-docker-builder/manifests/v3.10.15: unauthorized: authentication required
>[5] 18:23:15 [FAILURE] free-int-compute-ip-172-31-58-173.ec2.internal Exited with error code 1
>Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder ...
>Stderr: Warning: Permanently added 'free-int-compute-ip-172-31-58-173.ec2.internal,54.209.192.34' (ECDSA) to the list of known hosts.
>Get https://registry.reg-aws.openshift.com:443/v2/openshift3/ose-docker-builder/manifests/v3.10.15: unauthorized: authentication required
>[6] 18:23:15 [SUCCESS] free-int-node-compute-c1f05
>Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder ...
>v3.10.15: Pulling from registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder
>Digest: sha256:88dba9c1daaad9fbbbb34b23aceaf4645c23f2679b1b64b05a1a045cddc5e86c
>Status: Image is up to date for registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder:v3.10.15
>[7] 18:23:15 [SUCCESS] free-int-node-compute-1f8a0
>Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder ...
>v3.10.15: Pulling from registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder
>Digest: sha256:88dba9c1daaad9fbbbb34b23aceaf4645c23f2679b1b64b05a1a045cddc5e86c
>Status: Image is up to date for registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder:v3.10.15
>[8] 18:23:15 [SUCCESS] free-int-node-compute-b67eb
>Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder ...
>v3.10.15: Pulling from registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder
>Digest: sha256:88dba9c1daaad9fbbbb34b23aceaf4645c23f2679b1b64b05a1a045cddc5e86c
>Status: Image is up to date for registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder:v3.10.15
>[9] 18:23:15 [SUCCESS] free-int-node-compute-980ea
>Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder ...
>v3.10.15: Pulling from registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder
>Digest: sha256:88dba9c1daaad9fbbbb34b23aceaf4645c23f2679b1b64b05a1a045cddc5e86c
>Status: Image is up to date for registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder:v3.10.15
>[10] 18:23:15 [SUCCESS] free-int-node-compute-f0f4d
>Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder ...
>v3.10.15: Pulling from registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder
>Digest: sha256:88dba9c1daaad9fbbbb34b23aceaf4645c23f2679b1b64b05a1a045cddc5e86c
>Status: Image is up to date for registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder:v3.10.15
>[11] 18:23:15 [SUCCESS] free-int-node-compute-99204
>Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder ...
>v3.10.15: Pulling from registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder
>Digest: sha256:88dba9c1daaad9fbbbb34b23aceaf4645c23f2679b1b64b05a1a045cddc5e86c
>Status: Image is up to date for registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder:v3.10.15
>[12] 18:23:15 [SUCCESS] free-int-node-compute-30bae
>Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder ...
>v3.10.15: Pulling from registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder
>Digest: sha256:88dba9c1daaad9fbbbb34b23aceaf4645c23f2679b1b64b05a1a045cddc5e86c
>Status: Image is up to date for registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder:v3.10.15
>[13] 18:23:15 [SUCCESS] free-int-node-compute-117fe
>Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder ...
>v3.10.15: Pulling from registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder
>Digest: sha256:88dba9c1daaad9fbbbb34b23aceaf4645c23f2679b1b64b05a1a045cddc5e86c
>Status: Image is up to date for registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder:v3.10.15
>[14] 18:23:17 [SUCCESS] free-int-master-5470f
>Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder ...
>v3.10.15: Pulling from registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder
>d02c3bd49e78: Already exists
>475b0168c252: Already exists
>65895c0019c4: Already exists
>a7d9185f67df: Already exists
>3f00cef66704: Pulling fs layer
>3f00cef66704: Verifying Checksum
>3f00cef66704: Download complete
>3f00cef66704: Pull complete
>Digest: sha256:88dba9c1daaad9fbbbb34b23aceaf4645c23f2679b1b64b05a1a045cddc5e86c
>Status: Downloaded newer image for registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder:v3.10.15
>[15] 18:23:17 [SUCCESS] free-int-node-compute-15bd6
>Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder ...
>v3.10.15: Pulling from registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder
>d02c3bd49e78: Already exists
>475b0168c252: Already exists
>65895c0019c4: Already exists
>a7d9185f67df: Already exists
>3f00cef66704: Pulling fs layer
>3f00cef66704: Verifying Checksum
>3f00cef66704: Download complete
>3f00cef66704: Pull complete
>Digest: sha256:88dba9c1daaad9fbbbb34b23aceaf4645c23f2679b1b64b05a1a045cddc5e86c
>Status: Downloaded newer image for registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder:v3.10.15
>[16] 18:23:17 [SUCCESS] free-int-master-de987
>Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder ...
>v3.10.15: Pulling from registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder
>d02c3bd49e78: Already exists
>475b0168c252: Already exists
>65895c0019c4: Already exists
>a7d9185f67df: Already exists
>3f00cef66704: Pulling fs layer
>3f00cef66704: Verifying Checksum
>3f00cef66704: Download complete
>3f00cef66704: Pull complete
>Digest: sha256:88dba9c1daaad9fbbbb34b23aceaf4645c23f2679b1b64b05a1a045cddc5e86c
>Status: Downloaded newer image for registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder:v3.10.15
>[17] 18:23:17 [SUCCESS] free-int-master-3c664
>Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder ...
>v3.10.15: Pulling from registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder
>d02c3bd49e78: Already exists
>475b0168c252: Already exists
>65895c0019c4: Already exists
>a7d9185f67df: Already exists
>3f00cef66704: Pulling fs layer
>3f00cef66704: Verifying Checksum
>3f00cef66704: Download complete
>3f00cef66704: Pull complete
>Digest: sha256:88dba9c1daaad9fbbbb34b23aceaf4645c23f2679b1b64b05a1a045cddc5e86c
>Status: Downloaded newer image for registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder:v3.10.15
>[18] 18:23:17 [SUCCESS] free-int-node-infra-70a2b
>Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder ...
>v3.10.15: Pulling from registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder
>d02c3bd49e78: Already exists
>475b0168c252: Already exists
>65895c0019c4: Already exists
>a7d9185f67df: Already exists
>3f00cef66704: Pulling fs layer
>3f00cef66704: Verifying Checksum
>3f00cef66704: Download complete
>3f00cef66704: Pull complete
>Digest: sha256:88dba9c1daaad9fbbbb34b23aceaf4645c23f2679b1b64b05a1a045cddc5e86c
>Status: Downloaded newer image for registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder:v3.10.15
>[19] 18:23:18 [SUCCESS] free-int-node-infra-a5950
>Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder ...
>v3.10.15: Pulling from registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder
>d02c3bd49e78: Already exists
>475b0168c252: Already exists
>65895c0019c4: Already exists
>a7d9185f67df: Already exists
>3f00cef66704: Pulling fs layer
>3f00cef66704: Verifying Checksum
>3f00cef66704: Download complete
>3f00cef66704: Pull complete
>Digest: sha256:88dba9c1daaad9fbbbb34b23aceaf4645c23f2679b1b64b05a1a045cddc5e86c
>Status: Downloaded newer image for registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder:v3.10.15
>[20] 18:23:21 [SUCCESS] free-int-node-infra-bcf5a
>Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder ...
>v3.10.15: Pulling from registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder
>d02c3bd49e78: Already exists
>475b0168c252: Already exists
>65895c0019c4: Already exists
>a7d9185f67df: Already exists
>3f00cef66704: Pulling fs layer
>3f00cef66704: Verifying Checksum
>3f00cef66704: Download complete
>3f00cef66704: Pull complete
>Digest: sha256:88dba9c1daaad9fbbbb34b23aceaf4645c23f2679b1b64b05a1a045cddc5e86c
>Status: Downloaded newer image for registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder:v3.10.15
>[21] 18:23:25 [FAILURE] free-int-compute-ip-172-31-53-103.ec2.internal Exited with error code 1
>Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose-docker-builder ...
>Stderr: Warning: Permanently added 'free-int-compute-ip-172-31-53-103.ec2.internal,54.152.137.126' (ECDSA) to the list of known hosts.
>Get https://registry.reg-aws.openshift.com:443/v2/openshift3/ose-docker-builder/manifests/v3.10.15: unauthorized: authentication required
>[sedgar@bastion-nasa-2 openshift-ansible-ops (prod $%>)]$

Actions: View

Attachments on bug 1579675: 1470025