Jian, Can you please tell me why you create a secret with registry credentials? I'm failing to understand why you need to create that secret if the registry is not secured. Is this an issue with using the route? Or does your registry expect credentials? Maybe I'm missing something here. Are you saying that the created apb-run service account needs an associated secret specifying registry credentials?
Dylan, 1, From the logs, we can see the "ImagePullBackOff" errors when running the deploy pod. And, I noticed that you used the "imagePullSecrets:" mechanism in the deploy pod, so, I replaced the origin secret(apb-run-provision-postgresql-apb-dockercfg-jmshn) with the new secret(my-secret), it works! It could pull the image successfully! 2, The registry is the default cluster registry. In the `apb run` process, we can see the pushing image action works well, but the pulling image works failed. I create a new secret just in order to indicate the root cause of this bug is the origin secret used in "imagePullSecrets" is incorrect.
I confirmed I can recreate this problem when using MiniShift. I believe that it must be related to running on a remote cluster since I did not see it with catasb. Still unsure what the root cause is because the dockercfg secret being created appears fine.
Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 15s default-scheduler Successfully assigned apb-run-provision-mediawiki-apbv6gpx to localhost Normal SuccessfulMountVolume 15s kubelet, localhost MountVolume.SetUp succeeded for volume "apb-run-provision-mediawiki-apb-token-td25f" Normal Pulling 14s kubelet, localhost pulling image "docker-registry-default.192.168.42.232.nip.io/openshift/mediawiki-apb" Warning Failed 14s kubelet, localhost Failed to pull image "docker-registry-default.192.168.42.232.nip.io/openshift/mediawiki-apb": rpc error: code = 2 desc = Error response from daemon: {"message":"Get https://docker-registry-default.192.168.42.232.nip.io/v1/_ping: x509: certificate signed by unknown authority"} Warning Failed 14s kubelet, localhost Error: ErrImagePull Normal BackOff 13s kubelet, localhost Back-off pulling image "docker-registry-default.192.168.42.232.nip.io/openshift/mediawiki-apb" Warning Failed 13s kubelet, localhost Error: ImagePullBackOff
[dymurray@dymurray scripts]$ oc get secret apb-run-provision-mediawiki-apb-dockercfg-7twf5 -o json | jq .data |jq .[".dockercfg"] { ".dockercfg": "eyIxNzIuMzAuMS4xOjUwMDAiOnsidXNlcm5hbWUiOiJzZXJ2aWNlYWNjb3VudCIsInBhc3N3b3JkIjoiZXlKaGJHY2lPaUpTVXpJMU5pSXNJblI1Y0NJNklrcFhWQ0o5LmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUptYjI4aUxDSnJkV0psY201bGRHVnpMbWx2TDNObGNuWnBZMlZoWTJOdmRXNTBMM05sWTNKbGRDNXVZVzFsSWpvaVlYQmlMWEoxYmkxd2NtOTJhWE5wYjI0dGJXVmthV0YzYVd0cExXRndZaTEwYjJ0bGJpMTJZMnh4ZUNJc0ltdDFZbVZ5Ym1WMFpYTXVhVzh2YzJWeWRtbGpaV0ZqWTI5MWJuUXZjMlZ5ZG1salpTMWhZMk52ZFc1MExtNWhiV1VpT2lKaGNHSXRjblZ1TFhCeWIzWnBjMmx2YmkxdFpXUnBZWGRwYTJrdFlYQmlJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpYSjJhV05sTFdGalkyOTFiblF1ZFdsa0lqb2lNV0V3TjJFMk5qQXRNVEF3T0MweE1XVTRMV0V3WkRjdE5tRXhZMlprT0dWa1lUVmpJaXdpYzNWaUlqb2ljM2x6ZEdWdE9uTmxjblpwWTJWaFkyTnZkVzUwT21admJ6cGhjR0l0Y25WdUxYQnliM1pwYzJsdmJpMXRaV1JwWVhkcGEya3RZWEJpSW4wLkhCWXdqYXJSdmRnUmVTaXNkLUdOUWkxNU03TkpCeEhMaGx4QkI5YnVaMEtTOFRYZ2ZLdldlRmpTOEp2UUcyNTdlZk84Mzk5SFlWVk5UbnhNWWZLQndlbVcta2xNaXBMWWhuUFdSUmNxZWdQMHNmNTBoTkt5VDlGTkprWnhpVG1Ta1lZdVZuT3BOTVJQcXFlTGNFNTNMOEV4SGRWMXhGSnFTM01URHprZjdKVVY0RkY5OXprYng4dkhra2VpSDJxOEhkUUkwaDFmV19uMFAtM2VuNllabnc2N2JyLV9zRVNrNFhxblZfU2xjTXNGbFQ5dGtTU0NELXoxVmN3Z0VRSGNwVmh5WU1YcGpsN21KSVlDeldVSnhlSG01am1oZk9SR3lHWnF5WW9ZUmhRcmdHSE9HbFN3V0xQNnE0T1JMQUVCMWlnSnhicFhsZ011QkVOa1FWYUNKdyIsImVtYWlsIjoic2VydmljZWFjY291bnRAZXhhbXBsZS5vcmciLCJhdXRoIjoiYzJWeWRtbGpaV0ZqWTI5MWJuUTZaWGxLYUdKSFkybFBhVXBUVlhwSk1VNXBTWE5KYmxJMVkwTkpO a2xyY0ZoV1EwbzVMbVY1U25Cak0wMXBUMmxLY21SWFNteGpiVFZzWkVkV2Vrd3pUbXhqYmxwd1dUSldhRmt5VG5aa1Z6VXdTV2wzYVdFelZtbGFXRXAxV2xoU2JHTjVOWEJpZVRsNldsaEtNbUZYVG14WlYwNXFZak5XZFdSRE9YVlpWekZzWXpOQ2FGa3lWV2xQYVVwdFlqSTRhVXh EU25Ka1YwcHNZMjAxYkdSSFZucE1iV3gyVEROT2JHTnVXbkJaTWxab1dUSk9kbVJYTlRCTU0wNXNXVE5LYkdSRE5YVlpWekZzU1dwdmFWbFlRbWxNV0VveFlta3hkMk50T1RKaFdFNXdZakkwZEdKWFZtdGhWMFl6WVZkMGNFeFhSbmRaYVRFd1lqSjBiR0pwTVRKWk1uaDRaVU5KYz BsdGRERlpiVlo1WW0xV01GcFlUWFZoVnpoMll6SldlV1J0YkdwYVYwWnFXVEk1TVdKdVVYWmpNbFo1Wkcxc2FscFRNV2haTWs1MlpGYzFNRXh0TldoaVYxVnBUMmxLYUdOSFNYUmpibFoxVEZoQ2VXSXpXbkJqTW14Mllta3hkRnBYVW5CWldHUndZVEpyZEZsWVFtbEphWGRwWVROV 2FWcFlTblZhV0ZKc1kzazFjR0o1T1hwYVdFb3lZVmRPYkZsWFRtcGlNMVoxWkVNNWVscFlTakpoVjA1c1RGZEdhbGt5T1RGaWJsRjFaRmRzYTBscWIybE5WMFYzVGpKRk1rNXFRWFJOVkVGM1QwTXdlRTFYVlRSTVYwVjNXa1JqZEU1dFJYaFpNbHByVDBkV2ExbFVWbXBKYVhkcFl6 TldhVWxxYjJsak0yeDZaRWRXZEU5dVRteGpibHB3V1RKV2FGa3lUblprVnpVd1QyMWFkbUo2Y0doalIwbDBZMjVXZFV4WVFubGlNMXB3WXpKc2RtSnBNWFJhVjFKd1dWaGtjR0V5YTNSWldFSnBTVzR3TGtoQ1dYZHFZWEpTZG1SblVtVlRhWE5rTFVkT1VXa3hOVTAzVGtwQ2VFaE1 hR3g0UWtJNVluVmFNRXRUT0ZSWVoyWkxkbGRsUm1wVE9FcDJVVWN5TlRkbFprODRNems1U0ZsV1ZrNVVibmhOV1daTFFuZGxiVmN0YTJ4TmFYQk1XV2h1VUZkU1VtTnhaV2RRTUhObU5UQm9Ua3Q1VkRsR1RrcHJXbmhwVkcxVGExbFpkVlp1VDNCT1RWSlFjWEZsVEdORk5UTk1PRV Y0U0dSV01YaEdTbkZUTTAxVVJIcHJaamRLVlZZMFJrWTVPWHByWW5nNGRraHJhMlZwU0RKeE9FaGtVVWt3YURGbVYxOXVNRkF0TTJWdU5sbGFibmMyTjJKeUxWOXpSVk5yTkZoeGJsWmZVMnhqVFhOR2JGUTVkR3RUVTBORUxYb3hWbU4zWjBWUlNHTndWbWg1V1UxWWNHcHNOMjFLU 1ZsRGVsZFZTbmhsU0cwMWFtMW9aazlTUjNsSFduRjVXVzlaVW1oUmNtZEhTRTlIYkZOM1YweFFObkUwVDFKTVFVVkNNV2xuU25oaWNGaHNaMDExUWtWT2ExRldZVU5LZHc9PSJ9LCJkb2NrZXItcmVnaXN0cnkuZGVmYXVsdC5zdmM6NTAwMCI6eyJ1c2VybmFtZSI6InNlcnZpY2Vh Y2NvdW50IiwicGFzc3dvcmQiOiJleUpoYkdjaU9pSlNVekkxTmlJc0luUjVjQ0k2SWtwWFZDSjkuZXlKcGMzTWlPaUpyZFdKbGNtNWxkR1Z6TDNObGNuWnBZMlZoWTJOdmRXNTBJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5dVlXMWxjM0JoWTJVaU9 pSm1iMjhpTENKcmRXSmxjbTVsZEdWekxtbHZMM05sY25acFkyVmhZMk52ZFc1MEwzTmxZM0psZEM1dVlXMWxJam9pWVhCaUxYSjFiaTF3Y205MmFYTnBiMjR0YldWa2FXRjNhV3RwTFdGd1lpMTBiMnRsYmkxMlkyeHhlQ0lzSW10MVltVnlibVYwWlhNdWFXOHZjMlZ5ZG1salpXRm pZMjkxYm5RdmMyVnlkbWxqWlMxaFkyTnZkVzUwTG01aGJXVWlPaUpoY0dJdGNuVnVMWEJ5YjNacGMybHZiaTF0WldScFlYZHBhMmt0WVhCaUlpd2lhM1ZpWlhKdVpYUmxjeTVwYnk5elpYSjJhV05sWVdOamIzVnVkQzl6WlhKMmFXTmxMV0ZqWTI5MWJuUXVkV2xrSWpvaU1XRXdOM kUyTmpBdE1UQXdPQzB4TVdVNExXRXdaRGN0Tm1FeFkyWmtPR1ZrWVRWaklpd2ljM1ZpSWpvaWMzbHpkR1Z0T25ObGNuWnBZMlZoWTJOdmRXNTBPbVp2YnpwaGNHSXRjblZ1TFhCeWIzWnBjMmx2YmkxdFpXUnBZWGRwYTJrdFlYQmlJbjAuSEJZd2phclJ2ZGdSZVNpc2QtR05RaTE1 TTdOSkJ4SExobHhCQjlidVowS1M4VFhnZkt2V2VGalM4SnZRRzI1N2VmTzgzOTlIWVZWTlRueE1ZZktCd2VtVy1rbE1pcExZaG5QV1JSY3FlZ1Awc2Y1MGhOS3lUOUZOSmtaeGlUbVNrWVl1Vm5PcE5NUlBxcWVMY0U1M0w4RXhIZFYxeEZKcVMzTVREemtmN0pVVjRGRjk5emtieDh 2SGtrZWlIMnE4SGRRSTBoMWZXX24wUC0zZW42WVpudzY3YnItX3NFU2s0WHFuVl9TbGNNc0ZsVDl0a1NTQ0QtejFWY3dnRVFIY3BWaHlZTVhwamw3bUpJWUN6V1VKeGVIbTVqbWhmT1JHeUdacXlZb1lSaFFyZ0dIT0dsU3dXTFA2cTRPUkxBRUIxaWdKeGJwWGxnTXVCRU5rUVZhQ0 p3IiwiZW1haWwiOiJzZXJ2aWNlYWNjb3VudEBleGFtcGxlLm9yZyIsImF1dGgiOiJjMlZ5ZG1salpXRmpZMjkxYm5RNlpYbEthR0pIWTJsUGFVcFRWWHBKTVU1cFNYTkpibEkxWTBOSk5rbHJjRmhXUTBvNUxtVjVTbkJqTTAxcFQybEtjbVJYU214amJUVnNaRWRXZWt3elRteGpib HB3V1RKV2FGa3lUblprVnpVd1NXbDNhV0V6Vm1sYVdFcDFXbGhTYkdONU5YQmllVGw2V2xoS01tRlhUbXhaVjA1cVlqTldkV1JET1hWWlZ6RnNZek5DYUZreVZXbFBhVXB0WWpJNGFVeERTbkprVjBwc1kyMDFiR1JIVm5wTWJXeDJURE5PYkdOdVduQlpNbFpvV1RKT2RtUlhOVEJN TTA1c1dUTktiR1JETlhWWlZ6RnNTV3B2YVZsWVFtbE1XRW94WW1reGQyTnRPVEpoV0U1d1lqSTBkR0pYVm10aFYwWXpZVmQwY0V4WFJuZFphVEV3WWpKMGJHSnBNVEpaTW5oNFpVTkpjMGx0ZERGWmJWWjVZbTFXTUZwWVRYVmhWemgyWXpKV2VXUnRiR3BhVjBacVdUSTVNV0p1VVh aak1sWjVaRzFzYWxwVE1XaFpNazUyWkZjMU1FeHROV2hpVjFWcFQybEthR05IU1hSamJsWjFURmhDZVdJelduQmpNbXgyWW1reGRGcFhVbkJaV0dSd1lUSnJkRmxZUW1sSmFYZHBZVE5XYVZwWVNuVmFXRkpzWTNrMWNHSjVPWHBhV0VveVlWZE9iRmxYVG1waU0xWjFaRU01ZWxwWV NqSmhWMDVzVEZkR2Fsa3lPVEZpYmxGMVpGZHNhMGxxYjJsTlYwVjNUakpGTWs1cVFYUk5WRUYzVDBNd2VFMVhWVFJNVjBWM1drUmpkRTV0UlhoWk1scHJUMGRXYTFsVVZtcEphWGRwWXpOV2FVbHFiMmxqTTJ4NlpFZFdkRTl1VG14amJscHdXVEpXYUZreVRuWmtWelV3VDIxYWRtS jZjR2hqUjBsMFkyNVdkVXhZUW5saU0xcHdZekpzZG1KcE1YUmFWMUp3V1Zoa2NHRXlhM1JaV0VKcFNXNHdMa2hDV1hkcVlYSlNkbVJuVW1WVGFYTmtMVWRPVVdreE5VMDNUa3BDZUVoTWFHeDRRa0k1WW5WYU1FdFRPRlJZWjJaTGRsZGxSbXBUT0VwMlVVY3lOVGRsWms4NE16azVT RmxXVms1VWJuaE5XV1pMUW5kbGJWY3RhMnhOYVhCTVdXaHVVRmRTVW1OeFpXZFFNSE5tTlRCb1RrdDVWRGxHVGtwclduaHBWRzFUYTFsWmRWWnVUM0JPVFZKUWNYRmxUR05GTlROTU9FVjRTR1JXTVhoR1NuRlRNMDFVUkhwclpqZEtWVlkwUmtZNU9YcHJZbmc0ZGtocmEyVnBTREp 4T0Voa1VVa3dhREZtVjE5dU1GQXRNMlZ1TmxsYWJuYzJOMkp5TFY5elJWTnJORmh4YmxaZlUyeGpUWE5HYkZRNWRHdFRVME5FTFhveFZtTjNaMFZSU0dOd1ZtaDVXVTFZY0dwc04yMUtTVmxEZWxkVlNuaGxTRzAxYW0xb1prOVNSM2xIV25GNVdXOVpVbWhSY21kSFNFOUhiRk4zVj B4UU5uRTBUMUpNUVVWQ01XbG5TbmhpY0Zoc1owMTFRa1ZPYTFGV1lVTktkdz09In19" } [dymurray@dymurray scripts]$ echo "eyIxNzIuMzAuMS4xOjUwMDAiOnsidXNlcm5hbWUiOiJzZXJ2aWNlYWNjb3VudCIsInBhc3N3b3JkIjoiZXlKaGJHY2lPaUpTVXpJMU5pSXNJblI1Y0NJNklrcFhWQ0o5LmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJW aFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUptYjI4aUxDSnJkV0psY201bGRHVnpMbWx2TDNObGNuWnBZMlZoWTJOdmRXNTBMM05sWTNKbGRDNXVZVzFsSWpvaVlYQmlMWEoxYmkxd2NtOTJhWE5wYjI0dGJXVmthV0YzYVd0cExXRndZaTEwYjJ0bGJpMTJZMnh4ZUNJc0ltdDFZbVZ5Ym1WMFpYTXVhVzh2YzJWeWRtbGpaV0ZqWTI5MWJuUXZjMlZ5ZG1salpTMWhZMk52ZFc1MExtNWhiV1VpT2lKaGNHSXRjblZ1TFhCeWIzWnBjMmx2YmkxdFpXUnBZWGRwYTJrdFlYQmlJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpYSjJhV05sTFdGalkyOTFiblF1ZFdsa0lqb2lNV0V3TjJFMk5qQXRNVEF3T0MweE1XVTRMV0V3WkRjdE5tRXhZMlprT0dWa1lUVmpJaXdpYzNWaUlqb2ljM2x6ZEdWdE9uTmxjblpwWTJWaFkyTnZkVzUwT21admJ6cGhjR0l0Y25WdUxYQnliM1pwYzJsdmJpMXRaV1JwWVhkcGEya3RZWEJpSW4wLkhCWXdqYXJSdmRnUmVTaXNkLUdOUWkxNU03TkpCeEhMaGx4QkI5YnVaMEtTOFRYZ2ZLdldlRmpTOEp2UUcyNTdlZk84Mzk5SFlWVk5UbnhNWWZLQndlbVcta2xNaXBMWWhuUFdSUmNxZWdQMHNmNTBoTkt5VDlGTkprWnhpVG1Ta1lZdVZuT3BOTVJQcXFlTGNFNTNMOEV4SGRWMXhGSnFTM01URHprZjdKVVY0RkY5OXprYng4dkhra2VpSDJxOEhkUUkwaDFmV19uMFAtM2VuNllabnc2N2JyLV9zRVNrNFhxblZfU2xjTXNGbFQ5dGtTU0NELXoxVmN3Z0VRSGNwVmh5WU1YcGpsN21KSVlDeldVSnhlSG01am1oZk9SR3lHWnF5WW9ZUmhRcmdHSE9HbFN3V0xQNnE0T1JMQUVCMWlnSnhicFhsZ011QkVOa1FWYUNKdyIsImVtYWlsIjoic2VydmljZWFjY291bnRAZXhhbXBsZS5vcmciLCJhdXRoIjoiYzJWeWRtbGpaV0ZqWTI5MWJuUTZaWGxLYUdKSFkybFBhVXBUVlhwSk1VNXBTWE5KYmxJMVkwTkpOa2xyY0ZoV1EwbzVMbVY1U25Cak0wMXBUMmxLY21SWFNteGpiVFZzWkVkV2Vrd3pUbXhqYmxwd1dUSldhRmt5VG5aa1Z6VXdTV2wzYVdFelZtbGFXRXAxV2xoU2JHTjVOWEJpZVRsNldsaEtNbUZYVG14WlYwNXFZak5XZFdSRE9YVlpWekZzWXpOQ2FGa3lWV 2xQYVVwdFlqSTRhVXhEU25Ka1YwcHNZMjAxYkdSSFZucE1iV3gyVEROT2JHTnVXbkJaTWxab1dUSk9kbVJYTlRCTU0wNXNXVE5LYkdSRE5YVlpWekZzU1dwdmFWbFlRbWxNV0VveFlta3hkMk50T1RKaFdFNXdZakkwZEdKWFZtdGhWMFl6WVZkMGNFeFhSbmRaYVRFd1lqSjBiR0pw TVRKWk1uaDRaVU5KYzBsdGRERlpiVlo1WW0xV01GcFlUWFZoVnpoMll6SldlV1J0YkdwYVYwWnFXVEk1TVdKdVVYWmpNbFo1Wkcxc2FscFRNV2haTWs1MlpGYzFNRXh0TldoaVYxVnBUMmxLYUdOSFNYUmpibFoxVEZoQ2VXSXpXbkJqTW14Mllta3hkRnBYVW5CWldHUndZVEpyZEZ sWVFtbEphWGRwWVROV2FWcFlTblZhV0ZKc1kzazFjR0o1T1hwYVdFb3lZVmRPYkZsWFRtcGlNMVoxWkVNNWVscFlTakpoVjA1c1RGZEdhbGt5T1RGaWJsRjFaRmRzYTBscWIybE5WMFYzVGpKRk1rNXFRWFJOVkVGM1QwTXdlRTFYVlRSTVYwVjNXa1JqZEU1dFJYaFpNbHByVDBkV2 ExbFVWbXBKYVhkcFl6TldhVWxxYjJsak0yeDZaRWRXZEU5dVRteGpibHB3V1RKV2FGa3lUblprVnpVd1QyMWFkbUo2Y0doalIwbDBZMjVXZFV4WVFubGlNMXB3WXpKc2RtSnBNWFJhVjFKd1dWaGtjR0V5YTNSWldFSnBTVzR3TGtoQ1dYZHFZWEpTZG1SblVtVlRhWE5rTFVkT1VXa 3hOVTAzVGtwQ2VFaE1hR3g0UWtJNVluVmFNRXRUT0ZSWVoyWkxkbGRsUm1wVE9FcDJVVWN5TlRkbFprODRNems1U0ZsV1ZrNVVibmhOV1daTFFuZGxiVmN0YTJ4TmFYQk1XV2h1VUZkU1VtTnhaV2RRTUhObU5UQm9Ua3Q1VkRsR1RrcHJXbmhwVkcxVGExbFpkVlp1VDNCT1RWSlFj WEZsVEdORk5UTk1PRVY0U0dSV01YaEdTbkZUTTAxVVJIcHJaamRLVlZZMFJrWTVPWHByWW5nNGRraHJhMlZwU0RKeE9FaGtVVWt3YURGbVYxOXVNRkF0TTJWdU5sbGFibmMyTjJKeUxWOXpSVk5yTkZoeGJsWmZVMnhqVFhOR2JGUTVkR3RUVTBORUxYb3hWbU4zWjBWUlNHTndWbWg 1V1UxWWNHcHNOMjFLU1ZsRGVsZFZTbmhsU0cwMWFtMW9aazlTUjNsSFduRjVXVzlaVW1oUmNtZEhTRTlIYkZOM1YweFFObkUwVDFKTVFVVkNNV2xuU25oaWNGaHNaMDExUWtWT2ExRldZVU5LZHc9PSJ9LCJkb2NrZXItcmVnaXN0cnkuZGVmYXVsdC5zdmM6NTAwMCI6eyJ1c2Vybm FtZSI6InNlcnZpY2VhY2NvdW50IiwicGFzc3dvcmQiOiJleUpoYkdjaU9pSlNVekkxTmlJc0luUjVjQ0k2SWtwWFZDSjkuZXlKcGMzTWlPaUpyZFdKbGNtNWxkR1Z6TDNObGNuWnBZMlZoWTJOdmRXNTBJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5d VlXMWxjM0JoWTJVaU9pSm1iMjhpTENKcmRXSmxjbTVsZEdWekxtbHZMM05sY25acFkyVmhZMk52ZFc1MEwzTmxZM0psZEM1dVlXMWxJam9pWVhCaUxYSjFiaTF3Y205MmFYTnBiMjR0YldWa2FXRjNhV3RwTFdGd1lpMTBiMnRsYmkxMlkyeHhlQ0lzSW10MVltVnlibVYwWlhNdWFX OHZjMlZ5ZG1salpXRmpZMjkxYm5RdmMyVnlkbWxqWlMxaFkyTnZkVzUwTG01aGJXVWlPaUpoY0dJdGNuVnVMWEJ5YjNacGMybHZiaTF0WldScFlYZHBhMmt0WVhCaUlpd2lhM1ZpWlhKdVpYUmxjeTVwYnk5elpYSjJhV05sWVdOamIzVnVkQzl6WlhKMmFXTmxMV0ZqWTI5MWJuUXV kV2xrSWpvaU1XRXdOMkUyTmpBdE1UQXdPQzB4TVdVNExXRXdaRGN0Tm1FeFkyWmtPR1ZrWVRWaklpd2ljM1ZpSWpvaWMzbHpkR1Z0T25ObGNuWnBZMlZoWTJOdmRXNTBPbVp2YnpwaGNHSXRjblZ1TFhCeWIzWnBjMmx2YmkxdFpXUnBZWGRwYTJrdFlYQmlJbjAuSEJZd2phclJ2ZG dSZVNpc2QtR05RaTE1TTdOSkJ4SExobHhCQjlidVowS1M4VFhnZkt2V2VGalM4SnZRRzI1N2VmTzgzOTlIWVZWTlRueE1ZZktCd2VtVy1rbE1pcExZaG5QV1JSY3FlZ1Awc2Y1MGhOS3lUOUZOSmtaeGlUbVNrWVl1Vm5PcE5NUlBxcWVMY0U1M0w4RXhIZFYxeEZKcVMzTVREemtmN 0pVVjRGRjk5emtieDh2SGtrZWlIMnE4SGRRSTBoMWZXX24wUC0zZW42WVpudzY3YnItX3NFU2s0WHFuVl9TbGNNc0ZsVDl0a1NTQ0QtejFWY3dnRVFIY3BWaHlZTVhwamw3bUpJWUN6V1VKeGVIbTVqbWhmT1JHeUdacXlZb1lSaFFyZ0dIT0dsU3dXTFA2cTRPUkxBRUIxaWdKeGJw WGxnTXVCRU5rUVZhQ0p3IiwiZW1haWwiOiJzZXJ2aWNlYWNjb3VudEBleGFtcGxlLm9yZyIsImF1dGgiOiJjMlZ5ZG1salpXRmpZMjkxYm5RNlpYbEthR0pIWTJsUGFVcFRWWHBKTVU1cFNYTkpibEkxWTBOSk5rbHJjRmhXUTBvNUxtVjVTbkJqTTAxcFQybEtjbVJYU214amJUVnN aRWRXZWt3elRteGpibHB3V1RKV2FGa3lUblprVnpVd1NXbDNhV0V6Vm1sYVdFcDFXbGhTYkdONU5YQmllVGw2V2xoS01tRlhUbXhaVjA1cVlqTldkV1JET1hWWlZ6RnNZek5DYUZreVZXbFBhVXB0WWpJNGFVeERTbkprVjBwc1kyMDFiR1JIVm5wTWJXeDJURE5PYkdOdVduQlpNbF pvV1RKT2RtUlhOVEJNTTA1c1dUTktiR1JETlhWWlZ6RnNTV3B2YVZsWVFtbE1XRW94WW1reGQyTnRPVEpoV0U1d1lqSTBkR0pYVm10aFYwWXpZVmQwY0V4WFJuZFphVEV3WWpKMGJHSnBNVEpaTW5oNFpVTkpjMGx0ZERGWmJWWjVZbTFXTUZwWVRYVmhWemgyWXpKV2VXUnRiR3BhV jBacVdUSTVNV0p1VVhaak1sWjVaRzFzYWxwVE1XaFpNazUyWkZjMU1FeHROV2hpVjFWcFQybEthR05IU1hSamJsWjFURmhDZVdJelduQmpNbXgyWW1reGRGcFhVbkJaV0dSd1lUSnJkRmxZUW1sSmFYZHBZVE5XYVZwWVNuVmFXRkpzWTNrMWNHSjVPWHBhV0VveVlWZE9iRmxYVG1w aU0xWjFaRU01ZWxwWVNqSmhWMDVzVEZkR2Fsa3lPVEZpYmxGMVpGZHNhMGxxYjJsTlYwVjNUakpGTWs1cVFYUk5WRUYzVDBNd2VFMVhWVFJNVjBWM1drUmpkRTV0UlhoWk1scHJUMGRXYTFsVVZtcEphWGRwWXpOV2FVbHFiMmxqTTJ4NlpFZFdkRTl1VG14amJscHdXVEpXYUZreVR uWmtWelV3VDIxYWRtSjZjR2hqUjBsMFkyNVdkVXhZUW5saU0xcHdZekpzZG1KcE1YUmFWMUp3V1Zoa2NHRXlhM1JaV0VKcFNXNHdMa2hDV1hkcVlYSlNkbVJuVW1WVGFYTmtMVWRPVVdreE5VMDNUa3BDZUVoTWFHeDRRa0k1WW5WYU1FdFRPRlJZWjJaTGRsZGxSbXBUT0VwMlVVY3 lOVGRsWms4NE16azVTRmxXVms1VWJuaE5XV1pMUW5kbGJWY3RhMnhOYVhCTVdXaHVVRmRTVW1OeFpXZFFNSE5tTlRCb1RrdDVWRGxHVGtwclduaHBWRzFUYTFsWmRWWnVUM0JPVFZKUWNYRmxUR05GTlROTU9FVjRTR1JXTVhoR1NuRlRNMDFVUkhwclpqZEtWVlkwUmtZNU9YcHJZb mc0ZGtocmEyVnBTREp4T0Voa1VVa3dhREZtVjE5dU1GQXRNMlZ1TmxsYWJuYzJOMkp5TFY5elJWTnJORmh4YmxaZlUyeGpUWE5HYkZRNWRHdFRVME5FTFhveFZtTjNaMFZSU0dOd1ZtaDVXVTFZY0dwc04yMUtTVmxEZWxkVlNuaGxTRzAxYW0xb1prOVNSM2xIV25GNVdXOVpVbWhS Y21kSFNFOUhiRk4zVjB4UU5uRTBUMUpNUVVWQ01XbG5TbmhpY0Zoc1owMTFRa1ZPYTFGV1lVTktkdz09In19" | base64 -d | jq { "172.30.1.1:5000": { "username": "serviceaccount", "password": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJmb28iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3Jld C5uYW1lIjoiYXBiLXJ1bi1wcm92aXNpb24tbWVkaWF3aWtpLWFwYi10b2tlbi12Y2xxeCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJhcGItcnVuLXByb3Zpc2lvbi1tZWRpYXdpa2ktYXBiIiwia3ViZXJuZXRlcy5pby9zZXJ2 aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiMWEwN2E2NjAtMTAwOC0xMWU4LWEwZDctNmExY2ZkOGVkYTVjIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmZvbzphcGItcnVuLXByb3Zpc2lvbi1tZWRpYXdpa2ktYXBiIn0.HBYwjarRvdgReSisd-GNQi15M7N JBxHLhlxBB9buZ0KS8TXgfKvWeFjS8JvQG257efO8399HYVVNTnxMYfKBwemW-klMipLYhnPWRRcqegP0sf50hNKyT9FNJkZxiTmSkYYuVnOpNMRPqqeLcE53L8ExHdV1xFJqS3MTDzkf7JUV4FF99zkbx8vHkkeiH2q8HdQI0h1fW_n0P-3en6YZnw67br-_sESk4XqnV_SlcMsFlT 9tkSSCD-z1VcwgEQHcpVhyYMXpjl7mJIYCzWUJxeHm5jmhfORGyGZqyYoYRhQrgGHOGlSwWLP6q4ORLAEB1igJxbpXlgMuBENkQVaCJw", "email": "serviceaccount", "auth": "c2VydmljZWFjY291bnQ6ZXlKaGJHY2lPaUpTVXpJMU5pSXNJblI1Y0NJNklrcFhWQ0o5LmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaF kyVWlPaUptYjI4aUxDSnJkV0psY201bGRHVnpMbWx2TDNObGNuWnBZMlZoWTJOdmRXNTBMM05sWTNKbGRDNXVZVzFsSWpvaVlYQmlMWEoxYmkxd2NtOTJhWE5wYjI0dGJXVmthV0YzYVd0cExXRndZaTEwYjJ0bGJpMTJZMnh4ZUNJc0ltdDFZbVZ5Ym1WMFpYTXVhVzh2YzJWeWRtb GpaV0ZqWTI5MWJuUXZjMlZ5ZG1salpTMWhZMk52ZFc1MExtNWhiV1VpT2lKaGNHSXRjblZ1TFhCeWIzWnBjMmx2YmkxdFpXUnBZWGRwYTJrdFlYQmlJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpYSjJhV05sTFdGalkyOTFiblF1ZFdsa0lqb2lN V0V3TjJFMk5qQXRNVEF3T0MweE1XVTRMV0V3WkRjdE5tRXhZMlprT0dWa1lUVmpJaXdpYzNWaUlqb2ljM2x6ZEdWdE9uTmxjblpwWTJWaFkyTnZkVzUwT21admJ6cGhjR0l0Y25WdUxYQnliM1pwYzJsdmJpMXRaV1JwWVhkcGEya3RZWEJpSW4wLkhCWXdqYXJSdmRnUmVTaXNkLUd OUWkxNU03TkpCeEhMaGx4QkI5YnVaMEtTOFRYZ2ZLdldlRmpTOEp2UUcyNTdlZk84Mzk5SFlWVk5UbnhNWWZLQndlbVcta2xNaXBMWWhuUFdSUmNxZWdQMHNmNTBoTkt5VDlGTkprWnhpVG1Ta1lZdVZuT3BOTVJQcXFlTGNFNTNMOEV4SGRWMXhGSnFTM01URHprZjdKVVY0RkY5OX prYng4dkhra2VpSDJxOEhkUUkwaDFmV19uMFAtM2VuNllabnc2N2JyLV9zRVNrNFhxblZfU2xjTXNGbFQ5dGtTU0NELXoxVmN3Z0VRSGNwVmh5WU1YcGpsN21KSVlDeldVSnhlSG01am1oZk9SR3lHWnF5WW9ZUmhRcmdHSE9HbFN3V0xQNnE0T1JMQUVCMWlnSnhicFhsZ011QkVOa 1FWYUNKdw==" }, "docker-registry.default.svc:5000": { "username": "serviceaccount", "password": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJmb28iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3Jld C5uYW1lIjoiYXBiLXJ1bi1wcm92aXNpb24tbWVkaWF3aWtpLWFwYi10b2tlbi12Y2xxeCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJhcGItcnVuLXByb3Zpc2lvbi1tZWRpYXdpa2ktYXBiIiwia3ViZXJuZXRlcy5pby9zZXJ2 aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiMWEwN2E2NjAtMTAwOC0xMWU4LWEwZDctNmExY2ZkOGVkYTVjIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmZvbzphcGItcnVuLXByb3Zpc2lvbi1tZWRpYXdpa2ktYXBiIn0.HBYwjarRvdgReSisd-GNQi15M7N JBxHLhlxBB9buZ0KS8TXgfKvWeFjS8JvQG257efO8399HYVVNTnxMYfKBwemW-klMipLYhnPWRRcqegP0sf50hNKyT9FNJkZxiTmSkYYuVnOpNMRPqqeLcE53L8ExHdV1xFJqS3MTDzkf7JUV4FF99zkbx8vHkkeiH2q8HdQI0h1fW_n0P-3en6YZnw67br-_sESk4XqnV_SlcMsFlT 9tkSSCD-z1VcwgEQHcpVhyYMXpjl7mJIYCzWUJxeHm5jmhfORGyGZqyYoYRhQrgGHOGlSwWLP6q4ORLAEB1igJxbpXlgMuBENkQVaCJw", "email": "serviceaccount", "auth": "c2VydmljZWFjY291bnQ6ZXlKaGJHY2lPaUpTVXpJMU5pSXNJblI1Y0NJNklrcFhWQ0o5LmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaF kyVWlPaUptYjI4aUxDSnJkV0psY201bGRHVnpMbWx2TDNObGNuWnBZMlZoWTJOdmRXNTBMM05sWTNKbGRDNXVZVzFsSWpvaVlYQmlMWEoxYmkxd2NtOTJhWE5wYjI0dGJXVmthV0YzYVd0cExXRndZaTEwYjJ0bGJpMTJZMnh4ZUNJc0ltdDFZbVZ5Ym1WMFpYTXVhVzh2YzJWeWRtb GpaV0ZqWTI5MWJuUXZjMlZ5ZG1salpTMWhZMk52ZFc1MExtNWhiV1VpT2lKaGNHSXRjblZ1TFhCeWIzWnBjMmx2YmkxdFpXUnBZWGRwYTJrdFlYQmlJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpYSjJhV05sTFdGalkyOTFiblF1ZFdsa0lqb2lN V0V3TjJFMk5qQXRNVEF3T0MweE1XVTRMV0V3WkRjdE5tRXhZMlprT0dWa1lUVmpJaXdpYzNWaUlqb2ljM2x6ZEdWdE9uTmxjblpwWTJWaFkyTnZkVzUwT21admJ6cGhjR0l0Y25WdUxYQnliM1pwYzJsdmJpMXRaV1JwWVhkcGEya3RZWEJpSW4wLkhCWXdqYXJSdmRnUmVTaXNkLUd OUWkxNU03TkpCeEhMaGx4QkI5YnVaMEtTOFRYZ2ZLdldlRmpTOEp2UUcyNTdlZk84Mzk5SFlWVk5UbnhNWWZLQndlbVcta2xNaXBMWWhuUFdSUmNxZWdQMHNmNTBoTkt5VDlGTkprWnhpVG1Ta1lZdVZuT3BOTVJQcXFlTGNFNTNMOEV4SGRWMXhGSnFTM01URHprZjdKVVY0RkY5OX prYng4dkhra2VpSDJxOEhkUUkwaDFmV19uMFAtM2VuNllabnc2N2JyLV9zRVNrNFhxblZfU2xjTXNGbFQ5dGtTU0NELXoxVmN3Z0VRSGNwVmh5WU1YcGpsN21KSVlDeldVSnhlSG01am1oZk9SR3lHWnF5WW9ZUmhRcmdHSE9HbFN3V0xQNnE0T1JMQUVCMWlnSnhicFhsZ011QkVOa 1FWYUNKdw==" } }
Somehow the secret being created doesn't include the insecure registry config change for the minishift registry.
I think I finally have an explanation for what is occurring here and I'm not convinced this is something we can control. When a pod is created an imagePullSecret is created with default docker config settings for the registry. By default these entries are the ones shown above: One for the IP+Port of the registry service and one for the service_route+port. To get around this, we must create a new dockercfg secret which specifies this docker-server. This would be the following command (like Jian showed above): $ oc secrets new-dockercfg <pull_secret_name> \ --docker-server=<registry_server> --docker-username=<user_name> \ --docker-password=<password> --docker-email=<email> It feels like the only way to get around this would be to manually create the secret when using `apb run` with the `--registry-route` flag. Otherwise we simply must document how to fix the error when running on a remote cluster. All of this is due to not being able to read the Docker certs from a remote cluster (see here: https://bugzilla.redhat.com/show_bug.cgi?id=1526147). I will discuss with the team on how to proceed.
Jian, I tried testing by doing the same thing you did and actually was unable to reproduce the solution. I am getting error image pulls even after creating a new secret and adding it to the service account: [dymurray@dymurray mediawiki123-apb]$ oc get pods NAME READY STATUS RESTARTS AGE apb-run-provision-mediawiki-apbw9xll 0/1 ErrImagePull 0 31s [dymurray@dymurray mediawiki123-apb]$ oc get pod apb-run-provision-mediawiki-apbw9xll -o yaml --- imagePullSecrets: - name: apb-run-provision-mediawiki-apb-dockercfg-ntb5t - name: run-pull-secret [dymurray@dymurray mediawiki123-apb]$ oc get secret run-pull-secret -o yaml apiVersion: v1 data: .dockerconfigjson: eyJhdXRocyI6eyJkb2NrZXItcmVnaXN0cnktZGVmYXVsdC4xOTIuMTY4LjQyLjIzMi5uaXAuaW8iOnsidXNlcm5hbWUiOiJkZXZlbG9wZXIiLCJwYXNzd29yZCI6IlFuYXVjXzJlVF9TQTE0VUQ5eEJyREQ4bDJmMEZYenhtVnYwMEFmcEFTS0UiLCJlbWFpbCI6ImZvb0Bmb28uY29tIiwiYXV0aCI6IlpHVjJaV3h2Y0dWeU9sRnVZWFZqWHpKbFZGOVRRVEUwVlVRNWVFSnlSRVE0YkRKbU1FWlllbmh0Vm5Zd01FRm1jRUZUUzBVPSJ9fX0= kind: Secret metadata: creationTimestamp: 2018-02-14T15:34:30Z name: run-pull-secret namespace: run resourceVersion: "213003" selfLink: /api/v1/namespaces/run/secrets/run-pull-secret uid: 8c5ee053-119c-11e8-a0d7-6a1cfd8eda5c type: kubernetes.io/dockerconfigjson [dymurray@dymurray mediawiki123-apb]$ echo "eyJhdXRocyI6eyJkb2NrZXItcmVnaXN0cnktZGVmYXVsdC4xOTIuMTY4LjQyLjIzMi5uaXAuaW8iOnsidXNlcm5hbWUiOiJkZXZlbG9wZXIiLCJwYXNzd29yZCI6IlFuYXVjXzJlVF9TQTE0VUQ5eEJyREQ4bDJmMEZYenhtVnYwMEFmcEFTS0UiLCJlbWFpbCI6ImZvb0Bmb28uY29tIiwiYXV0aCI6IlpHVjJaV3h2Y0dWeU9sRnVZWFZqWHpKbFZGOVRRVEUwVlVRNWVFSnlSRVE0YkRKbU1FWlllbmh0Vm5Zd01FRm1jRUZUUzBVPSJ9fX0=" | base64 -d {"auths":{"docker-registry-default.192.168.42.232.nip.io":{"username":"developer","password":"Qnauc_2eT_SA14UD9xBrDD8l2f0FXzxmVv00AfpASKE","email":"foo","auth":"ZGV2ZWxvcGVyOlFuYXVjXzJlVF9TQTE0VUQ5eEJyREQ4bDJmMEZYenhtVnYwMEFmcEFTS0U="}}} Can you give me any advice how to resolve this? I have added the proper secret to the pods and they still get image pull errors
Dylan, The related steps in the above "Additional info" field. I created a docker-registry secret first, like below: [root@host-172-16-120-78 ~]# oc create secret docker-registry my-secret --docker-server="docker-registry-default.apps.0205-ehb.qe.rhcloud.com" --docker-username=jiazha --docker-password=a9kX-zoA44qFC4Aa_u9V8BKL_3n3oUyfpUJFAM-IEr4 --docker-email=jiazha -n test And then, I replace the origin secret with it. Like: --- imagePullSecrets: - name: my-secret Hope that helps!
*** Bug 1533318 has been marked as a duplicate of this bug. ***
I am having trouble testing this because of the docker cert problems explained in https://bugzilla.redhat.com/show_bug.cgi?id=1526147. I have workarounds to push the image onto the cluster but it then fails to test the path of this bug. The real problem ties back to the above bug that the APB tooling was not designed to work on remote clusters. This doesn't allow me to push the image onto the internal registry to test with. When I DO get the image there, the imagepull secret is already correct because I have to evaluate the Docker cert information from the remote host.
I get imagePull errors NOT due to the same problem described by Jian, but because the initial push never succeeds to begin with. [dymurray@dymurray mediawiki123-apb]$3-apb]$ apb run --project run --registry-route docker-registry-default.192.168.42.194.nip.io Finished writing dockerfile. Building APB using tag: [docker-registry-default.192.168.42.194.nip.io/openshift/mediawiki-apb] Successfully built APB image: docker-registry-default.192.168.42.194.nip.io/openshift/mediawiki-apb Pushing the image, this could take a minute... Successfully pushed image: docker-registry-default.192.168.42.194.nip.io/openshift/mediawiki-apb mediawiki_db_schema(required)[default: mediawiki]: mediawiki_site_name(required)[default: MediaWiki]: mediawiki_site_lang(required)[default: en]: mediawiki_admin_user(required)[default: admin]: mediawiki_admin_pass(required): changeme Creating project run Project run already exists Creating service account in run Service account apb-run-provision-mediawiki-apb already exists Creating role binding for apb-run-provision-mediawiki-apb in run Role binding apb-run-provision-mediawiki-apb already exists Creating pod with image docker-registry-default.192.168.42.194.nip.io/openshift/mediawiki-apb in run Created Pod APB run started Pod in phase: Pending Pod in phase: Pending Pod in phase: Pending APB run failed: (APB failed ImagePullBackOff - check name) Reason: None [dymurray@dymurray data]$ oc get images | grep apb [dymurray@dymurray data]$ As you can see the image doesn't exist because `apb push failed`.
Dylan, From the above log shows, the push action succeeded. You can get the image from the `openshift` namespace, like below: #oc get all -n openshift Then you can see your pushed images. Hope it helps!
We have documented a workaround for working with remote clusters here: https://github.com/ansibleplaybookbundle/ansible-playbook-bundle/blob/master/docs/developers.md#alternative-to-using-apb-push Instead of using `apb push` the developer can follow this documentation approach to populate their image onto the OpenShift cluster.
*** This bug has been marked as a duplicate of bug 1526147 ***
Dylan, I don't think this is a duplicate of bug 1526147. Your workaround solves the `push` issue, but, for this bug, the probelm is could NOT pull image. In other words, the `apb run` feature still cannot work against a remote host. That workaround did NOT solve this probelm. Or am I missing something? Like below:] [root@localhost hello-world-apb]#oc new-app . --name hello-world-apb -n openshift ... [root@localhost hello-world-apb]# apb run --project test --registry-route docker-registry-default.apps.0425-aus.qe.rhcloud.com Finished writing dockerfile. Building APB using tag: [docker-registry-default.apps.0425-aus.qe.rhcloud.com/openshift/hello-world-apb] Successfully built APB image: docker-registry-default.apps.0425-aus.qe.rhcloud.com/openshift/hello-world-apb Pushing the image, this could take a minute... Successfully pushed image: docker-registry-default.apps.0425-aus.qe.rhcloud.com/openshift/hello-world-apb Creating project test Created project Creating service account in test Created service account Creating role binding for apb-run-provision-hello-world-apb in test Created Role Binding Creating pod with image docker-registry-default.apps.0425-aus.qe.rhcloud.com/openshift/hello-world-apb in test Created Pod APB run started Pod in phase: Pending Pod in phase: Pending ... APB run failed: (APB failed ImagePullBackOff - check name) Reason: None Change status to "MODIFIED" since the `apb run` works failed. Correct me if I'm wrong.
Jian, I can update the other bug to include documentation around `apb run` as well but this bug should be a duplicate. The reason is that `apb run` (which is giving you this pull error) performs `apb push` under the hood. This means that on a remote cluster `apb run` will not work. The proper way to do this would be with an `oc run` command on the image you pushed to the registry using `oc new-app`. You are seeing a `imagePullBackOff` because the initial `apb push` never succeeded and the image is not populated in the registry. I will update the documentation bug to address this.
Dylan, As my steps show, I used `oc new-app` to push the destination image into the registry first(I confirm this image already stored in the registry), and then I run the `apb run` command, but it still failed. So, do you mean you will update the document to point out the `apb run` cannot work on a remote cluster? If yes, it's OK, but I think we should support it since other subcommands can work on a remote cluster.
Jian, The reason is that `apb run` is looking for a tagged image of the name: docker-registry-default.apps.0425-aus.qe.rhcloud.com/openshift/hello-world-apb. `oc new-app` will make the name of the image 172.30.1.1:5000/openshift/hello-world-apb. Again, since `apb run` is trying to perform `apb push` first, and then a simple `oc run` I think it makes more sense to simply document how a user can do `oc run` on their APB image which exists in the internal registry. There is no guarantee that running `apb run` will call an `oc run` on the proper image since you are changing the tag with `--registry-route`. Can you test with `oc new-app` and then `apb run` with*out* the --registry-route flag? I'm still not certain that will work but I would be curious if it's a workaround. The proper thing to do here is to document the limitations of `apb run, test, and push` on a remote host and provide workaround `oc` commands that would allow someone to do the same functionality.
I am again closing this in favor of the documentation bug. It includes the limitation of `apb run` and will include instruction to achieve the same functionality as `apb run`. *** This bug has been marked as a duplicate of bug 1526147 ***
Dylan, I have tried the `apb run` without the `--registry-route` flag, but still got errors, details as below: 1) run `oc new-app xxx` [root@localhost hello-world-apb]# oc new-app ./ --name hello-world-apb -n openshift --> Found Docker image 0b9692f (44 hours old) from Docker Hub for "ansibleplaybookbundle/apb-base" * An image stream will be created as "apb-base:latest" that will track the source image * A Docker build using source code from git:ansibleplaybookbundle/hello-world-apb.git#master will be created * The resulting image will be pushed to image stream "hello-world-apb:latest" * Every time "apb-base:latest" changes a new build will be triggered * WARNING: this source repository may require credentials. Create a secret with your git credentials and use 'set build-secret' to assign it to the build config. * This image will be deployed in deployment config "hello-world-apb" * The image does not expose any ports - if you want to load balance or send traffic to this component you will need to create a service with 'expose dc/hello-world-apb --port=[port]' later * WARNING: Image "ansibleplaybookbundle/apb-base" runs as the 'root' user which may not be permitted by your cluster administrator --> Creating resources ... imagestream "apb-base" created imagestream "hello-world-apb" created buildconfig "hello-world-apb" created deploymentconfig "hello-world-apb" created --> Success Build scheduled, use 'oc logs -f bc/hello-world-apb' to track its progress. Run 'oc status' to view your app. 2) Check images, but, why not is "172.30.127.220:5000/openshift/hello-world-apb"? Or "docker-registry.default.svc:5000/openshift/hello-world-apb"? [root@host-172-16-120-63 ~]# oc get imagestream -n openshift | grep hello hello-world-apb docker-registry.default.svc:5000/openshift/hello-world-apb [root@host-172-16-120-63 ~]# oc get images | grep hello 3) run the `apb run`, got below errors: [root@localhost hello-world-apb]# apb run --project jian Found registry IP at: 172.30.127.220:5000 Finished writing dockerfile. Building APB using tag: [172.30.127.220:5000/openshift/hello-world-apb] Successfully built APB image: 172.30.127.220:5000/openshift/hello-world-apb Exception occurred! UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60) the apb tool version: [root@localhost hello-world-apb]# apb version Version: apb-1.2.6
Jian, Thanks for testing that. It timed out on `apb push` as I expected it to. I have updated https://bugzilla.redhat.com/show_bug.cgi?id=1526147 to reflect documentation instructions on accomplishing `apb run` using `oc run` since we are blocked by the `apb push` limitation.
Dylan, Pull image still failed when run `apb run` command in cluster locally. The pushed image name is different with the `oc get image` image name. [root@qe-zhsun-611-310-auto-1master-etcd-1 hello-world-apb]# apb run --project test --registry-route docker-registry-default.apps.0610-q6m.qe.rhcloud.com Finished writing dockerfile. Building APB using tag: [docker-registry-default.apps.0610-q6m.qe.rhcloud.com/openshift/hello-world-apb] Successfully built APB image: docker-registry-default.apps.0610-q6m.qe.rhcloud.com/openshift/hello-world-apb Found image: docker-registry.default.svc:5000/openshift/hello-world-apb Warning: Tagged image registry prefix doesn't match. Deleting anyway. Given: docker-registry-default.apps.0610-q6m.qe.rhcloud.com; Found: docker-registry.default.svc:5000 Successfully deleted sha256:1f9a00ff391d955fcd8cd04e1c33db363c0344662d06fcc4cc80156cdc88330a Pushing the image, this could take a minute... Successfully pushed image: docker-registry-default.apps.0610-q6m.qe.rhcloud.com/openshift/hello-world-apb Creating project test Created project Creating service account in test Created service account Creating role binding for apb-run-provision-hello-world-apb in test Created Role Binding Creating pod with image docker-registry-default.apps.0610-q6m.qe.rhcloud.com/openshift/hello-world-apb in test Created Pod APB run started Pod in phase: Pending APB run failed: (APB failed ImagePullBackOff - check name) Reason: None [root@qe-zhsun-611-310-auto-1master-etcd-1 hello-world-apb]# oc get pod -n test NAME READY STATUS RESTARTS AGE apb-run-provision-hello-world-apbfdz45 0/1 ImagePullBackOff 0 4m [root@qe-zhsun-611-310-auto-1master-etcd-1 hello-world-apb]# oc describe pod apb-run-provision-hello-world-apbfdz45 -n test Name: apb-run-provision-hello-world-apbfdz45 Namespace: test Node: qe-zhsun-611-310-auto-1nrr-1/172.16.120.96 Start Time: Mon, 11 Jun 2018 02:04:01 -0400 Labels: <none> Annotations: openshift.io/scc=anyuid Status: Pending IP: 10.129.0.135 Containers: apb-run-provision-hello-world-apb: Container ID: Image: docker-registry-default.apps.0610-q6m.qe.rhcloud.com/openshift/hello-world-apb Image ID: Port: <none> Host Port: <none> Command: entrypoint.sh provision --extra-vars {"namespace": "test", "_apb_plan_id": "default"} State: Waiting Reason: ImagePullBackOff Ready: False Restart Count: 0 Environment: POD_NAME: apb-run-provision-hello-world-apbfdz45 (v1:metadata.name) POD_NAMESPACE: test (v1:metadata.namespace) Mounts: /var/run/secrets/kubernetes.io/serviceaccount from apb-run-provision-hello-world-apb-token-8cbqj (ro) Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: apb-run-provision-hello-world-apb-token-8cbqj: Type: Secret (a volume populated by a Secret) SecretName: apb-run-provision-hello-world-apb-token-8cbqj Optional: false QoS Class: BestEffort Node-Selectors: node-role.kubernetes.io/compute=true Tolerations: <none> Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 4m default-scheduler Successfully assigned apb-run-provision-hello-world-apbfdz45 to qe-zhsun-611-310-auto-1nrr-1 Normal Pulling 4m kubelet, qe-zhsun-611-310-auto-1nrr-1 pulling image "docker-registry-default.apps.0610-q6m.qe.rhcloud.com/openshift/hello-world-apb" Warning Failed 4m kubelet, qe-zhsun-611-310-auto-1nrr-1 Failed to pull image "docker-registry-default.apps.0610-q6m.qe.rhcloud.com/openshift/hello-world-apb": rpc error: code = Unknown desc = pinging docker registry returned: Get https://docker-registry-default.apps.0610-q6m.qe.rhcloud.com/v2/: x509: certificate signed by unknown authority Warning Failed 4m kubelet, qe-zhsun-611-310-auto-1nrr-1 Error: ErrImagePull Normal SandboxChanged 3m (x22 over 4m) kubelet, qe-zhsun-611-310-auto-1nrr-1 Pod sandbox changed, it will be killed and re-created. [root@qe-zhsun-611-310-auto-1master-etcd-1 hello-world-apb]# oc get image |grep hello sha256:1f9a00ff391d955fcd8cd04e1c33db363c0344662d06fcc4cc80156cdc88330a docker-registry.default.svc:5000/openshift/hello-world-apb@sha256:1f9a00ff391d955fcd8cd04e1c33db363c0344662d06fcc4cc80156cdc88330a
Sun, This unfortunately is a side affect of a misconfigured docker config. By default the openshift installer adds an entry `--insecure-registry 172.30.0.0/16`. When we push the image, we attempt to tag it with this IP as the prefix (172.30.1.1:5000/openshift/<apb_name>). If you are using a local cluster then you should have no need to use the `--registry-route` flag. This is what forces the tag to be <route>/openshift/<apb_name> which you probably don't have in your docker config as an `--insecure-registry` argument. $ cat /etc/sysconfig/docker I recommend either not using the --registry-route flag (since you are using the tooling in a local cluster as the tool expects, or you have to add an entry in your docker config: --insecure-registry docker-registry.default.svc:5000. Whenever you see: Failed to pull image "docker-registry-default.apps.0610-q6m.qe.rhcloud.com/openshift/hello-world-apb": rpc error: code = Unknown desc = pinging docker registry returned: Get https://docker-registry-default.apps.0610-q6m.qe.rhcloud.com/v2/: x509: certificate signed by unknown authority That means that the certs for the registry are failing, and since they use self signed certs we need to specify it as an insecure registry.