Bug 1618718

what namespace are you pushing to?  what account are you doing the oc+docker login with? are you sure the account you did a "docker login" with has permission to create imagestreams/tags in that namespace?

1 - Namespace?
geosolution-geosolutions.a3c1.starter-us-west-1.openshiftapps.com

2 - Account?
johan.holtman

3 - Persmissions? 
Yes, I pushed a lot of images. The current one is running ;-)

I don't see any requests related to that namespace in our registry logs, what is the specific docker image name you are pushing?

These are the commands I do: 

$ docker build -t geosolutions/geosolution:latest . 

$ docker tag geosolutions/geosolution registry.starter-us-west1.openshift.com/geosolutions/geosolution

From the start: 
* On windows10 home I start the Docker Quickstart Terminal
* oc login ... the command copied from the GUI
* docker login 
* I build with maven
* docker build -t geosolutions/geosolution:latest . 
* docker tag geosolutions/geosolution registry.starter-us-west1.openshift.com/geosolutions/geosolution

registry.starter-us-west1.openshift.com should be registry.starter-us-west-1.openshift.com

but i assume you had that right in your attempts.

Can you please try to push again so i can hope to grab some current logs that might show the push failure?

OK, give me a few secs

Done

ok, it seems like something is wrong w/ your account token.

can you rerun your docker login?

"docker login -u john.holtman -p `oc whoami -t`"

and then try to push again.

also can you show me the output of "oc whoami"?

$ oc whoami

jandroid1

$ rerun of docker login done. 
Login succeeded -- as previous times.

Rerun done again. 
I got a new token since last week. I copied from the GUI - as requested.

Can you remove your .docker/config.json and redo the login? i'm looking for a clean start here.

based on what i see in the logs the token your docker client is sending does not look right and i wonder if it's still sending the old token for some reason.

1 - Docker terminal: docker-machine stop. Exit. 
2 - Moved '.docker/config.json to a new sub folder. 
3 - Restarted docker quickstart terminal
4 - New oc login + new docker login + docker tag
5 - Docker push ... 

Same error

The logs are insisting the token your docker client is passing w/ the request is not a valid openshift token.

in your .docker/config.json, take a look at the "auth" block for the openshift registry entry.

base64 decode the content, it should look something like "username:tokenvalue" after you decode it.

Then see if you can do an "oc login --token tokenvalue"

Also if there are other fields in the registry entry, can you supply your (redacted) .docker/config.json?

? Base64 decode: shows johan.holtman:expected_token
This is what is expected. 

? oc login --token tokenvalue
This is how exactly login
$ oc login https://api.starter-us-west-1.openshift.com --token=expected_token

No problem. I get back: 
Logged into "https://api.starter-us-west-1.openshift.com:443" as "jandroid1" using the token provided.
You have one project on this server: "geosolutions"
Using project "geosolutions".

Docker login was als successful, etc. 
Same error. 

I guess using Docker quickstart terminal is no problem. 
I used to work with minishift, but that does not work anymore. WIthout any changes. 
Maybe this is of help: 
Within the Docker Quickstart terminal, can I start minishift? 

When I perform this command ($ minishift start --vm-driver=virtualbox --memory=3G) then I get a lot of "OK" but finally get: 

-- Starting Minishift VM .... FAIL E0818 11:14:41.501787     760 start.go:428] Error starting the VM: Error getting the state for host: machine does not exist. Retrying.
Error starting the VM: Error getting the state for host: machine does not exist

Can I do anything to continue deploying to Docker / CAAS?

I discussed this w/ my team this morning and their assessment is that the token docker is passing to the registry is invalid or expired (pretty much what i was seeing).

Since you have erased your .docker/config.json and generated a new token, I'm not sure how that is possible, but I have confirmed that I am able to push images using my account/token (on the same starter-us-west-1 cluster you are using), so it does seem to be a specific issue w/ your token or docker client.

I would still be interested in seeing your .docker/config.json file w/ the auth tokens redacted.

Also a bit of a long shot, but have you tried this workflow from any other machines?

I will try to replay the situation in about 9 hours. Hold on.

1 - the config.json

{
	"auths": {
		"registry.starter-us-west-1.openshift.com:443": {
			"auth": "am9oY***ZMkM0"
		}
	},
	"HttpHeaders": {
		"User-Agent": "Docker-Client/18.03.0-ce (windows)"
	}
}

2 - another machine? 

Not possible.

the only difference I see is that my .docker/config.json does not have the port:

"registry.starter-us-west-1.openshift.com": {
                        "auth": "xxxxxxxxx"
                }

but it does make me wonder if that is what is screwing things up for you, can you try removing it?

OK, I will remove it in 9 hours and make a retry. 

If that is not working, can you give a new token? I cannot deploy a new site since about 2 weeks.

I actually don't have a way to give you a new token, but doing an oc login should get you a new token if you are not already logged in.  (removing your ~/.kube/config will clean up all your local tokens and should allow you to re-login and get a new token).

you can also get a new token from the web console.

1 - removed the ~/.kube/config

DONE

2 - Relogin

The token from the user interface did not change. It is put back into the new ~/.kube/config file. 

Just to confirm: 
* I start Docker quickstart terminal. (I dont use minishift). 
* I oc login with the token (from the OC login)
* I docker login with the same token
* I push my tagged image ... and Yes, I get the same error.

and you removed the port from your .docker/config.json?

Removed kube config, docker config, stopped docker, stopped all, restarted docker quickstart terminal, oc login = ok, docker login = ok, docker push ... failed with same error.

Still not clear to me you've removed the port value from your .docker/config.json?

See above, I have removed the entire docker/.config 2 times. 
With stopping, starting, etc.

removing the entire config is not what i am asking you to do.

remove the PORT from your .docker/config.json.

make it look like:
{
	"auths": {
		"registry.starter-us-west-1.openshift.com": {
			"auth": "am9oY***ZMkM0"
		}
	},
	"HttpHeaders": {
		"User-Agent": "Docker-Client/18.03.0-ce (windows)"
	}
}

It works -- thank you!

I changed 2 things: 

In the .docker/config.json: 
From: 
   "registry.starter-us-west-1.openshift.com:443": {
To: 
   "registry.starter-us-west-1.openshift.com": {

The docker login command: 
From: 
docker login -u johan.holtman -p ***_5tgHY2C4 registry.starter-us-west-1.openshift.com:443

To:  
docker login -u johan.holtman -p ***_5tgHY2C4 registry.starter-us-west-1.openshift.com:443

Excellent!  I assume that second command was supposed to be pasted as:

docker login -u johan.holtman -p ***_5tgHY2C4 registry.starter-us-west-1.openshift.com

(no :443).

*** Bug 1622291 has been marked as a duplicate of this bug. ***