Bug 1401336

Summary: ansible fails on permissions without -vvvv to slow down, on raspi3
Product: [Fedora] Fedora Reporter: Ilkka Tengvall <ikke>
Component: ansibleAssignee: Kevin Fenzi <kevin>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 25CC: 707119577, a.badger, athmanem, kevin, kupo, mark, maxim, toromoti
Target Milestone: ---   
Target Release: ---   
Hardware: armv7hl   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-10-30 20:22:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Ilkka Tengvall 2016-12-04 22:20:39 UTC
Description of problem:

I installed Fedora 25 on my RasPi 3, and started configuring it via Ansible. It always fails on filesystem permissions error, unless I slow down the process e.g. by -vvvv to print debug to slow it down.

Version-Release number of selected component (if applicable):

on my F25 laptop:

ansible-2.2.0.0-3.fc25.noarch

On my F25 raspi3 (up to date as of 2016-12-05):

kernel-4.8.10-300.fc25.armv7hl
python-2.7.12-7.fc25.armv7hl
fedora-release-25-1.noarch


How reproducible:

Very often, unless slowed down by -vvvv. Almost every time.

Steps to Reproduce:
1. Do some ansible playbook, and run it. The ansible_user is root,
   ansible_became is false.:

ansible-playbook -i hosts site.yml --limit raspi


Actual results:

TASK [setup] *******************************************************************
fatal: [raspi]: UNREACHABLE! => {"changed": false, "msg": "Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the remote directory. Consider changing the remote temp path in ansible.cfg to a path rooted in \"/tmp\". Failed command was: ( umask 77 && mkdir -p \"` echo $HOME/.ansible/tmp/ansible-tmp-1480889344.21-18688229650665 `\" && echo ansible-tmp-1480889344.21-18688229650665=\"` echo $HOME/.ansible/tmp/ansible-tmp-1480889344.21-18688229650665 `\" ), exited with result 127: ansible-tmp-1480889344.21-18688229650665=/root/.ansible/tmp/ansible-tmp-1480889344.21-18688229650665\n", "unreachable": true}


Expected results:

ansible to progress further:

ansible-playbook -i hosts site.yml --limit raspi -vvvv

Loading callback plugin default of type stdout, v2.0 from /usr/lib/python2.7/site-packages/ansible/plugins/callback/__init__.pyc

PLAYBOOK: site.yml *************************************************************
8 plays in site.yml

PLAY [apply common configuration to all nodes] *********************************

TASK [setup] *******************************************************************
Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/system/setup.py
<raspi> ESTABLISH SSH CONNECTION FOR USER: root
<raspi> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/home/ikke/.ansible/cp/ansible-ssh-%C raspi '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1480889633.09-217626402117535 `" && echo ansible-tmp-1480889633.09-217626402117535="` echo $HOME/.ansible/tmp/ansible-tmp-1480889633.09-217626402117535 `" ) && sleep 0'"'"''
<raspi> PUT /tmp/tmpNnwhh0 TO /root/.ansible/tmp/ansible-tmp-1480889633.09-217626402117535/setup.py
<raspi> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/home/ikke/.ansible/cp/ansible-ssh-%C '[raspi]'
<raspi> ESTABLISH SSH CONNECTION FOR USER: root
<raspi> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/home/ikke/.ansible/cp/ansible-ssh-%C raspi '/bin/sh -c '"'"'chmod u+x /root/.ansible/tmp/ansible-tmp-1480889633.09-217626402117535/ /root/.ansible/tmp/ansible-tmp-1480889633.09-217626402117535/setup.py && sleep 0'"'"''
<raspi> ESTABLISH SSH CONNECTION FOR USER: root
<raspi> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/home/ikke/.ansible/cp/ansible-ssh-%C -tt raspi '/bin/sh -c '"'"'/usr/bin/python /root/.ansible/tmp/ansible-tmp-1480889633.09-217626402117535/setup.py; rm -rf "/root/.ansible/tmp/ansible-tmp-1480889633.09-217626402117535/" > /dev/null 2>&1 && sleep 0'"'"''
ok: [raspi]




Additional info:

Comment 1 Kevin Fenzi 2016-12-07 03:49:03 UTC
That is... quite odd. 

Is /root/ on the remote machine on any kind of odd partition? Or low on space?

Can you run the failing case with: 

ANSIBLE_KEEP_REMOTE_FILES=1

and then ssh over to the remote host and run the failing temp files with python and see what error(s) you get?

Comment 2 Kevin Fenzi 2017-06-01 22:55:45 UTC
Any news here? Does this still happen with the latest ansible version? 

Can you look at the remote files as indicated in the last comment?

Comment 3 Kevin Fenzi 2017-10-30 20:22:12 UTC
Closing this now. Please feel free to re-open or file a new bug if you are still seeing this issue.

Comment 4 Vincent Corleone 2018-10-12 02:24:10 UTC
(In reply to Kevin Fenzi from comment #1)
> That is... quite odd. 
> 
> Is /root/ on the remote machine on any kind of odd partition? Or low on
> space?
> 
> Can you run the failing case with: 
> 
> ANSIBLE_KEEP_REMOTE_FILES=1
> 
> and then ssh over to the remote host and run the failing temp files with
> python and see what error(s) you get?

What do you mean by 'run the failing case with: ANSIBLE_KEEP_REMOTE_FILES=1'?