Bug 1388531

Summary: [Errno 25] Inappropriate ioctl for device
Product: [Fedora] Fedora Reporter: Juraci Paixão Kröhling <jcosta>
Component: ansibleAssignee: Kevin Fenzi <kevin>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 24CC: a.badger, athmanem, kevin, kupo, mark, maxim, toromoti
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ansible-2.2.0.0-3.fc24 ansible-2.2.0.0-3.fc25 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-07 23:27:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Juraci Paixão Kröhling 2016-10-25 14:58:40 UTC
Description of problem:
Since the last updates, cron operations are broken, with the message "[Errno 25] Inappropriate ioctl for device".

Version-Release number of selected component (if applicable):
ansible 2.1.2.0
Both the target and the local machines are fully up to date (Fedora 24). Kernel on both machines: 4.7.9-200.fc24.x86_64

How reproducible:
Always

Steps to Reproduce:
1. With the latest Ansible on Fedora 24, run a playbook with a cron section.

Actual results:
The cron will be updated by Ansible

Expected results:
A fatal error is thrown.

Additional info:

On the console, the following can be seen:

fatal: [riacho]: FAILED! => {"changed": false, "failed": true, "module_stderr": "", "module_stdout": "\r\nTraceback (most recent call last):\r\n  File \"/tmp/ansible_oUdIn7/ansible_module_cron.py\", line 700, in <module>\r\n    main()\r\n  File \"/tmp/ansible_oUdIn7/ansible_module_cron.py\", line 575, in main\r\n    crontab = CronTab(module, user, cron_file)\r\n  File \"/tmp/ansible_oUdIn7/ansible_module_cron.py\", line 233, in __init__\r\n    self.read()\r\n  File \"/tmp/ansible_oUdIn7/ansible_module_cron.py\", line 251, in read\r\n    (rc, out, err) = self.module.run_command(self._read_user_execute(), use_unsafe_shell=True)\r\n  File \"/tmp/ansible_oUdIn7/ansible_module_cron.py\", line 480, in _read_user_execute\r\n    elif os.getlogin() != self.user:\r\nOSError: [Errno 25] Inappropriate ioctl for device\r\n", "msg": "MODULE FAILURE"}


Sample Ansible instructions that will trigger this:

- name: Backup emails
  cron: name="backup emails" user=root special_time=daily job="/usr/bin/rsync -av /var/spool/mail/vhosts/ /mnt/backups/emails/ && touch /mnt/backups/emails/lastbackup"

- name: Renew certificates
  cron: name="Renew certificate for {{item}}" user=root special_time=monthly job="/usr/local/bin/update-letsencrypt-cert.sh {{item}}"
  with_items:
  - sub.domain1.tld
  - sub.domain2.tld


Manually ssh'ing into the machine works, sudo'ing to root (sudo -s) works, as well as "sudo crontab -l" and "sudo crontab -e"+changes.

Comment 1 Juraci Paixão Kröhling 2016-10-25 15:00:40 UTC
Additional info: this was working fine until at least Sept 22nd.

Comment 2 Kevin Fenzi 2016-10-25 15:25:53 UTC
I cannot duplicate this here. 

What is 'pipelining' set to in your ansible.cfg ? If it's True, does changing it to false make it work?

Comment 3 Juraci Paixão Kröhling 2016-10-25 15:30:34 UTC
It was originally commented out and I changed it to False after seeing a post on Stackoverflow, with no effect. I just changed to True, ran the playbook again, and got the same failure.

Comment 4 Kevin Fenzi 2016-10-25 16:10:18 UTC
Can you run the module by itself with debug and attach that?

ansible -vvvv -m cron -a 'name="backup emails" user=root special_time=daily job="/usr/bin/rsync -av /var/spool/mail/vhosts/ /mnt/backups/emails/ && touch /mnt/backups/emails/lastbackup"' hostname

Comment 5 Juraci Paixão Kröhling 2016-10-25 16:25:43 UTC
Sure: https://paste.fedoraproject.org/460873/47741270/

Comment 6 Juraci Paixão Kröhling 2016-10-25 16:27:45 UTC
Not sure it's relevant, but the host is available behind a bastion host. The ssh_config is as follows:

Host riacho
Hostname bastion.domain.tld
ForwardAgent yes
ControlMaster auto
ControlPersist yes
ControlPath ~/.ssh/socket-%r@%h:%p
Compression yes
AddressFamily inet

Comment 7 Kevin Fenzi 2016-10-31 17:17:56 UTC
I am pretty sure this is related to your bastion host setup. 

If you disable the Control* options does it work?

If you replace with: 

Host riacho
   HostName %h
   ProxyCommand ssh -q bastion.domain.tld /usr/bin/nc %h 22

Does it work?

Comment 8 Juraci Paixão Kröhling 2016-10-31 17:33:39 UTC
Sorry, it seems I was wrong: this host is one of the only ones where I don't have a bastion in front :)

In any case, I removed pretty much all custom settings from ~/.ssh/config related to this host, leaving only the "Host" and "Hostname" parts, and I still have the problem. It's worth noting that there are steps happening before this, so, the SSH connection is successful for those steps.

Logs for a successful scenario:
https://paste.fedoraproject.org/466918/93517214/

Logs for a failure scenario:
https://paste.fedoraproject.org/466919/47793520/

Comment 9 Juraci Paixão Kröhling 2016-10-31 17:46:50 UTC
I also managed to get the `cron` python script that Ansible copies to the remote host, and ran it manually, getting the same error.

Python script: https://paste.fedoraproject.org/466921/93570914/
Manual runs, with the same ioctl issue: https://paste.fedoraproject.org/466922/35952147/

Comment 10 Kevin Fenzi 2016-10-31 18:05:20 UTC
Looks like this is 

https://github.com/ansible/ansible-modules-core/pull/4777

which should be fixed in 2.1.3 (which should be out very soon) and 2.2.0...

Comment 11 Juraci Paixão Kröhling 2016-11-02 09:48:16 UTC
+1, that's indeed the issue. Patching cron.py made it work.

Comment 12 Fedora Update System 2016-11-02 17:30:35 UTC
ansible-2.2.0.0-2.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-e348c4e0e3

Comment 13 Fedora Update System 2016-11-02 17:31:30 UTC
ansible-2.2.0.0-2.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-a5f9956e38

Comment 14 Fedora Update System 2016-11-05 03:33:01 UTC
ansible-2.2.0.0-2.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-e348c4e0e3

Comment 15 Fedora Update System 2016-11-05 16:01:15 UTC
ansible-2.2.0.0-3.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-3113e71193

Comment 16 Fedora Update System 2016-11-05 16:02:00 UTC
ansible-2.2.0.0-3.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-3ccb098630

Comment 17 Fedora Update System 2016-11-05 19:01:13 UTC
ansible-2.2.0.0-3.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-3ccb098630

Comment 18 Fedora Update System 2016-11-06 01:53:22 UTC
ansible-2.2.0.0-3.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-3113e71193

Comment 19 Fedora Update System 2016-11-07 23:27:40 UTC
ansible-2.2.0.0-3.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.

Comment 20 Fedora Update System 2016-11-19 21:12:16 UTC
ansible-2.2.0.0-3.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.