Bug 1540845

Summary: [Ceph-ansible] Failure on TASK [ceph-iscsi-gw : lock ssl file access to root only]
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Pratik Surve <prsurve>
Component: Ceph-AnsibleAssignee: Sébastien Han <shan>
Status: CLOSED ERRATA QA Contact: Madhavi Kasturi <mkasturi>
Severity: urgent Docs Contact: Erin Donnelly <edonnell>
Priority: medium    
Version: 3.0CC: adeza, agunn, anharris, aschoen, asriram, ceph-eng-bugs, edonnell, gmeno, hnallurv, jdillama, kdreyer, nthomas, sankarshan, shan, tchandra, vakulkar
Target Milestone: z2Keywords: Regression
Target Release: 3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: RHEL: ceph-ansible-3.0.29-1.el7cp Ubuntu: ceph-ansible_3.0.29-2redhat1 Doc Type: Bug Fix
Doc Text:
Previously, deployment of only one Ceph iSCSI Gateway node was supported because SSL certificated were only generated for one Ceph iSCSI Gateway node when using the "ceph-ansible" utility. This issue has been fixed, the certificates are distributed across all the gateways and more than one iSCSI Gateway node can now be deployed.
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-26 17:38:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1544643, 1557269, 1600697    
Attachments:
Description Flags
The zip contains logs of ansible, iscsi-gws.yml,all.yml none

Description Pratik Surve 2018-02-01 06:41:22 UTC
Created attachment 1389346 [details]
The zip contains logs of ansible, iscsi-gws.yml,all.yml

Description of problem:
i was setting-up iscsi-gateways using ceph ansible with 2 node collocated with osd ceph-ansible failed saying 

TASK [ceph-iscsi-gw : lock ssl file access to root only] **********************************************************************************************************************************************************
 [WARNING]: when statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found: ceph_release_num.{{ ceph_release }} >= ceph_release_num.luminous

 [WARNING]: when statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found: ceph_release_num.{{ ceph_release }} >= ceph_release_num.luminous

changed: [magna075] => (item=/etc/ceph/iscsi-gateway.crt)
failed: [magna078] (item=/etc/ceph/iscsi-gateway.crt) => {"changed": false, "item": "/etc/ceph/iscsi-gateway.crt", "msg": "file (/etc/ceph/iscsi-gateway.crt) is absent, cannot continue", "path": "/etc/ceph/iscsi-gateway.crt", "state": "absent"}
changed: [magna075] => (item=/etc/ceph/iscsi-gateway.key)
failed: [magna078] (item=/etc/ceph/iscsi-gateway.key) => {"changed": false, "item": "/etc/ceph/iscsi-gateway.key", "msg": "file (/etc/ceph/iscsi-gateway.key) is absent, cannot continue", "path": "/etc/ceph/iscsi-gateway.key", "state": "absent"}
changed: [magna075] => (item=/etc/ceph/iscsi-gateway.pem)
failed: [magna078] (item=/etc/ceph/iscsi-gateway.pem) => {"changed": false, "item": "/etc/ceph/iscsi-gateway.pem", "msg": "file (/etc/ceph/iscsi-gateway.pem) is absent, cannot continue", "path": "/etc/ceph/iscsi-gateway.pem", "state": "absent"}
changed: [magna075] => (item=/etc/ceph/iscsi-gateway-pub.key)
failed: [magna078] (item=/etc/ceph/iscsi-gateway-pub.key) => {"changed": false, "item": "/etc/ceph/iscsi-gateway-pub.key", "msg": "file (/etc/ceph/iscsi-gateway-pub.key) is absent, cannot continue", "path": "/etc/ceph/iscsi-gateway-pub.key", "state": "absent"}

TASK [ceph-iscsi-gw : igw_lun | configure luns (create/map rbds and add to lio)] **********************************************************************************************************************************
 [WARNING]: when statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found: ceph_release_num.{{ ceph_release }} >= ceph_release_num.luminous

changed: [magna075] => (item={u'host': u'magna075', u'image': u'ansible1', u'state': u'present', u'pool': u'rbd', u'size': u'30G'})
failed: [magna075] (item={u'host': u'magna078', u'image': u'ansible2', u'state': u'present', u'pool': u'rbd', u'size': u'15G'}) => {"changed": false, "item": {"host": "magna078", "image": "ansible2", "pool": "rbd", "size": "15G", "state": "present"}, "msg": "host name given for ansible2 is not a valid gateway name, listed in the config"}


Version-Release number of selected component (if applicable):
ceph-ansible-3.0.14-1.el7cp.noarch
ceph-iscsi-config-2.3-12.el7cp.noarch	


How reproducible:
always

Steps to Reproduce:
1.configure iscsi-gws.yml as per documentation
2.run the playbook

Actual results:


Expected results:


Additional info:
Attaching the playbook log.

Comment 4 Andrew Schoen 2018-02-12 16:34:18 UTC
The problem here is that even if you have multiple iscsi-gw nodes the playbook is only creating keys on one of the nodes. This happens because of ``run_once: True`` here: https://github.com/ceph/ceph-ansible/blob/stable-3.0/roles/ceph-iscsi-gw/tasks/generate_crt.yml#L5

However, when you get to the 'lock ssl file access to root only' task it tries to run on both iscsi-gw nodes and one fails because keys where not generated on it. Notice the lack of ``run_once: True`` here: https://github.com/ceph/ceph-ansible/blob/stable-3.0/roles/ceph-iscsi-gw/tasks/generate_crt.yml#L20

Comment 13 Sébastien Han 2018-03-13 16:46:14 UTC
Can we just then add a doc text and move ahead with this?
Thanks

Comment 19 Sébastien Han 2018-03-28 15:57:16 UTC
Tejas, this is surprising, there hasn't been any chances since 3.0.25-1.el7cp.noarch, see:

git show --name-only --pretty="" v3.0.25..v3.0.28
roles/ceph-common/tasks/misc/system_tuning.yml
roles/ceph-rgw/handlers/main.yml
tests/functional/centos/7/docker/group_vars/rgws
roles/ceph-rgw/tasks/docker/copy_configs.yml
roles/ceph-rgw/handlers/main.yml
tox.ini
roles/ceph-client/tasks/pre_requisite.yml
roles/ceph-mds/tasks/non_containerized.yml
roles/ceph-mon/tasks/openstack_config.yml
tests/functional/centos/7/cluster/ceph-override.json
tests/functional/centos/7/ooo-collocation/Vagrantfile
tests/functional/centos/7/ooo-collocation/ceph-override.json
tests/functional/centos/7/ooo-collocation/hosts
tests/functional/centos/7/ooo-collocation/vagrant_variables.yml
tox.ini
roles/ceph-defaults/handlers/main.yml
roles/ceph-common/tasks/misc/system_tuning.yml
infrastructure-playbooks/rolling_update.yml

The role ceph-iscsi hasn't been touched, so I'm wondering why the workaround doesn't work.

Are we talking about the original issue or something new just came up?
Thanks.

Comment 21 Sébastien Han 2018-03-29 11:47:08 UTC
Unfortunately, it is not reproducible anywhere, can we access an env where the issue is reproducible 100%? Thanks.

Comment 24 Sébastien Han 2018-04-03 13:20:57 UTC
Testing a patch upstream.

Comment 29 Sébastien Han 2018-04-12 16:36:38 UTC
Erin, edited the text, feel free to re-phrase :)

Comment 33 errata-xmlrpc 2018-04-26 17:38:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1259