Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1702498

Summary:	Re-running overcloud deploy command with Ceph nodes fails on TASK [unpack downloaded ceph-ansible fetch tarball to fetch directory
Product:	Red Hat OpenStack	Reporter:	Marius Cornea <mcornea>
Component:	openstack-tripleo-heat-templates	Assignee:	John Fulton <johfulto>
Status:	CLOSED ERRATA	QA Contact:	Yogev Rabl <yrabl>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	15.0 (Stein)	CC:	dbecker, emacchi, gcharot, gfidente, johfulto, jvisser, mburns, morazi, sclewis, tenobreg
Target Milestone:	beta	Keywords:	Triaged
Target Release:	15.0 (Stein)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-tripleo-heat-templates-10.5.1-0.20190509160423.4dac4dc.el8ost.noarch	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-09-21 11:21:34 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1705248
Bug Blocks:

Description Marius Cornea 2019-04-23 23:18:17 UTC

Description of problem:

Re-running overcloud deploy command with Ceph nodes fails on TASK [unpack downloaded ceph-ansible fetch tarball to fetch directory:

2019-04-23 22:56:10,754 p=21491 u=mistral |  TASK [unpack downloaded ceph-ansible fetch tarball to fetch directory] *********
2019-04-23 22:56:10,754 p=21491 u=mistral |  Tuesday 23 April 2019  22:56:10 +0000 (0:00:00.114)       0:15:27.699 ********* 
2019-04-23 22:56:11,066 p=21491 u=mistral |  fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "/usr/bin/gtar --gzip --extract --file /tmp/temporary_dir_old.tar.gz -C /var/lib/mistral/overcloud/ceph-ansible/fetch_dir", "delta": "0:00:00.008969", "end": "2019-04-23 22:56:11.043733", "msg": "non-zero return code", "rc": 2, "start": "2019-04-23 22:56:11.034764", "stderr": "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc: Cannot utime: Operation not permitted\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var: Cannot utime: Operation not permitted\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.mgr.controller-0.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.mgr.controller-1.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.mgr.controller-2.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.client.admin.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.client.openstack.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.client.manila.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.client.radosgw.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph: Cannot utime: Operation not permitted\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-osd: Cannot utime: Operation not permitted\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rgw: Cannot utime: Operation not permitted\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-mds: Cannot utime: Operation not permitted\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rbd: Cannot utime: Operation not permitted\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rbd-mirror: Cannot utime: Operation not permitted\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-osd/ceph.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rgw/ceph.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-mds/ceph.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rbd/ceph.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rbd-mirror/ceph.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph: Cannot utime: Operation not permitted\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib: Cannot utime: Operation not permitted\n/usr/bin/gtar: Exiting with failure status due to previous errors", "stderr_lines": ["/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc: Cannot utime: Operation not permitted", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var: Cannot utime: Operation not permitted", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.mgr.controller-0.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.mgr.controller-1.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.mgr.controller-2.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.client.admin.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.client.openstack.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.client.manila.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.client.radosgw.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph: Cannot utime: Operation not permitted", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-osd: Cannot utime: Operation not permitted", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rgw: Cannot utime: Operation not permitted", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-mds: Cannot utime: Operation not permitted", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rbd: Cannot utime: Operation not permitted", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rbd-mirror: Cannot utime: Operation not permitted", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-osd/ceph.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rgw/ceph.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-mds/ceph.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rbd/ceph.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rbd-mirror/ceph.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph: Cannot utime: Operation not permitted", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib: Cannot utime: Operation not permitted", "/usr/bin/gtar: Exiting with failure status due to previous errors"], "stdout": "", "stdout_lines": []}


Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-10.5.1-0.20190423085106.3f148c4.el8ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy overcloud with ceph enabled
2. Re-run overcloud deploy with --config-download-only

Actual results:
Fails on TASK [unpack downloaded ceph-ansible fetch tarball to fetch directory

Expected results:
No failures on re-run.

Additional info:

Comment 1 John Fulton 2019-04-24 13:20:45 UTC

WORKAROUND:

If you're using the defaults:

  source stackrc
  swift list
  swift list ceph_ansible_fetch_dir
  swift download ceph_ansible_fetch_dir <tar file from last command>

If you're using LocalCephAnsibleFetchDirectoryBackup

 rm -rf /tmp/fetch_dir/temporary_dir.tar.gz

Or remove whatever you set LocalCephAnsibleFetchDirectoryBackup to. E.g. the above example assumes it was set to /tmp/fetch_dir/. 

This should let you redeploy, but if you're scaling monitors the above actions will cause monitor scaling to not work.

Comment 2 John Fulton 2019-04-24 13:22:40 UTC

 swift download ceph_ansible_fetch_dir <tar file from last command>(In reply to John Fulton from comment #1)
> WORKAROUND:
> 
> If you're using the defaults:
> 
>   source stackrc
>   swift list
>   swift list ceph_ansible_fetch_dir
>   swift download ceph_ansible_fetch_dir <tar file from last command>

AND (and this step is important): 

swift delete ceph_ansible_fetch_dir <tar file from last command>

Comment 8 Yogev Rabl 2019-06-13 17:32:23 UTC

Verified

Comment 12 errata-xmlrpc 2019-09-21 11:21:34 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811

Comment 13 John Fulton 2019-09-23 13:28:55 UTC

Clearing needinfo. No doctext necessary.