Bug 1702498 - Re-running overcloud deploy command with Ceph nodes fails on TASK [unpack downloaded ceph-ansible fetch tarball to fetch directory
Summary: Re-running overcloud deploy command with Ceph nodes fails on TASK [unpack dow...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 15.0 (Stein)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: beta
: 15.0 (Stein)
Assignee: John Fulton
QA Contact: Yogev Rabl
URL:
Whiteboard:
Depends On: 1705248
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-04-23 23:18 UTC by Marius Cornea
Modified: 2019-09-26 10:49 UTC (History)
11 users (show)

Fixed In Version: openstack-tripleo-heat-templates-10.5.1-0.20190509160423.4dac4dc.el8ost.noarch
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-09-21 11:21:34 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1823229 0 None None None 2019-04-24 14:16:10 UTC
OpenStack gerrit 655568 0 None MERGED Remove ceph-ansible fetch directory after it is saved 2020-07-01 07:06:38 UTC
OpenStack gerrit 656351 0 None MERGED Remove ceph-ansible fetch directory after it is saved 2020-07-01 07:06:38 UTC
Red Hat Product Errata RHEA-2019:2811 0 None None None 2019-09-21 11:21:48 UTC

Description Marius Cornea 2019-04-23 23:18:17 UTC
Description of problem:

Re-running overcloud deploy command with Ceph nodes fails on TASK [unpack downloaded ceph-ansible fetch tarball to fetch directory:

2019-04-23 22:56:10,754 p=21491 u=mistral |  TASK [unpack downloaded ceph-ansible fetch tarball to fetch directory] *********
2019-04-23 22:56:10,754 p=21491 u=mistral |  Tuesday 23 April 2019  22:56:10 +0000 (0:00:00.114)       0:15:27.699 ********* 
2019-04-23 22:56:11,066 p=21491 u=mistral |  fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "/usr/bin/gtar --gzip --extract --file /tmp/temporary_dir_old.tar.gz -C /var/lib/mistral/overcloud/ceph-ansible/fetch_dir", "delta": "0:00:00.008969", "end": "2019-04-23 22:56:11.043733", "msg": "non-zero return code", "rc": 2, "start": "2019-04-23 22:56:11.034764", "stderr": "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc: Cannot utime: Operation not permitted\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var: Cannot utime: Operation not permitted\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.mgr.controller-0.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.mgr.controller-1.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.mgr.controller-2.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.client.admin.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.client.openstack.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.client.manila.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.client.radosgw.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph: Cannot utime: Operation not permitted\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-osd: Cannot utime: Operation not permitted\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rgw: Cannot utime: Operation not permitted\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-mds: Cannot utime: Operation not permitted\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rbd: Cannot utime: Operation not permitted\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rbd-mirror: Cannot utime: Operation not permitted\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-osd/ceph.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rgw/ceph.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-mds/ceph.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rbd/ceph.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rbd-mirror/ceph.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph: Cannot utime: Operation not permitted\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib: Cannot utime: Operation not permitted\n/usr/bin/gtar: Exiting with failure status due to previous errors", "stderr_lines": ["/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc: Cannot utime: Operation not permitted", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var: Cannot utime: Operation not permitted", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.mgr.controller-0.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.mgr.controller-1.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.mgr.controller-2.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.client.admin.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.client.openstack.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.client.manila.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.client.radosgw.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph: Cannot utime: Operation not permitted", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-osd: Cannot utime: Operation not permitted", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rgw: Cannot utime: Operation not permitted", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-mds: Cannot utime: Operation not permitted", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rbd: Cannot utime: Operation not permitted", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rbd-mirror: Cannot utime: Operation not permitted", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-osd/ceph.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rgw/ceph.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-mds/ceph.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rbd/ceph.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rbd-mirror/ceph.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph: Cannot utime: Operation not permitted", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib: Cannot utime: Operation not permitted", "/usr/bin/gtar: Exiting with failure status due to previous errors"], "stdout": "", "stdout_lines": []}


Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-10.5.1-0.20190423085106.3f148c4.el8ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy overcloud with ceph enabled
2. Re-run overcloud deploy with --config-download-only

Actual results:
Fails on TASK [unpack downloaded ceph-ansible fetch tarball to fetch directory

Expected results:
No failures on re-run.

Additional info:

Comment 1 John Fulton 2019-04-24 13:20:45 UTC
WORKAROUND:

If you're using the defaults:

  source stackrc
  swift list
  swift list ceph_ansible_fetch_dir
  swift download ceph_ansible_fetch_dir <tar file from last command>

If you're using LocalCephAnsibleFetchDirectoryBackup

 rm -rf /tmp/fetch_dir/temporary_dir.tar.gz

Or remove whatever you set LocalCephAnsibleFetchDirectoryBackup to. E.g. the above example assumes it was set to /tmp/fetch_dir/. 

This should let you redeploy, but if you're scaling monitors the above actions will cause monitor scaling to not work.

Comment 2 John Fulton 2019-04-24 13:22:40 UTC
 swift download ceph_ansible_fetch_dir <tar file from last command>(In reply to John Fulton from comment #1)
> WORKAROUND:
> 
> If you're using the defaults:
> 
>   source stackrc
>   swift list
>   swift list ceph_ansible_fetch_dir
>   swift download ceph_ansible_fetch_dir <tar file from last command>

AND (and this step is important): 

swift delete ceph_ansible_fetch_dir <tar file from last command>

Comment 8 Yogev Rabl 2019-06-13 17:32:23 UTC
Verified

Comment 12 errata-xmlrpc 2019-09-21 11:21:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811

Comment 13 John Fulton 2019-09-23 13:28:55 UTC
Clearing needinfo. No doctext necessary.


Note You need to log in before you can comment on or make changes to this bug.