Description of problem: Re-running overcloud deploy command with Ceph nodes fails on TASK [unpack downloaded ceph-ansible fetch tarball to fetch directory: 2019-04-23 22:56:10,754 p=21491 u=mistral | TASK [unpack downloaded ceph-ansible fetch tarball to fetch directory] ********* 2019-04-23 22:56:10,754 p=21491 u=mistral | Tuesday 23 April 2019 22:56:10 +0000 (0:00:00.114) 0:15:27.699 ********* 2019-04-23 22:56:11,066 p=21491 u=mistral | fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "/usr/bin/gtar --gzip --extract --file /tmp/temporary_dir_old.tar.gz -C /var/lib/mistral/overcloud/ceph-ansible/fetch_dir", "delta": "0:00:00.008969", "end": "2019-04-23 22:56:11.043733", "msg": "non-zero return code", "rc": 2, "start": "2019-04-23 22:56:11.034764", "stderr": "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc: Cannot utime: Operation not permitted\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var: Cannot utime: Operation not permitted\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.mgr.controller-0.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.mgr.controller-1.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.mgr.controller-2.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.client.admin.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.client.openstack.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.client.manila.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.client.radosgw.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph: Cannot utime: Operation not permitted\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-osd: Cannot utime: Operation not permitted\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rgw: Cannot utime: Operation not permitted\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-mds: Cannot utime: Operation not permitted\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rbd: Cannot utime: Operation not permitted\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rbd-mirror: Cannot utime: Operation not permitted\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-osd/ceph.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rgw/ceph.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-mds/ceph.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rbd/ceph.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rbd-mirror/ceph.keyring: Cannot open: File exists\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph: Cannot utime: Operation not permitted\n/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib: Cannot utime: Operation not permitted\n/usr/bin/gtar: Exiting with failure status due to previous errors", "stderr_lines": ["/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc: Cannot utime: Operation not permitted", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var: Cannot utime: Operation not permitted", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.mgr.controller-0.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.mgr.controller-1.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.mgr.controller-2.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.client.admin.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.client.openstack.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.client.manila.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph/ceph.client.radosgw.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/etc/ceph: Cannot utime: Operation not permitted", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-osd: Cannot utime: Operation not permitted", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rgw: Cannot utime: Operation not permitted", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-mds: Cannot utime: Operation not permitted", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rbd: Cannot utime: Operation not permitted", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rbd-mirror: Cannot utime: Operation not permitted", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-osd/ceph.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rgw/ceph.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-mds/ceph.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rbd/ceph.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph/bootstrap-rbd-mirror/ceph.keyring: Cannot open: File exists", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib/ceph: Cannot utime: Operation not permitted", "/usr/bin/gtar: 126e4394-6606-11e9-b9fb-5254007e41bb/var/lib: Cannot utime: Operation not permitted", "/usr/bin/gtar: Exiting with failure status due to previous errors"], "stdout": "", "stdout_lines": []} Version-Release number of selected component (if applicable): openstack-tripleo-heat-templates-10.5.1-0.20190423085106.3f148c4.el8ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy overcloud with ceph enabled 2. Re-run overcloud deploy with --config-download-only Actual results: Fails on TASK [unpack downloaded ceph-ansible fetch tarball to fetch directory Expected results: No failures on re-run. Additional info:
WORKAROUND: If you're using the defaults: source stackrc swift list swift list ceph_ansible_fetch_dir swift download ceph_ansible_fetch_dir <tar file from last command> If you're using LocalCephAnsibleFetchDirectoryBackup rm -rf /tmp/fetch_dir/temporary_dir.tar.gz Or remove whatever you set LocalCephAnsibleFetchDirectoryBackup to. E.g. the above example assumes it was set to /tmp/fetch_dir/. This should let you redeploy, but if you're scaling monitors the above actions will cause monitor scaling to not work.
swift download ceph_ansible_fetch_dir <tar file from last command>(In reply to John Fulton from comment #1) > WORKAROUND: > > If you're using the defaults: > > source stackrc > swift list > swift list ceph_ansible_fetch_dir > swift download ceph_ansible_fetch_dir <tar file from last command> AND (and this step is important): swift delete ceph_ansible_fetch_dir <tar file from last command>
Verified
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:2811
Clearing needinfo. No doctext necessary.