Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 1927685

Summary: [ceph-ansible] : Rolling update failed in TASK [ceph-nfs : create rgw nfs user "cephnfs"] while upgrading from 4.x to 5.x
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Manasa <mgowri>
Component: Ceph-AnsibleAssignee: Guillaume Abrioux <gabrioux>
Status: CLOSED CANTFIX QA Contact: Manasa <mgowri>
Severity: high Docs Contact:
Priority: unspecified    
Version: 5.0CC: aschoen, ceph-eng-bugs, gabrioux, gmeno, nthomas, vashastr, vereddy, ykaul
Target Milestone: ---Keywords: Regression, UpgradeBlocker
Target Release: 5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-03-10 14:17:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 499109, 1917530, 1918424    

Description Manasa 2021-02-11 10:33:29 UTC
Description of problem:
Rolling update to 5.0 on an existing 4.2 cluster failed in TASK [ceph-nfs : create rgw nfs user "cephnfs"]


Version-Release number of selected component (if applicable):

ceph-ansible-6.0.0-0.1.alpha7.el8cp.noarch
ansible-2.9.17-1.el8ae.noarch


How reproducible:
Tried once


Steps to Reproduce:
1. Deploy a ceph 4.2 cluster on rhel-8 with an nfs node
2. Upgrade it to 5.0 using rolling_update.yml playbook

Actual results:
The upgrade fails in TASK [ceph-nfs : create rgw nfs user "cephnfs"] with error:

stderr: |-
    2021-02-11T09:06:38.528+0000 7ffaa1b86200 -1 int rgw::cls::fifo::list_part(librados::v14_2_0::IoCtx&, const string&, std::optional<std::basic_string_view<char> >, uint64_t, uint64_t, std::vector<rados::cls::fifo::part_list_entry>, bool, bool*, std::__cxx11::string*, uint64_t, optional_yield):228 fifo::op::LIST_PART failed r=-1 tid=1
    2021-02-11T09:06:38.528+0000 7ffaa1b86200 -1 int rgw::cls::fifo::FIFO::list(int, std::optional<std::basic_string_view<char> >, std::vector<rgw::cls::fifo::list_entry>, bool, optional_yield):1229 list_entries failed: r=-1 tid= 1
    2021-02-11T09:06:38.528+0000 7ffaa1b86200 -1 static int RGWDataChangesFIFO::exists(ceph::common::CephContext*, librados::v14_2_0::Rados*, const rgw_pool&, bool*, bool*): unable to list entries: default.rgw.log/data_log.2: (1) Operation not permitted
    2021-02-11T09:06:38.529+0000 7ffaa1b86200 -1 int rgw::cls::fifo::get_meta(librados::v14_2_0::IoCtx&, const string&, std::optional<rados::cls::fifo::objv>, rados::cls::fifo::info*, uint32_t*, uint32_t*, uint64_t, optional_yield, bool):105 fifo::op::GET_META failed r=-1 tid=0
    2021-02-11T09:06:38.529+0000 7ffaa1b86200 -1 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr<rgw::cls::fifo::FIFO>*, optional_yield, std::optional<rados::cls::fifo::objv>, bool):884 get_meta failed: r=-1
    2021-02-11T09:06:38.529+0000 7ffaa1b86200 -1 static int RGWDataChangesFIFO::exists(ceph::common::CephContext*, librados::v14_2_0::Rados*, const rgw_pool&, bool*, bool*): unable to open FIFO: default.rgw.log/data_log.3: (1) Operation not permitted
    2021-02-11T09:06:38.529+0000 7ffaa1b86200 -1 int RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*, librados::v14_2_0::Rados*): Error when checking for existing FIFO datalog backend: (1) Operation not permitted
    2021-02-11T09:06:38.576+0000 7ffaa1b86200 -1 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr<rgw::cls::fifo::FIFO>*, optional_yield, std::optional<rados::cls::fifo::objv>, std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925 create_meta failed: r=-1
    2021-02-11T09:06:38.576+0000 7ffaa1b86200 -1 int RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*, librados::v14_2_0::Rados*): Error when starting backend: Operation not permitted
    2021-02-11T09:06:38.576+0000 7ffaa1b86200  0 ERROR: failed to start datalog_rados service ((1) Operation not permitted
    2021-02-11T09:06:38.576+0000 7ffaa1b86200  0 ERROR: failed to init services (ret=(1) Operation not permitted)
    couldn't init storage provider
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>


Expected results:

cluster gets upgraded to ceph 5.0


Additional info: log files and all.yml information attached.