Bug 2130434 - CephFS: mgr/volumes: Intermittent ParsingError failure in mgr/volumes module during "clone cancel"
Summary: CephFS: mgr/volumes: Intermittent ParsingError failure in mgr/volumes module ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: CephFS
Version: 5.2
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
: 6.0
Assignee: Kotresh HR
QA Contact: Amarnath
Masauso Lungu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-09-28 07:24 UTC by Kotresh HR
Modified: 2023-03-20 18:59 UTC (History)
6 users (show)

Fixed In Version: ceph-17.2.3-45.el9cp
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-03-20 18:58:27 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Ceph Project Bug Tracker 55583 0 None None None 2022-09-28 07:24:14 UTC
Red Hat Issue Tracker RHCEPH-5371 0 None None None 2022-09-28 07:36:17 UTC
Red Hat Product Errata RHBA-2023:1360 0 None None None 2023-03-20 18:59:18 UTC

Description Kotresh HR 2022-09-28 07:24:14 UTC
This bug was initially created as a copy of Bug #2120491

I am copying this bug because: 



Description of problem:
A test runs that attempt to cancel a clone. Sometimes when this test runs it fails with an python traceback in the mgr response. This traceback indicates a configparser.ParsingError was raised.

rados: ret=-22, Invalid argument: "Traceback (most recent call last):\n
  File \"/usr/share/ceph/mgr/mgr_module.py\", line 1701, in _handle_command\n
    return self.handle_command(inbuf, cmd)\n
  File \"/usr/share/ceph/mgr/volumes/module.py\", line 409, in handle_command\n
    return handler(inbuf, cmd)\n
  File \"/usr/share/ceph/mgr/volumes/module.py\", line 38, in wrap\n
    return f(self, inbuf, cmd)\n
  File \"/usr/share/ceph/mgr/volumes/module.py\", line 636, in _cmd_fs_clone_cancel\n
    vol_name=cmd['vol_name'], clone_name=cmd['clone_name'], group_name=cmd.get('group_name', None))\n
  File \"/usr/share/ceph/mgr/volumes/fs/volume.py\", line 582, in clone_cancel\n
    self.cloner.cancel_job(volname, (clonename, groupname))\n
  File \"/usr/share/ceph/mgr/volumes/fs/async_cloner.py\", line 389, in cancel_job\n
    with open_subvol(self.fs_client.mgr, fs_handle, self.vc.volspec, group, clonename, SubvolumeOpType.CLONE_CANCEL) as clone_subvolume:\n
  File \"/usr/lib64/python3.6/contextlib.py\", line 81, in __enter__\n
    return next(self.gen)\n
  File \"/usr/share/ceph/mgr/volumes/fs/operations/subvolume.py\", line 72, in open_subvol\n
    subvolume = loaded_subvolumes.get_subvolume_object(mgr, fs, vol_spec, group, subvolname)\n
  File \"/usr/share/ceph/mgr/volumes/fs/operations/versions/__init__.py\", line 95, in get_subvolume_object\n
    subvolume.discover()\n
  File \"/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_base.py\", line 319, in discover\n
    self.metadata_mgr.refresh()\n
  File \"/usr/share/ceph/mgr/volumes/fs/operations/versions/metadata_manager.py\", line 52, in refresh\n
    self.config.readfp(conf_data)\n
  File \"/usr/lib64/python3.6/configparser.py\", line 763, in readfp\n
    self.read_file(fp, source=filename)\n
  File \"/usr/lib64/python3.6/configparser.py\", line 718, in read_file\n
    self._read(f, source)\n
  File \"/usr/lib64/python3.6/configparser.py\", line 1111, in _read\n
    raise e\nconfigparser.ParsingError: Source contains parsing errors: '<???>'\n\t[line 13]: 'a0\\n'\n" 

For more info, please look into upstream tracker https://tracker.ceph.com/issues/55583

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 9 Amarnath 2022-10-12 19:33:45 UTC
Verifed with different clone names and clone cancel operation.
did not observer any parser error.

Version tested : 
[root@ceph-amk-bz-o4n2a9-node7 ~]# ceph versions
{
    "mon": {
        "ceph version 17.2.3-49.el9cp (70ade2732bfe4eb64129e97862bf0b1744c800d2) quincy (stable)": 3
    },
    "mgr": {
        "ceph version 17.2.3-49.el9cp (70ade2732bfe4eb64129e97862bf0b1744c800d2) quincy (stable)": 2
    },
    "osd": {
        "ceph version 17.2.3-49.el9cp (70ade2732bfe4eb64129e97862bf0b1744c800d2) quincy (stable)": 12
    },
    "mds": {
        "ceph version 17.2.3-49.el9cp (70ade2732bfe4eb64129e97862bf0b1744c800d2) quincy (stable)": 3
    },
    "overall": {
        "ceph version 17.2.3-49.el9cp (70ade2732bfe4eb64129e97862bf0b1744c800d2) quincy (stable)": 20
    }
}
[root@ceph-amk-bz-o4n2a9-node7 ~]# 

[root@ceph-amk-bz-o4n2a9-node7 ~]# ceph fs subvolume snapshot create cephfs subvol_1 snap_1 --group_name subvolgroup_1
[root@ceph-amk-bz-o4n2a9-node7 ~]# ceph fs subvolume snapshot clone cephfs subvol_1 snap_1 "123**\\//@clone_1112312" --group_name subvolgroup_1
Error ENOTSUP: operation 'clone_internal' is not allowed on subvolume '123**\//@clone_1112312' of type subvolume
[root@ceph-amk-bz-o4n2a9-node7 ~]# ceph fs subvolume snapshot clone cephfs subvol_1 snap_1 "clone_1" --group_name subvolgroup_1
[root@ceph-amk-bz-o4n2a9-node7 ~]# ceph fs subvolume snapshot info cephfs subvol_1 snap_1 --group_name subvolgroup_1
{
    "created_at": "2022-10-12 19:17:18.005749",
    "data_pool": "cephfs.cephfs.data",
    "has_pending_clones": "yes",
    "pending_clones": [
        {
            "name": "clone_1"
        }
    ]
}
[root@ceph-amk-bz-o4n2a9-node7 ~]# ceph fs clone cancel cephfs clone_1
[root@ceph-amk-bz-o4n2a9-node7 ~]# ceph fs subvolume snapshot info cephfs subvol_1 snap_1 --group_name subvolgroup_1
{
    "created_at": "2022-10-12 19:17:18.005749",
    "data_pool": "cephfs.cephfs.data",
    "has_pending_clones": "no"
}
[root@ceph-amk-bz-o4n2a9-node7 ~]# ceph fs subvolume snapshot clone cephfs subvol_1 snap_1 "123**\\//@clone_1112312" --group_name subvolgroup_1
Error EEXIST: subvolume '123**\//@clone_1112312' exists
[root@ceph-amk-bz-o4n2a9-node7 ~]# ceph fs subvolume ls cephfs 
[
    {
        "name": "clone_1"
    },
    {
        "name": "123**\\"
    }
]
[root@ceph-amk-bz-o4n2a9-node7 ~]# ceph fs clone cancel cephfs clone_1
Error EINVAL: cannot cancel -- clone finished (check clone status)
[root@ceph-amk-bz-o4n2a9-node7 ~]# ceph fs clone status cephfs clone_1
{
  "status": {
    "state": "canceled",
    "source": {
      "volume": "cephfs",
      "subvolume": "subvol_1",
      "snapshot": "snap_1",
      "group": "subvolgroup_1"
    },
    "failure": {
      "errno": "4",
      "error_msg": "user interrupted clone operation"
    }
  }
}
[root@ceph-amk-bz-o4n2a9-node7 ~]# ceph fs subvolume snapshot clone cephfs subvol_1 snap_1 "123**\\@clone_1112312" --group_name subvolgroup_1
[root@ceph-amk-bz-o4n2a9-node7 ~]# ceph fs subvolume snapshot info cephfs subvol_1 snap_1 --group_name subvolgroup_1
{
    "created_at": "2022-10-12 19:17:18.005749",
    "data_pool": "cephfs.cephfs.data",
    "has_pending_clones": "yes",
    "pending_clones": [
        {
            "name": "123**\\@clone_1112312"
        }
    ]
}
[root@ceph-amk-bz-o4n2a9-node7 ~]# ceph fs clone status cephfs 123**\\@clone_1112312
{
  "status": {
    "state": "in-progress",
    "source": {
      "volume": "cephfs",
      "subvolume": "subvol_1",
      "snapshot": "snap_1",
      "group": "subvolgroup_1"
    }
  }
}
[root@ceph-amk-bz-o4n2a9-node7 ~]# ceph fs subvolume snapshot clone cephfs subvol_1 snap_1 "123**\\ @clone_1112312" --group_name subvolgroup_1
[root@ceph-amk-bz-o4n2a9-node7 ~]# ceph fs clone status cephfs "123**\\ @clone_1112312"
{
  "status": {
    "state": "in-progress",
    "source": {
      "volume": "cephfs",
      "subvolume": "subvol_1",
      "snapshot": "snap_1",
      "group": "subvolgroup_1"
    }
  }
}
[root@ceph-amk-bz-o4n2a9-node7 ~]# ceph fs clone cancel cephfs "123**\\ @clone_1112312"
[root@ceph-amk-bz-o4n2a9-node7 ~]# 
[root@ceph-amk-bz-o4n2a9-node7 ~]# ceph fs clone cancel cephfs "123**\\@clone_1112312"
[root@ceph-amk-bz-o4n2a9-node7 ~]# ceph fs clone status cephfs 123**\\@clone_1112312
{
  "status": {
    "state": "canceled",
    "source": {
      "volume": "cephfs",
      "subvolume": "subvol_1",
      "snapshot": "snap_1",
      "group": "subvolgroup_1"
    },
    "failure": {
      "errno": "4",
      "error_msg": "user interrupted clone operation"
    }
  }
}
[root@ceph-amk-bz-o4n2a9-node7 ~]# ceph fs subvolume ls cephfs
[
    {
        "name": "clone_1"
    },
    {
        "name": "123**\\@clone_1112312"
    },
    {
        "name": "123**\\"
    },
    {
        "name": "123**\\ @clone_1112312"
    }
]
[root@ceph-amk-bz-o4n2a9-node7 ~]# ceph fs subvolume snapshot clone cephfs subvol_1 snap_1 "123**\\n \t @clone_1112312" --group_name subvolgroup_1
[root@ceph-amk-bz-o4n2a9-node7 ~]# ceph fs clone cancel cephfs "123**\\n \t @clone_1112312"
[root@ceph-amk-bz-o4n2a9-node7 ~]# ceph fs clone status cephfs "123**\\n \t @clone_1112312"
{
  "status": {
    "state": "canceled",
    "source": {
      "volume": "cephfs",
      "subvolume": "subvol_1",
      "snapshot": "snap_1",
      "group": "subvolgroup_1"
    },
    "failure": {
      "errno": "4",
      "error_msg": "user interrupted clone operation"
    }
  }
}
[root@ceph-amk-bz-o4n2a9-node7 ~]# ceph fs subvolume ls cephfs
[
    {
        "name": "clone_1"
    },
    {
        "name": "123**\\@clone_1112312"
    },
    {
        "name": "123**\\n \\t @clone_1112312"
    },
    {
        "name": "123**\\"
    },
    {
        "name": "123**\\ @clone_1112312"
    }
]
[root@ceph-amk-bz-o4n2a9-node7 ~]# 

Regards,
Amarnath

Comment 18 errata-xmlrpc 2023-03-20 18:58:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 6.0 Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:1360


Note You need to log in before you can comment on or make changes to this bug.