Back to bug 2086419
| Who | When | What | Removed | Added |
|---|---|---|---|---|
| Red Hat One Jira (issues.redhat.com) | 2022-05-16 06:31:52 UTC | Link ID | Red Hat Issue Tracker RHCEPH-4311 | |
| Neha Ojha | 2022-05-16 16:40:14 UTC | Assignee | nojha | sseshasa |
| Sridhar Seshasayee | 2022-05-19 13:36:01 UTC | Flags | needinfo?(sseshasa) | |
| Neha Ojha | 2022-05-19 21:18:46 UTC | Status | NEW | ASSIGNED |
| Sridhar Seshasayee | 2022-05-23 15:24:41 UTC | Flags | needinfo?(sseshasa) | |
| Vikhyat Umrao | 2022-05-23 22:45:27 UTC | Flags | needinfo?(pdhiran) | |
| Vikhyat Umrao | 2022-05-23 23:21:19 UTC | Summary | [All-Monitors-Crash] set_mon_vals failed to set cluster_network = 172.16.0.0/16: Configuration option 'cluster_network' may not be modified at runtime | [All-Monitors-Crashed in 16.2.7/src/mon/PaxosService.cc: 193: FAILED ceph_assert(have_pending)] set_mon_vals failed to set cluster_network = 172.16.0.0/16: Configuration option 'cluster_network' may not be modified at runtime |
| Vikhyat Umrao | 2022-05-23 23:24:18 UTC | Dependent Products | Red Hat OpenShift Data Foundation | |
| Sunil Kumar Nagaraju | 2022-05-24 03:54:09 UTC | Flags | needinfo?(pdhiran) | |
| Red Hat Bugzilla | 2022-05-26 08:30:43 UTC | CC | ceph-qe-bugs | |
| Vikhyat Umrao | 2022-05-26 18:20:04 UTC | Link ID | Ceph Project Bug Tracker 55773 | |
| Neha Ojha | 2022-05-31 20:02:30 UTC | Link ID | Github ceph/ceph/pull/46428 | |
| Vikhyat Umrao | 2022-05-31 20:07:55 UTC | Summary | [All-Monitors-Crashed in 16.2.7/src/mon/PaxosService.cc: 193: FAILED ceph_assert(have_pending)] set_mon_vals failed to set cluster_network = 172.16.0.0/16: Configuration option 'cluster_network' may not be modified at runtime | [All-Monitors-Crashed in 16.2.7/src/mon/PaxosService.cc: 193: FAILED ceph_assert(have_pending)] |
| Mudit Agarwal | 2022-06-07 13:03:56 UTC | Blocks | 2083074 | |
| Vikhyat Umrao | 2022-06-15 21:17:29 UTC | Link ID | Github ceph/ceph/pull/46691 | |
| Sridhar Seshasayee | 2022-06-17 16:36:44 UTC | Status | ASSIGNED | POST |
| errata-xmlrpc | 2022-06-17 22:59:51 UTC | CC | tserlin | |
| Status | POST | MODIFIED | ||
| Fixed In Version | ceph-16.2.8-49.el8cp | |||
| Status | MODIFIED | ON_QA | ||
| Sunil Kumar Nagaraju | 2022-06-21 13:05:05 UTC | QA Contact | pdhiran | sunnagar |
| Status | ON_QA | VERIFIED | ||
| Keywords | Automation | |||
| Akash Raj | 2022-07-29 06:00:51 UTC | Blocks | 2102272 | |
| Akash Raj | 2022-07-29 06:01:38 UTC | Flags | needinfo?(sseshasa) | |
| CC | akraj | |||
| Sridhar Seshasayee | 2022-07-29 14:26:03 UTC | Doc Type | If docs needed, set a value | Bug Fix |
| Doc Text | Cause: The KVMonitor paxos services manages the keys being added/removed/modified when performing changes to the cluster. In this case new OSDs were being added using the OSD deployment tool. The keys in this case were attempted to be added without verifying whether the service could write to it. Consequence: Assertion failure in the paxos code causing the monitor to crash. Fix: Ensure that the KVMonitor service is able to write before proceeding further with adding the new OSD. Otherwise, push the command back into the relevant queue to be retried at a later point. Result: The OSD deployment tool was successful in deploying all the OSDs without any issue after applying the fix. | |||
| Flags | needinfo?(sseshasa) | |||
| Akash Raj | 2022-08-03 14:14:55 UTC | Docs Contact | akraj | |
| Doc Text | Cause: The KVMonitor paxos services manages the keys being added/removed/modified when performing changes to the cluster. In this case new OSDs were being added using the OSD deployment tool. The keys in this case were attempted to be added without verifying whether the service could write to it. Consequence: Assertion failure in the paxos code causing the monitor to crash. Fix: Ensure that the KVMonitor service is able to write before proceeding further with adding the new OSD. Otherwise, push the command back into the relevant queue to be retried at a later point. Result: The OSD deployment tool was successful in deploying all the OSDs without any issue after applying the fix. | .OSD deployment tool successfully deploys all the OSDs while making changes to the cluster The KVMonitor paxos services manages the keys being added, removed, or modified when performing changes to the cluster, Previously, while adding new OSDs using the OSD deployment tool, the keys would be added without verifying whether the service could write to it. Due to this, assertion failure would occur in the paxos code causing the monitor to crash. With this fix, ensure that the KVMonitor service is able to write prior to adding new OSDs, failing which, you need to push the command back into the relevant queue to be retried at a later point. The OSD deployment tool successfully deploys all the OSDs without any issues. | ||
| Flags | needinfo?(sseshasa) | |||
| Sridhar Seshasayee | 2022-08-03 14:32:23 UTC | Flags | needinfo?(sseshasa) | |
| Akash Raj | 2022-08-05 07:21:31 UTC | Doc Text | .OSD deployment tool successfully deploys all the OSDs while making changes to the cluster The KVMonitor paxos services manages the keys being added, removed, or modified when performing changes to the cluster, Previously, while adding new OSDs using the OSD deployment tool, the keys would be added without verifying whether the service could write to it. Due to this, assertion failure would occur in the paxos code causing the monitor to crash. With this fix, ensure that the KVMonitor service is able to write prior to adding new OSDs, failing which, you need to push the command back into the relevant queue to be retried at a later point. The OSD deployment tool successfully deploys all the OSDs without any issues. | .OSD deployment tool successfully deploys all the OSDs while making changes to the cluster The KVMonitor paxos services manages the keys being added, removed, or modified when performing changes to the cluster, Previously, while adding new OSDs using the OSD deployment tool, the keys would be added without verifying whether the service could write to it. Due to this, assertion failure would occur in the paxos code causing the monitor to crash. The latest fix ensures that the KVMonitor service is able to write prior to adding new OSDs. otherwise, the command back into the relevant queue to be retried at a later point. The OSD deployment tool successfully deploys all the OSDs without any issues. |
| Akash Raj | 2022-08-09 07:31:29 UTC | Doc Text | .OSD deployment tool successfully deploys all the OSDs while making changes to the cluster The KVMonitor paxos services manages the keys being added, removed, or modified when performing changes to the cluster, Previously, while adding new OSDs using the OSD deployment tool, the keys would be added without verifying whether the service could write to it. Due to this, assertion failure would occur in the paxos code causing the monitor to crash. The latest fix ensures that the KVMonitor service is able to write prior to adding new OSDs. otherwise, the command back into the relevant queue to be retried at a later point. The OSD deployment tool successfully deploys all the OSDs without any issues. | .OSD deployment tool successfully deploys all the OSDs while making changes to the cluster The KVMonitor paxos services manages the keys being added, removed, or modified when performing changes to the cluster, Previously, while adding new OSDs using the OSD deployment tool, the keys would be added without verifying whether the service could write to it. Due to this, assertion failure would occur in the paxos code causing the monitor to crash. The latest fix ensures that the KVMonitor service is able to write prior to adding new OSDs, otherwise, the command back is pushed back into the relevant queue to be retried at a later point. The OSD deployment tool successfully deploys all the OSDs without any issues. |
| errata-xmlrpc | 2022-08-09 09:59:36 UTC | Status | VERIFIED | RELEASE_PENDING |
| errata-xmlrpc | 2022-08-09 17:38:23 UTC | Status | RELEASE_PENDING | CLOSED |
| Resolution | --- | ERRATA | ||
| Last Closed | 2022-08-09 17:38:23 UTC | |||
| errata-xmlrpc | 2022-08-09 17:39:04 UTC | Link ID | Red Hat Product Errata RHSA-2022:5997 |
Back to bug 2086419