RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1949369 - path devices are not suddenly removed after flushing a multipath device
Summary: path devices are not suddenly removed after flushing a multipath device
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: device-mapper-multipath
Version: 8.2
Hardware: Unspecified
OS: Unspecified
low
urgent
Target Milestone: beta
: ---
Assignee: Ben Marzinski
QA Contact: Lin Li
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-14 06:51 UTC by Takashi Kajinami
Modified: 2024-06-14 01:14 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-10-27 01:07:25 UTC
Type: Bug
Target Upstream Version:
Embargoed:
pm-rhel: mirror+

Attachments (Terms of Use)

Description Takashi Kajinami 2021-04-14 06:51:31 UTC 
Description of problem:

In RHOSP, we run the following steps when powering on an instance
to ensure that all volume devices are available.

1. Flush multipath device
2. Remove each iscsi devices by
   $ echo 1 | sudo tee -a /sys/block/<device>/device/delete
3. Logout from all iscsi portals
4. Relogin to iscsi portals
5. Set node.startup=manual
6. Scan iscsi devices
7. Set up a multipath devices using available iscsi devices by
   $ multipathd -a /dev/<device>

A problem here is that the path device was not properly removed from multipathd
after 1-3, and step 7 fails because of orphan paths left.

This issue was initially observed in RHOSP16.1 deployment with the following packages
installed.
 - device-mapper-multipath-0.8.3-3.el8_2.3.x86_64
 - device-mapper-multipath-libs-0.8.3-3.el8_2.3.x86_64
 - kpartx-0.8.3-3.el8_2.3.x86_64

We confirmed that updating the packages to the following version solves the issue.
 - device-mapper-multipath-0.8.4-5.el8.x86_64
 - device-mapper-multipath-libs-0.8.4-5.el8.x86_64
 - kpartx-0.8.4-5.el8.x86_64

This version is available in normal rhel8 repo but is not available in eus repo.
Since RHOSP16.1 depends on RHEL8.2 we do need the fixed version in RHEL8.2


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 2 Jon Magrini 2021-04-14 13:36:28 UTC 
---
1. Flush multipath device
---

What procedure is used to flush the device, ie: multipath -f? Are you able to recreate this issue using multipathd cli vs multipath -f? IE: multipathd remove|del map|multipath $map. From the changelog of 0.8.4-3 I believe the following patches added from bz #1845875 address this issue: 

- Add 0024-libmultipath-make-dm_get_map-status-return-codes-sym.patch
- Add 0025-multipathd-fix-check_path-errors-with-removed-map.patch
- Add 0026-libmultipath-make-dm_flush_maps-only-return-0-on-suc.patch
- Add 0027-multipathd-add-del-maps-multipathd-command.patch
- Add 0028-multipath-make-flushing-maps-work-like-other-command.patch
- Add 0029-multipath-delegate-flushing-maps-to-multipathd.patch
- Add 0030-multipath-add-option-to-skip-multipathd-delegation.patch
  * The above 7 patches fix bz #1845875. Multipath now attempts to
    delegate device removal to multipathd, and multipathd handles
    external device removal better.

Further details are noted here:
https://bugzilla.redhat.com/show_bug.cgi?id=1845875#c3 


If not too invasive possibly backport those to EUS.
Comment 3 Jon Magrini 2021-04-14 13:45:18 UTC 
A zstream/eus was cloned from bz #1845875 as bz #1856944 : https://bugzilla.redhat.com/show_bug.cgi?id=1856944#c1 though doesn't seem to contain what is needed to address the issue described here:

- Add 0020-libmultipath-make-dm_get_map-status-return-codes-sym.patch
- Add 0021-multipathd-fix-check_path-errors-with-removed-map.patch
  * The above 2 patches fix bz #1856944. multipathd handles external
    device removal better.
Comment 4 Takashi Kajinami 2021-04-14 13:49:42 UTC 
> What procedure is used to flush the device, ie: multipath -f? 

yes. multipath -f is used to flush the device.

> Are you able to recreate this issue using multipathd cli vs multipath -f? IE: multipathd remove|del map|multipath $map.

We have not tried usage of only multipathd cli.
What we've tried before updating the device-mapper-multipath package was is to modify implementation in OpenStack
to execute "mulipathd del path /dev/<device>" after "multipath -f" but before removing iscsi device.
This was tested in the customer's deployment and we confirmed this also solves the problem.
Comment 16 Takashi Kajinami 2021-10-27 01:07:25 UTC 
Sorry I forgot to update this bug after we reached to the conclusion in RHOSP side.

I'm closing this bug as notabug because the issue will be fixed in RHOSP (os-brick).
Note You need to log in before you can comment on or make changes to this bug.