Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1591498

Summary: bcache io error messages during random commands
Product: Red Hat Enterprise Linux 7 Reporter: Corey Marthaler <cmarthal>
Component: lvm2Assignee: David Teigland <teigland>
lvm2 sub component: Command-line tools QA Contact: cluster-qe <cluster-qe>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: unspecified CC: agk, heinzm, jbrassow, mcsontos, msnitzer, prajnoha, teigland, thornber, zkabelac
Version: 7.6   
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: lvm2-2.02.179-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-10-30 11:03:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
verbose lvcreate attempt none

Description Corey Marthaler 2018-06-14 22:01:14 UTC 
Description of problem:
This appears to be easiest to reproduce when creating a raid10 volume, but happens with other cmds as well.

 host-093.virt.lab.msp.redhat.com: lvcreate  --type raid10 -i 2 -n remove_final_meta -L 20M raid_sanity /dev/sda2 /dev/sdg1 /dev/sdg2 /dev/sde2
   bcache io error -5 fd 4
   dev_write_zeros /dev/raid_sanity/remove_final_meta at 0 bcache flush failed invalidate fd 4
   io_submit failed: Success
   bcache io error -5 fd 4


 host-093.virt.lab.msp.redhat.com: lvcreate  --type raid10 -i 3 -n raid_on_4Kextent_vg -L 80k -R 64k raid_sanity
   bcache io error -5 fd 4
   dev_write_zeros /dev/raid_sanity/raid_on_4Kextent_vg at 0 bcache flush failed invalidate fd 4
   io_submit failed: Success
   bcache io error -5 fd 4


[root@host-073 ~]# vgremove -f snapper_thinp_stack
  bcache io error -5 fd 6
  dev_write_bytes /dev/snapper_thinp/POOL at 22528 bcache flush failed invalidate fd 6
  Failed to write metadata to /dev/snapper_thinp/POOL fd -1
  WARNING: Failed to write an MDA of VG snapper_thinp_stack.
  Failed to write VG snapper_thinp_stack.
  io_submit failed: No such device or address
  bcache io error -5 fd 6



Version-Release number of selected component (if applicable):
3.10.0-905.el7.x86_64

lvm2-2.02.178-2.el7    BUILT: Thu Jun 14 06:17:42 CDT 2018
lvm2-libs-2.02.178-2.el7    BUILT: Thu Jun 14 06:17:42 CDT 2018
lvm2-cluster-2.02.178-2.el7    BUILT: Thu Jun 14 06:17:42 CDT 2018
lvm2-lockd-2.02.178-2.el7    BUILT: Thu Jun 14 06:17:42 CDT 2018
lvm2-python-boom-0.8.5-5.el7    BUILT: Thu Jun 14 06:21:09 CDT 2018
cmirror-2.02.178-2.el7    BUILT: Thu Jun 14 06:17:42 CDT 2018
device-mapper-1.02.147-2.el7    BUILT: Thu Jun 14 06:17:42 CDT 2018
device-mapper-libs-1.02.147-2.el7    BUILT: Thu Jun 14 06:17:42 CDT 2018
device-mapper-event-1.02.147-2.el7    BUILT: Thu Jun 14 06:17:42 CDT 2018
device-mapper-event-libs-1.02.147-2.el7    BUILT: Thu Jun 14 06:17:42 CDT 2018
device-mapper-persistent-data-0.7.3-3.el7    BUILT: Tue Nov 14 05:07:18 CST 2017


How reproducible:
Often
Comment 2 Corey Marthaler 2018-06-14 22:03:42 UTC 
Created attachment 1451620 [details]
verbose lvcreate attempt
Comment 3 Marian Csontos 2018-06-15 12:34:00 UTC 
Joe, is this a known issue? Fixed upstream?
Comment 4 Joe Thornber 2018-06-15 13:23:42 UTC 
The bcache messages will improve soon when the io_engine starts handling
the opening/closing of devices, since it will know the path not just
the file descriptor.

As for why this is failing, I have no idea.  Handing over to Dave.
Comment 5 David Teigland 2018-06-15 14:50:29 UTC 
io error and device failure issues were fixed some weeks ago, which I believe will fix most or all of this, we just need to get 2.02.179 built for testing.

I'm thinking of

https://sourceware.org/git/?p=lvm2.git;a=commit;h=28c8e95d197bf512a39b561281162ff4d93a598e
https://sourceware.org/git/?p=lvm2.git;a=commit;h=6d14d5d16b92c520b5f4ee464f171684cac40735
https://sourceware.org/git/?p=lvm2.git;a=commit;h=c67bd8b47b755a33104e942abc49e516d2a6dace
Comment 7 Corey Marthaler 2018-06-18 16:46:14 UTC 
Still some unexpected output/debugging in the latest, both the word "Error" and "failed".

[root@host-093 ~]# lvcreate --type raid10 -m 1 -L 400M -n my_raid test
  Using default stripesize 64.00 KiB.
  Error writing device /dev/test/my_raid at 0 length 4096.
  Logical volume "my_raid" created.
  io_submit failed: Success
Comment 8 Corey Marthaler 2018-06-18 16:46:57 UTC 
3.10.0-906.el7.x86_64

lvm2-2.02.179-1.el7    BUILT: Mon Jun 18 01:12:41 CDT 2018
lvm2-libs-2.02.179-1.el7    BUILT: Mon Jun 18 01:12:41 CDT 2018
lvm2-cluster-2.02.179-1.el7    BUILT: Mon Jun 18 01:12:41 CDT 2018
lvm2-lockd-2.02.179-1.el7    BUILT: Mon Jun 18 01:12:41 CDT 2018
lvm2-python-boom-0.8.5-6.el7    BUILT: Mon Jun 18 01:16:13 CDT 2018
cmirror-2.02.179-1.el7    BUILT: Mon Jun 18 01:12:41 CDT 2018
device-mapper-1.02.148-1.el7    BUILT: Mon Jun 18 01:12:41 CDT 2018
device-mapper-libs-1.02.148-1.el7    BUILT: Mon Jun 18 01:12:41 CDT 2018
device-mapper-event-1.02.148-1.el7    BUILT: Mon Jun 18 01:12:41 CDT 2018
device-mapper-event-libs-1.02.148-1.el7    BUILT: Mon Jun 18 01:12:41 CDT 2018
Comment 9 David Teigland 2018-06-18 17:03:26 UTC 
hadn't noticed that one, removed here

https://sourceware.org/git/?p=lvm2.git;a=commit;h=bfb904af1cb0a48a4ac44eb72179f628fda4fd03
Comment 10 Corey Marthaler 2018-08-06 19:31:56 UTC 
We're no longer seeing these errors with the latest kernel/lvm rpms. Marking verified.

3.10.0-931.el7.x86_64

lvm2-2.02.180-2.el7    BUILT: Wed Aug  1 11:22:48 CDT 2018
lvm2-libs-2.02.180-2.el7    BUILT: Wed Aug  1 11:22:48 CDT 2018
lvm2-cluster-2.02.180-2.el7    BUILT: Wed Aug  1 11:22:48 CDT 2018
lvm2-lockd-2.02.180-2.el7    BUILT: Wed Aug  1 11:22:48 CDT 2018
lvm2-python-boom-0.9-5.el7    BUILT: Wed Aug  1 11:24:13 CDT 2018
cmirror-2.02.180-2.el7    BUILT: Wed Aug  1 11:22:48 CDT 2018
device-mapper-1.02.149-2.el7    BUILT: Wed Aug  1 11:22:48 CDT 2018
device-mapper-libs-1.02.149-2.el7    BUILT: Wed Aug  1 11:22:48 CDT 2018
device-mapper-event-1.02.149-2.el7    BUILT: Wed Aug  1 11:22:48 CDT 2018
device-mapper-event-libs-1.02.149-2.el7    BUILT: Wed Aug  1 11:22:48 CDT 2018
device-mapper-persistent-data-0.7.3-3.el7    BUILT: Tue Nov 14 05:07:18 CST 2017
Comment 12 errata-xmlrpc 2018-10-30 11:03:47 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3193