Bug 1510585
| Summary: | vdo creation fails on multiple partitioned devices | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Corey Marthaler <cmarthal> |
| Component: | vdo | Assignee: | corwin <corwin> |
| Status: | CLOSED ERRATA | QA Contact: | Jakub Krysl <jkrysl> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.5 | CC: | awalsh, bgurney, cmarthal, jkrysl, kzak, limershe, msnitzer, sweettea |
| Target Milestone: | rc | Keywords: | TestBlocker |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | 6.1.0.44 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-04-10 15:47:46 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Corey Marthaler
2017-11-07 17:46:33 UTC
# Works on base mpath volume: [root@harding-02 ~]# vdo create --name vdo_a1 --device /dev/mapper/mpatha Creating VDO vdo_a1 Starting VDO vdo_a1 Starting compression on VDO vdo_a1 VDO instance 0 volume is ready at /dev/mapper/vdo_a1 (In reply to Corey Marthaler from comment #0) > Description of problem: > > [root@harding-02 ~]# vdo create --name vdo_a1 --device /dev/mapper/mpatha1 > Creating VDO vdo_a1 > vdo: ERROR - vdoformat: physical size must be a multiple of block size 4096 > > > Nov 7 11:40:34 harding-02 kernel: kvdo: modprobe: loaded version > Nov 7 11:40:34 harding-02 vdo: ERROR - vdoformat: physical size must be a > multiple of block size 4096 > Nov 7 11:40:34 harding-02 systemd: Started /usr/sbin/lvm pvscan --cache > 253:20. What did you use to create the partition? Is the partition's start aligned on a 4K boundary? If not that could be why vdoformat is failing. Please share a dmsetup table output for the multipath device and the partition's linear (kpartx created?) DM device ontop. It appears to have a problem w/ any device with multiple partitions (and nothing to do w/ mpath). # Mpath [root@harding-02 ~]# dmsetup table | grep mpathb mpathb: 0 524288000 multipath 2 queue_if_no_path retain_attached_hw_handler 1 alua 2 1 service-time 0 2 2 65:64 1 1 8:192 1 1 service-time 0 2 2 65:192 1 1 8:64 1 1 mpathb2: 0 262132605 linear 253:2 262148670 mpathb1: 0 262148607 linear 253:2 63 [root@harding-02 ~]# kpartx /dev/mapper/mpathb mpathb1 : 0 262148607 /dev/mapper/mpathb 63 mpathb2 : 0 262132605 /dev/mapper/mpathb 262148670 [root@harding-02 ~]# vdo create --name foo --device /dev/mapper/mpathb1 Creating VDO foo vdo: ERROR - vdoformat: physical size must be a multiple of block size 4096 [root@harding-02 ~]# vdo create --name foo --device /dev/mapper/mpathb2 Creating VDO foo vdo: ERROR - vdoformat: physical size must be a multiple of block size 4096 # Phys device [root@host-026 ~]# cat /proc/partitions | grep sda 8 0 26214400 sda 8 1 13109008 sda1 8 2 13101007 sda2 [root@host-026 ~]# vdo create --name foo --device /dev/sda1 Creating VDO foo vdo: ERROR - vdoformat: physical size must be a multiple of block size 4096 vdo: ERROR - vdoformat: physical size must be a multiple of block size 4096 [root@host-026 ~]# kpartx /dev/sda sda1 : 0 26218017 /dev/sda 63 sda2 : 0 26202015 /dev/sda 26218080 (In reply to Corey Marthaler from comment #5) > It appears to have a problem w/ any device with multiple partitions (and > nothing to do w/ mpath). > > > # Mpath > [root@harding-02 ~]# dmsetup table | grep mpathb > mpathb: 0 524288000 multipath 2 queue_if_no_path retain_attached_hw_handler > 1 alua 2 1 service-time 0 2 2 65:64 1 1 8:192 1 1 service-time 0 2 2 65:192 > 1 1 8:64 1 1 > mpathb2: 0 262132605 linear 253:2 262148670 > mpathb1: 0 262148607 linear 253:2 63 > [root@harding-02 ~]# kpartx /dev/mapper/mpathb > mpathb1 : 0 262148607 /dev/mapper/mpathb 63 > mpathb2 : 0 262132605 /dev/mapper/mpathb 262148670 ... > # Phys device > [root@host-026 ~]# cat /proc/partitions | grep sda > 8 0 26214400 sda > 8 1 13109008 sda1 > 8 2 13101007 sda2 > > [root@host-026 ~]# kpartx /dev/sda > sda1 : 0 26218017 /dev/sda 63 > sda2 : 0 26202015 /dev/sda 26218080 You've created partitions (likely with fdisk?) that are using the old MSDOS layout where the first partition starts at sector 63. parted should default to aligning the partition to 1MB unless the optimal_io_size et al published for the device warrant the parted tool to start the partitions at a different alignment. Honestly this is user error. (Mixed with our willingness to have fdisk default to creating partitions that are misaligned!!! that is the most alarming aspect of this report.. cannot believe linux partitioning tools still suck this badly). Cc'ing kzak to get his take (again). One could make a device of size, say, 128G+1K, I think, if one uses LVM with extent size 1K. A VDO on such a device would fail in a similar fashion without the use of fdisk. The VDO side is done and merged (see fix version); the question is whether the comments here indicate other tools/bugs/etc. need work. I reproduced this using fdisk on 512K block size on our storageqe-73 server:
# fdisk /dev/mapper/mpathb
--snip--
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p):
Using default response p
Partition number (1-4, default 1):
First sector (2048-20971519, default 2048): 63
Value out of range.
First sector (2048-20971519, default 2048): 2049
Last sector, +sectors or +size{K,M,G} (2049-20971519, default 20971519): 20971518
Partition 1 of type Linux and of size 10 GiB is set
vdo-6.1.0.34-8:
# vdo create --name vdo --device /dev/mapper/mpathb1
Creating VDO vdo
vdo: ERROR - vdoformat: physical size must be a multiple of block size 4096
vdo-6.1.0.55-10:
# vdo create --name vdo --device /dev/mapper/mpathb1
Creating VDO vdo
Starting VDO vdo
Starting compression on VDO vdo
VDO instance 0 volume is ready at /dev/mapper/vdo
[ 1938.982240] uds: modprobe: loaded version 6.1.0.31
[ 1939.008869] kvdo: modprobe: loaded version 6.1.0.55
[ 1939.191978] kvdo0:dmsetup: starting device 'vdo' device instantiation 0 (ti=ffffbfbac0f39040) write policy sync
[ 1939.237289] kvdo0:dmsetup: zones: 1 logical, 1 physical, 1 hash; base threads: 5
[ 1939.378236] kvdo0:dmsetup: Setting UDS index target state to online
[ 1939.406448] kvdo0:dmsetup: uds: kvdo0:dedupeQ: loading or rebuilding index: dev=/dev/mapper/mpathb1 offset=4096 size=2781704192
[ 1939.407207] uds: kvdo0:dedupeQ: Failed loading or rebuilding index: UDS Error: No index found (1061)
uds: kvdo0:dedupeQ: creating index: dev=/dev/mapper/mpathb1 offset=4096 size=2781704192
uds: kvdo0:dedupeQ: Using 6 indexing zones for concurrency.
[ 1939.562739] device 'vdo' started
[ 1939.577886] device-mapper: table: 253:8: adding target device (start sect 0 len 20969464) caused an alignment inconsistency
[ 1939.627898] kvdo0:dmsetup: resuming device 'vdo'
[ 1939.648616] kvdo0:dmsetup: device 'vdo' resumed
[ 1939.692555] kvdo0:packerQ: compression is enabled
Please note the device-mapper alignment inconsistency warning. I believe this is to be expected and the presence of this warning is a nice feature. But to be sure this is OK I created filesystem, mounted and wrote some data from /dev/urandom.
I started getting I/O Buffer Errors on the VDO device. First using XFS (created with discards):
# dd if=/dev/urandom of=vdo/random count=10000 bs=1M status=progress
8049917952 bytes (8.0 GB) copied, 64.411219 s, 125 MB/s
dd: error writing ‘vdo/random’: Input/output error
7727+0 records in
7726+0 records out
8101298176 bytes (8.1 GB) copied, 64.8483 s, 125 MB/s
[ 2329.776346] Buffer I/O error on dev dm-8, logical block 1682150, lost async page write
[ 2329.819027] Buffer I/O error on dev dm-8, logical block 1682151, lost async page write
[ 2329.856170] Buffer I/O error on dev dm-8, logical block 1682152, lost async page write
[ 2329.892801] Buffer I/O error on dev dm-8, logical block 1682153, lost async page write
[ 2329.910608] Buffer I/O error on dev dm-8, logical block 1708774, lost async page write
[ 2329.910613] Buffer I/O error on dev dm-8, logical block 1708775, lost async page write
[ 2329.910616] Buffer I/O error on dev dm-8, logical block 1708776, lost async page write
[ 2329.910618] Buffer I/O error on dev dm-8, logical block 1708777, lost async page write
[ 2329.910620] Buffer I/O error on dev dm-8, logical block 1708778, lost async page write
[ 2329.910621] Buffer I/O error on dev dm-8, logical block 1708779, lost async page write
[ 2336.110791] buffer_io_error: 207590 callbacks suppressed
[ 2336.137096] Buffer I/O error on dev dm-8, logical block 1876199, lost async page write
[ 2336.176722] Buffer I/O error on dev dm-8, logical block 1876200, lost async page write
[ 2336.213219] Buffer I/O error on dev dm-8, logical block 1876201, lost async page write
[ 2336.249112] Buffer I/O error on dev dm-8, logical block 1876202, lost async page write
[ 2336.259195] Buffer I/O error on dev dm-8, logical block 1904871, lost async page write
[ 2336.259201] Buffer I/O error on dev dm-8, logical block 1904872, lost async page write
[ 2336.259203] Buffer I/O error on dev dm-8, logical block 1904873, lost async page write
[ 2336.259205] Buffer I/O error on dev dm-8, logical block 1904874, lost async page write
[ 2336.259207] Buffer I/O error on dev dm-8, logical block 1904875, lost async page write
[ 2336.259209] Buffer I/O error on dev dm-8, logical block 1904876, lost async page write
[ 2338.202447] XFS (dm-8): metadata I/O error: block 0x9ffc58 ("xlog_iodone") error 28 numblks 64
[ 2338.209229] XFS (dm-8): metadata I/O error: block 0x8 ("xfs_buf_iodone_callback_error") error 28 numblks 8
[ 2338.252338] XFS (dm-8): metadata I/O error: block 0x9ffc60 ("xlog_iodone") error 28 numblks 64
[ 2338.252370] XFS (dm-8): Log I/O Error Detected. Shutting down filesystem
[ 2338.252371] XFS (dm-8): Please umount the filesystem and rectify the problem(s)
[ 2338.252376] XFS (dm-8): Failing async write on buffer block 0x20. Retrying async write.
[ 2338.252378] XFS (dm-8): Failing async write on buffer block 0x28. Retrying async write.
[ 2338.252378] XFS (dm-8): Failing async write on buffer block 0x8. Retrying async write.
So to eliminate error on XFS side, I tested with EXT4 (again created with discards):
# dd if=/dev/urandom of=vdo/random count=10000 bs=1M status=progress
7377780736 bytes (7.4 GB) copied, 59.257049 s, 125 MB/s
dd: error writing ‘vdo/random’: Read-only file system
7143+0 records in
7142+0 records out
7488929792 bytes (7.5 GB) copied, 60.1745 s, 124 MB/s
[ 2572.304654] EXT4-fs warning (device dm-8): ext4_end_bio:316: I/O error -28 writing to inode 12 (offset 6425673728 size 4489216 starting block 1749861)
[ 2572.372411] Buffer I/O error on device dm-8, logical block 1749861
[ 2572.400162] EXT4-fs warning (device dm-8): ext4_end_bio:316: I/O error -28 writing to inode 12 (offset 6425673728 size 4763648 starting block 1749862)
[ 2572.459905] Buffer I/O error on device dm-8, logical block 1749862
[ 2572.487692] EXT4-fs warning (device dm-8): ext4_end_bio:316: I/O error -28 writing to inode 12 (offset 6425673728 size 4763648 starting block 1749863)
[ 2572.548701] Buffer I/O error on device dm-8, logical block 1749863
[ 2572.576416] EXT4-fs warning (device dm-8): ext4_end_bio:316: I/O error -28 writing to inode 12 (offset 6425673728 size 4763648 starting block 1749864)
[ 2572.636868] Buffer I/O error on device dm-8, logical block 1749864
[ 2572.664199] EXT4-fs warning (device dm-8): ext4_end_bio:316: I/O error -28 writing to inode 12 (offset 6425673728 size 4763648 starting block 1749865)
[ 2572.724801] Buffer I/O error on device dm-8, logical block 1749865
[ 2572.752564] EXT4-fs warning (device dm-8): ext4_end_bio:316: I/O error -28 writing to inode 12 (offset 6425673728 size 4763648 starting block 1749866)
[ 2572.814471] Buffer I/O error on device dm-8, logical block 1749866
[ 2572.844917] EXT4-fs warning (device dm-8): ext4_end_bio:316: I/O error -28 writing to inode 12 (offset 6425673728 size 4763648 starting block 1749867)
[ 2572.907179] Buffer I/O error on device dm-8, logical block 1749867
[ 2572.934908] EXT4-fs warning (device dm-8): ext4_end_bio:316: I/O error -28 writing to inode 12 (offset 6425673728 size 4763648 starting block 1749868)
[ 2572.995345] Buffer I/O error on device dm-8, logical block 1749868
[ 2573.023158] EXT4-fs warning (device dm-8): ext4_end_bio:316: I/O error -28 writing to inode 12 (offset 6425673728 size 4763648 starting block 1749869)
[ 2573.083747] Buffer I/O error on device dm-8, logical block 1749869
[ 2573.111821] EXT4-fs warning (device dm-8): ext4_end_bio:316: I/O error -28 writing to inode 12 (offset 6425673728 size 4763648 starting block 1749870)
[ 2573.172606] Buffer I/O error on device dm-8, logical block 1749870
[ 2575.794617] JBD2: Detected IO errors while flushing file data on dm-8-8
[ 2575.828750] Aborting journal on device dm-8-8.
[ 2575.851007] buffer_io_error: 194548 callbacks suppressed
[ 2575.875206] Buffer I/O error on dev dm-8, logical block 1081344, lost sync page write
[ 2575.910402] JBD2: Error -5 detected when updating journal superblock for dm-8-8.
[ 2578.976738] Buffer I/O error on dev dm-8, logical block 0, lost sync page write
[ 2579.014578] EXT4-fs error (device dm-8): ext4_journal_check_start:56: Detected aborted journal
[ 2579.056630] EXT4-fs (dm-8): Remounting filesystem read-only
[ 2579.081702] EXT4-fs (dm-8): previous I/O error to superblock detected
[ 2579.111212] Buffer I/O error on dev dm-8, logical block 0, lost sync page write
[ 2579.143909] EXT4-fs (dm-8): ext4_writepages: jbd2_start: 57344 pages, ino 12; err -30
I managed to get these errors on other configurations as well. I am investigating that now and will probably create a new BZ when I have enough info and I am sure this is not related. Till than I leave this open.
Created BZ 1517911. As it is not related to this one (reproduced it on other setups), setting as verified. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:0871 |