Bug 1532682
| Summary: | Overwriting data in VDO leads to I/O errors | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Jakub Krysl <jkrysl> |
| Component: | vdo | Assignee: | corwin <corwin> |
| Status: | CLOSED NOTABUG | QA Contact: | Jakub Krysl <jkrysl> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.5 | CC: | awalsh, bgurney, dkeefe, jkrysl, limershe, pasik, sweettea |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-02-03 15:41:19 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Attachments: | |||
When VDO writes unique data, it requires a free block -- it will only free the block previously used for that logical address after the new data is completely committed to disk. A downside of that is that if you fill up your VDO, you can only overwrite with zero blocks or duplicate data to free up space. Jakub, Can you add the VDO status or vdostats --verbose output before and after the test. Created attachment 1383237 [details]
vdo status and vdostats --verbose before (clean VDO)
Created attachment 1383238 [details]
vdo status and vdostats --verbose after 1st fill (dd)
# dd if=/dev/urandom of=/dev/mapper/vdo bs=4k count=522989 status=progress
2048585728 bytes (2.0 GB) copied, 11.000145 s, 186 MB/s
522989+0 records in
522989+0 records out
2142162944 bytes (2.1 GB) copied, 52.3088 s, 41.0 MB/s
Created attachment 1383240 [details]
vdo status and vdostats --verbose after 2nd fill (dd)
# dd if=/dev/urandom of=/dev/mapper/vdo bs=4k count=522989 status=progress
2055692288 bytes (2.1 GB) copied, 11.000114 s, 187 MB/s
522989+0 records in
522989+0 records out
2142162944 bytes (2.1 GB) copied, 20.5383 s, 104 MB/s
/var/log/messages:
[83188.556680] Buffer I/O error on dev dm-3, logical block 120, lost async page write
[83188.564263] Buffer I/O error on dev dm-3, logical block 122, lost async page write
[83188.571830] Buffer I/O error on dev dm-3, logical block 121, lost async page write
[83188.579392] Buffer I/O error on dev dm-3, logical block 123, lost async page write
[83188.586950] Buffer I/O error on dev dm-3, logical block 124, lost async page write
[83188.594508] Buffer I/O error on dev dm-3, logical block 125, lost async page write
[83188.602066] Buffer I/O error on dev dm-3, logical block 126, lost async page write
[83188.609637] Buffer I/O error on dev dm-3, logical block 127, lost async page write
[83188.617200] Buffer I/O error on dev dm-3, logical block 128, lost async page write
[83188.624765] Buffer I/O error on dev dm-3, logical block 129, lost async page write
[83193.565024] buffer_io_error: 287836 callbacks suppressed
[83193.570343] Buffer I/O error on dev dm-3, logical block 296165, lost async page write
[83193.578165] Buffer I/O error on dev dm-3, logical block 296164, lost async page write
[83193.586007] Buffer I/O error on dev dm-3, logical block 296167, lost async page write
[83193.593850] Buffer I/O error on dev dm-3, logical block 296166, lost async page write
[83193.601711] Buffer I/O error on dev dm-3, logical block 296168, lost async page write
[83193.609532] Buffer I/O error on dev dm-3, logical block 296169, lost async page write
[83193.617357] Buffer I/O error on dev dm-3, logical block 296170, lost async page write
[83193.625177] Buffer I/O error on dev dm-3, logical block 296171, lost async page write
[83193.633001] Buffer I/O error on dev dm-3, logical block 296172, lost async page write
[83193.640822] Buffer I/O error on dev dm-3, logical block 296163, lost async page write
[83197.616725] VFS: Dirty inode writeback failed for block device dm-3 (err=-5).
Created attachment 1383241 [details]
/var/log/messages
I loaded uds and kvdo modules beforehand to be sure to have the newest version.
Based on the discussion in this ticket, I am closing as NOTABUG, since this is expected behavior. |
Description of problem: Filling VDO 2nd time using dd with exactly the amount of data as there is logical space on VDO leads to I/O errors. # vdo create --name vdo --device /dev/mapper/test-small Creating VDO vdo Starting VDO vdo Starting compression on VDO vdo VDO instance 13 volume is ready at /dev/mapper/vdo # vdo status | grep logical logical blocks: 522989 logical blocks used: 0 # dd if=/dev/urandom of=/dev/mapper/vdo bs=4k count=522989 status=progress 2076041216 bytes (2.1 GB) copied, 11.000138 s, 189 MB/s 522989+0 records in 522989+0 records out 2142162944 bytes (2.1 GB) copied, 35.7177 s, 60.0 MB/s # vdostats Device 1K-blocks Used Available Use% Space saving% /dev/mapper/vdo 5242880 5242400 480 99% 0% Note: default VDO is under-provisioned in this version, hence the available 480 1-K blocks. With this VDO size (3G backing device) there will always be 480 1-K blocks left "available". Doing the same dd results in I/O errors in /var/log/messages now: # dd if=/dev/urandom of=/dev/mapper/vdo bs=4k count=522989 status=progress 2082357248 bytes (2.1 GB) copied, 11.000125 s, 189 MB/s 522989+0 records in 522989+0 records out 2142162944 bytes (2.1 GB) copied, 20.0007 s, 107 MB/s /var/log/messages here: [107605.525671] buffer_io_error: 199636 callbacks suppressed [107605.531075] Buffer I/O error on dev dm-5, logical block 120, lost async page write [107605.538731] Buffer I/O error on dev dm-5, logical block 121, lost async page write [107605.546378] Buffer I/O error on dev dm-5, logical block 122, lost async page write [107605.554022] Buffer I/O error on dev dm-5, logical block 123, lost async page write [107605.561676] Buffer I/O error on dev dm-5, logical block 124, lost async page write [107605.569344] Buffer I/O error on dev dm-5, logical block 125, lost async page write [107605.576989] Buffer I/O error on dev dm-5, logical block 126, lost async page write [107605.584631] Buffer I/O error on dev dm-5, logical block 127, lost async page write [107605.592279] Buffer I/O error on dev dm-5, logical block 128, lost async page write [107605.599928] Buffer I/O error on dev dm-5, logical block 130, lost async page write [107610.539335] buffer_io_error: 292574 callbacks suppressed [107610.544736] Buffer I/O error on dev dm-5, logical block 310883, lost async page write [107610.552647] Buffer I/O error on dev dm-5, logical block 310884, lost async page write [107610.560562] Buffer I/O error on dev dm-5, logical block 310885, lost async page write [107610.568469] Buffer I/O error on dev dm-5, logical block 310886, lost async page write [107610.576372] Buffer I/O error on dev dm-5, logical block 310889, lost async page write [107610.584276] Buffer I/O error on dev dm-5, logical block 310890, lost async page write [107610.592180] Buffer I/O error on dev dm-5, logical block 310895, lost async page write [107610.600097] Buffer I/O error on dev dm-5, logical block 310897, lost async page write [107610.608012] Buffer I/O error on dev dm-5, logical block 310899, lost async page write [107610.615920] Buffer I/O error on dev dm-5, logical block 310900, lost async page write [107614.204312] VFS: Dirty inode writeback failed for block device dm-5 (err=-5). At this point writing (overwriting) only 1000 4K blocks to VDO leads to errors again: # dd if=/dev/urandom of=/dev/mapper/vdo bs=4k count=1000 status=progress 1000+0 records in 1000+0 records out 4096000 bytes (4.1 MB) copied, 0.118492 s, 34.6 MB/s same /var/log/messages as above. Zeroing the vdo and filling it up again leads to no I/O errors: # dd if=/dev/urandom of=/dev/mapper/vdo bs=4k count=522989 status=progress 2072612864 bytes (2.1 GB) copied, 11.000083 s, 188 MB/s 522989+0 records in 522989+0 records out 2142162944 bytes (2.1 GB) copied, 35.5554 s, 60.2 MB/s # dd if=/dev/zero of=/dev/mapper/vdo bs=4k count=522989 status=progress 522989+0 records in 522989+0 records out 2142162944 bytes (2.1 GB) copied, 2.6893 s, 797 MB/s # dd if=/dev/urandom of=/dev/mapper/vdo bs=4k count=522989 status=progress 2089074688 bytes (2.1 GB) copied, 11.000131 s, 190 MB/s 522989+0 records in 522989+0 records out 2142162944 bytes (2.1 GB) copied, 28.0129 s, 76.5 MB/s Version-Release number of selected component (if applicable): vdo-6.1.0.106 How reproducible: 100% Steps to Reproduce: 1. vdo create --name vdo --device /dev/sdc 2. vdo growLogical --name vdo --vdoLogicalSize (logical_size + 4K) 3. dd if=/dev/urandom of=/dev/mapper/vdo bs=4K count=522989 status=progress 3. repeat step 3 Actual results: I/O errors in /var/log/messages when overwriting data in VDO Expected results: no I/O errors Additional info: