Bug 1883643
| Summary: | Using MD RAID on a cinder volume exposes the data to the compute host | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | David Vallee Delisle <dvd> |
| Component: | openstack-tripleo-heat-templates | Assignee: | Alan Bishop <abishop> |
| Status: | CLOSED DEFERRED | QA Contact: | David Rosenfeld <drosenfe> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 10.0 (Newton) | CC: | abishop, acanan, agk, akaris, astillma, broose, cschwede, eharney, gcharot, gfidente, jbrassow, jpittman, jraju, jvisser, jwaterwo, lmarsh, mabrams, marjones, mburns, nkshirsa, nsoffer, nwolf, pablo.iranzo, pgrist, pmorey, sclewis, scohen, sgotliv, spower, sputhenp, srevivo, teigland, tkajinam, tshefi, tvignaud, tvvcox, vcojot, vfarias, zkabelac |
| Target Milestone: | --- | Keywords: | Triaged |
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | 1261083 | Environment: | |
| Last Closed: | 2020-11-04 15:20:04 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1261083, 1884225 | ||
| Bug Blocks: | 1518969 | ||
|
Description
David Vallee Delisle
2020-09-29 18:51:32 UTC
I'm setting up an env where I can study the behavior. Why all the private comments? Thanks for the advice, folks! I now have an env in which I can reproduce the problem, and the next step is to run a series of experiments to determine the most effective way of filtering devices without breaking other things. Hello,
A concern was raised by the fact that we documented some possible filesystem corruption after adding AUTO -all in /etc/mdadm.conf. I revisited the sosreport and here's the logs and the presumed timeline.
[1] We had recurring "tur checker reports path is down" messages returned by multipath
[2] At some point, during the troubleshooting of the case, we ended up with these kernel messages, like if the underlying block devices were pulled out without proper unmounting.
It's worth nothing that the second batch of errors happened at the time the sosreport was taken, so it might be caused by a read operation like listing the partitions.
Also, regarding the 20:29 errors, this happened during a troubleshooting session. We did a lot of operations to try to flush these devices like dmsetup remove, multipath -f.
One last thing, at the time of taking sosreport on that specific compute node, I don't see /etc/mdadm.conf in there, and mdadm still has the device mapper devices [3]. I can only presume that /etc/mdadm.conf didn't include AUTO -all at the time the sosreport was taken.
I'm no storage expert, but at this point, I'd be surprised if "AUTO -all" would have caused said corruption.
Thanks,
DVD
[1]
~~~
2020-09-23T20:29:42.818521+00:00 overcloud-compute-0 multipathd: 3600601607b703600d3008296faabe911: sdj - tur checker reports path is down
2020-09-23T20:29:42.818708+00:00 overcloud-compute-0 multipathd: 3600601607b703600d3008296faabe911: sdm - tur checker reports path is down
2020-09-23T20:29:42.818888+00:00 overcloud-compute-0 multipathd: 3600601607b703600d3008296faabe911: sdk - tur checker reports path is down
2020-09-23T20:29:42.819137+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdn - tur checker reports path is down
2020-09-23T20:29:42.819380+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdq - tur checker reports path is down
2020-09-23T20:29:42.819621+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdt - tur checker reports path is down
2020-09-23T20:29:42.819865+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdr - tur checker reports path is down
2020-09-23T20:29:42.820089+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdv - tur checker reports path is down
2020-09-23T20:29:43.820706+00:00 overcloud-compute-0 multipathd: 3600601607b703600d3008296faabe911: sdf - tur checker reports path is down
2020-09-23T20:29:43.821015+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdx - tur checker reports path is down
2020-09-23T20:29:43.821232+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdab - tur checker reports path is down
2020-09-23T20:29:43.821448+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdad - tur checker reports path is down
2020-09-23T20:29:43.822583+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdaf - tur checker reports path is down
2020-09-23T20:29:43.822846+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdal - tur checker reports path is down
2020-09-23T20:29:43.824198+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdap - tur checker reports path is down
2020-09-23T20:29:43.825539+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdar - tur checker reports path is down
2020-09-23T20:29:43.826730+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdau - tur checker reports path is down
2020-09-23T20:29:43.826957+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdaw - tur checker reports path is down
2020-09-23T20:29:44.827305+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdae - tur checker reports path is down
2020-09-23T20:29:44.827588+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdam - tur checker reports path is down
2020-09-23T20:29:44.827817+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdan - tur checker reports path is down
2020-09-23T20:29:44.828023+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdaq - tur checker reports path is down
2020-09-23T20:29:44.828237+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdav - tur checker reports path is down
2020-09-23T20:29:45.830050+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdu - tur checker reports path is down
2020-09-23T20:29:45.830378+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdy - tur checker reports path is down
2020-09-23T20:29:45.830575+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdw - tur checker reports path is down
2020-09-23T20:29:45.830768+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdac - tur checker reports path is down
2020-09-23T20:29:45.830930+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdaa - tur checker reports path is down
2020-09-23T20:29:45.831103+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdag - tur checker reports path is down
2020-09-23T20:29:45.831264+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdao - tur checker reports path is down
2020-09-23T20:29:45.831447+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdas - tur checker reports path is down
2020-09-23T20:29:45.831622+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdax - tur checker reports path is down
2020-09-23T20:29:46.831874+00:00 overcloud-compute-0 multipathd: 3600601607b703600d3008296faabe911: sdd - tur checker reports path is down
2020-09-23T20:29:46.832109+00:00 overcloud-compute-0 multipathd: 3600601607b703600d3008296faabe911: sdl - tur checker reports path is down
2020-09-23T20:29:46.832307+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdp - tur checker reports path is down
2020-09-23T20:29:46.832485+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdo - tur checker reports path is down
2020-09-23T20:29:46.832658+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sds - tur checker reports path is down
2020-09-23T20:29:46.832842+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdz - tur checker reports path is down
~~~
[2]
~~~
2020-09-23T20:29:46.991585+00:00 overcloud-compute-0 kernel: blk_update_request: I/O error, dev dm-0, sector 8
2020-09-23T20:29:46.991733+00:00 overcloud-compute-0 kernel: buffer_io_error: 909 callbacks suppressed
2020-09-23T20:29:46.991833+00:00 overcloud-compute-0 kernel: Buffer I/O error on dev dm-0, logical block 1, async page read
2020-09-23T20:29:46.991889+00:00 overcloud-compute-0 kernel: blk_update_request: I/O error, dev dm-0, sector 8
2020-09-23T20:29:47.113830+00:00 overcloud-compute-0 kernel: Buffer I/O error on dev dm-0, logical block 1, async page read
2020-09-23T20:29:47.113898+00:00 overcloud-compute-0 kernel: blk_update_request: I/O error, dev dm-0, sector 8
2020-09-23T20:29:47.113924+00:00 overcloud-compute-0 kernel: Buffer I/O error on dev dm-0, logical block 1, async page read
2020-09-23T20:29:47.113973+00:00 overcloud-compute-0 kernel: blk_update_request: I/O error, dev dm-0, sector 8
2020-09-23T20:29:47.174240+00:00 overcloud-compute-0 kernel: Buffer I/O error on dev dm-0, logical block 1, async page read
2020-09-23T20:29:47.201817+00:00 overcloud-compute-0 kernel: blk_update_request: I/O error, dev dm-0, sector 8
2020-09-23T20:29:47.201881+00:00 overcloud-compute-0 kernel: Buffer I/O error on dev dm-0, logical block 1, async page read
2020-09-23T20:29:47.262283+00:00 overcloud-compute-0 kernel: blk_update_request: I/O error, dev dm-0, sector 8
2020-09-23T20:29:47.262334+00:00 overcloud-compute-0 kernel: Buffer I/O error on dev dm-0, logical block 1, async page read
2020-09-23T20:29:47.355168+00:00 overcloud-compute-0 kernel: blk_update_request: I/O error, dev dm-0, sector 8
2020-09-23T20:29:47.355212+00:00 overcloud-compute-0 kernel: Buffer I/O error on dev dm-0, logical block 1, async page read
2020-09-23T20:29:47.355237+00:00 overcloud-compute-0 kernel: blk_update_request: I/O error, dev dm-0, sector 8
2020-09-23T20:29:47.383049+00:00 overcloud-compute-0 kernel: Buffer I/O error on dev dm-0, logical block 1, async page read
2020-09-23T20:29:47.479099+00:00 overcloud-compute-0 kernel: blk_update_request: I/O error, dev dm-0, sector 128
2020-09-23T20:29:47.479170+00:00 overcloud-compute-0 kernel: Buffer I/O error on dev dm-0, logical block 16, async page read
2020-09-23T20:29:47.479211+00:00 overcloud-compute-0 kernel: blk_update_request: I/O error, dev dm-0, sector 64
2020-09-23T20:29:47.539936+00:00 overcloud-compute-0 kernel: Buffer I/O error on dev dm-0, logical block 8, async page read
2020-09-23T20:29:47.655758+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2020-09-23T20:29:47.656088+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.656294+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.656469+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
2020-09-23T20:29:47.656669+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2020-09-23T20:29:47.656862+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.657021+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.657181+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
2020-09-23T20:29:47.657313+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2020-09-23T20:29:47.657505+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.657624+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.657736+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] CDB: Read(10) 28 00 00 00 00 00 00 00 20 00
2020-09-23T20:29:47.657921+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2020-09-23T20:29:47.658063+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.658227+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.658384+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
2020-09-23T20:29:47.658558+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2020-09-23T20:29:47.658729+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.658864+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.659010+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
2020-09-23T20:29:47.659180+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2020-09-23T20:29:47.659302+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.659433+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.659538+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
2020-09-23T20:29:47.659645+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2020-09-23T20:29:47.659770+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.659892+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.660013+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
2020-09-23T20:29:47.660116+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2020-09-23T20:29:47.660219+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.660334+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.660469+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
2020-09-23T20:29:47.660578+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2020-09-23T20:29:47.660682+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.660791+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.660885+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
2020-09-23T20:29:47.661011+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2020-09-23T20:29:47.661108+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.661264+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.661448+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
2020-09-23T20:29:47.661592+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] Unit Not Ready
2020-09-23T20:29:47.661717+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.661888+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.662007+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2020-09-23T20:29:47.662135+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.662236+00:00 overcloud-compute-0 kernel: sd 5:0:0:63: [sdad] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.662281+00:00 overcloud-compute-0 kernel: sdad: detected capacity change from 1073741824000 to 0
2020-09-23T20:29:47.683541+00:00 overcloud-compute-0 kernel: sd 6:0:0:63: [sdae] Unit Not Ready
2020-09-23T20:29:47.683781+00:00 overcloud-compute-0 kernel: sd 6:0:0:63: [sdae] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.683967+00:00 overcloud-compute-0 kernel: sd 6:0:0:63: [sdae] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.684122+00:00 overcloud-compute-0 kernel: sd 6:0:0:63: [sdae] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2020-09-23T20:29:47.684311+00:00 overcloud-compute-0 kernel: sd 6:0:0:63: [sdae] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.684451+00:00 overcloud-compute-0 kernel: sd 6:0:0:63: [sdae] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.684500+00:00 overcloud-compute-0 kernel: sdae: detected capacity change from 1073741824000 to 0
2020-09-23T20:29:47.732545+00:00 overcloud-compute-0 kernel: sd 7:0:0:63: [sdaf] Unit Not Ready
2020-09-23T20:29:47.732710+00:00 overcloud-compute-0 kernel: sd 7:0:0:63: [sdaf] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.732846+00:00 overcloud-compute-0 kernel: sd 7:0:0:63: [sdaf] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.732976+00:00 overcloud-compute-0 kernel: sd 7:0:0:63: [sdaf] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2020-09-23T20:29:47.733125+00:00 overcloud-compute-0 kernel: sd 7:0:0:63: [sdaf] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.733226+00:00 overcloud-compute-0 kernel: sd 7:0:0:63: [sdaf] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.733287+00:00 overcloud-compute-0 kernel: sdaf: detected capacity change from 1073741824000 to 0
2020-09-23T20:29:47.739513+00:00 overcloud-compute-0 kernel: sd 8:0:0:63: [sdag] Unit Not Ready
2020-09-23T20:29:47.739681+00:00 overcloud-compute-0 kernel: sd 8:0:0:63: [sdag] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.739849+00:00 overcloud-compute-0 kernel: sd 8:0:0:63: [sdag] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.739982+00:00 overcloud-compute-0 kernel: sd 8:0:0:63: [sdag] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2020-09-23T20:29:47.740138+00:00 overcloud-compute-0 kernel: sd 8:0:0:63: [sdag] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.740296+00:00 overcloud-compute-0 kernel: sd 8:0:0:63: [sdag] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.740347+00:00 overcloud-compute-0 kernel: sdag: detected capacity change from 1073741824000 to 0
2020-09-23T20:29:47.763549+00:00 overcloud-compute-0 kernel: sd 5:0:0:207: [sdal] Unit Not Ready
2020-09-23T20:29:47.763756+00:00 overcloud-compute-0 kernel: sd 5:0:0:207: [sdal] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.763939+00:00 overcloud-compute-0 kernel: sd 5:0:0:207: [sdal] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.764065+00:00 overcloud-compute-0 kernel: sd 5:0:0:207: [sdal] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2020-09-23T20:29:47.764162+00:00 overcloud-compute-0 kernel: sd 5:0:0:207: [sdal] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.764266+00:00 overcloud-compute-0 kernel: sd 5:0:0:207: [sdal] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.764307+00:00 overcloud-compute-0 kernel: sdal: detected capacity change from 1073741824000 to 0
2020-09-23T20:29:47.811560+00:00 overcloud-compute-0 kernel: sd 6:0:0:207: [sdam] Unit Not Ready
2020-09-23T20:29:47.811726+00:00 overcloud-compute-0 kernel: sd 6:0:0:207: [sdam] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.811879+00:00 overcloud-compute-0 kernel: sd 6:0:0:207: [sdam] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.811991+00:00 overcloud-compute-0 kernel: sd 6:0:0:207: [sdam] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2020-09-23T20:29:47.812103+00:00 overcloud-compute-0 kernel: sd 6:0:0:207: [sdam] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.812199+00:00 overcloud-compute-0 kernel: sd 6:0:0:207: [sdam] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.812249+00:00 overcloud-compute-0 kernel: sdam: detected capacity change from 1073741824000 to 0
2020-09-23T20:29:47.834493+00:00 overcloud-compute-0 multipathd: 3600601607b703600d3008296faabe911: sdb - tur checker reports path is down
2020-09-23T20:29:47.834844+00:00 overcloud-compute-0 multipathd: 3600601607b703600d3008296faabe911: sde - tur checker reports path is down
2020-09-23T20:29:47.835025+00:00 overcloud-compute-0 multipathd: 3600601607b703600d3008296faabe911: sdc - tur checker reports path is down
2020-09-23T20:29:47.835214+00:00 overcloud-compute-0 multipathd: 3600601607b703600d3008296faabe911: sdg - tur checker reports path is down
2020-09-23T20:29:47.835396+00:00 overcloud-compute-0 multipathd: 3600601607b703600d3008296faabe911: sdh - tur checker reports path is down
2020-09-23T20:29:47.835593+00:00 overcloud-compute-0 multipathd: 3600601607b703600d3008296faabe911: sdi - tur checker reports path is down
2020-09-23T20:29:47.835785+00:00 overcloud-compute-0 multipathd: 3600601607b703600d3008296faabe911: sdj - tur checker reports path is down
2020-09-23T20:29:47.835957+00:00 overcloud-compute-0 multipathd: 3600601607b703600d3008296faabe911: sdm - tur checker reports path is down
2020-09-23T20:29:47.836122+00:00 overcloud-compute-0 multipathd: 3600601607b703600d3008296faabe911: sdk - tur checker reports path is down
2020-09-23T20:29:47.836303+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdn - tur checker reports path is down
2020-09-23T20:29:47.836496+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdq - tur checker reports path is down
2020-09-23T20:29:47.836676+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdt - tur checker reports path is down
2020-09-23T20:29:47.836873+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdr - tur checker reports path is down
2020-09-23T20:29:47.837045+00:00 overcloud-compute-0 multipathd: 3600601607b7036000aefd29a4aabe911: sdv - tur checker reports path is down
2020-09-23T20:29:47.857542+00:00 overcloud-compute-0 kernel: sd 7:0:0:207: [sdan] Unit Not Ready
2020-09-23T20:29:47.857765+00:00 overcloud-compute-0 kernel: sd 7:0:0:207: [sdan] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.857929+00:00 overcloud-compute-0 kernel: sd 7:0:0:207: [sdan] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.858079+00:00 overcloud-compute-0 kernel: sd 7:0:0:207: [sdan] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2020-09-23T20:29:47.858211+00:00 overcloud-compute-0 kernel: sd 7:0:0:207: [sdan] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.858337+00:00 overcloud-compute-0 kernel: sd 7:0:0:207: [sdan] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.858379+00:00 overcloud-compute-0 kernel: sdan: detected capacity change from 1073741824000 to 0
2020-09-23T20:29:47.880567+00:00 overcloud-compute-0 kernel: sd 8:0:0:207: [sdao] Unit Not Ready
2020-09-23T20:29:47.880798+00:00 overcloud-compute-0 kernel: sd 8:0:0:207: [sdao] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.880930+00:00 overcloud-compute-0 kernel: sd 8:0:0:207: [sdao] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.881040+00:00 overcloud-compute-0 kernel: sd 8:0:0:207: [sdao] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2020-09-23T20:29:47.881157+00:00 overcloud-compute-0 kernel: sd 8:0:0:207: [sdao] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.881267+00:00 overcloud-compute-0 kernel: sd 8:0:0:207: [sdao] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.881326+00:00 overcloud-compute-0 kernel: sdao: detected capacity change from 1073741824000 to 0
2020-09-23T20:29:47.903543+00:00 overcloud-compute-0 kernel: sd 5:0:0:225: [sdap] Unit Not Ready
2020-09-23T20:29:47.903747+00:00 overcloud-compute-0 kernel: sd 5:0:0:225: [sdap] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.903888+00:00 overcloud-compute-0 kernel: sd 5:0:0:225: [sdap] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.904018+00:00 overcloud-compute-0 kernel: sd 5:0:0:225: [sdap] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2020-09-23T20:29:47.904153+00:00 overcloud-compute-0 kernel: sd 5:0:0:225: [sdap] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.904270+00:00 overcloud-compute-0 kernel: sd 5:0:0:225: [sdap] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.904316+00:00 overcloud-compute-0 kernel: sdap: detected capacity change from 1073741824000 to 0
2020-09-23T20:29:47.926540+00:00 overcloud-compute-0 kernel: sd 6:0:0:225: [sdaq] Unit Not Ready
2020-09-23T20:29:47.926755+00:00 overcloud-compute-0 kernel: sd 6:0:0:225: [sdaq] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.926929+00:00 overcloud-compute-0 kernel: sd 6:0:0:225: [sdaq] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.927069+00:00 overcloud-compute-0 kernel: sd 6:0:0:225: [sdaq] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2020-09-23T20:29:47.927212+00:00 overcloud-compute-0 kernel: sd 6:0:0:225: [sdaq] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.927330+00:00 overcloud-compute-0 kernel: sd 6:0:0:225: [sdaq] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.927396+00:00 overcloud-compute-0 kernel: sdaq: detected capacity change from 1073741824000 to 0
2020-09-23T20:29:47.948536+00:00 overcloud-compute-0 kernel: sd 7:0:0:225: [sdar] Unit Not Ready
2020-09-23T20:29:47.948682+00:00 overcloud-compute-0 kernel: sd 7:0:0:225: [sdar] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.948801+00:00 overcloud-compute-0 kernel: sd 7:0:0:225: [sdar] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.948913+00:00 overcloud-compute-0 kernel: sd 7:0:0:225: [sdar] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2020-09-23T20:29:47.949035+00:00 overcloud-compute-0 kernel: sd 7:0:0:225: [sdar] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.949138+00:00 overcloud-compute-0 kernel: sd 7:0:0:225: [sdar] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.949189+00:00 overcloud-compute-0 kernel: sdar: detected capacity change from 1073741824000 to 0
2020-09-23T20:29:47.971558+00:00 overcloud-compute-0 kernel: sd 8:0:0:225: [sdas] Unit Not Ready
2020-09-23T20:29:47.971798+00:00 overcloud-compute-0 kernel: sd 8:0:0:225: [sdas] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.971957+00:00 overcloud-compute-0 kernel: sd 8:0:0:225: [sdas] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.972081+00:00 overcloud-compute-0 kernel: sd 8:0:0:225: [sdas] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2020-09-23T20:29:47.972210+00:00 overcloud-compute-0 kernel: sd 8:0:0:225: [sdas] Sense Key : Illegal Request [current]
2020-09-23T20:29:47.972327+00:00 overcloud-compute-0 kernel: sd 8:0:0:225: [sdas] Add. Sense: Logical unit not supported
2020-09-23T20:29:47.972414+00:00 overcloud-compute-0 kernel: sdas: detected capacity change from 1073741824000 to 0
2020-09-23T20:29:48.022572+00:00 overcloud-compute-0 kernel: sd 5:0:0:214: [sdau] Unit Not Ready
2020-09-23T20:29:48.022721+00:00 overcloud-compute-0 kernel: sd 5:0:0:214: [sdau] Sense Key : Illegal Request [current]
2020-09-23T20:29:48.022853+00:00 overcloud-compute-0 kernel: sd 5:0:0:214: [sdau] Add. Sense: Logical unit not supported
2020-09-23T20:29:48.022985+00:00 overcloud-compute-0 kernel: sd 5:0:0:214: [sdau] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2020-09-23T20:29:48.023119+00:00 overcloud-compute-0 kernel: sd 5:0:0:214: [sdau] Sense Key : Illegal Request [current]
2020-09-23T20:29:48.023207+00:00 overcloud-compute-0 kernel: sd 5:0:0:214: [sdau] Add. Sense: Logical unit not supported
2020-09-23T20:29:48.023251+00:00 overcloud-compute-0 kernel: sdau: detected capacity change from 1073741824000 to 0
2020-09-23T20:29:48.044557+00:00 overcloud-compute-0 kernel: sd 6:0:0:214: [sdav] Unit Not Ready
2020-09-23T20:29:48.044779+00:00 overcloud-compute-0 kernel: sd 6:0:0:214: [sdav] Sense Key : Illegal Request [current]
2020-09-23T20:29:48.044962+00:00 overcloud-compute-0 kernel: sd 6:0:0:214: [sdav] Add. Sense: Logical unit not supported
2020-09-23T20:29:48.045116+00:00 overcloud-compute-0 kernel: sd 6:0:0:214: [sdav] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2020-09-23T20:29:48.045292+00:00 overcloud-compute-0 kernel: sd 6:0:0:214: [sdav] Sense Key : Illegal Request [current]
2020-09-23T20:29:48.045436+00:00 overcloud-compute-0 kernel: sd 6:0:0:214: [sdav] Add. Sense: Logical unit not supported
2020-09-23T20:29:48.045492+00:00 overcloud-compute-0 kernel: sdav: detected capacity change from 1073741824000 to 0
2020-09-23T20:29:48.073542+00:00 overcloud-compute-0 kernel: sd 7:0:0:214: [sdaw] Unit Not Ready
2020-09-23T20:29:48.073740+00:00 overcloud-compute-0 kernel: sd 7:0:0:214: [sdaw] Sense Key : Illegal Request [current]
2020-09-23T20:29:48.073882+00:00 overcloud-compute-0 kernel: sd 7:0:0:214: [sdaw] Add. Sense: Logical unit not supported
2020-09-23T20:29:48.074063+00:00 overcloud-compute-0 kernel: sd 7:0:0:214: [sdaw] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2020-09-23T20:29:48.074207+00:00 overcloud-compute-0 kernel: sd 7:0:0:214: [sdaw] Sense Key : Illegal Request [current]
2020-09-23T20:29:48.074393+00:00 overcloud-compute-0 kernel: sd 7:0:0:214: [sdaw] Add. Sense: Logical unit not supported
2020-09-23T20:29:48.074449+00:00 overcloud-compute-0 kernel: sdaw: detected capacity change from 1073741824000 to 0
2020-09-23T20:29:48.093553+00:00 overcloud-compute-0 kernel: sd 8:0:0:214: [sdax] Unit Not Ready
2020-09-23T20:29:48.093702+00:00 overcloud-compute-0 kernel: sd 8:0:0:214: [sdax] Sense Key : Illegal Request [current]
2020-09-23T20:29:48.093842+00:00 overcloud-compute-0 kernel: sd 8:0:0:214: [sdax] Add. Sense: Logical unit not supported
[...]
2020-09-23T21:20:11.448513+00:00 overcloud-compute-0 kernel: blk_update_request: I/O error, dev dm-0, sector 8
2020-09-23T21:20:11.448584+00:00 overcloud-compute-0 kernel: buffer_io_error: 581 callbacks suppressed
2020-09-23T21:20:11.448614+00:00 overcloud-compute-0 kernel: Buffer I/O error on dev dm-0, logical block 1, async page read
2020-09-23T21:20:11.448636+00:00 overcloud-compute-0 kernel: blk_update_request: I/O error, dev dm-0, sector 8
2020-09-23T21:20:11.509030+00:00 overcloud-compute-0 kernel: Buffer I/O error on dev dm-0, logical block 1, async page read
2020-09-23T21:20:11.509086+00:00 overcloud-compute-0 kernel: blk_update_request: I/O error, dev dm-0, sector 8
2020-09-23T21:20:11.536946+00:00 overcloud-compute-0 kernel: Buffer I/O error on dev dm-0, logical block 1, async page read
2020-09-23T21:20:11.569442+00:00 overcloud-compute-0 kernel: blk_update_request: I/O error, dev dm-0, sector 8
2020-09-23T21:20:11.631674+00:00 overcloud-compute-0 kernel: Buffer I/O error on dev dm-0, logical block 1, async page read
2020-09-23T21:20:11.631727+00:00 overcloud-compute-0 kernel: blk_update_request: I/O error, dev dm-0, sector 8
2020-09-23T21:20:11.659570+00:00 overcloud-compute-0 kernel: Buffer I/O error on dev dm-0, logical block 1, async page read
2020-09-23T21:20:11.720400+00:00 overcloud-compute-0 kernel: blk_update_request: I/O error, dev dm-0, sector 8
2020-09-23T21:20:11.720472+00:00 overcloud-compute-0 kernel: Buffer I/O error on dev dm-0, logical block 1, async page read
2020-09-23T21:20:11.781559+00:00 overcloud-compute-0 kernel: blk_update_request: I/O error, dev dm-0, sector 8
2020-09-23T21:20:11.781599+00:00 overcloud-compute-0 kernel: Buffer I/O error on dev dm-0, logical block 1, async page read
2020-09-23T21:20:11.842215+00:00 overcloud-compute-0 kernel: blk_update_request: I/O error, dev dm-0, sector 8
2020-09-23T21:20:11.842264+00:00 overcloud-compute-0 kernel: Buffer I/O error on dev dm-0, logical block 1, async page read
2020-09-23T21:20:11.903762+00:00 overcloud-compute-0 kernel: blk_update_request: I/O error, dev dm-0, sector 128
2020-09-23T21:20:11.903813+00:00 overcloud-compute-0 kernel: Buffer I/O error on dev dm-0, logical block 16, async page read
2020-09-23T21:20:11.969855+00:00 overcloud-compute-0 kernel: Buffer I/O error on dev dm-0, logical block 8, async page read
~~~
[3]
~~~
02755983]$ cat sosreport-20200923-212117/overcloud-compute-0/sos_commands/md/mdadm_-D_.dev.md
/dev/md125:
Version : 1.1
Raid Level : raid0
Total Devices : 0
Persistence : Superblock is persistent
State : inactive
Number Major Minor RaidDevice
/dev/md126:
Version : 1.1
Raid Level : raid0
Total Devices : 1
Persistence : Superblock is persistent
State : inactive
Working Devices : 1
Name : disc
UUID : 5914375c:972d27e0:fad157e6:e2b33ef4
Events : 125751
Number Major Minor RaidDevice
- 253 1 - /dev/dm-1
/dev/md127:
Version : 1.1
Raid Level : raid0
Total Devices : 1
Persistence : Superblock is persistent
State : inactive
Working Devices : 1
Number Major Minor RaidDevice
- 253 0 - /dev/dm-0
I'm closing this based on feedback received, and the details I outline in comment #7. Fixing the problem in older OSP releases (10 and 13) will be difficult, and later releases based on rhel-8 (e.g. OSP-16) are not vulnerable. Let's use a KCS to cover this one. |