Bug 1494361

Summary: Introspection is unable to detect hardware raid on Wiwynn SV7221G2-V
Product: Red Hat OpenStack Reporter: Chen <cchen>
Component: rhosp-directorAssignee: Angus Thomas <athomas>
Status: CLOSED NOTABUG QA Contact: Amit Ugol <augol>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 10.0 (Newton)CC: aschultz, bfournie, cchen, dbecker, dtantsur, mburns, mkovacik, morazi, rhel-osp-director-maint
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-10-03 20:13:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
The introspection data
none
Hardware raid disk could be detected by RHEL7 installation program
none
ramdisk log with raid and without raid to compare none

Description Chen 2017-09-22 06:34:20 UTC
Description of problem:

Introspection is unable to detect hardware raid on Wiwynn SV7221G2-V 

Version-Release number of selected component (if applicable):

OSP10
rhosp-director-images-ipa-10.0-20170727.1.el7ost.noarch

How reproducible:

100%

Steps to Reproduce:
1. Start introspection
2. Examine the introspection data
3.

Actual results:

No hardware raid disk is detected

Expected results:

Hardware raid disks should be able to be detected

Additional info:

Comment 1 Chen 2017-09-22 06:35:31 UTC
Created attachment 1329387 [details]
The introspection data

Comment 2 Chen 2017-09-22 06:36:33 UTC
Created attachment 1329388 [details]
Hardware raid disk could be detected by RHEL7 installation program

Comment 4 Bob Fournier 2017-09-25 13:48:14 UTC
This looks like a missing driver in IPA.  Would it be possible to get the introspection ramdisk logs so we can narrow this down? 

To do this...
1. Modify /etc/ironic-inspector/inspector.conf to set:
# Whether to store ramdisk logs even if it did not return an error
# message (dependent upon "ramdisk_logs_dir" option being set).
# (boolean value)
always_store_ramdisk_logs = true

2. restart ironic-inspector -
sudo systemctl restart openstack-ironic-inspector.service

3. rerun introspection

4. Provide logs for node in /var/log/ironic-inspector/ramdisk/
(or whatever ramdisk_logs_dir is set to in /etc/ironic-inspector/inspector.conf 

Thank you.

Comment 5 Chen 2017-09-27 09:33:08 UTC
Created attachment 1331350 [details]
ramdisk log with raid and without raid to compare

Hi Bob,

Thank you for your reply. The attachment is the ramdisk log and please let me know if anything else if needed.

Best Regards,
Chen

Comment 6 Bob Fournier 2017-09-27 19:30:05 UTC
Thanks, I do the Wiwynn device recognized in the RAID logs:
Sep 26 08:11:22 localhost.localdomain kernel: scsi host0: Avago SAS based MegaRAID driver
Sep 26 08:11:22 localhost.localdomain kernel: scsi 0:0:8:0: Enclosure         Wiwynn   Knox12G          0808 PQ: 0 ANSI: 5

Sep 26 08:11:22 localhost.localdomain kernel: scsi 0:0:9:0: Enclosure         Wiwynn   Knox12G          0808 PQ: 0 ANSI: 5

Still trying to determine why disks not seen.

Comment 7 Bob Fournier 2017-09-27 19:58:21 UTC

I do see these errors in the raid log case which is probably the failure trying to detect the RAID:
Sep 26 08:13:02 localhost.localdomain kernel: sd 0:0:13:0: [sdd] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Sep 26 08:13:02 localhost.localdomain kernel: sd 0:0:13:0: [sdd] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
Sep 26 08:13:02 localhost.localdomain kernel: sd 0:0:13:0: [sdd] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Sep 26 08:13:02 localhost.localdomain kernel: sd 0:0:13:0: [sdd] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
Sep 26 08:13:02 localhost.localdomain ironic-python-agent[1001]: 2017-09-26 08:13:01.419 1001 DEBUG root [-] TYPE did not match. Wanted: 'disk' but found: '' list_all_block_devices /usr/lib/python2.7/site-packages/ironic_python_agent/hardware.py:132
Sep 26 08:13:02 localhost.localdomain ironic-python-agent[1001]: 2017-09-26 08:13:01.420 1001 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): dmidecode --type 17 | grep Size execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:349
Sep 26 08:13:02 localhost.localdomain ironic-python-agent[1001]: 2017-09-26 08:13:01.424 1001 DEBUG oslo_concurrency.processutils [-] CMD "dmidecode --type 17 | grep Size" returned: 0 in 0.004s execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:379
Sep 26 08:13:02 localhost.localdomain ironic-python-agent[1001]: 2017-09-26 08:13:01.424 1001 DEBUG ironic_lib.utils [-] Execution completed, command line is "dmidecode --type 17 | grep Size" execute /usr/lib/python2.7/site-packages/ironic_lib/utils.py:92
Sep 26 08:13:02 localhost.localdomain ironic-python-agent[1001]: 2017-09-26 08:13:01.424 1001 DEBUG ironic_lib.utils [-] Command stdout is: "        Size: 32 GB
Sep 26 08:13:02 localhost.localdomain ironic-python-agent[1001]: Size: No Module Installed
Sep 26 08:13:02 localhost.localdomain ironic-python-agent[1001]: Size: No Module Installed
Sep 26 08:13:02 localhost.localdomain ironic-python-agent[1001]: Size: No Module Installed
Sep 26 08:13:02 localhost.localdomain ironic-python-agent[1001]: Size: No Module Installed
Sep 26 08:13:02 localhost.localdomain ironic-python-agent[1001]: Size: No Module Installed
Sep 26 08:13:02 localhost.localdomain ironic-python-agent[1001]: Size: No Module Installed
Sep 26 08:13:02 localhost.localdomain ironic-python-agent[1001]: Size: No Module Installed
Sep 26 08:13:02 localhost.localdomain ironic-python-agent[1001]: Size: No Module Installed
Sep 26 08:13:02 localhost.localdomain ironic-python-agent[1001]: Size: No Module Installed
Sep 26 08:13:02 localhost.localdomain ironic-python-agent[1001]: Size: No Module Installed
Sep 26 08:13:02 localhost.localdomain ironic-python-agent[1001]: Size: No Module Installed
Sep 26 08:13:02 localhost.localdomain ironic-python-agent[1001]: Size: No Module Installed
Sep 26 08:13:02 localhost.localdomain ironic-python-agent[1001]: Size: No Module Installed
Sep 26 08:13:02 localhost.localdomain ironic-python-agent[1001]: Size: No Module Installed
Sep 26 08:13:02 localhost.localdomain ironic-python-agent[1001]: Size: No Module Installed

Comment 8 Chen 2017-09-30 05:36:15 UTC
Hi Bob,

Thank you for your help and confirm.

Please let me know if there could be progress that can be shared with the customer.

Best Regards,
Chen

Comment 9 Bob Fournier 2017-10-02 20:40:03 UTC
I'm looking at logs for Ceph2 (node e686cee2-75ef-4475-9d33-014d46249a42) along with the png file for this node (raid_e686cee2-75ef-4475-9d33-014d46249a42.png).  It looks like the RAID is being configured at the System BIOS at the BMC address 192.168.128.207, not in a hardware RAID controller, is that correct?  This is considered software RAID, not hardware RAID and as such introspection will return the individual disks not the entire RAIDed disk, which is what is happening.

As can be seen in the ramdisk logs for e686cee2-75ef-4475-9d33-014d46249a42 individual disks are detected for sde and sdf:

journal:Sep 26 08:11:18 localhost.localdomain kernel: ata5.00: ATA-10: ST1000NX0313, SN02, max UDMA/133
journal:Sep 26 08:11:18 localhost.localdomain kernel: ata6.00: ATA-10: ST1000NX0313, SN02, max UDMA/133
journal:Sep 26 08:11:19 localhost.localdomain kernel: scsi 5:0:0:0: Direct-Access     ATA      ST1000NX0313     SN02 PQ: 0 ANSI: 5
journal:Sep 26 08:11:19 localhost.localdomain kernel: scsi 6:0:0:0: Direct-Access     ATA      ST1000NX0313     SN02 PQ: 0 ANSI: 5
journal:Sep 26 08:11:24 localhost.localdomain ironic-python-agent[1008]: KNAME="sde" MODEL="ST1000NX0313    " SIZE="1000204886016" ROTA="1" TYPE="disk"
journal:Sep 26 08:11:24 localhost.localdomain ironic-python-agent[1008]: KNAME="sdf" MODEL="ST1000NX0313    " SIZE="1000204886016" ROTA="1" TYPE="disk"

This was a recent discussion on rhos-dev mailing list, see the archives here (if you can't access the archives I will post the conversation) - http://post-office.corp.redhat.com/archives/rh-openstack-dev/2017-September/msg00183.html

Comment 10 Bob Fournier 2017-10-03 20:13:16 UTC
Per the previous comment, introspection will not be able to detect the RAID, only the individual disks because the RAID configuration was done in BIOS (software RAID) and not by using a dedicated hardware RAID device.  IPA, and therefore introspection, is reporting what the kernel reports, which is separate disks, as shown in the logs in Comment 9.