Bug 1903143

Created attachment 1735228 [details]
VGA console screenshot

Created attachment 1735228 [details]
VGA console screenshot

Created attachment 1735228 [details]
VGA console screenshot

Description of problem:
Introspection times out.


Version-Release number of selected component (if applicable):

RHOS-17.0-RHEL-8-20201128.n.1

failed to run hardware-detect utility: Unexpected error while running command.


...
...

File: "/usr/bin/hardware-retect", line 10 in <module>\n
    sys.exit(main())\n
File: "/usr/lib/python3.6/site-packages/hardware/detect.py", line 731, in  main
    if not detect_system(hrdw):
File: "/usr/lib/python3.6/site-packages/hardware/detect.py", line 250 in detect_system
   mobo_id = _get_value(hw_list, 'system', 'motherboard, 'serial')
File: 
File: "/usr/lib/python3.6/site-packages/hardware/detect.py", line 194, in _get_value
    if i[0:3] == vect:
'bool' object is not subscriptable: 
ironic_python_agent.error.IntrospectionError: The following errors were encountered:
INFO  No ipa-api-url  configured. Heartbeat and lookup skipped for introspection


The package versions in the repos, the IPA image supposed to use the same:
python3-hardware-detect                                   noarch                                   0.25.1-0.20201010081933.9bc8b1f.el8ost
openstack-ironic-python-agent                                  noarch                                  6.5.0-0.20201013113532.80b0a9a.el8ost                                    rhelosp-17.0


undercloud-0/var/log/containers/ironic-inspector/ironic-inspector.log.gz:

2020-11-30 15:03:46.823 7 DEBUG ironic_inspector.main [req-c79f1aa1-56cc-4ec9-a0dd-82fa1f3d7244 - - - - -] [node: MAC 52:54:00:8e:07:8f] Received data from the ramdisk: {'inventory': {'interfaces': [{'name': 'ens5', 'mac_address': '52:54:00:e6:97:30', 'ipv4_address': '10.0.0.35', 'ipv6_address': '2620:52:0:13b8::fe:87', 'has_carrier': True, 'lldp': [], 'vendor': '0x1af4', 'product': '0x0001', 'client_id': None, 'biosdevname': None}, {'name': 'ens3', 'mac_address': '52:54:00:8e:07:8f', 'ipv4_address': '192.168.24.100', 'ipv6_address': 'fe80::c0bc:1cf0:ac86:94de%ens3', 'has_carrier': True, 'lldp': [], 'vendor': '0x1af4', 'product': '0x0001', 'client_id': None, 'biosdevname': None}, {'name': 'ens4', 'mac_address': '52:54:00:8b:34:d6', 'ipv4_address': '172.16.0.39', 'ipv6_address': 'fe80::31d7:5d54:7f1b:c4c2%ens4', 'has_carrier': True, 'lldp': [], 'vendor': '0x1af4', 'product': '0x0001', 'client_id': None, 'biosdevname': None}], 'cpu': {'model_name': 'Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz', 'frequency': '2199.996', 'count': 2, 'architecture': 'x86_64', 'flags': ['fpu', 'vme', 'de', 'pse', 'tsc', 'msr', 'pae', 'mce', 'cx8', 'apic', 'sep', 'mtrr', 'pge', 'mca', 'cmov', 'pat', 'pse36', 'clflush', 'mmx', 'fxsr', 'sse', 'sse2', 'ss', 'syscall', 'nx', 'pdpe1gb', 'rdtscp', 'lm', 'constant_tsc', 'arch_perfmon', 'rep_good', 'nopl', 'xtopology', 'cpuid', 'tsc_known_freq', 'pni', 'pclmulqdq', 'vmx', 'ssse3', 'fma', 'cx16', 'pcid', 'sse4_1', 'sse4_2', 'x2apic', 'movbe', 'popcnt', 'tsc_deadline_timer', 'aes', 'xsave', 'avx', 'f16c', 'rdrand', 'hypervisor', 'lahf_lm', 'abm', '3dnowprefetch', 'cpuid_fault', 'invpcid_single', 'pti', 'tpr_shadow', 'vnmi', 'flexpriority', 'ept', 'vpid', 'ept_ad', 'fsgsbase', 'tsc_adjust', 'bmi1', 'hle', 'avx2', 'smep', 'bmi2', 'erms', 'invpcid', 'rtm', 'rdseed', 'adx', 'smap', 'xsaveopt', 'arat', 'umip', 'arch_capabilities']}, 'disks': [{'name': '/dev/vda', 'model': '', 'size': 11811160064, 'rotational': True, 'wwn': None, 'serial': None, 'vendor': '0x1af4', 'wwn_with_extension': None, 'wwn_vendor_extension': None, 'hctl': None, 'by_path': '/dev/disk/by-path/pci-0000:00:08.0'}, {'name': '/dev/vdb', 'model': '', 'size': 12884901888, 'rotational': True, 'wwn': None, 'serial': None, 'vendor': '0x1af4', 'wwn_with_extension': None, 'wwn_vendor_extension': None, 'hctl': None, 'by_path': '/dev/disk/by-path/pci-0000:00:09.0'}, {'name': '/dev/vdc', 'model': '', 'size': 12884901888, 'rotational': True, 'wwn': None, 'serial': None, 'vendor': '0x1af4', 'wwn_with_extension': None, 'wwn_vendor_extension': None, 'hctl': None, 'by_path': '/dev/disk/by-path/pci-0000:00:0a.0'}, {'name': '/dev/vdd', 'model': '', 'size': 12884901888, 'rotational': True, 'wwn': None, 'serial': None, 'vendor': '0x1af4', 'wwn_with_extension': None, 'wwn_vendor_extension': None, 'hctl': None, 'by_path': '/dev/disk/by-path/pci-0000:00:0b.0'}, {'name': '/dev/vde', 'model': '', 'size': 12884901888, 'rotational': True, 'wwn': None, 'serial': None, 'vendor': '0x1af4', 'wwn_with_extension': None, 'wwn_vendor_extension': None, 'hctl': None, 'by_path': '/dev/disk/by-path/pci-0000:00:0c.0'}, {'name': '/dev/vdf', 'model': '', 'size': 12884901888, 'rotational': True, 'wwn': None, 'serial': None, 'vendor': '0x1af4', 'wwn_with_extension': None, 'wwn_vendor_extension': None, 'hctl': None, 'by_path': '/dev/disk/by-path/pci-0000:00:0d.0'}], 'memory': {'total': 4119384064, 'physical_mb': 4096}, 'bmc_address': '0.0.0.0', 'bmc_v6address': '::/0', 'system_vendor': {'product_name': 'KVM', 'serial_number': '', 'manufacturer': 'Red Hat'}, 'boot': {'current_boot_mode': 'bios', 'pxe_interface': '52:54:00:8e:07:8f'}, 'hostname': 'localhost.localdomain'}, 'root_disk': {'name': '/dev/vda', 'model': '', 'size': 11811160064, 'rotational': True, 'wwn': None, 'serial': None, 'vendor': '0x1af4', 'wwn_with_extension': None, 'wwn_vendor_extension': None, 'hctl': None, 'by_path': '/dev/disk/by-path/pci-0000:00:08.0'}, 'boot_interface': '52:54:00:8e:07:8f', 'configuration': {'collectors': ['default', 'extra-hardware', 'numa-topology', 'logs'], 'managers': [{'name': 'generic_hardware_manager', 'version': '1.1'}]}, 'numa_topology': {'ram': [{'numa_node': 0, 'size_kb': 4018736}], 'cpus': [{'thread_siblings': [1, 0], 'cpu': 0, 'numa_node': 0}], 'nics': []}, 'logs': '<hidden>', 'error': 'The following errors were encountered:\n* failed to run hardware-detect utility: Unexpected error while running command.\nCommand: hardware-detect\nExit code: 1\nStdout: \'\'\nStderr: \'/bin/sh: cli64: command not found\\nInfo: No Areca controller found\\nCannot find megacli on the system\\nFailed retrieving disk information device/model for vdf: [Errno 20] Not a directory: \\\'/sys/block/vdf/device/device/model\\\'\\nFailed retrieving disk information device/revision for vdf: [Errno 20] Not a directory: \\\'/sys/block/vdf/device/device/revision\\\'\\nFailed at getting disk information at /sys/block/vdf/device/scsi_disk/virtio9/cache_type: [Errno 2] No such file or directory: \\\'/sys/block/vdf/device/scsi_disk/virtio9/cache_type\\\'\\nread_smart: Reading S.M.A.R.T information on /dev/vdf\\nread_smart: Reading S.M.A.R.T information on /dev/vdf with -d ata\\nread_smart: no device /dev/vdf\\nFailed retrieving disk information device/model for vdd: [Errno 20] Not a directory: \\\'/sys/block/vdd/device/device/model\\\'\\nFailed retrieving disk information device/revision for vdd: [Errno 20] Not a directory: \\\'/sys/block/vdd/device/device/revision\\\'\\nFailed at getting disk information at /sys/block/vdd/device/scsi_disk/virtio7/cache_type: [Errno 2] No such file or directory: \\\'/sys/block/vdd/device/scsi_disk/virtio7/cache_type\\\'\\nread_smart: Reading S.M.A.R.T information on /dev/vdd\\nread_smart: Reading S.M.A.R.T information on /dev/vdd with -d ata\\nread_smart: no device /dev/vdd\\nFailed retrieving disk information device/model for vdb: [Errno 20] Not a directory: \\\'/sys/block/vdb/device/device/model\\\'\\nFailed retrieving disk information device/revision for vdb: [Errno 20] Not a directory: \\\'/sys/block/vdb/device/device/revision\\\'\\nFailed at getting disk information at /sys/block/vdb/device/scsi_disk/virtio5/cache_type: [Errno 2] No such file or directory: \\\'/sys/block/vdb/device/scsi_disk/virtio5/cache_type\\\'\\nread_smart: Reading S.M.A.R.T information on /dev/vdb\\nread_smart: Reading S.M.A.R.T information on /dev/vdb with -d ata\\nread_smart: no device /dev/vdb\\nFailed retrieving disk information device/model for vde: [Errno 20] Not a directory: \\\'/sys/block/vde/device/device/model\\\'\\nFailed retrieving disk information device/revision for vde: [Errno 20] Not a directory: \\\'/sys/block/vde/device/device/revision\\\'\\nFailed at getting disk information at /sys/block/vde/device/scsi_disk/virtio8/cache_type: [Errno 2] No such file or directory: \\\'/sys/block/vde/device/scsi_disk/virtio8/cache_type\\\'\\nread_smart: Reading S.M.A.R.T information on /dev/vde\\nread_smart: Reading S.M.A.R.T information on /dev/vde with -d ata\\nread_smart: no device /dev/vde\\nFailed retrieving disk information device/model for vdc: [Errno 20] Not a directory: \\\'/sys/block/vdc/device/device/model\\\'\\nFailed retrieving disk information device/revision for vdc: [Errno 20] Not a directory: \\\'/sys/block/vdc/device/device/revision\\\'\\nFailed at getting disk information at /sys/block/vdc/device/scsi_disk/virtio6/cache_type: [Errno 2] No such file or directory: \\\'/sys/block/vdc/device/scsi_disk/virtio6/cache_type\\\'\\nread_smart: Reading S.M.A.R.T information on /dev/vdc\\nread_smart: Reading S.M.A.R.T information on /dev/vdc with -d ata\\nread_smart: no device /dev/vdc\\nFailed retrieving disk information device/model for vda: [Errno 20] Not a directory: \\\'/sys/block/vda/device/device/model\\\'\\nFailed retrieving disk information device/revision for vda: [Errno 20] Not a directory: \\\'/sys/block/vda/device/device/revision\\\'\\nFailed at getting disk information at /sys/block/vda/device/scsi_disk/virtio4/cache_type: [Errno 2] No such file or directory: \\\'/sys/block/vda/device/scsi_disk/virtio4/cache_type\\\'\\nread_smart: Reading S.M.A.R.T information on /dev/vda\\nread_smart: Reading S.M.A.R.T information on /dev/vda with -d ata\\nread_smart: no device /dev/vda\\nTraceback (most recent call last):\\n  File "/usr/bin/hardware-detect", line 10, in <module>\\n    sys.exit(main())\\n  File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 731, in main\\n    if not detect_system(hrdw):\\n  File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 250, in detect_system\\n    mobo_id = _get_value(hw_lst, \\\'system\\\', \\\'motherboard\\\', \\\'serial\\\')\\n  File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 194, in _get_value\\n    if i[0:3] == vect:\\nTypeError: \\\'bool\\\' object is not subscriptable\\n\''} api_continue /usr/lib/python3.6/site-packages/ironic_inspector/main.py:329
2020-11-30 15:03:46.829 7 DEBUG ironic_inspector.process [-] [node: MAC 52:54:00:8e:07:8f] Running pre-processing hook ramdisk_error _run_pre_hooks /usr/lib/python3.6/site-packages/ironic_inspector/process.py:129
2020-11-30 15:03:46.829 7 ERROR ironic_inspector.utils [-] [node: MAC 52:54:00:8e:07:8f] Ramdisk reported error: The following errors were encountered:
* failed to run hardware-detect utility: Unexpected error while running command.
Command: hardware-detect
Exit code: 1
Stdout: ''
Stderr: '/bin/sh: cli64: command not found\nInfo: No Areca controller found\nCannot find megacli on the system\nFailed retrieving disk information device/model for vdf: [Errno 20] Not a directory: \'/sys/block/vdf/device/device/model\'\nFailed retrieving disk information device/revision for vdf: [Errno 20] Not a directory: \'/sys/block/vdf/device/device/revision\'\nFailed at getting disk information at /sys/block/vdf/device/scsi_disk/virtio9/cache_type: [Errno 2] No such file or directory: \'/sys/block/vdf/device/scsi_disk/virtio9/cache_type\'\nread_smart: Reading S.M.A.R.T information on /dev/vdf\nread_smart: Reading S.M.A.R.T information on /dev/vdf with -d ata\nread_smart: no device /dev/vdf\nFailed retrieving disk information device/model for vdd: [Errno 20] Not a directory: \'/sys/block/vdd/device/device/model\'\nFailed retrieving disk information device/revision for vdd: [Errno 20] Not a directory: \'/sys/block/vdd/device/device/revision\'\nFailed at getting disk information at /sys/block/vdd/device/scsi_disk/virtio7/cache_type: [Errno 2] No such file or directory: \'/sys/block/vdd/device/scsi_disk/virtio7/cache_type\'\nread_smart: Reading S.M.A.R.T information on /dev/vdd\nread_smart: Reading S.M.A.R.T information on /dev/vdd with -d ata\nread_smart: no device /dev/vdd\nFailed retrieving disk information device/model for vdb: [Errno 20] Not a directory: \'/sys/block/vdb/device/device/model\'\nFailed retrieving disk information device/revision for vdb: [Errno 20] Not a directory: \'/sys/block/vdb/device/device/revision\'\nFailed at getting disk information at /sys/block/vdb/device/scsi_disk/virtio5/cache_type: [Errno 2] No such file or directory: \'/sys/block/vdb/device/scsi_disk/virtio5/cache_type\'\nread_smart: Reading S.M.A.R.T information on /dev/vdb\nread_smart: Reading S.M.A.R.T information on /dev/vdb with -d ata\nread_smart: no device /dev/vdb\nFailed retrieving disk information device/model for vde: [Errno 20] Not a directory: \'/sys/block/vde/device/device/model\'\nFailed retrieving disk information device/revision for vde: [Errno 20] Not a directory: \'/sys/block/vde/device/device/revision\'\nFailed at getting disk information at /sys/block/vde/device/scsi_disk/virtio8/cache_type: [Errno 2] No such file or directory: \'/sys/block/vde/device/scsi_disk/virtio8/cache_type\'\nread_smart: Reading S.M.A.R.T information on /dev/vde\nread_smart: Reading S.M.A.R.T information on /dev/vde with -d ata\nread_smart: no device /dev/vde\nFailed retrieving disk information device/model for vdc: [Errno 20] Not a directory: \'/sys/block/vdc/device/device/model\'\nFailed retrieving disk information device/revision for vdc: [Errno 20] Not a directory: \'/sys/block/vdc/device/device/revision\'\nFailed at getting disk information at /sys/block/vdc/device/scsi_disk/virtio6/cache_type: [Errno 2] No such file or directory: \'/sys/block/vdc/device/scsi_disk/virtio6/cache_type\'\nread_smart: Reading S.M.A.R.T information on /dev/vdc\nread_smart: Reading S.M.A.R.T information on /dev/vdc with -d ata\nread_smart: no device /dev/vdc\nFailed retrieving disk information device/model for vda: [Errno 20] Not a directory: \'/sys/block/vda/device/device/model\'\nFailed retrieving disk information device/revision for vda: [Errno 20] Not a directory: \'/sys/block/vda/device/device/revision\'\nFailed at getting disk information at /sys/block/vda/device/scsi_disk/virtio4/cache_type: [Errno 2] No such file or directory: \'/sys/block/vda/device/scsi_disk/virtio4/cache_type\'\nread_smart: Reading S.M.A.R.T information on /dev/vda\nread_smart: Reading S.M.A.R.T information on /dev/vda with -d ata\nread_smart: no device /dev/vda\nTraceback (most recent call last):\n  File "/usr/bin/hardware-detect", line 10, in <module>\n    sys.exit(main())\n  File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 731, in main\n    if not detect_system(hrdw):\n  File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 250, in detect_system\n    mobo_id = _get_value(hw_lst, \'system\', \'motherboard\', \'serial\')\n  File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 194, in _get_value\n    if i[0:3] == vect:\nTypeError: \'bool\' object is not subscriptable\n'
2020-11-30 15:03:46.829 7 ERROR ironic_inspector.process [-] [node: MAC 52:54:00:8e:07:8f] Hook ramdisk_error failed, delaying error report until node look up: Ramdisk reported error: The following errors were encountered:
* failed to run hardware-detect utility: Unexpected error while running command.