Bug 671059
| Summary: | bad performance of ipmitool sdr by update to 2.0.16 | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 5 | Reporter: | Masahiro Matsuya <mmatsuya> |
| Component: | OpenIPMI | Assignee: | Jan Safranek <jsafrane> |
| Status: | CLOSED ERRATA | QA Contact: | qe-baseos-daemons |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 5.5 | CC: | azelinka, cww, jwest, kvolny, msvoboda, ovasik, rick.beldin, rvokal |
| Target Milestone: | rc | Keywords: | Regression, ZStream |
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
When IPMI-enabled devices reported SDR (sensor data record) records under a different owner than the BMC (Baseboard Management Controller), the IPMItool utility tried to retrieve these SDR records from the IPMI bus instead of the BMC bus. Due to timeout setting for the SDR read attempt, serious performance issues occurred and no sensor data was shown. This issue has been fixed and IPMItool now correctly reads these SDR records from the BMC and shows the correct sensor data on these platforms.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2013-01-08 04:52:14 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 722462 | ||
RH needs someone at HP to respond: "Reading ipmitool code and IPMI 2.0 specs, there is something I don't understand. The HP hardware reports owner of some sensors as 'system software' with ID=0 (=BIOS). See this Sensor Record (generated by freeipmi, it has better output of sent/received messages): SDR Compact Sensor Record ===================================================== [ 40h] = record_id[16b] [ 1h] = sdr_version_major[ 4b] [ 5h] = sdr_version_minor[ 4b] [ 2h] = record_type[ 8b] [ 2Bh] = record_length[ 8b] [ 1h] = sensor_owner_id.type[ 1b] <---!!! [ 0h] = sensor_owner_id[ 7b] <---!!! [ 0h] = sensor_owner_lun[ 2b] [ 0h] = sensor_owner_lun.reserved[ 2b] [ 0h] = channel_number[ 4b] [ 0h] = sensor_number[ 8b] [ 22h] = entity_id[ 8b] . . . So, what ipmitool tries to do is to bridge the sensor reading from address 0x20 (=BMC) to the 0x01 (=BIOS), which fails as you describe. Question is, what's correct? Is the ipmitool wrong when it tries to bridge the sensor reading or the HP hardware, which reports the sensor is owned by someone-else-not-BMC? The IPMI spec does not give me a clue if the messages should be bridged or not. The ipmitool code is pretty self-explaining and suggests, that bridging is the only thing that can be done with these sensors and the guys who wrote it put it there for some reason. On the other hand, freeipmi (=another IPMI implementation in RHEL 5) does not bridge it. I'd like to hear HP statement on this. I think I could add new command line option to turn off bridging, but ipmitool upstream is not responding much, so it would be probably Red Hat specific extension." The poor performance comes from IPMITool trying to bridge the GET_SENSOR_READING command over IPMB for SDR entries that have “System Software” (instead of “IPMB Address”) defined as their “Sensor Owner ID”. This is documented under “5.4 Sensor Owner ID” in IPMI 2.0 SPEC. Because IPMITool does not distinguish sensor owner with system software ID from sensor owner with IPMB address and always bridges the command over IPMB, the target with system software ID defined does not (and should not) return a sensor reading and therefore, based on OpenIPMI driver’s default behavior, there will be a 5 seconds delay before a response is sent back for such sensor. So what it means by defining an SDR entry having a “System Software ID” is unclear based on my understanding of IPMI; normally the sensor should be owned by BMC (ID=0x20). However, I think IPMITool should check that sensor owner ID field first to see if it should use that IPMB slave address before bridging the command over IPMB. If the sensor is owned by system software, then just don’t bridge the command from BMC (ID=0x20).
Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.
New Contents:
When IPMI-enabled devices reported SDR (sensor data record) records under a different owner than the BMC (Baseboard Management Controller), the IPMItool utility tried to retrieve these SDR records from the IPMI bus instead of the BMC bus. Due to timeout setting for the SDR read attempt, serious performance issues occurred and no sensor data was shown. This issue has been fixed and IPMItool now correctly reads these SDR records from the BMC and shows the correct sensor data on these platforms.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2013-0123.html |
Description of problem: OpenIPMI-tools-2.0.16-* causes big performance problems with "ipmitool -c sdr". The command takes 60+ seconds. OpenIPMI-tools-2.0.6-* takes less than a second. The slowdown occurs just prior to each "No Reading" sensor, so it appears the new code has a ~5 second timeout it never had before. The delay happens when IPMB (not System Interface) is used to probe the sensor information. Kernel (driver handling ipmi request) tries to query the information 5 times (first try and 4 reties). The timeout of each query is 1 second. So, it takes 5 seconds (1 second x 5 times) until replying timeout from kernel. To decide if use IPMB in 2.0.16, local IPMB addr and target IPMB addr are checked. If those are not same (meaning that the IPMB target is on remote), IPMB will be used and this delay occur. The target IPMB address looks decided upon information provided by kernel (or firmware). Can you add a method to force to use System Interface? Or, can you add a method to configure the number of retries and the time to timeout in kernel. Version-Release number of selected component (if applicable): OpenIPMI-tools-2.0.16-5 OpenIPMI-tools-2.0.16-7 OpenIPMI-tools-2.0.16-11 How reproducible: Always on hardware enabled IPMI Steps to Reproduce: 1. service ipmi start 2. time ipmitool -c sdr Actual results: Bad performance on "ipmitool -c sdr" Expected results: "ipmitool -c sdr" outputs the results without such a long delay. Additional info: The following 2 lines commented out created forcing to use IPMB in 2.0.16. struct ipmi_rs * ipmi_sdr_get_sensor_reading_ipmb(struct ipmi_intf *intf, uint8_t sensor, uint8_t target, uint8_t lun) { struct ipmi_rq req; struct ipmi_rs *rsp; uint8_t save_addr; // if ((strncmp(intf->name, "ipmb", 4)) != 0) // return ipmi_sdr_get_sensor_reading(intf, sensor);