Bug 1679174
| Summary: | baremetal node cleaning may render disk drives unusable | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Vagner Farias <vfarias> |
| Component: | openstack-ironic-python-agent | Assignee: | RHOS Maint <rhos-maint> |
| Status: | CLOSED ERRATA | QA Contact: | mlammon |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 13.0 (Queens) | CC: | akaris, alcides_coelho, alsilva, bfournie, dtantsur, jkreger, mburns, pkundal, pmannidi, pveiga, racedoro, rhos-maint, slinaber, vfarias |
| Target Milestone: | z6 | Keywords: | Triaged, ZStream |
| Target Release: | 13.0 (Queens) | Flags: | alsilva:
automate_bug+
alsilva: needinfo+ alsilva: needinfo+ |
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-ironic-python-agent-3.2.2-2.el7ost rhosp-director-images-ipa-13.0-20190409.1.el7ost.noarch | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-04-30 17:24:46 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1679786 | ||
Hello folks, I found following document during my researches regarding this issue: https://docs.openstack.org/releasenotes/ironic-python-agent/rocky.html#relnotes-3-3-0-stable-rocky Once it mentions lots of errors related to ATA Secure Erase mechanism similar to what customer faced on his environment, I would suggest to consider it and check if Red Hat already implemented those changes to the ironic-python-agent currently running on customer. Thank you, Alcides Coelho Junior Technology Strategist, LATAM Solutions Support Team (SST) Dell EMC | Remote Services and Solutions I'm looking into providing backports. Hi, Is it possible for the customer to try our OSP 14 IPA images and see if the problem is fixed with them? I understand that the risk is to brick the disks again, but that's the only way I can know for sure (unless somebody with similar hardware can reproduce it as well). Meanwhile, the backports have been proposed upstream. (In reply to Dmitry Tantsur from comment #4) > Is it possible for the customer to try our OSP 14 IPA images and see if the > problem is fixed with them? I understand that the risk is to brick the disks > again, but that's the only way I can know for sure (unless somebody with > similar hardware can reproduce it as well). Before doing it, would it make sense to try to "unlock" current failed disks, as in [1]? Or the goal of [2] is to prevent ATA Secure Erase from being issued because in some situations as it can't be reverted? We'd try running hdparm directly after deploying RHEL 7 in one of the servers. If this works, risk of loosing disks is highly mitigated. [1] https://review.openstack.org/#/c/638368/1/ironic_python_agent/hardware.py [2] https://review.openstack.org/#/c/638369/2/ironic_python_agent/hardware.py I think it makes sense to try unlocking the disks first. I'm CC'ing Julia who worked on some of the upstream patches. Allessandro - I was under the impression from Comment 23 that there would be some testing done to verify these patches fix the problem? If not we can assume that this fix resolves the issue. These fixes are scheduled to be imported from upstream by ReleaseDelivery on March 18, we should be able to have a pkg ready by the end of that week and hot_fix available shortly after that. An import for 13z6 is being done on 20Mar19, the puddle with these fixes will be ready early next week at which point we'll move the bug to MODIFIED and can provide the hotfix. Allessandro - that update to the whiteboard is just the ship date for the 13z6 release. The puddle with the ironic-python-agent image built from the openstack-ironic-python-agent package should be be available today at which point we can provide the hotfix. (In reply to Bob Fournier from comment #33) > Allessandro - that update to the whiteboard is just the ship date for the > 13z6 release. The puddle with the ironic-python-agent image built from the > openstack-ironic-python-agent package should be be available today at which > point we can provide the hotfix. OK. We are waiting for the hotfix then. Updated FIV with ironic-python-agent images package. Maybe we need to use rhosp-director-images-13.0-20190409.1.el7ost.noarch instead, which includes rhosp-director-images-ipa? I'm assuming that the IPA image needs to be provided as part of the hotfix? Thank you. We don't have access to this hardware for verification so will just be a code verification. Please re-open if issue not resolved. Environment: yum info openstack-ironic-python-agent Loaded plugins: search-disabled-repos Available Packages Name : openstack-ironic-python-agent Arch : noarch Version : 3.3.1 Release : 0.20181224074744.aab9f53.el7ost Size : 11 k Please ignore, https://bugzilla.redhat.com/show_bug.cgi?id=1679174#c46 Verification: yum info openstack-ironic-python-agent Loaded plugins: search-disabled-repos Available Packages Name : openstack-ironic-python-agent Arch : noarch Version : 3.2.2 Release : 2.el7ost Size : 11 k Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0937 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |
Description of problem: While executing a full node cleaning in Ceph nodes, some of the disks became unusable. These disks are shown as in failed state through server out-of-band management platform. Hardware vendor claims disks are now shown as having 0 KB. Server model: Dell PowerEdge R730xd Version-Release number of selected component (if applicable): openstack-ironic-api-10.1.6-4.el7ost.noarch openstack-ironic-common-10.1.6-4.el7ost.noarch openstack-ironic-conductor-10.1.6-4.el7ost.noarch openstack-ironic-inspector-7.2.1-5.el7ost.noarch openstack-ironic-staging-drivers-0.9.1-1.el7ost.noarch puppet-ironic-12.4.0-4.el7ost.noarch python2-ironicclient-2.2.1-1.el7ost.noarch python2-ironic-neutron-agent-1.0.0-1.el7ost.noarch python-ironic-inspector-client-3.1.1-2.el7ost.noarch python-ironic-lib-2.12.1-2.el7ost.noarch rhosp-director-images-13.0-20180905.1.el7ost.noarch rhosp-director-images-13.0-20190109.1.el7ost.noarch rhosp-director-images-ipa-13.0-20180905.1.el7ost.noarch rhosp-director-images-ipa-13.0-20190109.1.el7ost.noarch rhosp-director-images-ipa-x86_64-13.0-20180905.1.el7ost.noarch rhosp-director-images-ipa-x86_64-13.0-20190109.1.el7ost.noarch rhosp-director-images-x86_64-13.0-20180905.1.el7ost.noarch rhosp-director-images-x86_64-13.0-20190109.1.el7ost.noarch How reproducible: Always in this hardware. Steps to Reproduce: 1. ironic --ironic-api-version 1.15 node-set-provision-state <node_id> clean --clean-steps '[{"interface": "deploy", "step": "erase_devices"}]' Actual results: Node cleaning eventually fails and some disks become unusable. Expected results: Node cleaning should erase disks data and exit with success. Additional info: According to [1], ATA Secure Erase may indeed render disks unusable. [1] https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase