Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1679174

Summary: baremetal node cleaning may render disk drives unusable
Product: Red Hat OpenStack Reporter: Vagner Farias <vfarias>
Component: openstack-ironic-python-agentAssignee: RHOS Maint <rhos-maint>
Status: CLOSED ERRATA QA Contact: mlammon
Severity: high Docs Contact:
Priority: high    
Version: 13.0 (Queens)CC: akaris, alcides_coelho, alsilva, bfournie, dtantsur, jkreger, mburns, pkundal, pmannidi, pveiga, racedoro, rhos-maint, slinaber, vfarias
Target Milestone: z6Keywords: Triaged, ZStream
Target Release: 13.0 (Queens)Flags: alsilva: automate_bug+
alsilva: needinfo+
alsilva: needinfo+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-ironic-python-agent-3.2.2-2.el7ost rhosp-director-images-ipa-13.0-20190409.1.el7ost.noarch Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-04-30 17:24:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1679786    

Description Vagner Farias 2019-02-20 14:21:38 UTC
Description of problem:
While executing a full node cleaning in Ceph nodes, some of the disks became unusable. These disks are shown as in failed state through server out-of-band management platform. Hardware vendor claims disks are now shown as having 0 KB.

Server model: Dell PowerEdge R730xd

Version-Release number of selected component (if applicable):
openstack-ironic-api-10.1.6-4.el7ost.noarch
openstack-ironic-common-10.1.6-4.el7ost.noarch
openstack-ironic-conductor-10.1.6-4.el7ost.noarch
openstack-ironic-inspector-7.2.1-5.el7ost.noarch
openstack-ironic-staging-drivers-0.9.1-1.el7ost.noarch
puppet-ironic-12.4.0-4.el7ost.noarch
python2-ironicclient-2.2.1-1.el7ost.noarch
python2-ironic-neutron-agent-1.0.0-1.el7ost.noarch
python-ironic-inspector-client-3.1.1-2.el7ost.noarch
python-ironic-lib-2.12.1-2.el7ost.noarch
rhosp-director-images-13.0-20180905.1.el7ost.noarch
rhosp-director-images-13.0-20190109.1.el7ost.noarch
rhosp-director-images-ipa-13.0-20180905.1.el7ost.noarch
rhosp-director-images-ipa-13.0-20190109.1.el7ost.noarch
rhosp-director-images-ipa-x86_64-13.0-20180905.1.el7ost.noarch
rhosp-director-images-ipa-x86_64-13.0-20190109.1.el7ost.noarch
rhosp-director-images-x86_64-13.0-20180905.1.el7ost.noarch
rhosp-director-images-x86_64-13.0-20190109.1.el7ost.noarch

How reproducible:
Always in this hardware.

Steps to Reproduce:
1. ironic --ironic-api-version 1.15 node-set-provision-state <node_id> clean --clean-steps '[{"interface": "deploy", "step": "erase_devices"}]'


Actual results:
Node cleaning eventually fails and some disks become unusable.

Expected results:
Node cleaning should erase disks data and exit with success.

Additional info:
According to [1], ATA Secure Erase may indeed render disks unusable.

[1] https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase

Comment 1 Alcides Coelho 2019-02-20 17:46:18 UTC
Hello folks,

I found following document during my researches regarding this issue: 
https://docs.openstack.org/releasenotes/ironic-python-agent/rocky.html#relnotes-3-3-0-stable-rocky

Once it mentions lots of errors related to ATA Secure Erase mechanism similar to what customer faced on his environment, I would suggest to consider it and check if Red Hat already implemented those changes to the ironic-python-agent currently running on customer.

Thank you,
Alcides Coelho Junior
Technology Strategist, LATAM Solutions Support Team (SST)
Dell EMC | Remote Services and Solutions

Comment 3 Dmitry Tantsur 2019-02-21 09:03:04 UTC
I'm looking into providing backports.

Comment 4 Dmitry Tantsur 2019-02-21 09:19:07 UTC
Hi,

Is it possible for the customer to try our OSP 14 IPA images and see if the problem is fixed with them? I understand that the risk is to brick the disks again, but that's the only way I can know for sure (unless somebody with similar hardware can reproduce it as well).

Meanwhile, the backports have been proposed upstream.

Comment 12 Vagner Farias 2019-02-21 13:20:16 UTC
(In reply to Dmitry Tantsur from comment #4)
> Is it possible for the customer to try our OSP 14 IPA images and see if the
> problem is fixed with them? I understand that the risk is to brick the disks
> again, but that's the only way I can know for sure (unless somebody with
> similar hardware can reproduce it as well).

Before doing it, would it make sense to try to "unlock" current failed disks, as in [1]? Or the goal of [2] is to prevent ATA Secure Erase from being issued because in some situations as it can't be reverted?

We'd try running hdparm directly after deploying RHEL 7 in one of the servers.

If this works, risk of loosing disks is highly mitigated.

[1] https://review.openstack.org/#/c/638368/1/ironic_python_agent/hardware.py
[2] https://review.openstack.org/#/c/638369/2/ironic_python_agent/hardware.py

Comment 13 Dmitry Tantsur 2019-02-21 13:59:12 UTC
I think it makes sense to try unlocking the disks first. I'm CC'ing Julia who worked on some of the upstream patches.

Comment 25 Bob Fournier 2019-03-12 20:31:05 UTC
Allessandro - I was under the impression from Comment 23 that there would be some testing done to verify these patches fix the problem?  If not we can assume that this fix resolves the issue.  These fixes are scheduled to be imported from upstream by ReleaseDelivery on March 18, we should be able to have a pkg ready by the end of that week and hot_fix available shortly after that.

Comment 28 Bob Fournier 2019-03-20 13:15:27 UTC
An import for 13z6 is being done on 20Mar19, the puddle with these fixes will be ready early next week at which point we'll move the bug to MODIFIED and can provide the hotfix.

Comment 33 Bob Fournier 2019-04-08 14:59:40 UTC
Allessandro - that update to the whiteboard is just the ship date for the 13z6 release. The puddle with the ironic-python-agent image built from the openstack-ironic-python-agent package should be be available today at which point we can provide the hotfix.

Comment 34 Alessandro Silva 2019-04-09 13:49:54 UTC
(In reply to Bob Fournier from comment #33)
> Allessandro - that update to the whiteboard is just the ship date for the
> 13z6 release. The puddle with the ironic-python-agent image built from the
> openstack-ironic-python-agent package should be be available today at which
> point we can provide the hotfix.

OK. We are waiting for the hotfix then.

Comment 39 Bob Fournier 2019-04-11 13:14:47 UTC
Updated FIV with ironic-python-agent images package.

Comment 42 Bob Fournier 2019-04-11 14:41:04 UTC
Maybe we need to use rhosp-director-images-13.0-20190409.1.el7ost.noarch instead, which includes rhosp-director-images-ipa?  I'm assuming that the IPA image needs to be provided as part of the hotfix?

Comment 45 Bob Fournier 2019-04-11 16:29:35 UTC
Thank you.

Comment 46 mlammon 2019-04-16 13:47:33 UTC
We don't have access to this hardware for verification so will just be a code verification.
Please re-open if issue not resolved.

Environment:
yum info openstack-ironic-python-agent
Loaded plugins: search-disabled-repos
Available Packages
Name        : openstack-ironic-python-agent
Arch        : noarch
Version     : 3.3.1
Release     : 0.20181224074744.aab9f53.el7ost
Size        : 11 k

Comment 47 mlammon 2019-04-16 14:00:39 UTC
Please ignore, https://bugzilla.redhat.com/show_bug.cgi?id=1679174#c46

Verification:
yum info openstack-ironic-python-agent
Loaded plugins: search-disabled-repos
Available Packages
Name        : openstack-ironic-python-agent
Arch        : noarch
Version     : 3.2.2
Release     : 2.el7ost
Size        : 11 k

Comment 50 errata-xmlrpc 2019-04-30 17:24:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0937

Comment 51 Red Hat Bugzilla 2023-09-14 05:24:07 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days