Bug 1575945 - Heat in DEBUG logs private keys when a template creates a keypair
Summary: Heat in DEBUG logs private keys when a template creates a keypair
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-oslo-utils
Version: 11.0 (Ocata)
Hardware: Unspecified
OS: Unspecified
Target Milestone: async
: 11.0 (Ocata)
Assignee: Matthew Booth
QA Contact: Shai Revivo
Depends On:
Blocks: 1578343 1578346 1578347
TreeView+ depends on / blocked
Reported: 2018-05-08 11:08 UTC by ojanas
Modified: 2021-06-10 16:06 UTC (History)
22 users (show)

Fixed In Version: python-oslo-utils-3.22.1-2.el7ost
Doc Type: Bug Fix
Doc Text:
If Nova or Heat is configured to log at the DEBUG log level, private keys were logged as clear text when a keypair was created. oslo.utils now hides private keys in logs.
Clone Of:
: 1578343 1578346 1578347 (view as bug list)
Last Closed: 2018-06-08 12:41:11 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Launchpad 1770683 0 None None None 2018-05-11 16:07:10 UTC
OpenStack gerrit 567887 0 None None None 2018-05-11 16:20:37 UTC
OpenStack gerrit 568557 0 None None None 2018-05-15 11:27:53 UTC

Description ojanas 2018-05-08 11:08:30 UTC
Description of problem:

Customer noticed that Heat, when in DEBUG, creates a log entry containing private key when creating a keypair.

2018-05-08 09:49:13.198 742250 DEBUG novaclient.v2.client [req-12f8d12d-c94f-4d05-afd1-95dd16824dfd - crf - - -] RESP: [200] Content-Length: 2324 Content-Type: applica
tion/json Openstack-Api-Version: compute 2.1 X-Openstack-Nova-Api-Version: 2.1 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version X-Compute-Request-Id: req-947a
b50d-b915-4396-803e-55bf66a5f1e0 Date: Tue, 08 May 2018 06:49:13 GMT
RESP BODY: {"keypair": {"public_key": "ssh-rsa ... Generated-by-Nova", "private_key": "-----BEGIN RSA PRIVATE KEY-----\n ...

In my opinion, this is fine, since it happens during DEBUG only. I mean that is the DEBUG's purpose, right?

Customer brought this up because of the recent events with twitter leaking plain text passwords in the logs [1] and asked us at least consider whether we should do something about this or not.

Thank you.


[1] https://www.bleepingcomputer.com/news/security/twitter-admits-recording-plaintext-passwords-in-internal-logs-just-like-github/

Comment 1 Zane Bitter 2018-05-08 17:57:42 UTC
Getting Nova to generate a keypair is a fairly insecure way of doing things no matter how you look at it (since the private key gets stored unencrypted in Heat's DB, and is accessible through the API). The only secure method is to create the keypair locally and upload only the public key to Nova.

That said, it is definitely not OK to log the private key in my opinion, logging private keys is most definitely not the purpose of DEBUG-level logging, and practically all operators run heat with debug logging enabled.

Comment 3 Zane Bitter 2018-05-08 18:02:27 UTC
Changing component to python-novaclient, since that's where it needs to be sanitised. It's likely that e.g. Horizon and Mistral could also be affected in the same way that Heat is.

Comment 6 Matthew Booth 2018-05-11 15:46:47 UTC
Following this, the offending log is actually in keystoneclient's Session._http_log_response().

Comment 7 Matthew Booth 2018-05-11 15:54:35 UTC
I think this needs to be fixed in oslo.utils' mask_password. Specifically we should additionally mask "private_key": "...".

Comment 9 Matthew Booth 2018-05-14 13:44:05 UTC
The fix has now merged upstream.

Comment 10 Matthew Booth 2018-05-15 11:27:53 UTC
Submitted a backport for upstream Ocata: https://review.openstack.org/568557

Comment 11 Victor Stinner 2018-06-05 13:08:01 UTC
I requested Release oslo.utils 3.22.3 of Ocata:

Comment 12 Victor Stinner 2018-06-06 15:29:29 UTC
python-oslo-utils-3.22.1-2.el7ost is now ready for tests.

Comment 13 Chris Jones 2018-06-08 12:41:11 UTC
OSP11 is now EOL. This fix is in progress for OSP12 and newer.

Note You need to log in before you can comment on or make changes to this bug.