Bug 1346089

Summary: Secrets are not filtered when logging requests in the session code
Product: Red Hat OpenStack Reporter: Tim Suter <tsuter>
Component: openstack-ironicAssignee: Derek Higgins <derekh>
Status: CLOSED ERRATA QA Contact: mlammon
Severity: low Docs Contact:
Priority: low    
Version: unspecifiedCC: apevec, bfournie, derekh, dtantsur, gmollett, jjoyce, jschluet, lhh, lruzicka, mburns, mlammon, rhel-osp-director-maint, srevivo, tsuter
Target Milestone: z6Keywords: TestOnly, Triaged, ZStream
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: All   
Whiteboard:
Fixed In Version: openstack-ironic-6.2.4-1.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-15 13:48:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Tim Suter 2016-06-13 23:23:35 UTC 
Description of problem:
private keys are logged with debug loglevel set

https://github.com/openstack/python-ironicclient/blob/11a3f12c3adb4835e92ec0e8aaff270e2c54ab29/ironicclient/common/http.py#L280

after speaking with bnemec, its recommended that the output be sanitized before logging to remove private key
Comment 3 Derek Higgins 2016-09-19 14:33:18 UTC 
Do you have an example of which private keys are being logged? What command where you running at the time?
Comment 4 Dmitry Tantsur 2016-10-03 09:23:31 UTC 
Assuming the bug is talking about OSP8/Liberty, we should probably backport https://github.com/openstack/python-ironicclient/commit/eeb327b590b489a7f6b0e2176b288f7201de6aa9 (upstream as well, as we still can backport security issues to stable/liberty iirc).

Derek, this might be SSH private keys we're using for power management (also IPMI passwords).
Comment 8 Dmitry Tantsur 2016-11-01 10:04:48 UTC 
Hmm, I can confirm that the node creation request does not sanitize its debugging output even in OSP 10 (python-ironicclient-1.7.0):

DEBUG (session:337) REQ: curl -g -i -X POST http://172.21.64.1:6385/v1/nodes -H "X-OpenStack-Ironic-API-Version: 1.9" -H "User-Agent: python-ironicclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}b91de03dcf2e8c00d1781f33a94977613f33ac1b" -d '{"driver": "fake", "driver_info": {"ipmi_password": 12345}}'

I'm not exactly sure why the code above does not work. Probably a problem with mask_password from oslo_utils?
Comment 9 Dmitry Tantsur 2016-11-01 10:21:38 UTC 
I've found the problem: it's in keystoneauth, not in ironicclient: https://github.com/openstack/keystoneauth/blob/ab39cf066784dc01869a7e22a9289e154410e4e5/keystoneauth1/session.py#L340. We should have the same masking there, then backport all patches to older releases.
Comment 10 Dmitry Tantsur 2016-11-01 14:45:54 UTC 
UPD: the command I used: ironic --debug node-create -d fake -i ipmi_password=a12345
Comment 11 Derek Higgins 2016-11-01 15:01:02 UTC 
@tsuter Do you have an example of which private keys are being logged? What command where you running at the time?

In the mean time I'll progress on the assumption that the command you ran was the one that Dmitry used
$ ironic --debug node-create -d fake -i ipmi_password=a12345
Comment 13 Dmitry Tantsur 2016-11-02 09:31:31 UTC 
Ok, so there may be the second part to this bug: in oslo.utils ssh_key_contents is not included in the list of secrets: https://github.com/openstack/oslo.utils/blob/master/oslo_utils/strutils.py#L56-L60. We may want to fix it as well.
Comment 14 Derek Higgins 2016-11-03 13:09:18 UTC 
There are a number of different instances of secrets being displayed in this bug, to keep the objective of this bug clear I'm going to concentrate on the the case that tsuter originally reported

Problem: ssh private keys are being displayed when running "ironic node-show"

secrets have been masked from being displayed by the client since Jan 2016,
https://review.openstack.org/#/c/263822/

so they are masked in OSP 9 and 10 but not earlier versions

but the ssh private key isn't include in the list of things to mask, it needs to be added the _SANITIZE_KEYS list in oslo_utils/strutils.py

this will cover OSP 10 but will need to be backported to OSP 9 

in order to cover earlier versions we'll need to backport both the patch to 
ironicclient/common/http.py and oslo_utils/strutils.py

ironic also uses the _SANITIZE_KEYS list, adding ssh_key_contents to it will also prevent the key being sent to the client in the first place.
Comment 18 Derek Higgins 2016-11-29 15:30:44 UTC 
Fix merged into upstream newton
https://review.openstack.org/#/c/401160/2/ironic/api/controllers/v1/node.py
Comment 20 Lon Hohberger 2017-10-10 18:08:19 UTC 
According to our records, this should be resolved by openstack-ironic-6.2.4-1.el7ost.  This build is available now.
Comment 23 mlammon 2017-11-01 19:08:02 UTC 
installed latest osp10 undercloud with puddle 2017-10-30.3 on 11/1/2017
register node with debug and check output
This bug has been verified

environment
openstack-ironic-api-6.2.4-2.el7ost.noarch
openstack-ironic-conductor-6.2.4-2.el7ost.noarch
openstack-ironic-common-6.2.4-2.el7ost.noarch
openstack-ironic-inspector-4.2.2-3.el7ost.noarch

cat node.yaml
nodes:
    - name: overcloud-2-node5
      driver: pxe_ssh
      driver_info:
        ssh_username: stack
        ssh_key_contents:  |
          -----BEGIN RSA PRIVATE KEY-----
          MIIEogIBAAKCAQEAxc0a2u18EgTy5y9JvaExDXP2pWuE8Ebyo24AOo1iQoWR7D5n
          fNjkgCeKZRbABhsdoMBmbDMtn0PO3lzI2HnZQBB4BdBZprAiQ1NwKKotUv9puTeY
          [..]
          7DsSKAL4EDqjufY3h+4fRwOcD+EFqlUTDG1sjsSDKjdiHyYMzjcrg8nbaj/M9kAs
          xXnSm9686KxUiCDXO5FWKun204B18mPH1UP20aYw098t6aAQwm4=
          -----END RSA PRIVATE KEY-----
        ssh_virt_type: virsh
        ssh_address: 10.0.0.1
      properties:
        cpus: 4
        memory_mb: 12288
        local_gb: 60
        #boot_option: local (it doesn't set 'capabilities')
        root_device:
          # vendor: "0x1af4"
          # model: "QEMU HARDDISK"
          # size: 64424509440
          wwn: "0x0000000000000001"
          # serial: "0000000000000001"
          # vendor: QEMU
          # name: /dev/sda
      ports:
        - address: 52:54:00:a0:af:da


ironic --debug node-create -d fake -i ipmi_password=a12345



[stack@undercloud ~]$ ironic --debug node-show overcloud-2-node5
DEBUG (extension:157) found extension EntryPoint.parse('v2token = keystoneauth1.loading._plugins.identity.v2:Token')
DEBUG (extension:157) found extension EntryPoint.parse('v3oauth1 = keystoneauth1.extras.oauth1._loading:V3OAuth1')
DEBUG (extension:157) found extension EntryPoint.parse('admin_token = keystoneauth1.loading._plugins.admin_token:AdminToken')
DEBUG (extension:157) found extension EntryPoint.parse('v3oidcauthcode = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAuthorizationCode')
DEBUG (extension:157) found extension EntryPoint.parse('v2password = keystoneauth1.loading._plugins.identity.v2:Password')
DEBUG (extension:157) found extension EntryPoint.parse('v3samlpassword = keystoneauth1.extras._saml2._loading:Saml2Password')
DEBUG (extension:157) found extension EntryPoint.parse('v3password = keystoneauth1.loading._plugins.identity.v3:Password')
DEBUG (extension:157) found extension EntryPoint.parse('v3oidcaccesstoken = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAccessToken')
DEBUG (extension:157) found extension EntryPoint.parse('v3oidcpassword = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectPassword')
DEBUG (extension:157) found extension EntryPoint.parse('v3kerberos = keystoneauth1.extras.kerberos._loading:Kerberos')
DEBUG (extension:157) found extension EntryPoint.parse('token = keystoneauth1.loading._plugins.identity.generic:Token')
DEBUG (extension:157) found extension EntryPoint.parse('v3oidcclientcredentials = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectClientCredentials')
DEBUG (extension:157) found extension EntryPoint.parse('v3tokenlessauth = keystoneauth1.loading._plugins.identity.v3:TokenlessAuth')
DEBUG (extension:157) found extension EntryPoint.parse('v3token = keystoneauth1.loading._plugins.identity.v3:Token')
DEBUG (extension:157) found extension EntryPoint.parse('v3totp = keystoneauth1.loading._plugins.identity.v3:TOTP')
DEBUG (extension:157) found extension EntryPoint.parse('password = keystoneauth1.loading._plugins.identity.generic:Password')
DEBUG (extension:157) found extension EntryPoint.parse('v3fedkerb = keystoneauth1.extras.kerberos._loading:MappedKerberos')
DEBUG (extension:157) found extension EntryPoint.parse('token_endpoint = openstackclient.api.auth_plugin:TokenEndpoint')
DEBUG (extension:157) found extension EntryPoint.parse('aodh-noauth = aodhclient.noauth:AodhNoAuthLoader')
DEBUG (extension:157) found extension EntryPoint.parse('gnocchi-basic = gnocchiclient.auth:GnocchiBasicLoader')
DEBUG (extension:157) found extension EntryPoint.parse('gnocchi-noauth = gnocchiclient.auth:GnocchiNoAuthLoader')
DEBUG (session:342) REQ: curl -g -i -X GET http://192.168.0.1:5000/v2.0 -H "Accept: application/json" -H "User-Agent: ironic keystoneauth1/2.12.3 python-requests/2.11.1 CPython/2.7.5"
INFO (connectionpool:214) Starting new HTTP connection (1): 192.168.0.1
DEBUG (connectionpool:401) "GET /v2.0 HTTP/1.1" 200 230
DEBUG (session:390) RESP: [200] Date: Sat, 21 Oct 2017 01:14:54 GMT Server: Apache/2.4.6 (Red Hat Enterprise Linux) Vary: X-Auth-Token,Accept-Encoding x-openstack-request-id: req-fd331491-f18e-4e01-b0d2-1deae86519ac Content-Encoding: gzip Content-Length: 230 Connection: close Content-Type: application/json
RESP BODY: {"version": {"status": "deprecated", "updated": "2016-08-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://192.168.0.1:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

DEBUG (v2:63) Making authentication request to http://192.168.0.1:5000/v2.0/tokens
INFO (connectionpool:249) Resetting dropped connection: 192.168.0.1
DEBUG (connectionpool:401) "POST /v2.0/tokens HTTP/1.1" 200 1097
DEBUG (session:342) REQ: curl -g -i -X GET http://192.168.0.1:6385/v1/nodes/overcloud-2-node5 -H "X-OpenStack-Ironic-API-Version: 1.9" -H "User-Agent: python-ironicclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}10d1283f542c7700b2a7f7b2e821f64f00edb890"
INFO (connectionpool:214) Starting new HTTP connection (1): 192.168.0.1
DEBUG (connectionpool:401) "GET /v1/nodes/overcloud-2-node5 HTTP/1.1" 200 1246
DEBUG (session:390) RESP: [200] X-Openstack-Ironic-Api-Minimum-Version: 1.1 X-Openstack-Ironic-Api-Maximum-Version: 1.22 X-Openstack-Ironic-Api-Version: 1.9 Content-Length: 1246 Content-Type: application/json; charset=UTF-8 Openstack-Request-Id: req-6e4a3981-4918-4429-aa8d-133ea2004c29 Date: Sat, 21 Oct 2017 01:14:54 GMT Connection: keep-alive
RESP BODY: {"target_power_state": null, "links": [{"href": "http://192.168.0.1:6385/v1/nodes/7de84d3f-22aa-4cd9-b482-50a80f971284", "rel": "self"}, {"href": "http://192.168.0.1:6385/nodes/7de84d3f-22aa-4cd9-b482-50a80f971284", "rel": "bookmark"}], "extra": {}, "last_error": null, "updated_at": null, "maintenance_reason": null, "provision_state": "enroll", "clean_step": {}, "uuid": "7de84d3f-22aa-4cd9-b482-50a80f971284", "console_enabled": false, "target_provision_state": null, "provision_updated_at": null, "power_state": null, "inspection_started_at": null, "inspection_finished_at": null, "maintenance": false, "driver": "pxe_ssh", "reservation": null, "properties": {"memory_mb": 12288, "root_device": {"wwn": "0x0000000000000001"}, "local_gb": 60, "cpus": 4}, "instance_uuid": null, "name": "overcloud-2-node5", "driver_info": {"ssh_username": "stack", "ssh_virt_type": "virsh", "ssh_address": "10.0.0.1", "ssh_key_contents": "******"}, "created_at": "2017-10-21T01:13:53+00:00", "ports": [{"href": "http://192.168.0.1:6385/v1/nodes/7de84d3f-22aa-4cd9-b482-50a80f971284/ports", "rel": "self"}, {"href": "http://192.168.0.1:6385/nodes/7de84d3f-22aa-4cd9-b482-50a80f971284/ports", "rel": "bookmark"}], "driver_internal_info": {}, "instance_info": {}}

+------------------------+-------------------------------------------------------------------------+
| Property               | Value                                                                   |
+------------------------+-------------------------------------------------------------------------+
| chassis_uuid           |                                                                         |
| clean_step             | {}                                                                      |
| console_enabled        | False                                                                   |
| created_at             | 2017-10-21T01:13:53+00:00                                               |
| driver                 | pxe_ssh                                                                 |
| driver_info            | {u'ssh_username': u'stack', u'ssh_virt_type': u'virsh', u'ssh_address': |
|                        | u'10.0.0.1', u'ssh_key_contents': u'******'}                            |
| driver_internal_info   | {}                                                                      |
| extra                  | {}                                                                      |
| inspection_finished_at | None                                                                    |
| inspection_started_at  | None                                                                    |
| instance_info          | {}                                                                      |
| instance_uuid          | None                                                                    |
| last_error             | None                                                                    |
| maintenance            | False                                                                   |
| maintenance_reason     | None                                                                    |
| name                   | overcloud-2-node5                                                       |
| network_interface      |                                                                         |
| power_state            | None                                                                    |
| properties             | {u'memory_mb': 12288, u'root_device': {u'wwn': u'0x0000000000000001'},  |
|                        | u'local_gb': 60, u'cpus': 4}                                            |
| provision_state        | enroll                                                                  |
| provision_updated_at   | None                                                                    |
| raid_config            |                                                                         |
| reservation            | None                                                                    |
| resource_class         |                                                                         |
| target_power_state     | None                                                                    |
| target_provision_state | None                                                                    |
| target_raid_config     |                                                                         |
| updated_at             | None                                                                    |
| uuid                   | 7de84d3f-22aa-4cd9-b482-50a80f971284                                    |
+------------------------+-------------------------------------------------------------------------+
Comment 26 errata-xmlrpc 2017-11-15 13:48:19 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3235