Bug 1846154 - hard to tell if encryption is on or off
Summary: hard to tell if encryption is on or off
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: RADOS
Version: 4.1
Hardware: x86_64
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 7.0
Assignee: Radoslaw Zarzynski
QA Contact: skanta
Rivka Pollack
URL:
Whiteboard:
Depends On:
Blocks: 2237662
TreeView+ depends on / blocked
 
Reported: 2020-06-10 22:39 UTC by Ben England
Modified: 2023-12-13 15:18 UTC (History)
12 users (show)

Fixed In Version: ceph-18.2.0-1
Doc Type: Enhancement
Doc Text:
.New performance counters introduced for messenger v2 Previously, there were no dedicated performance counters for accounting encrypted traffic in messenger v2. With this enhancement, `msgr_recv_encrypted_bytes` and `msgr_send_encrypted_bytes`, are introduced to account for receiving and sending bytes respectively. These new performance counters facilitate rough validation of the encryption status.
Clone Of:
Environment:
Last Closed: 2023-12-13 15:18:29 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github ceph ceph pull 43791 0 None open auth, msg/async: add perf counters for encrypted traffic 2021-11-09 00:32:30 UTC
Red Hat Product Errata RHBA-2023:7780 0 None None None 2023-12-13 15:18:32 UTC

Description Ben England 2020-06-10 22:39:29 UTC
Description of problem:

The user interface AFAICT does not show whether data is being encrypted or not by the Ceph Messenger V2 component of librados, OSDs and other services.   Specifically there are no counters and there is no way of knowing other than brute-force methods that sysadmins do not want to use.  I'm not confident that encryption is happening.

This could be very good or very bad.   If encryption is happening and it is very low-overhead, that could explain some of this, but I doubt that based on Radek's observations, which show significant time spent in places like "ceph::crypto::onwire::AES128GCM_OnWireTxHandler::authenticated_encrypt_update"

https://gist.github.com/rzarzynski/0cef145c8fe2f85344653fd7377d64c8

However, if it is not happening, then customers may think they have enabled encryption when they really haven't.


Version-Release number of selected component (if applicable):

RHCS 4.1 http://download.eng.bos.redhat.com/rhel-8/composes/auto/ceph-4.1-rhel-8/RHCEPH-4.1-RHEL-8-20200603.ci.1
ceph-base-14.2.8-60.el8cp.x86_64


How reproducible:

every time


Steps to Reproduce: (details below)
1. install RHCS using ceph-ansible
2. run Ceph/CBT benchmark to put a heavy network load on Ceph
3. measure system throughput, resource utilization and profiles


Actual results:

the "perf" profiler shows no encryption activity after the following was set on all mons.   OSD logging, etc. shows no clear sign that encryption is enabled.


Expected results:

The mons and OSDs should tell you upon startup that they are trying to encrypt all connections and should warn you if they cannot.  There should be perf counters that indicate what fraction of data is encrypted.


Additional info:

https://docs.google.com/document/d/1iSL5PPXVn_6aDBcKGjvHVp5w0o3E9ztmSEDAQAisYEI/edit#

here is a CBT run tree that includes pbench resource utilization data and a "perf" profile generated as Radek describes

http://perf1.perf.lab.eng.bos.redhat.com/pub/bengland/tmp/encryption/pbench-user-benchmark__2020.06.10T22.25.44/

perf data is in the subdirectory 2020-06-10-22-25/results/00000000/

The ceph-ansible install run was logged here:

http://perf1.perf.lab.eng.bos.redhat.com/pub/bengland/tmp/encryption/ceph-ansible-site-yml-06101321.log

Comment 3 Ben England 2020-06-11 12:38:13 UTC
retried with [global], that fixed it, I guess that was stupid.   But it would still be nice if the OSDs and MONs logged something that said "i'm encrypting traffic" or not, as a sanity check.   So the original title of the bz still stands.  Thanks for your help.

Comment 37 errata-xmlrpc 2023-12-13 15:18:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 7.0 Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:7780


Note You need to log in before you can comment on or make changes to this bug.