Bug 1846154

Summary:	hard to tell if encryption is on or off
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	Ben England <bengland>
Component:	RADOS	Assignee:	Radoslaw Zarzynski <rzarzyns>
Status:	CLOSED ERRATA	QA Contact:	skanta
Severity:	medium	Docs Contact:	Rivka Pollack <rpollack>
Priority:	unspecified
Version:	4.1	CC:	akraj, akupczyk, asriram, bhubbard, ceph-eng-bugs, kdreyer, nojha, pdhiran, rzarzyns, sseshasa, vereddy, vumrao
Target Milestone:	---
Target Release:	7.0
Hardware:	x86_64
OS:	Unspecified
Whiteboard:
Fixed In Version:	ceph-18.2.0-1	Doc Type:	Enhancement
Doc Text:	.New performance counters introduced for messenger v2 Previously, there were no dedicated performance counters for accounting encrypted traffic in messenger v2. With this enhancement, `msgr_recv_encrypted_bytes` and `msgr_send_encrypted_bytes`, are introduced to account for receiving and sending bytes respectively. These new performance counters facilitate rough validation of the encryption status.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-12-13 15:18:29 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2237662

Description Ben England 2020-06-10 22:39:29 UTC

Description of problem:

The user interface AFAICT does not show whether data is being encrypted or not by the Ceph Messenger V2 component of librados, OSDs and other services. Specifically there are no counters and there is no way of knowing other than brute-force methods that sysadmins do not want to use. I'm not confident that encryption is happening.

This could be very good or very bad. If encryption is happening and it is very low-overhead, that could explain some of this, but I doubt that based on Radek's observations, which show significant time spent in places like "ceph::crypto::onwire::AES128GCM_OnWireTxHandler::authenticated_encrypt_update"

https://gist.github.com/rzarzynski/0cef145c8fe2f85344653fd7377d64c8

However, if it is not happening, then customers may think they have enabled encryption when they really haven't.

Version-Release number of selected component (if applicable):

RHCS 4.1 http://download.eng.bos.redhat.com/rhel-8/composes/auto/ceph-4.1-rhel-8/RHCEPH-4.1-RHEL-8-20200603.ci.1
ceph-base-14.2.8-60.el8cp.x86_64

How reproducible:

every time

Steps to Reproduce: (details below)
1. install RHCS using ceph-ansible
2. run Ceph/CBT benchmark to put a heavy network load on Ceph
3. measure system throughput, resource utilization and profiles

Actual results:

the "perf" profiler shows no encryption activity after the following was set on all mons. OSD logging, etc. shows no clear sign that encryption is enabled.

Expected results:

The mons and OSDs should tell you upon startup that they are trying to encrypt all connections and should warn you if they cannot. There should be perf counters that indicate what fraction of data is encrypted.

Additional info:

https://docs.google.com/document/d/1iSL5PPXVn_6aDBcKGjvHVp5w0o3E9ztmSEDAQAisYEI/edit#

here is a CBT run tree that includes pbench resource utilization data and a "perf" profile generated as Radek describes

http://perf1.perf.lab.eng.bos.redhat.com/pub/bengland/tmp/encryption/pbench-user-benchmark__2020.06.10T22.25.44/

perf data is in the subdirectory 2020-06-10-22-25/results/00000000/

The ceph-ansible install run was logged here:

http://perf1.perf.lab.eng.bos.redhat.com/pub/bengland/tmp/encryption/ceph-ansible-site-yml-06101321.log

Comment 3 Ben England 2020-06-11 12:38:13 UTC

retried with [global], that fixed it, I guess that was stupid.   But it would still be nice if the OSDs and MONs logged something that said "i'm encrypting traffic" or not, as a sanity check.   So the original title of the bz still stands.  Thanks for your help.

Comment 37 errata-xmlrpc 2023-12-13 15:18:29 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 7.0 Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:7780