Bug 1868754

Summary: java-11-openjdk / rhel-8 / FIPS: Ciphers remain in broken state (unusable), after supplied with wrongly sized buffer
Product: Red Hat Enterprise Linux 8 Reporter: zzambers
Component: java-11-openjdkAssignee: Martin Balao <mbalao>
Status: CLOSED ERRATA QA Contact: OpenJDK QA <java-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 8.3CC: ahughes, jandrlik, lmiksik, mbalao, pmikova
Target Milestone: rc   
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: java-11-openjdk-11.0.8.10-6.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1868759 (view as bug list) Environment:
Last Closed: 2020-11-04 02:43:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1868759    
Attachments:
Description Flags
CipherBreak.java none

Description zzambers 2020-08-13 17:57:39 UTC 
Ciphers remain in broken state (unusable), after supplied with wrongly sized buffer
- problem happens when wrongly sized data buffer is supplied to be encrypted, exception is thrown (CKR_DATA_LEN_RANGE), which is expected, but what is not expected is that ciphers became unusable afterwards, subsequent attempts to use ciphers end up with exception (CKR_OPERATION_ACTIVE), even on different instance/ different cipher with correct params. It seems that pkcs11 provider stays in some invalid state...
- affects all jdks (8, 11, latest-upstream)


Stack traces:

Alg 1: AES/CBC/NoPadding
Provider 1: SunPKCS11-NSS-FIPS version 1.8
Key 1: SunPKCS11-NSS-FIPS AES secret key, 128 bitssession object, sensitive, extractable)
java.security.ProviderException: update() failed
	at sun.security.pkcs11.P11Cipher.implUpdate(P11Cipher.java:629)
	at sun.security.pkcs11.P11Cipher.engineUpdate(P11Cipher.java:526)
	at sun.security.pkcs11.P11Cipher.engineDoFinal(P11Cipher.java:555)
	at sun.security.pkcs11.P11Cipher.engineDoFinal(P11Cipher.java:541)
	at javax.crypto.Cipher.doFinal(Cipher.java:2168)
	at CipherBreak.main(CipherBreak.java:24)
Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_DATA_LEN_RANGE
	at sun.security.pkcs11.wrapper.PKCS11.C_EncryptUpdate(Native Method)
	at sun.security.pkcs11.P11Cipher.implUpdate(P11Cipher.java:581)
	... 5 more
Alg 2: DES/CBC/NoPadding
Provider 2: SunPKCS11-NSS-FIPS version 1.8
Key 2: SunPKCS11-NSS-FIPS DES secret key, 56 bitssession object, sensitive, extractable)
Exception in thread "main" java.security.InvalidKeyException: Could not initialize cipher
	at sun.security.pkcs11.P11Cipher.implInit(P11Cipher.java:399)
	at sun.security.pkcs11.P11Cipher.engineInit(P11Cipher.java:299)
	at javax.crypto.Cipher.init(Cipher.java:1249)
	at javax.crypto.Cipher.init(Cipher.java:1189)
	at CipherBreak.main(CipherBreak.java:37)
Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_OPERATION_ACTIVE
	at sun.security.pkcs11.wrapper.PKCS11.C_EncryptInit(Native Method)
	at sun.security.pkcs11.P11Cipher.initialize(P11Cipher.java:468)
	at sun.security.pkcs11.P11Cipher.implInit(P11Cipher.java:397)
Comment 1 zzambers 2020-08-13 17:58:50 UTC 
Created attachment 1711353 [details]
CipherBreak.java

Reproducer
Comment 2 zzambers 2020-08-19 15:41:46 UTC 
I looked a bit at this issue, and I think the problem could actually be in NSS library iteself.

Exception (CKR_DATA_LEN_RANGE) is thrown by C_EncryptUpdate [1]. Documentation I have found on the internet says [2]:
"A call to C_EncryptUpdate which results in an error other than CKR_BUFFER_TOO_SMALL terminates the current encryption operation."

I may be reading it wrong, but it seems to me, that if CKR_DATA_LEN_RANGE should have terminated the current operation, subsequent operations should not be showing CKR_OPERATION_ACTIVE.

I was just wondering if calling reset with true would make any difference... [3]

[1] https://hg.openjdk.java.net/jdk8u/jdk8u/jdk/file/4687075d8ccf/src/share/classes/sun/security/pkcs11/P11Cipher.java#l581
[2] https://www.cryptsoft.com/pkcs11doc/v220/pkcs11__all_8h.html#aC_EncryptUpdate
[3] https://hg.openjdk.java.net/jdk8u/jdk8u/jdk/file/4687075d8ccf/src/share/classes/sun/security/pkcs11/P11Cipher.java#l628
Comment 30 errata-xmlrpc 2020-11-04 02:43:55 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (java-11-openjdk bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4657