Bug 710298

Summary: intel_aes_decrypt_cbc_256 doesn't work correctly when input and output buffers are the same
Product: Red Hat Enterprise Linux 6 Reporter: Elio Maldonado Batiz <emaldona>
Component: nssAssignee: Elio Maldonado Batiz <emaldona>
Status: CLOSED ERRATA QA Contact: Aleš Mareček <amarecek>
Severity: medium Docs Contact:
Priority: urgent    
Version: 6.2CC: amarecek, emaldona, iboverma, jrieden, jwest, kengert, nalin, rrelyea
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: nss-softokn-3.12.9-5.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 709517 Environment:
Last Closed: 2011-12-06 12:10:42 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 661821, 668887, 716532    

Description Elio Maldonado Batiz 2011-06-02 22:53:18 UTC 
+++ This bug was initially created as a clone of Bug #709517 +++

Description of problem:
It looks as though intel_aes_decrypt_cbc_256() doesn't work right when the input and output buffers are the same, though this appears to be something that PK11_CipherOp (which eventually calls this function, on processors that support it) is expected to be able to handle.

Version-Release number of selected component (if applicable):
nss-softokn-freebl-3.12.10-1.fc15.x86_64

How reproducible:
Always, provided you're on a 64-bit processor with native AES instruction support (the "flags:" line in /proc/cpuinfo will include "aes").  I've tested this on an i5 processor and also had it checked on an i7.

Steps to Reproduce:
1. Run reproducer with the "-i" flag, which will cause it to use the same buffer for input and output and default to a 256-bit key.
  
Actual results:
"data mismatch"

Expected results:
"Encrypted/recovered 32 bytes."

Additional info:
While NSS itself doesn't seem to trigger this behavior, the nascent support for using NSS as the backend for krb5's libk5crypto does.

--- Additional comment from nalin on 2011-05-31 17:19:59 EDT ---

Created attachment 502120 [details]
test

Compile:
  gcc -o aest aest.c `pkg-config --cflags --libs nss`
Run:
  ./aest -i                          (uses in-place encryption, errors out)
  env NSS_DISABLE_HW_AES=1 ./aest -i (uses unoptimized implementation, no error)
  ./aest                             (doesn't do encryption in place, no error)
Comment 1 Elio Maldonado Batiz 2011-06-02 22:55:32 UTC 
A patch for this bug has been approved upstream.
Comment 9 errata-xmlrpc 2011-12-06 12:10:42 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1584.html