Bug 678011

Summary: cryptsetup luksOpen causes ** stack smashing detected *** for large key
Product: Red Hat Enterprise Linux 5 Reporter: Milan Broz <mbroz>
Component: cryptsetup-luksAssignee: Milan Broz <mbroz>
Status: CLOSED ERRATA QA Contact: Release Test Team <release-test-team-automation>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 5.6CC: agk, atodorov, mbroz, prajnoha, prockai, pvrabec, zkabelac
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: cryptsetup-luks-1.0.3-6.el5 Doc Type: Bug Fix
Doc Text:
Previously, the cryptsetup utility could have terminated unexpectedly when the key size was larger than 256 bits. The cryptsetup utility now properly supports keys longer than 256 bits, fixing the problem.
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-07-21 07:04:16 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Milan Broz 2011-02-16 13:49:43 UTC
Description of problem:
When testing XTS mode support in RHEL5.6...

# cryptsetup luksOpen /dev/sdb x
Enter LUKS passphrase for /dev/sdb: 
Enter LUKS passphrase for /dev/sdb: 
key slot 0 unlocked.
*** stack smashing detected ***: cryptsetup terminated
Aborted
# cryptsetup luksDump /dev/sdb
LUKS header information for /dev/sdb

Version:        1
Cipher name:    aes
Cipher mode:    xts-plain64
Hash spec:      sha1
Payload offset: 4040
MK bits:        512
MK digest:      15 94 ed c8 e0 a5 2d 76 d3 33 08 6e 15 f5 78 44 59 54 13 2a 
MK salt:        97 54 22 c9 d6 cf 12 ee f0 44 87 fa c9 cd 09 0b 
                bb f6 e3 1a 3e 65 9c 8b de 5c f8 00 04 05 28 dd 
MK iterations:  10
UUID:           3fa26e48-4bcb-4146-a5eb-8c3d911e5aab

Key Slot 0: ENABLED
        Iterations:             407518
        Salt:                   e2 d1 74 eb ec 0f 31 58 6f 62 40 54 70 c3 de b6 
                                82 74 f8 7f 88 f1 9f 37 bb d7 2d e7 e7 c7 58 89 
        Key material offset:    8
        AF stripes:             4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

Version-Release number of selected component (if applicable):
cryptsetup-luks-1.0.3-5.el5.x86_64

Comment 1 Alexander Todorov 2011-02-23 11:28:17 UTC
Milan,
can you explain what's the issue here and provide detailed steps to reproduce?

Comment 2 Milan Broz 2011-02-23 15:31:58 UTC
Using large key (with some cipher supporting it) should be enough:

# echo xxx | cryptsetup luksFormat -c blowfish-cbc-essiv:sha256 -s 448 /dev/sdb
Command successful.

# echo xxx | cryptsetup luksOpen /dev/sdb x
key slot 0 unlocked.
*** stack smashing detected ***: cryptsetup terminated
Aborted

(I did not yet analysed code, this bug is not present in upstream version.)

Comment 3 Alexander Todorov 2011-02-24 13:47:06 UTC
What do you mean by large key? How large?

Comment 4 Milan Broz 2011-02-24 13:55:08 UTC
see comment #2

Comment 5 Alexander Todorov 2011-02-24 16:33:52 UTC
I see now: 
--key-size, -s 
 set key size in bits.

Comment 6 Milan Broz 2011-03-08 11:10:27 UTC
Fixed in
cryptsetup-luks-1.0.3-6.el5

Comment 8 Alexander Todorov 2011-05-10 10:01:40 UTC
Tested with cryptsetup-luks-1.0.3-8.el5:

# losetup /dev/loop0 disk.img 
# echo xxx | cryptsetup luksFormat -c blowfish-cbc-essiv:sha256 -s 448 /dev/loop0 
Command successful.
# echo xxx | cryptsetup luksOpen /dev/loop0 x
key slot 0 unlocked.
Command successful.
# ls -l /dev/mapper/x 
brw-rw---- 1 root disk 253, 2 May 10 06:00 /dev/mapper/x

Comment 9 Eliska Slobodova 2011-07-07 16:07:43 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Previously, the cryptsetup utility could have terminated unexpectedly when the key size was larger than 256 bits. The cryptsetup utility now properly supports keys longer than 256 bits, fixing the problem.

Comment 10 errata-xmlrpc 2011-07-21 07:04:16 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0987.html