Bug 678011 - cryptsetup luksOpen causes ** stack smashing detected *** for large key
cryptsetup luksOpen causes ** stack smashing detected *** for large key
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: cryptsetup-luks (Show other bugs)
5.6
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Milan Broz
Release Test Team
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2011-02-16 08:49 EST by Milan Broz
Modified: 2013-02-28 23:10 EST (History)
7 users (show)

See Also:
Fixed In Version: cryptsetup-luks-1.0.3-6.el5
Doc Type: Bug Fix
Doc Text:
Previously, the cryptsetup utility could have terminated unexpectedly when the key size was larger than 256 bits. The cryptsetup utility now properly supports keys longer than 256 bits, fixing the problem.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-07-21 03:04:16 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Milan Broz 2011-02-16 08:49:43 EST
Description of problem:
When testing XTS mode support in RHEL5.6...

# cryptsetup luksOpen /dev/sdb x
Enter LUKS passphrase for /dev/sdb: 
Enter LUKS passphrase for /dev/sdb: 
key slot 0 unlocked.
*** stack smashing detected ***: cryptsetup terminated
Aborted
# cryptsetup luksDump /dev/sdb
LUKS header information for /dev/sdb

Version:        1
Cipher name:    aes
Cipher mode:    xts-plain64
Hash spec:      sha1
Payload offset: 4040
MK bits:        512
MK digest:      15 94 ed c8 e0 a5 2d 76 d3 33 08 6e 15 f5 78 44 59 54 13 2a 
MK salt:        97 54 22 c9 d6 cf 12 ee f0 44 87 fa c9 cd 09 0b 
                bb f6 e3 1a 3e 65 9c 8b de 5c f8 00 04 05 28 dd 
MK iterations:  10
UUID:           3fa26e48-4bcb-4146-a5eb-8c3d911e5aab

Key Slot 0: ENABLED
        Iterations:             407518
        Salt:                   e2 d1 74 eb ec 0f 31 58 6f 62 40 54 70 c3 de b6 
                                82 74 f8 7f 88 f1 9f 37 bb d7 2d e7 e7 c7 58 89 
        Key material offset:    8
        AF stripes:             4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

Version-Release number of selected component (if applicable):
cryptsetup-luks-1.0.3-5.el5.x86_64
Comment 1 Alexander Todorov 2011-02-23 06:28:17 EST
Milan,
can you explain what's the issue here and provide detailed steps to reproduce?
Comment 2 Milan Broz 2011-02-23 10:31:58 EST
Using large key (with some cipher supporting it) should be enough:

# echo xxx | cryptsetup luksFormat -c blowfish-cbc-essiv:sha256 -s 448 /dev/sdb
Command successful.

# echo xxx | cryptsetup luksOpen /dev/sdb x
key slot 0 unlocked.
*** stack smashing detected ***: cryptsetup terminated
Aborted

(I did not yet analysed code, this bug is not present in upstream version.)
Comment 3 Alexander Todorov 2011-02-24 08:47:06 EST
What do you mean by large key? How large?
Comment 4 Milan Broz 2011-02-24 08:55:08 EST
see comment #2
Comment 5 Alexander Todorov 2011-02-24 11:33:52 EST
I see now: 
--key-size, -s 
 set key size in bits.
Comment 6 Milan Broz 2011-03-08 06:10:27 EST
Fixed in
cryptsetup-luks-1.0.3-6.el5
Comment 8 Alexander Todorov 2011-05-10 06:01:40 EDT
Tested with cryptsetup-luks-1.0.3-8.el5:

# losetup /dev/loop0 disk.img 
# echo xxx | cryptsetup luksFormat -c blowfish-cbc-essiv:sha256 -s 448 /dev/loop0 
Command successful.
# echo xxx | cryptsetup luksOpen /dev/loop0 x
key slot 0 unlocked.
Command successful.
# ls -l /dev/mapper/x 
brw-rw---- 1 root disk 253, 2 May 10 06:00 /dev/mapper/x
Comment 9 Eliska Slobodova 2011-07-07 12:07:43 EDT
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Previously, the cryptsetup utility could have terminated unexpectedly when the key size was larger than 256 bits. The cryptsetup utility now properly supports keys longer than 256 bits, fixing the problem.
Comment 10 errata-xmlrpc 2011-07-21 03:04:16 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0987.html

Note You need to log in before you can comment on or make changes to this bug.