Bug 678011 - cryptsetup luksOpen causes ** stack smashing detected *** for large key
Summary: cryptsetup luksOpen causes ** stack smashing detected *** for large key
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: cryptsetup-luks
Version: 5.6
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Milan Broz
QA Contact: Release Test Team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-02-16 13:49 UTC by Milan Broz
Modified: 2013-03-01 04:10 UTC (History)
7 users (show)

Fixed In Version: cryptsetup-luks-1.0.3-6.el5
Doc Type: Bug Fix
Doc Text:
Previously, the cryptsetup utility could have terminated unexpectedly when the key size was larger than 256 bits. The cryptsetup utility now properly supports keys longer than 256 bits, fixing the problem.
Clone Of:
Environment:
Last Closed: 2011-07-21 07:04:16 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2011:0987 0 normal SHIPPED_LIVE cryptsetup-luks bug fix update 2011-07-20 15:45:04 UTC

Description Milan Broz 2011-02-16 13:49:43 UTC
Description of problem:
When testing XTS mode support in RHEL5.6...

# cryptsetup luksOpen /dev/sdb x
Enter LUKS passphrase for /dev/sdb: 
Enter LUKS passphrase for /dev/sdb: 
key slot 0 unlocked.
*** stack smashing detected ***: cryptsetup terminated
Aborted
# cryptsetup luksDump /dev/sdb
LUKS header information for /dev/sdb

Version:        1
Cipher name:    aes
Cipher mode:    xts-plain64
Hash spec:      sha1
Payload offset: 4040
MK bits:        512
MK digest:      15 94 ed c8 e0 a5 2d 76 d3 33 08 6e 15 f5 78 44 59 54 13 2a 
MK salt:        97 54 22 c9 d6 cf 12 ee f0 44 87 fa c9 cd 09 0b 
                bb f6 e3 1a 3e 65 9c 8b de 5c f8 00 04 05 28 dd 
MK iterations:  10
UUID:           3fa26e48-4bcb-4146-a5eb-8c3d911e5aab

Key Slot 0: ENABLED
        Iterations:             407518
        Salt:                   e2 d1 74 eb ec 0f 31 58 6f 62 40 54 70 c3 de b6 
                                82 74 f8 7f 88 f1 9f 37 bb d7 2d e7 e7 c7 58 89 
        Key material offset:    8
        AF stripes:             4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

Version-Release number of selected component (if applicable):
cryptsetup-luks-1.0.3-5.el5.x86_64

Comment 1 Alexander Todorov 2011-02-23 11:28:17 UTC
Milan,
can you explain what's the issue here and provide detailed steps to reproduce?

Comment 2 Milan Broz 2011-02-23 15:31:58 UTC
Using large key (with some cipher supporting it) should be enough:

# echo xxx | cryptsetup luksFormat -c blowfish-cbc-essiv:sha256 -s 448 /dev/sdb
Command successful.

# echo xxx | cryptsetup luksOpen /dev/sdb x
key slot 0 unlocked.
*** stack smashing detected ***: cryptsetup terminated
Aborted

(I did not yet analysed code, this bug is not present in upstream version.)

Comment 3 Alexander Todorov 2011-02-24 13:47:06 UTC
What do you mean by large key? How large?

Comment 4 Milan Broz 2011-02-24 13:55:08 UTC
see comment #2

Comment 5 Alexander Todorov 2011-02-24 16:33:52 UTC
I see now: 
--key-size, -s 
 set key size in bits.

Comment 6 Milan Broz 2011-03-08 11:10:27 UTC
Fixed in
cryptsetup-luks-1.0.3-6.el5

Comment 8 Alexander Todorov 2011-05-10 10:01:40 UTC
Tested with cryptsetup-luks-1.0.3-8.el5:

# losetup /dev/loop0 disk.img 
# echo xxx | cryptsetup luksFormat -c blowfish-cbc-essiv:sha256 -s 448 /dev/loop0 
Command successful.
# echo xxx | cryptsetup luksOpen /dev/loop0 x
key slot 0 unlocked.
Command successful.
# ls -l /dev/mapper/x 
brw-rw---- 1 root disk 253, 2 May 10 06:00 /dev/mapper/x

Comment 9 Eliska Slobodova 2011-07-07 16:07:43 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Previously, the cryptsetup utility could have terminated unexpectedly when the key size was larger than 256 bits. The cryptsetup utility now properly supports keys longer than 256 bits, fixing the problem.

Comment 10 errata-xmlrpc 2011-07-21 07:04:16 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0987.html


Note You need to log in before you can comment on or make changes to this bug.