Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
For bugs related to Red Hat Enterprise Linux 5 product line. The current stable release is 5.10. For Red Hat Enterprise Linux 6 and above, please visit Red Hat JIRA https://issues.redhat.com/secure/CreateIssue!default.jspa?pid=12332745 to report new issues.

Bug 461537

Summary: crypto: hmac(md5) self-test panics system
Product: Red Hat Enterprise Linux 5 Reporter: Jarod Wilson <jarod>
Component: kernelAssignee: Neil Horman <nhorman>
Status: CLOSED ERRATA QA Contact: Martin Jenner <mjenner>
Severity: high Docs Contact:
Priority: medium    
Version: 5.3CC: herbert.xu, lwang
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-01-20 20:18:17 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
work-in-progress test module
none
[CRYPTO] hmac: Avoid calling virt_to_page on key
none
[CRYPTO] hmac: Avoid calling virt_to_page on key
none
[CRYPTO] hmac: Avoid calling virt_to_page on key none

Description Jarod Wilson 2008-09-08 21:48:48 UTC
Description of problem:
Running a variant of the tcrypt do_test() function in another module being developed for fips cavs testing, all tests but the one for hmac(md5) successfully pass. The hmac(md5) test panics the system.


Version-Release number of selected component (if applicable):
kernel-2.6.18-109.el5 + the big 5.3 crypto patch series


How reproducible:
Run crypto/tcrypt.c:do_test(100);


Actual results:
kernel panic

backtrace coming eventually, if I can ever get kdump and crash to both behave. First, kdump wasn't dumping, now it is, but crash says the vmcore doesn't match the vmlinux, when it most definitely should...


Expected results:
test passes

Comment 1 Herbert Xu 2008-09-09 00:02:23 UTC
Please attach the modified test code that you're running.  Thanks!

Comment 2 Jarod Wilson 2008-09-09 14:32:45 UTC
Created attachment 316194 [details]
work-in-progress test module

Here's the work-in-progress test module, wired up sufficiently to panic the box I'm working on over here. Note that it presently relies on a copy of crypto/internal.h for the definition of digest_test(), which if you ask me, maybe ought to be in include/linux/ncrypto.h along side where alg_test() is...

The panicking kernel is 2.6.18-109.el5 + the crypto patch series dated 8/22, can also provide my srpm, if so desired.

Comment 3 Neil Horman 2008-09-09 14:51:38 UTC
do you have the backtrace of your panic handy?

Comment 4 Jarod Wilson 2008-09-09 18:29:04 UTC
Yeah, I finally broke down and hooked up serial:

fips_cavs_test: loaded
fips_cavs_test: testing algorithm hmac(md5)
Unable to handle kernel NULL pointer dereference at 0000000000000000 RIP:
 [<ffffffff8862c2e1>] :hmac:hmac_setkey+0xa4/0x137
PGD 0
Oops: 0000 [1] SMP
last sysfs file: /devices/pci0000:00/0000:00:03.0/0000:04:00.0/irq
CPU 0
Modules linked in: md5 hmac crypto_hash fips_cavs_test(U) testmgr aead crypto_blkcipher cryptomgr crypto_algapi ipv6 xfrm_nalgo crypto_api autofs4 hidp l2cap bluetooth nfs lockd fscache nfs_acl sunrpc ip_conntrack_netbios_ns ipt_REJECT xt_state ip_conntrack nfnetlink xt_tcpudp iptable_filter ip_tables x_tables dm_mirror dm_multipath dm_mod video sbs backlight i2c_ec button battery asus_acpi acpi_memhotplug ac lp snd_intel8x0 sg snd_ac97_codec ac97_bus snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq ide_cd snd_seq_device i2c_i801 snd_pcm_oss serio_raw floppy i2c_core e752x_edac pcspkr netxen_nic snd_mixer_oss snd_pcm e1000 snd_timer snd soundcore edac_mc snd_page_alloc cdrom parport_pc shpchp parport ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 2899, comm: cryptomgr_test Tainted: G      2.6.18-109.el5.fips2 #1
RIP: 0010:[<ffffffff8862c2e1>]  [<ffffffff8862c2e1>] :hmac:hmac_setkey+0xa4/0x137
RSP: 0000:ffff81003324dc50  EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff810034a2b5d0 RCX: 00000000000fdff1
RDX: 00000007eff88607 RSI: ffffffff88607b87 RDI: 0000000000000000
RBP: 0000000000000040 R08: 0000000000000050 R09: ffff810037b411c0
R10: ffff810037b41210 R11: ffffffff8862c23d R12: ffff810034a2b610
R13: ffff810034a2b650 R14: 0000000000000040 R15: 0000000000000010
FS:  0000000000000000(0000) GS:ffffffff803b6000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000000201000 CR4: 00000000000006e0
Process cryptomgr_test (pid: 2899, threadinfo ffff81003324c000, task ffff81003db11820)
Stack:  0000000000000000 0000000000000000 0000000000000000 0000000000000000
 ffff8100092c4968 0000000000000000 ffffffff8860cc68 ffff810034a2b580
 0000000000000005 ffff81003324ddd0 ffff81003dccf278 ffffffff88603798
Call Trace:
 [<ffffffff88603798>] :testmgr:alg_test_hash+0x1ec/0x3fb
 [<ffffffff8001a204>] vsnprintf+0x559/0x59e
 [<ffffffff8008b02a>] enqueue_task+0x41/0x56
 [<ffffffff8008b066>] __activate_task+0x27/0x39
 [<ffffffff88603536>] :testmgr:alg_test+0x10f/0x13a
 [<ffffffff88603561>] :testmgr:cryptomgr_test+0x0/0x4b
 [<ffffffff8009e7b3>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8860358c>] :testmgr:cryptomgr_test+0x2b/0x4b
 [<ffffffff800327e6>] kthread+0xfe/0x132
 [<ffffffff8005dfb1>] child_rip+0xa/0x11
 [<ffffffff8009e7b3>] keventd_create_kthread+0x0/0xc4
 [<ffffffff800326e8>] kthread+0x0/0x132
 [<ffffffff8005dfa7>] child_rip+0x0/0x11


Code: 48 8b 07 44 89 44 24 0c 48 6b d2 38 48 83 e0 fc 48 01 d0 48
RIP  [<ffffffff8862c2e1>] :hmac:hmac_setkey+0xa4/0x137
 RSP <ffff81003324dc50>

Comment 5 Neil Horman 2008-09-09 19:28:49 UTC
Ok, thanks, I'll get to this as soon as I can

Comment 6 Herbert Xu 2008-09-09 23:27:13 UTC
Weird, it works on my machine:

fips_cavs_test: loaded
fips_cavs_test: testing algorithm hmac(md5)
fips_cavs_test: algorithm hmac(md5) testing complete, ret 0

Can I login into the machine where you've got the tree that does this to check
whether the crypto patch series is identical to what I've got?

Thanks!

Comment 7 Herbert Xu 2008-09-09 23:31:12 UTC
Oh wait, I see what's going on.  We need this patch from upstream:

commit 67412f0e78dfbbbcb36e631d9df70c6c559d60d4
Author: Herbert Xu <herbert.org.au>
Date:   Tue May 6 20:46:49 2008 +0800

    [CRYPTO] hmac: Avoid calling virt_to_page on key

Comment 8 Herbert Xu 2008-09-09 23:38:16 UTC
Created attachment 316262 [details]
[CRYPTO] hmac: Avoid calling virt_to_page on key

Here is the back-port.

    [CRYPTO] hmac: Avoid calling virt_to_page on key

    When HMAC gets a key longer than the block size of the hash, it needs
    to feed it as input to the hash to reduce it to a fixed length.  As
    it is HMAC converts the key to a scatter and gather list.  However,
    this doesn't work on certain platforms if the key is not allocated
    via kmalloc.  For example, the keys from tcrypt are stored in the
    rodata section and this causes it to fail with HMAC on x86-64.

    This patch fixes this by copying the key to memory obtained via
    kmalloc before hashing it.

    Signed-off-by: Herbert Xu <herbert.org.au>

Comment 9 Neil Horman 2008-09-10 00:15:39 UTC
cool, thanks herbert, I'll get that posted tomorrow.

Comment 10 Herbert Xu 2008-09-10 00:25:25 UTC
Thanks Neil!

Comment 11 Jarod Wilson 2008-09-10 17:31:30 UTC
Even with that patch:

fips_cavs_test: loaded
fips_cavs_test: testing algorithm hmac(md5)
Unable to handle kernel NULL pointer dereference at 0000000000000005 RIP: 
 [<ffffffff801341e7>] update+0x4f/0x141
PGD 0 
Oops: 0000 [1] SMP 
last sysfs file: /module/libata/version
CPU 0 
Modules linked in: md5 hmac crypto_hash fips_cavs_test(U) testmgr aead crypto_blkcipher cryptomgr crypto_algapi ipv6 xfrm_nalgo crypto_api autofs4 hidp
 l2cap bluetooth nfs lockd fscache nfs_acl sunrpc ip_conntrack_netbios_ns ipt_REJECT xt_state ip_conntrack nfnetlink xt_tcpudp iptable_filter ip_tables
 x_tables dm_mirror dm_multipath dm_mod video sbs backlight i2c_ec button battery asus_acpi acpi_memhotplug ac lp sg snd_intel8x0 snd_ac97_codec ac97_b
us snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd floppy parport_pc pcspkr e1000 s
oundcore e752x_edac i2c_i801 snd_page_alloc edac_mc serio_raw parport netxen_nic ide_cd i2c_core cdrom shpchp ata_piix libata sd_mod scsi_mod ext3 jbd 
uhci_hcd ohci_hcd ehci_hcd
Pid: 4318, comm: cryptomgr_test Tainted: G      2.6.18-109.el5.fips3 #1
RIP: 0010:[<ffffffff801341e7>]  [<ffffffff801341e7>] update+0x4f/0x141
RSP: 0000:ffff8100302abbe0  EFLAGS: 00010286
RAX: ffff8100302abc90 RBX: ffff81003cdf1410 RCX: 00000000302abdd0
RDX: ffff81003cdf1910 RSI: ffff81003cdf1000 RDI: ffff81003cdf1930
RBP: ffff81003cdf1000 R08: 0000000094c15b42 R09: 00000000c391f680
R10: 000000000d275a80 R11: 000000002933dacf R12: 00000000ffff8100
R13: 0000000000001000 R14: ffff81003cdf18c0 R15: 0000000000000005
FS:  0000000000000000(0000) GS:ffffffff803a9000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000005 CR3: 0000000000201000 CR4: 00000000000006e0
Process cryptomgr_test (pid: 4318, threadinfo ffff8100302aa000, task ffff810037ebd080)
Stack:  000000503cdf1920 ffff8100302abc50 0000000000000002 ffff81003cdf1410
 ffff81003cdf18c0 0000000000000050 ffffffff8860eb87 0000000000000090
 0000000000000040 ffffffff88633722 00000010302abdd0 ffff81003cdf1450
Call Trace:
 [<ffffffff88633722>] :hmac:hmac_setkey+0x173/0x1ed
 [<ffffffff8860a798>] :testmgr:alg_test_hash+0x1ec/0x3fb
 [<ffffffff8008b02b>] enqueue_task+0x41/0x56
 [<ffffffff8008b067>] __activate_task+0x27/0x39
 [<ffffffff8860a536>] :testmgr:alg_test+0x10f/0x13a
 [<ffffffff8860a561>] :testmgr:cryptomgr_test+0x0/0x4b
 [<ffffffff8009e7b4>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8860a58c>] :testmgr:cryptomgr_test+0x2b/0x4b
 [<ffffffff800327e6>] kthread+0xfe/0x132
 [<ffffffff8005dfb1>] child_rip+0xa/0x11
 [<ffffffff8009e7b4>] keventd_create_kthread+0x0/0xc4
 [<ffffffff800326e8>] kthread+0x0/0x132
 [<ffffffff8005dfa7>] child_rip+0x0/0x11


Code: 49 8b 17 41 29 cd 45 39 e5 45 0f 47 ec 48 c1 ea 33 31 f6 48 
RIP  [<ffffffff801341e7>] update+0x4f/0x141
 RSP <ffff8100302abbe0>


On the bright side, backing out some xen pv-on-fv driver patches got kdump working again for me, so I do have a full vmcore this time...

Comment 12 Neil Horman 2008-09-10 20:32:28 UTC
Jarod, Is it ok If I hop on the machine your testing on?  I think this will be easier If I can build a few debug kernels and walk through this.  Thanks!

Comment 13 Herbert Xu 2008-09-11 00:40:58 UTC
Created attachment 316379 [details]
[CRYPTO] hmac: Avoid calling virt_to_page on key

Sorry, my back-port was buggy.  Here is the corrected version.

Comment 14 Herbert Xu 2008-09-11 00:43:08 UTC
Created attachment 316380 [details]
[CRYPTO] hmac: Avoid calling virt_to_page on key

That was bad too.  Here is the right one.

Comment 15 Jarod Wilson 2008-09-11 03:34:52 UTC
(In reply to comment #12)
> Jarod, Is it ok If I hop on the machine your testing on?  I think this will be
> easier If I can build a few debug kernels and walk through this.  Thanks!

I'd absolutely let ya hop on Neil, but I think there's no need. :)

I just threw together a kernel w/Herbert's updated patch in comment #14:

fips_cavs_test: loaded
fips_cavs_test: testing algorithm hmac(md5)
fips_cavs_test: algorithm hmac(md5) testing complete, ret 0

Looks like we're good to go there.

Comment 16 RHEL Program Management 2008-09-11 03:48:57 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

Comment 17 Neil Horman 2008-09-11 12:25:27 UTC
cool, thanks jarod.

Comment 18 Don Zickus 2008-09-15 14:18:59 UTC
in kernel-2.6.18-115.el5
You can download this test kernel from http://people.redhat.com/dzickus/el5

Comment 22 errata-xmlrpc 2009-01-20 20:18:17 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2009-0225.html