They should be available in the next rawhide build.

Testing of an image built with this kernel in openQA, it looks like this change often breaks the boot:

https://openqa.stg.fedoraproject.org/tests/overview?version=Rawhide&groupid=1&build=Fedora-Rawhide-20180801.n.0-KERNELFIX-NOREPORT&distri=fedora

all the failures in '_boot_to_anaconda' (but not failures in '_console_wait_login' or '_do_install_and_reboot') look to be caused by a kernel issue that's clearly related to this. e.g.:

https://openqa.stg.fedoraproject.org/tests/335896#step/_boot_to_anaconda/12

and:

https://openqa.stg.fedoraproject.org/tests/335898#step/_boot_to_anaconda/13

and:

https://openqa.stg.fedoraproject.org/tests/335902#step/_boot_to_anaconda/14

and so on. Extracting the clean trace is actually rather difficult, I'm afraid - I can't just direct the kernel logs out over the serial line because if you boot with console=ttyS0 , anaconda notices, and goes down a different boot path which confuses openQA...

I'm trying to reproduce this locally, but haven't yet, I've booted a VM configured similarly to the openQA ones several times and haven't hit the bug yet. Trying to figure out what the difference is now. (It could *possibly* have to do with running multiple VMs similtaneously, I guess?)

These AEAD modules are not used anywhere yet (and they should not been even included in initramfs then).

This change alone cannot cause any breakage. I guess some related change in kernel in the same build is the isuue.

Ok, seems it is the Aegis code...

There are two problems then:

 - why is that module loaded at all when nobody use it yet

 - it crashes in in aegis128_aesni_init
   https://openqa.stg.fedoraproject.org/tests/335896#step/_boot_to_anaconda/12

Ondra, any idea what is going here?

(In reply to Milan Broz from comment #4)
> Ok, seems it is the Aegis code...
> 
> There are two problems then:
> 
>  - why is that module loaded at all when nobody use it yet

That's a good question... Perhaps openQA has some scripts to detect and attempt loading of all crypto modules, in order to force their testmgr tests to be run (to discover problems)? It apparently goes in alphabetic order, since there are almost no other crypto modules loaded at the point of the crash.

> 
>  - it crashes in in aegis128_aesni_init
>   
> https://openqa.stg.fedoraproject.org/tests/335896#step/_boot_to_anaconda/12
> 
> Ondra, any idea what is going here?

Hm... it is crashing at the first AESENC instruction, so my initial guess is that this is some problem with QEMU and CPUID flags. The AEGIS AES-NI modules check the CPU flags upon load and error out if required CPU features (in this case SSE2 and AES-NI) are not supported.

In that case other AES-NI crypto modules should also cause the same crash (if AEGIS didn't beat them in the sorting...). If not, then I must have messed something up in the checks probably (but I don't see how)... Perhaps there is some blacklist somewhere internally that prevents the loading of (known) AES-NI modules at boot, because the openQA people weren't able to setup their QEMU VMs properly?

Not sure if important, but seems that aes-ni is the only module compiled-in on x86_64, all other aes-ni modules are real modules.
 CONFIG_CRYPTO_AES_NI_INTEL=y

(In reply to Ondrej Mosnacek from comment #5)
> In that case other AES-NI crypto modules should also cause the same crash
> (if AEGIS didn't beat them in the sorting...). If not, then I must have
> messed something up in the checks probably (but I don't see how)... Perhaps
> there is some blacklist somewhere internally that prevents the loading of
> (known) AES-NI modules at boot, because the openQA people weren't able to
> setup their QEMU VMs properly?

No, wait, I really messed something up... On my kernel testing VM (only) the AEGIS AES-NI algs also load even when AES-NI is not supported... I'll look into this.

(In reply to Milan Broz from comment #6)
> Not sure if important, but seems that aes-ni is the only module compiled-in
> on x86_64, all other aes-ni modules are real modules.
>  CONFIG_CRYPTO_AES_NI_INTEL=y

I don't think that matters, I have almost everything set to =y on my test kernel and it still affects only AEGIS.

Right, I see where the problem is now. I misunderstood the x86_match_cpu() function. It matches when *any* of the items match, not when *all* match. I'll try to come up with a fix and post it upstream. In the meantime, the following configs should be set back to N to avoid crashes:

CONFIG_CRYPTO_AEGIS128_AESNI_SSE2=n
CONFIG_CRYPTO_AEGIS128L_AESNI_SSE2=n
CONFIG_CRYPTO_AEGIS256_AESNI_SSE2=n

I posted a patch that should fix the issue upstream:

https://patchwork.kernel.org/patch/10553369/

Reopening it, this simple change apparenty uncovered more serious bug and we need also the patch mentioned in comment#10.

Should we wait until the upstream merges it or Fedora kernel can temporarily use the patch above together with config change?

Given it broke things once, I'd like to wait until it's at least been taken into the maintainers tree and gotten some acks. Once that happens we can see about bringing it in.

"That's a good question... Perhaps openQA has some scripts to detect and attempt loading of all crypto modules,"

There is nothing like this. At the time this crash occurred, literally all openQA had done was boot up the system with the test ISO attached and select the 'Install Fedora' option from the bootloader menu. That's *it*. The traceback then occurred during anaconda startup.

I don't know what the 'cryptomgr_test' referred to in the traceback is, but it's nothing specific to openQA. It's something that, for some reason, runs as part of the process of booting a normal Fedora installation image, normally.

Looking through kernel source, cryptomgr_test is part of the kernel:

https://github.com/torvalds/linux/blob/abc36be236358162202e86ad88616ff95a755101/crypto/algboss.c#L210-L230

and is called through crypto_wait_for_test -> cryptomgr_notify -> cryptomgr_schedule_test -> cryptomgr_test whenever a crypto algorithm is registered by crypto_register_alg in https://github.com/torvalds/linux/blob/abc36be236358162202e86ad88616ff95a755101/crypto/algapi.c#L354-L374 .

> traceback then occurred during anaconda startup.

I wonder if anaconda loads them for HW crypto acceleration for on disk encryption or something to do with cryptsetup in prep for if a user will want to install to an encrypted disk.

Sorry, to be clearer: I can't say that it happens *specifically when anaconda is starting up*. Just that's the thing that happens when you boot an installer image.

I actually think it's just part of normal kernel init, and this would likely happen booting a live image with the same kernel on the same VMs, for instance.

No, it should not be tun in kernel init (unless it is compiled-in to monolitic kernel).

These Aegis/Morus AEAD (AEAD = authenticated encryption with additional data) are NOT used anywhere yet. Disk encryption uses solely length-preserving modes (AES with XTS,CBC mode). (Yes, I want to use them for disk encryption in future, that is actually the reason for this bug. But for now, NOTHING should load it.)

That's why I am so puzzled by this crash on all systems - these modules should not be loaded at all! Kernel have cipher vectors for tests, but these are run once the algorithm is requested, while module is being initialized, not on kernel init.

I guess we just see another bug here :) It would be interesting to trace it, but for now we should perhaps wait fore that AES-NI init patch.

Anyway, thanks for your time here!

Did you examine the codepath I just described? I didn't go back any further than that, but just looking at the function names, it looks a lot like the thing that's crashing (cryptomgr_test) is run whenever a crypto algorithm is 'registered', i.e. crypto_register_alg() in algapi.c . Which it seems reasonable to guess is something done for all crypto algorithms.

Note that when the bug happens, there is no formatted storage on the system at all. There is only the 'CD' it's booting from, and a 'disk' that is at this point blank and unformatted.

I think I figured out why the modules are loading automatically at boot...

When writing the x86_64 optimized implementations, I looked at how some of the existing modules are implemented and tried to adapt the code I found there. The modules I looked at (mainly aesni_intel) contained 'MODULE_DEVICE_TABLE' declarations with structures describing the required CPU features. Assuming this is common boilerplate, I tried to replicate the same for AEGIS and MORUS.

However, it seems the semantic of that declaration is 'automatically load this module at boot if the CPU features match' and that only some modules actually specify this declaration. It doesn't seem to be related to the on-demand module auto-loading performed by the crypto API (which uses just MODULE_ALIAS_CRYPTO).

So I should go back and remove those declarations completely, as they would just cause premature and unnecessary preloading like here...

Sorry about this whole mess and pointing fingers at openQA, it was my fault after all :)

In the meantime I have submitted a new patch that also fixes the autoloading issue (which is present in MORUS, too). [1]

Since Linus has put off the 4.18 release by another week [2], there is a good chance this will get fixed within this cycle and will not need to go through the stable release backporting process.

[1] https://patchwork.kernel.org/patch/10555031/
[2] https://lkml.org/lkml/2018/8/5/150

I tested it with locally built Fedora kernel with patch from comment#20 and it indeed works as expected now (all ciphers built as modules).

So let's wait for upstream to merge the patch and then try to enable it again, it should work now ;-)

Patch is upstream https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=877ccce7cbe8409256616f5e6bdedb08ce2e82db

Once it hits Fedora package source, could you please enable these modules again?

Thanks!

I see that 4.18 and 4.19-rc0 is already built, could we please reenable these cipher options? All problems mentioned earlier should be fixed in the 4.18 final.

There are crucial for authenticated disk encryption in future.

Thanks!

I've turned them back on for Rawhide and the F29 branch.