Marking as security because unauthenticated users can crash the server.

Version of 389-ds-base: 389-ds-base-1.2.11.15-60.el6.x86_64

Do we have the version info of NSS?

nss-3.19.1-3.el6_6.x86_64
nss-softokn-3.14.3-22.el6_6.x86_64

Hello Andrew,

1) We are interested in the processor you are running the Directory Server on.  Could you please provide the output of `uname -a` and the content of /proc/cpuinfo?

2) We would like to know setting the following environment variable(s) changes the behaviour.
2-1)
Open /etc/sysconfig/dirsrv and add the following line:
export NSS_DISABLE_HW_GCM=1
Restart the Directory Server.
Does the LDAP/TLS request crash the server?
2-2)
If the server crashes, add another variable to /etc/sysconfig/dirsrv:
export NSS_DISABLE_HW_AES=1
Restart the Directory Server.
Does the LDAP/TLS request crash the server?

Thanks.

1)
[root@zldap1 ~]# uname -a
Linux zldap1.zwiegnet.local 2.6.32-504.8.1.el6.x86_64 #1 SMP Wed Jan 28 21:11:36 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[root@zldap1 ~]# cat /proc/cpuinfo
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 21
model           : 1
model name      : AMD Ftm)-8120 Eight-Core Processor          
stepping        : 2
cpu MHz         : 3110.448
cache size      : 2048 KB
physical id     : 0
siblings        : 1
core id         : 4
cpu cores       : 1
apicid          : 0
initial apicid  : 4
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu de tsc msr pae cx8 cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm rep_good unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch xop perfctr_core perfctr_nb
bogomips        : 6220.89
TLB size        : 1536 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : AuthenticAMD
cpu family      : 21
model           : 1
model name      : AMD Ftm)-8120 Eight-Core Processor          
stepping        : 2
cpu MHz         : 3110.448
cache size      : 2048 KB
physical id     : 0
siblings        : 1
core id         : 4
cpu cores       : 1
apicid          : 0
initial apicid  : 4
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu de tsc msr pae cx8 cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm rep_good unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch xop perfctr_core perfctr_nb
bogomips        : 6220.89
TLB size        : 1536 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management:



2-1 no longer crashes the dir server
NSS_DISABLE_HW_GCM=1

2-2 no longer crashes the dir server
NSS_DISABLE_HW_AES=1

Thank you, Andrew.

The NSS expert gave us the explanation and the above instructions:

> The processor on the machine is claiming it can do AVX when it can't.
> We can get the user working securely by setting NSS_DISABLE_HW_GCM=1.
> That will keep NSS away from using any AVX instructions.

Please see also 
> http://www.felixcloutier.com/x86/MOVDQU.html

It seems "AMD Ftm)-8120 Eight-Core Processor" has the issue.  Since setting "NSS_DISABLE_HW_GCM=1" solves the crash issue, could you keep it in your sysconfig dirsrv file?  Please note that setting NSS_DISABLE_HW_AES=1 is not necessary.

It doesn't appear that the processor is advertising avx support.  That is, why does NSS think it has avx support?  For example, here is my laptop info which advertises aes, avx, and clmul:

flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt

It has aes, avx, avx2, and pclmulqdq

We're reading bit 28 of ecx from the cpuid command (numbered low to high starting at 0) for avx:

            freebl_cpuid(1, &eax, &ebx, &ecx, &edx);
            has_intel_aes = (ecx & (1 << 25)) != 0 ? 1 : -1;
#ifdef USE_HW_GCM
            disable_hw_gcm = getenv("NSS_DISABLE_HW_GCM");
            if (disable_hw_gcm == NULL) {
                has_intel_clmul = (ecx & (1 << 1)) != 0 ? 1 : -1;
                has_intel_avx = (ecx & (1 << 28)) != 0 ? 1 : -1;
            }


   use_hw_gcm = (PRBool)
                (use_hw_aes && has_intel_avx>0 && has_intel_clmul>0);

"Since setting "NSS_DISABLE_HW_GCM=1" solves the crash issue, could you keep it in your sysconfig dirsrv file?" Yes I can.

Will there be an official patch to fix this issue? I assume others may be using the processor, and likely could have the same issue.

Created attachment 1067827 [details]
Sample test program to see what cpuid is returning.

If the reporter could compile this program and paste it's outputs, we could get an idea what NSS is seeing from the cpu:

cc -o raw_cpuid raw_cpuid.c

./raw_cpuid

According to http://support.amd.com/TechDocs/25481.pdf we are checking the correct bit.

According to https://en.wikipedia.org/wiki/Bulldozer_%28microarchitecture%29 This processor should support AVX, I wonder why 1) it's failing, and 2) why the kernel knows to turn off AVX support on it.

bob

The server in question is a VM, running on XenServer 6.1

./raw_cpuid 
eax = 0x00600f12
ebx = 0x03080800
ecx = 0x1698220b
edx = 0x178bfbff

has_intel_aes = 1
has_intel_clmul = 1
has_intel_avx = 1
has_intel_avx = 1

use_hw_aes = 1
use_hw_gcm = 1

OK, so the processor can do avx, but Xen turns it off, but doesn't catch the cpu_id command. This is looking like a Xen issue.

OK, I've researched this some more, and it looks like we need to check whether or not not OS supports save and restore. Now RHEL 6 always supports save and restore, but it looks like Xen doesn't always support it.

The issue here is the code in question is FIPS, so the bar to change it is pretty high, and we do have a work around. I recommend fixing the issue upstream, creating a knowledge base doc describing the issue and the work around. When and if we require a softoken refresh with the relevant FIPS validation, we would then pick up the fix.

bob

So part 1 of my recommendation is already done. There is an upstream patch that handles this case. It's also in RHEL 7. When we do a RHEL 6 FIPS refresh, we could pick it up then. In the meantime we can release not the work around for RHEL 6 running under as a client under Xen.

bob

> release not the work around

That should be release note the work around

I was now able to duplicate this on ANOTHER PROCESSOR, seperate server. Same way to resolve, add  export NSS_DISABLE_HW_GCM=1 to /etc/sysconfig/dirsrv

I think there needs to be a RHEL/389 patch for this

[root@zldap2 ~]# cat /proc/cpuinfo 
processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 21
model		: 1
model name	: AMD FX(tm)-4130 Quad-Core Processor            
stepping	: 2
cpu MHz		: 3825.752
cache size	: 2048 KB
physical id	: 0
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu de tsc msr pae cx8 cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm rep_good unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch xop fma4 perfctr_core perfctr_nb
bogomips	: 7651.50
TLB size	: 1536 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 48 bits physical, 48 bits virtual
power management:

different XenServer Version as well, it could impact a broad range of hardware/software

(In reply to andyzwieg from comment #26)
> I was now able to duplicate this on ANOTHER PROCESSOR, seperate server. Same
> way to resolve, add  export NSS_DISABLE_HW_GCM=1 to /etc/sysconfig/dirsrv
> 
> I think there needs to be a RHEL/389 patch for this

There needs to be a Xenserver patch to return the correct processor codes.

There already is an NSS patch for Fedora and RHEL7 that will eventually make its way into RHEL6.

This isn't a 389 issue, it's just that 389 is one of the few servers that uses NSS for crypto.

Can we open this bug to the general public?  I don't think this is a CVE issue, and we have another 389 community user who is running into the same issue.

Yes, this bug should be opened so that others can find it until there's a release note on this.

It also should be assigned to NSS, though right now the only fix is the environment work around.

The environments that fail are RHEL6 running under Xen earilier than I think 4.5 and running on a processor with AVX support.