Bug 990006

Summary: nfs: openssl compilation fails
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Saurabh <saujain>
Component: glusterdAssignee: santosh pradhan <spradhan>
Status: CLOSED CURRENTRELEASE QA Contact: Sudhir D <sdharane>
Severity: high Docs Contact:
Priority: medium    
Version: 2.1CC: mzywusko, rhs-bugs, rjoseph, saujain, spradhan, vagarwal, vbellur
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-09-02 11:11:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Saurabh 2013-07-30 09:05:12 UTC
Description of problem:

openssl compilation fails 
openssl is part of fs-sanity test suite

[root@nfs2 ~]# gluster volume status dist-rep
Status of volume: dist-rep
Gluster process						Port	Online	Pid
------------------------------------------------------------------------------
Brick 10.70.37.123:/rhs/bricks/d1r1			49155	Y	17319
Brick 10.70.37.121:/rhs/bricks/d1r2			49155	Y	22164
Brick 10.70.37.196:/rhs/bricks/d2r1			49155	Y	22065
Brick 10.70.37.118:/rhs/bricks/d2r2			49155	Y	21948
Brick 10.70.37.123:/rhs/bricks/d3r1			49156	Y	17330
Brick 10.70.37.121:/rhs/bricks/d3r2			49156	Y	22175
Brick 10.70.37.196:/rhs/bricks/d4r1			49156	Y	22076
Brick 10.70.37.118:/rhs/bricks/d4r2			49156	Y	21959
Brick 10.70.37.123:/rhs/bricks/d5r1			49157	Y	17341
Brick 10.70.37.121:/rhs/bricks/d5r2			49157	Y	22186
Brick 10.70.37.196:/rhs/bricks/d6r1			49157	Y	22087
Brick 10.70.37.118:/rhs/bricks/d6r2			49157	Y	21970
NFS Server on localhost					2049	Y	22155
Self-heal Daemon on localhost				N/A	Y	22162
NFS Server on 10.70.37.123				2049	Y	17571
Self-heal Daemon on 10.70.37.123			N/A	Y	17579
NFS Server on 10.70.37.196				2049	Y	22269
Self-heal Daemon on 10.70.37.196			N/A	Y	22276
NFS Server on 10.70.37.121				2049	Y	22368
Self-heal Daemon on 10.70.37.121			N/A	Y	22376
 
There are no active volume tasks

Version-Release number of selected component (if applicable):
[root@nfs2 ~]# rpm -qa | grep glusterfs
glusterfs-3.4.0.13rhs-1.el6rhs.x86_64
glusterfs-fuse-3.4.0.13rhs-1.el6rhs.x86_64
glusterfs-server-3.4.0.13rhs-1.el6rhs.x86_64
[root@nfs2 ~]# 


How reproducible:
already happened thrice on the same built

Steps to Reproduce:
1. create a 6x2 volume, start it
2. mount over nfs on an RHEL 6.1 client
3. execute the openssl compilation script from fs-sanity test suite


Actual results:
make[1]: Entering directory `/mnt/nfs-test/run3888/openssl-1.0.0d/test'
gcc -I.. -I../include  -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -m64 -DL_ENDIAN -DTERMIO -O3 -Wall -DMD32_REG_T=int -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DWHIRLPOOL_ASM   -c -o bntest.o bntest.c
make[2]: Entering directory `/mnt/nfs-test/run3888/openssl-1.0.0d/test'
( :; LIBDEPS="${LIBDEPS:--L.. -lssl -L.. -lcrypto  -ldl}"; LDCMD="${LDCMD:-gcc}"; LDFLAGS="${LDFLAGS:--DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -m64 -DL_ENDIAN -DTERMIO -O3 -Wall -DMD32_REG_T=int -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DWHIRLPOOL_ASM}"; LIBPATH=`for x in $LIBDEPS; do echo $x; done | sed -e 's/^ *-L//;t' -e d | uniq`; LIBPATH=`echo $LIBPATH | sed -e 's/ /:/g'`; LD_LIBRARY_PATH=$LIBPATH:$LD_LIBRARY_PATH ${LDCMD} ${LDFLAGS} -o ${APPNAME:=bntest} bntest.o ${LIBDEPS} )
/usr/lib/gcc/x86_64-redhat-linux/4.4.5/../../../../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: ld returned 1 exit status
make[2]: *** [link_app.] Error 1
make[2]: Leaving directory `/mnt/nfs-test/run3888/openssl-1.0.0d/test'
make[1]: *** [bntest] Error 2
make[1]: Leaving directory `/mnt/nfs-test/run3888/openssl-1.0.0d/test'
make: *** [build_tests] Error 1


Expected results:
The issue should not happen, as linux kernl untar and compilation passes on same build and mount point

Additional info:

Presently I have seen this issue with RHEL 6.1 client, not tried the same on other RHEL clients while keeping the glusterfs rpms same on server side.

Comment 2 santosh pradhan 2013-07-30 10:50:17 UTC
Thanks Saurabh for sharing your setup. I could compile the openssl bundle in the NFS mount point successfully. Could you check it again?

Comment 3 rjoseph 2013-07-31 06:24:34 UTC
Also make sure that all the build dependencies are taken care in the test setup. You can verify this by just running openssl build on the local filesystem.

Comment 4 santosh pradhan 2013-07-31 11:24:37 UTC
Agree with Rajesh.

The error is something with GCC compiler flags i.e. whenever the program does not have the "main"/_start routine, crt ll catch it unless the proper flag (nostart or so) is passed to gcc.

If the compilation passes with native/local file system, we need to look into it.

Comment 7 santosh pradhan 2013-08-01 11:17:29 UTC
In rhsauto036, Inside /mnt/nfs-test/openssl-1.0.0d, I ran the make test and passes. So the NFS looks safe :-)

Comment 8 santosh pradhan 2013-08-01 15:35:10 UTC
This does not look like a NFS issue. What do you say?

Comment 9 rjoseph 2013-08-02 05:12:24 UTC
From the initial look it looks like during the untar operation on NFS mount some of the files created are empty (zero byte) with no file access attributes.

Is it similar to the bug we opened for untar?

Comment 10 santosh pradhan 2013-08-06 11:50:36 UTC
When I ran the test with the script (time /opt/qa/tools/system_light/run.sh -w /mnt/glnfs/ -t openssl -l /home/openssl.log), I found some of the files are of 0 size which causes the compilation failure.

But when I do it manually using following command, no FAILURE is seen. 

command for manual test:

rm -rf /mnt/glnfs/openssl-1.0.1e*; umount -f /mnt/glnfs; mount -t nfs -o vers=3,tcp,noatime 10.70.43.16:/nfstest /mnt/glnfs; cd /mnt/glnfs; wget http://www.openssl.org/source/openssl-1.0.1e.tar.gz; tar xvf openssl-1.0.1e.tar.gz; cd openssl-1.0.1e; ./config; make; make test

Does not look like a NFS issue. May be the script or the open-ssl bundle being used is problematic? I am not sure.

What do you say?

Comment 11 rjoseph 2013-09-02 07:07:38 UTC
Ran the openssl script as shown below:

/opt/qa/tools/system_light/run.sh -w /mnt/nfs/openssl/ -t openssl -l /mnt/openssl.log

The test passed without any error.

Server version: glusterfs 3.4.0.17rhs

Comment 12 santosh pradhan 2013-09-02 11:05:28 UTC
I tried with glusterfs latest build i.e.3.4.0.30rhs-2 and RHEL 6.1 client. Test PASSED. 

[root@localhost ~]# /opt/qa/tools/system_light/run.sh -w /mnt/glnfs/ -t openssl -l /home/openssl.log
/opt/qa/tools/system_light/scripts
/opt/qa/tools/system_light
Tests available:
arequal
bonnie
compile_kernel
dbench
dd
ffsb
fileop
fs_mark
fsx
glusterfs_build
iozone
locks
ltp
multiple_files
openssl
posix_compliance
postmark
read_large
rpc
syscallbench
tiobench
===========================TESTS RUNNING===========================
Changing to the specified mountpoint
/mnt/glnfs/run1136
executing openssl
untarring the openssl tarball
start:21:05:12

real	0m12.363s
user	0m0.427s
sys	0m0.948s
executing ./config:21:05:25

real	0m5.680s
user	0m3.056s
sys	0m0.896s
executing make:21:05:30

real	11m23.625s
user	1m53.461s
sys	0m43.078s
executing make test:21:16:54

real	1m39.244s
user	0m27.438s
sys	0m13.188s
removed openssl directories and files
1
Total 1 tests were successful
Switching over to the previous working directory
Removing /mnt/glnfs//run1136/


Saurabh,
Could you have a look at this?

Thanks,
Santosh

Comment 13 santosh pradhan 2013-09-02 11:11:31 UTC
As this works with the latest build (The test combination: Gluster FS version 3.4.0.30rhs-2, RHEL 6.1 client and test ran through QA script i.e. /opt/qa/tools/system_light/run.sh -w /mnt/glnfs/ -t openssl -l /home/openssl.log). I am closing the defect with "Works with current release".