Bug 771459

Summary: [abrt] librdmacm-utils-1.0.10-4.fc15: load_driver: Process /usr/bin/rping was killed by signal 11 (SIGSEGV)
Product: [Fedora] Fedora Reporter: Doug Ledford <dledford>
Component: librdmacmAssignee: Doug Ledford <dledford>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 16CC: dledford, narf
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard: abrt_hash:0d9ba1b3d4570b3fa6de3c341273c860a6d97363
Fixed In Version: libibmad-1.3.8-1.fc16 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 740133 Environment:
Last Closed: 2012-01-15 20:03:48 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 740133    
Bug Blocks:    

Description Doug Ledford 2012-01-03 20:09:08 UTC
+++ This bug was initially created as a clone of Bug #740133 +++

abrt version: 2.0.3
architecture:   x86_64
backtrace_rating: 4
cmdline:        rping -c -a 192.168.231.1
component:      librdmacm
crash_function: load_driver
executable:     /usr/bin/rping
kernel:         2.6.40.4-5.fc15.x86_64
os_release:     Fedora release 15 (Lovelock)
package:        librdmacm-utils-1.0.10-4.fc15
reason:         Process /usr/bin/rping was killed by signal 11 (SIGSEGV)
time:           Tue Sep 20 23:59:22 2011
uid:            500
username:       narf

backtrace:
:[New LWP 2762]
:[Thread debugging using libthread_db enabled]
:Core was generated by `rping -c -a xxx.xxx.xxx.xxx'.
:Program terminated with signal 11, Segmentation fault.
:#0  load_driver (name=0x3 <Address 0x3 out of bounds>) at src/init.c:185
:185	src/init.c: No such file or directory.
:	in src/init.c
:
:Thread 1 (Thread 0x7fc9fe58d720 (LWP 2762)):
:#0  load_driver (name=0x3 <Address 0x3 out of bounds>) at src/init.c:185
:        so_name = <optimized out>
:        dlhandle = <optimized out>
:#1  0x0000003ec6006d90 in load_drivers () at src/init.c:231
:        name = 0x76c610
:        next_name = 0x0
:        env = <optimized out>
:        list = <optimized out>
:        env_name = <optimized out>
:#2  0x0000003ec600760a in ibverbs_init (list=0x3ec620bab0) at src/init.c:512
:        sysfs_path = <optimized out>
:        sysfs_dev = 0x0
:        next_dev = <optimized out>
:        device = <optimized out>
:        num_devices = 0
:        list_size = 0
:        statically_linked = 0
:        no_driver = 1
:        ret = <optimized out>
:#3  0x0000003ec6006474 in __ibv_get_device_list (num=0x7fff54301acc) at src/device.c:67
:        l = 0x0
:        i = <optimized out>
:#4  0x0000003ec5c01fb7 in ucma_init () at src/cma.c:226
:        dev_list = <optimized out>
:        cma_dev = <optimized out>
:        i = <optimized out>
:        ret = <optimized out>
:        dev_cnt = <optimized out>
:        attr = {fw_ver = "\000\000\000\000\000\000\000\000\020@v\000\000\000\000\000\310\035\060T\377\177\000\000\004", '\000' <repeats 15 times>, "\020\306v", '\000' <repeats 13 times>"\341, !\300\305>\000\000", node_guid = 262269962264, sys_image_guid = 284535810, max_mr_size = 140734606537320, page_size_cap = 262276349782, vendor_id = 0, vendor_part_id = 0, hw_ver = 4, max_qp = 0, max_qp_wr = 1412439072, device_cap_flags = 32767, max_sge = 286876800, max_sge_rd = 61, max_cq = 0, max_cqe = 0, max_mr = 7750040, max_pd = 0, max_qp_rd_atom = 0, max_ee_rd_atom = 0, max_res_rd_atom = 283928608, max_qp_init_rd_atom = 61, max_ee_init_rd_atom = 1412439312, atomic_cap = 32767, max_ee = 284002958, max_rdd = 61, max_mw = 286872032, max_raw_ipv6_qp = 61, max_raw_ethy_qp = 624, max_mcast_grp = 0, max_mcast_qp_attach = 0, max_total_mcast_qp_attach = 0, max_ah = 283627497, max_fmr = 61, max_map_per_fmr = 0, max_srq = 0, max_srq_wr = 1412439496, max_srq_sge = 32767, max_pkeys = 0, local_ca_ack_delay = 0 '\000', phys_port_cnt = 0 '\000'}
:#5  ucma_init () at src/cma.c:209
:        ret = 0
:#6  0x0000003ec5c0290d in rdma_create_id (channel=0x76c610, id=0x764260, context=0x764010, ps=RDMA_PS_TCP) at src/cma.c:397
:        resp = <optimized out>
:        cmd = <optimized out>
:        id_priv = <optimized out>
:        msg = <optimized out>
:        ret = <optimized out>
:        size = <optimized out>
:#7  0x00000000004018bc in main (argc=4, argv=0x7fff54301dc8) at examples/rping.c:1198
:        cb = 0x764010
:        op = <optimized out>
:        ret = 0
:        persistent_server = 0
:From                To                  Syms Read   Shared Object Library
:0x0000003d11605660  0x0000003d116103cc  Yes         /lib64/libpthread.so.0
:0x0000003ec5c01610  0x0000003ec5c040ec  Yes         /usr/lib64/librdmacm.so.1
:0x0000003ec6003340  0x0000003ec60090dc  Yes         /usr/lib64/libibverbs.so.1
:0x0000003d10e1ece0  0x0000003d10f4338c  Yes         /lib64/libc.so.6
:0x0000003d10600b20  0x0000003d10618cca  Yes         /lib64/ld-linux-x86-64.so.2
:0x0000003d11200de0  0x0000003d1120191c  Yes         /lib64/libdl.so.2
:0x00007fc9fe388550  0x00007fc9fe38b94c  Yes         /usr/lib64/libmlx4-rdmav2.so
:$1 = 0x0
:No symbol "__glib_assert_msg" in current context.
:rax            0x3ec6009180	269609898368
:rbx            0x3	3
:rcx            0x3	3
:rdx            0x3ec6009183	269609898371
:rsi            0x1	1
:rdi            0x3	3
:rbp            0x7fff54301970	0x7fff54301970
:rsp            0x7fff54301930	0x7fff54301930
:r8             0x90000002	2415919106
:r9             0x0	0
:r10            0x3d11200f26	262280318758
:r11            0x3d1081f000	262269956096
:r12            0x0	0
:r13            0x7	7
:r14            0x7642b0	7750320
:r15            0x7648d0	7751888
:rip            0x3ec6006c9e	0x3ec6006c9e <load_driver+30>
:eflags         0x10206	[ PF IF RF ]
:cs             0x33	51
:ss             0x2b	43
:ds             0x0	0
:es             0x0	0
:fs             0x0	0
:gs             0x0	0
:Dump of assembler code for function load_driver:
:   0x0000003ec6006c80 <+0>:	push   %rbx
:   0x0000003ec6006c81 <+1>:	lea    0x24f8(%rip),%rax        # 0x3ec6009180
:   0x0000003ec6006c88 <+8>:	lea    0x24f4(%rip),%rdx        # 0x3ec6009183
:   0x0000003ec6006c8f <+15>:	mov    %rdi,%rcx
:   0x0000003ec6006c92 <+18>:	mov    %rdi,%rbx
:   0x0000003ec6006c95 <+21>:	mov    $0x1,%esi
:   0x0000003ec6006c9a <+26>:	sub    $0x10,%rsp
:=> 0x0000003ec6006c9e <+30>:	cmpb   $0x2f,(%rdi)
:   0x0000003ec6006ca1 <+33>:	lea    0x8(%rsp),%rdi
:   0x0000003ec6006ca6 <+38>:	cmovne %rax,%rdx
:   0x0000003ec6006caa <+42>:	xor    %eax,%eax
:   0x0000003ec6006cac <+44>:	callq  0x3ec6002fe0 <__asprintf_chk@plt>
:   0x0000003ec6006cb1 <+49>:	test   %eax,%eax
:   0x0000003ec6006cb3 <+51>:	js     0x3ec6006ce0 <load_driver+96>
:   0x0000003ec6006cb5 <+53>:	mov    0x8(%rsp),%rdi
:   0x0000003ec6006cba <+58>:	mov    $0x2,%esi
:   0x0000003ec6006cbf <+63>:	callq  0x3ec6003210 <dlopen@plt>
:   0x0000003ec6006cc4 <+68>:	test   %rax,%rax
:   0x0000003ec6006cc7 <+71>:	je     0x3ec6006d10 <load_driver+144>
:   0x0000003ec6006cc9 <+73>:	mov    0x8(%rsp),%rdi
:   0x0000003ec6006cce <+78>:	callq  0x3ec6002fa0 <free@plt>
:   0x0000003ec6006cd3 <+83>:	add    $0x10,%rsp
:   0x0000003ec6006cd7 <+87>:	pop    %rbx
:   0x0000003ec6006cd8 <+88>:	retq   
:   0x0000003ec6006cd9 <+89>:	nopl   0x0(%rax)
:   0x0000003ec6006ce0 <+96>:	mov    0x204a79(%rip),%rax        # 0x3ec620b760
:   0x0000003ec6006ce7 <+103>:	lea    0x259a(%rip),%rdx        # 0x3ec6009288
:   0x0000003ec6006cee <+110>:	mov    %rbx,%rcx
:   0x0000003ec6006cf1 <+113>:	mov    $0x1,%esi
:   0x0000003ec6006cf6 <+118>:	mov    (%rax),%rdi
:   0x0000003ec6006cf9 <+121>:	xor    %eax,%eax
:   0x0000003ec6006cfb <+123>:	callq  0x3ec6002e90 <__fprintf_chk@plt>
:   0x0000003ec6006d00 <+128>:	add    $0x10,%rsp
:   0x0000003ec6006d04 <+132>:	pop    %rbx
:   0x0000003ec6006d05 <+133>:	retq   
:   0x0000003ec6006d06 <+134>:	nopw   %cs:0x0(%rax,%rax,1)
:   0x0000003ec6006d10 <+144>:	callq  0x3ec6003060 <dlerror@plt>
:   0x0000003ec6006d15 <+149>:	mov    %rax,%r8
:   0x0000003ec6006d18 <+152>:	mov    0x204a41(%rip),%rax        # 0x3ec620b760
:   0x0000003ec6006d1f <+159>:	lea    0x259a(%rip),%rdx        # 0x3ec60092c0
:   0x0000003ec6006d26 <+166>:	mov    %rbx,%rcx
:   0x0000003ec6006d29 <+169>:	mov    $0x1,%esi
:   0x0000003ec6006d2e <+174>:	mov    (%rax),%rdi
:   0x0000003ec6006d31 <+177>:	xor    %eax,%eax
:   0x0000003ec6006d33 <+179>:	callq  0x3ec6002e90 <__fprintf_chk@plt>
:   0x0000003ec6006d38 <+184>:	jmp    0x3ec6006cc9 <load_driver+73>
:End of assembler dump.

build_ids:
:00caa79d9b9399c8cecfe1e6b9b0937c74a24a17
:90fe13fac734263f663981ce4d1691cf926ced25
:a8eb20c22ee7a1fb41d7f54054348d960d8b377f
:180a52229d80449b98afbc5ce07be44ba029edd1
:aed03f94157330fb4df28843d1a20c5f13ad2b5c
:1d87720659528ad80e8870da2b2bb4af4470cf66
:06c8b720124f1072b9ab3bfac874757f4ed4e5a7
:c3bf0640bc3345ce7d088a9e8917ed5d7471e1b2

comment:
:set up two systems with infiniband
:run rping server on one system
:run rping client on the other, connect to server, it crashes

dso_list:
:/lib64/libdl-2.14.so glibc-2.14-5.x86_64 (Fedora Project) 1316295321
:/usr/lib64/librdmacm.so.1.0.0 librdmacm-1.0.10-4.fc15.x86_64 (Fedora Project) 1316554438
:/lib64/ld-2.14.so glibc-2.14-5.x86_64 (Fedora Project) 1316295321
:/usr/lib64/libmlx4-rdmav2.so libmlx4-1.0.1-8.fc15.x86_64 (Fedora Project) 1316554437
:/lib64/libpthread-2.14.so glibc-2.14-5.x86_64 (Fedora Project) 1316295321
:/lib64/libc-2.14.so glibc-2.14-5.x86_64 (Fedora Project) 1316295321
:/usr/bin/rping librdmacm-utils-1.0.10-4.fc15.x86_64 (Fedora Project) 1316554438
:/usr/lib64/libibverbs.so.1.0.0 libibverbs-1.1.3-5.fc15.x86_64 (Fedora Project) 1316554437

environ:
:ORBIT_SOCKETDIR=/tmp/orbit-narf
:XDG_SESSION_ID=1
:HOSTNAME=scruffy.tuneology.com
:IMSETTINGS_INTEGRATE_DESKTOP=yes
:GPG_AGENT_INFO=/tmp/keyring-RnS090/gpg:0:1
:TERM=xterm
:SHELL=/bin/bash
:HISTSIZE=1000
:XDG_SESSION_COOKIE=960cafd7769342edf580cd0a00000011-1316577044.924530-86909652
:GJS_DEBUG_OUTPUT=stderr
:WINDOWID=33554801
:GNOME_KEYRING_CONTROL=/tmp/keyring-RnS090
:QTDIR=/usr/lib64/qt-3.3
:QTINC=/usr/lib64/qt-3.3/include
:'GJS_DEBUG_TOPICS=JS ERROR;JS LOG'
:IMSETTINGS_MODULE=none
:USER=narf
:LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
:SSH_AUTH_SOCK=/tmp/keyring-RnS090/ssh
:USERNAME=narf
:SESSION_MANAGER=local/unix:@/tmp/.ICE-unix/2223,unix/unix:/tmp/.ICE-unix/2223
:PATH=/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/narf/bin
:MAIL=/var/spool/mail/narf
:DESKTOP_SESSION=gnome
:QT_IM_MODULE=xim
:PWD=/roberto/narf/opencv/demo
:XMODIFIERS=@im=none
:GNOME_KEYRING_PID=2215
:LANG=en_US.UTF-8
:KDE_IS_PRELINKED=1
:GDM_LANG=
:KDEDIRS=/usr
:GDMSESSION=gnome
:HISTCONTROL=ignoredups
:HOME=/home/narf
:SHLVL=2
:GNOME_DESKTOP_SESSION_ID=this-is-deprecated
:LOGNAME=narf
:QTLIB=/usr/lib64/qt-3.3/lib
:CVS_RSH=ssh
:DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-b3HGuXBeRw,guid=a784b9dbfe48b5ebbba140350000002a
:'LESSOPEN=||/usr/bin/lesspipe.sh %s'
:WINDOWPATH=1
:XDG_RUNTIME_DIR=/run/user/narf
:DISPLAY=:0
:G_BROKEN_FILENAMES=1
:COLORTERM=gnome-terminal
:XAUTHORITY=/var/run/gdm/auth-for-narf-y0GuPI/database
:_=/usr/bin/rping
:OLDPWD=/home/narf

maps:
:00400000-00405000 r-xp 00000000 fd:01 1459450                            /usr/bin/rping
:00604000-00606000 rw-p 00004000 fd:01 1459450                            /usr/bin/rping
:00764000-00785000 rw-p 00000000 00:00 0                                  [heap]
:3d10600000-3d1061f000 r-xp 00000000 fd:01 1051198                        /lib64/ld-2.14.so
:3d1081e000-3d1081f000 r--p 0001e000 fd:01 1051198                        /lib64/ld-2.14.so
:3d1081f000-3d10820000 rw-p 0001f000 fd:01 1051198                        /lib64/ld-2.14.so
:3d10820000-3d10821000 rw-p 00000000 00:00 0 
:3d10e00000-3d10f8f000 r-xp 00000000 fd:01 1051199                        /lib64/libc-2.14.so
:3d10f8f000-3d1118f000 ---p 0018f000 fd:01 1051199                        /lib64/libc-2.14.so
:3d1118f000-3d11193000 r--p 0018f000 fd:01 1051199                        /lib64/libc-2.14.so
:3d11193000-3d11194000 rw-p 00193000 fd:01 1051199                        /lib64/libc-2.14.so
:3d11194000-3d1119a000 rw-p 00000000 00:00 0 
:3d11200000-3d11202000 r-xp 00000000 fd:01 1051202                        /lib64/libdl-2.14.so
:3d11202000-3d11402000 ---p 00002000 fd:01 1051202                        /lib64/libdl-2.14.so
:3d11402000-3d11403000 r--p 00002000 fd:01 1051202                        /lib64/libdl-2.14.so
:3d11403000-3d11404000 rw-p 00003000 fd:01 1051202                        /lib64/libdl-2.14.so
:3d11600000-3d11616000 r-xp 00000000 fd:01 1051200                        /lib64/libpthread-2.14.so
:3d11616000-3d11815000 ---p 00016000 fd:01 1051200                        /lib64/libpthread-2.14.so
:3d11815000-3d11816000 r--p 00015000 fd:01 1051200                        /lib64/libpthread-2.14.so
:3d11816000-3d11817000 rw-p 00016000 fd:01 1051200                        /lib64/libpthread-2.14.so
:3d11817000-3d1181b000 rw-p 00000000 00:00 0 
:3ec5c00000-3ec5c05000 r-xp 00000000 fd:01 1463524                        /usr/lib64/librdmacm.so.1.0.0
:3ec5c05000-3ec5e05000 ---p 00005000 fd:01 1463524                        /usr/lib64/librdmacm.so.1.0.0
:3ec5e05000-3ec5e06000 rw-p 00005000 fd:01 1463524                        /usr/lib64/librdmacm.so.1.0.0
:3ec6000000-3ec600c000 r-xp 00000000 fd:01 1485121                        /usr/lib64/libibverbs.so.1.0.0
:3ec600c000-3ec620b000 ---p 0000c000 fd:01 1485121                        /usr/lib64/libibverbs.so.1.0.0
:3ec620b000-3ec620c000 rw-p 0000b000 fd:01 1485121                        /usr/lib64/libibverbs.so.1.0.0
:7fc9fe387000-7fc9fe38d000 r-xp 00000000 fd:01 1463522                    /usr/lib64/libmlx4-rdmav2.so
:7fc9fe38d000-7fc9fe58c000 ---p 00006000 fd:01 1463522                    /usr/lib64/libmlx4-rdmav2.so
:7fc9fe58c000-7fc9fe58d000 rw-p 00005000 fd:01 1463522                    /usr/lib64/libmlx4-rdmav2.so
:7fc9fe58d000-7fc9fe591000 rw-p 00000000 00:00 0 
:7fc9fe5c4000-7fc9fe5c5000 rw-p 00000000 00:00 0 
:7fff542e3000-7fff54304000 rw-p 00000000 00:00 0                          [stack]
:7fff543af000-7fff543b0000 r-xp 00000000 00:00 0                          [vdso]
:ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

var_log_messages:
:Sep 20 23:59:22 scruffy kernel: [  559.433198] rping[2762]: segfault at 3 ip 0000003ec6006c9e sp 00007fff54301930 error 4 in libibverbs.so.1.0.0[3ec6000000+c000]
:Sep 20 23:59:22 scruffy abrt[2763]: saved core dump of pid 2762 (/usr/bin/rping) to /var/spool/abrt/ccpp-2011-09-20-23:59:22-2762.new/coredump (446464 bytes)

Comment 1 Fedora Update System 2012-01-03 20:11:45 UTC
libibmad-1.3.8-1.fc16, libmlx4-1.0.2-3.fc16, libcxgb3-1.3.1-1.fc16, librdmacm-1.0.15-1.fc16, libibverbs-1.1.6-1.fc16, libnes-1.1.3-1.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/libibmad-1.3.8-1.fc16,libmlx4-1.0.2-3.fc16,libcxgb3-1.3.1-1.fc16,librdmacm-1.0.15-1.fc16,libibverbs-1.1.6-1.fc16,libnes-1.1.3-1.fc16

Comment 2 Fedora Update System 2012-01-04 01:59:56 UTC
Package libibmad-1.3.8-1.fc16, libmlx4-1.0.2-3.fc16, libcxgb3-1.3.1-1.fc16, librdmacm-1.0.15-1.fc16, libibverbs-1.1.6-1.fc16, libnes-1.1.3-1.fc16, opensm-3.3.12-1.fc16:
* should fix your issue,
* was pushed to the Fedora 16 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing libibmad-1.3.8-1.fc16 libmlx4-1.0.2-3.fc16 libcxgb3-1.3.1-1.fc16 librdmacm-1.0.15-1.fc16 libibverbs-1.1.6-1.fc16 libnes-1.1.3-1.fc16 opensm-3.3.12-1.fc16'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-0092/libibmad-1.3.8-1.fc16,libmlx4-1.0.2-3.fc16,libcxgb3-1.3.1-1.fc16,librdmacm-1.0.15-1.fc16,libibverbs-1.1.6-1.fc16,libnes-1.1.3-1.fc16,opensm-3.3.12-1.fc16
then log in and leave karma (feedback).

Comment 3 Fedora Update System 2012-01-15 20:03:48 UTC
libibmad-1.3.8-1.fc16, libmlx4-1.0.2-3.fc16, libcxgb3-1.3.1-1.fc16, librdmacm-1.0.15-1.fc16, libibverbs-1.1.6-1.fc16, libnes-1.1.3-1.fc16, opensm-3.3.12-1.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.