Bug 1403971

Summary: /usr/bin/more crash in end_it due to double free
Product: Red Hat Enterprise Linux 7 Reporter: Paulo Andrade <pandrade>
Component: util-linuxAssignee: Karel Zak <kzak>
Status: CLOSED ERRATA QA Contact: Radka Brychtova <rskvaril>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 7.2CC: bblaskov, pandrade
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: util-linux-2.23.2-36.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-01 21:41:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Paulo Andrade 2016-12-12 19:00:28 UTC
Backtrace looks like this:

Core was generated by `more ld.so.cache'.
Program terminated with signal 6, Aborted.
#0  0x00007fd618cc45f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56	  return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt
#0  0x00007fd618cc45f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007fd618cc5ce8 in __GI_abort () at abort.c:90
#2  0x00007fd618d04317 in __libc_message (do_abort=do_abort@entry=2, 
    fmt=fmt@entry=0x7fd618e0da28 "*** Error in `%s': %s: 0x%s ***\n")
    at ../sysdeps/unix/sysv/linux/libc_fatal.c:196
#3  0x00007fd618d0c023 in malloc_printerr (ar_ptr=0x7fd619049760 <main_arena>, ptr=<optimized out>, 
    str=0x7fd618e0db28 "double free or corruption (!prev)", action=3) at malloc.c:5018
#4  _int_free (av=0x7fd619049760 <main_arena>, p=<optimized out>, have_lock=0) at malloc.c:3842
#5  0x0000000000403afe in end_it (dummy=dummy@entry=0) at text-utils/more.c:777
#6  0x0000000000405e87 in command (filename=filename@entry=0x0, f=f@entry=0xa2dda0) at text-utils/more.c:1263
#7  0x0000000000406664 in screen (f=f@entry=0xa2dda0, num_lines=<optimized out>, num_lines@entry=23)
    at text-utils/more.c:708
#8  0x0000000000402365 in main (argc=<optimized out>, argv=<optimized out>) at text-utils/more.c:503

  The end_it function should either have a double call
guard in case on high system load, or set to NULL global
variables it free.

Comment 3 Paulo Andrade 2016-12-14 11:34:49 UTC
Steps to reproduce 1$ stands for terminal 1, 2$ terminal 2:

1$ cd /etc
1$ /usr/bin/more ld.so.cache

2$ gdb -p $(pidof more)
(gdb) b end_it
(gdb) handle SIGINT nostop noprint pass
(gdb) c

1$ <<Press q>>

--- screenshot of 2$ ---
Breakpoint 1, end_it (dummy=0) at text-utils/more.c:680
680	{
(gdb) n
681		reset_tty();
(gdb) 
682		if (clreol) {
(gdb) 
686		} else if (!clreol && (promptlen > 0)) {
(gdb) 
687			kill_line();
(gdb) 
688			fflush(stdout);
(gdb) 
691		free(previousre);
(gdb) 
692		free(Line);
(gdb) 
---

1$ <<Press ^C>>

2$ Tell gdb to continue

What will happen is that end_it will execute again, from
the signal handler.

Below from quick fc25 test:
---
(gdb) b end_it
Breakpoint 1 at 0x55d8a2727d00: file text-utils/more.c, line 680.
(gdb) handle SIGINT nostop noprint pass
SIGINT is used by the debugger.
Are you sure you want to change it? (y or n) y
Signal        Stop	Print	Pass to program	Description
SIGINT        No	No	Yes		Interrupt
(gdb) c
Continuing.

Breakpoint 1, end_it (dummy=0) at text-utils/more.c:680
680	{
(gdb) n
681		reset_tty();
(gdb) 
682		if (clreol) {
(gdb) 
686		} else if (!clreol && (promptlen > 0)) {
(gdb) 
687			kill_line();
(gdb) 
688			fflush(stdout);
(gdb) 
691		free(previousre);
(gdb) 
692		free(Line);
(gdb) 
693		_exit(EXIT_SUCCESS);
(gdb) c
Continuing.

Breakpoint 1, end_it (dummy=2) at text-utils/more.c:680
680	{
(gdb) n
681		reset_tty();
(gdb) 
682		if (clreol) {
(gdb) 
686		} else if (!clreol && (promptlen > 0)) {
(gdb) 
690			putcerr('\n');
(gdb) 
691		free(previousre);
(gdb) 
692		free(Line);
(gdb) 

Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:58
58	}


and on the first terminal:
^C
*** Error in `/usr/bin/more': double free or corruption (!prev): 0x000055d8a3fb7200 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7925b)[0x7f300f3b525b]
/lib64/libc.so.6(+0x828ea)[0x7f300f3be8ea]
/lib64/libc.so.6(cfree+0x4c)[0x7f300f3c231c]
/usr/bin/more(+0x3d4b)[0x55d8a2727d4b]
/lib64/libc.so.6(+0x359a0)[0x7f300f3719a0]
/usr/bin/more(+0x3d4b)[0x55d8a2727d4b]
/usr/bin/more(+0x623f)[0x55d8a272a23f]
/usr/bin/more(+0x6a66)[0x55d8a272aa66]
/usr/bin/more(+0x23a0)[0x55d8a27263a0]
/lib64/libc.so.6(__libc_start_main+0xf1)[0x7f300f35c401]
/usr/bin/more(+0x24ca)[0x55d8a27264ca]
======= Memory map: ========
55d8a2724000-55d8a272d000 r-xp 00000000 fd:00 1967010                    /usr/bin/more
55d8a292c000-55d8a292d000 r--p 00008000 fd:00 1967010                    /usr/bin/more
55d8a292d000-55d8a292e000 rw-p 00009000 fd:00 1967010                    /usr/bin/more
55d8a3fb6000-55d8a3fd7000 rw-p 00000000 00:00 0                          [heap]
7f3004000000-7f3004021000 rw-p 00000000 00:00 0 
7f3004021000-7f3008000000 ---p 00000000 00:00 0 
7f300858c000-7f30085a2000 r-xp 00000000 fd:00 1975230                    /usr/lib64/libgcc_s-6.2.1-20160916.so.1
7f30085a2000-7f30087a1000 ---p 00016000 fd:00 1975230                    /usr/lib64/libgcc_s-6.2.1-20160916.so.1
7f30087a1000-7f30087a2000 r--p 00015000 fd:00 1975230                    /usr/lib64/libgcc_s-6.2.1-20160916.so.1
7f30087a2000-7f30087a3000 rw-p 00016000 fd:00 1975230                    /usr/lib64/libgcc_s-6.2.1-20160916.so.1
7f30087a3000-7f300f33c000 r--p 00000000 fd:00 1967684                    /usr/lib/locale/locale-archive
7f300f33c000-7f300f4f9000 r-xp 00000000 fd:00 1975160                    /usr/lib64/libc-2.24.so
7f300f4f9000-7f300f6f8000 ---p 001bd000 fd:00 1975160                    /usr/lib64/libc-2.24.so
7f300f6f8000-7f300f6fc000 r--p 001bc000 fd:00 1975160                    /usr/lib64/libc-2.24.so
7f300f6fc000-7f300f6fe000 rw-p 001c0000 fd:00 1975160                    /usr/lib64/libc-2.24.so
7f300f6fe000-7f300f702000 rw-p 00000000 00:00 0 
7f300f702000-7f300f729000 r-xp 00000000 fd:00 1975976                    /usr/lib64/libtinfo.so.6.0
7f300f729000-7f300f929000 ---p 00027000 fd:00 1975976                    /usr/lib64/libtinfo.so.6.0
7f300f929000-7f300f92d000 r--p 00027000 fd:00 1975976                    /usr/lib64/libtinfo.so.6.0
7f300f92d000-7f300f92e000 rw-p 0002b000 fd:00 1975976                    /usr/lib64/libtinfo.so.6.0
7f300f92e000-7f300f953000 r-xp 00000000 fd:00 1977178                    /usr/lib64/ld-2.24.so
7f300face000-7f300fb35000 r--p 00000000 fd:00 267760                     /usr/share/locale/pt_BR/LC_MESSAGES/util-linux.mo
7f300fb35000-7f300fb37000 rw-p 00000000 00:00 0 
7f300fb49000-7f300fb4a000 rw-p 00000000 00:00 0 
7f300fb4a000-7f300fb51000 r--s 00000000 fd:00 2361838                    /usr/lib64/gconv/gconv-modules.cache
7f300fb51000-7f300fb53000 rw-p 00000000 00:00 0 
7f300fb53000-7f300fb54000 r--p 00025000 fd:00 1977178                    /usr/lib64/ld-2.24.so
7f300fb54000-7f300fb55000 rw-p 00026000 fd:00 1977178                    /usr/lib64/ld-2.24.so
7f300fb55000-7f300fb56000 rw-p 00000000 00:00 0 
7fffabbe2000-7fffabc03000 rw-p 00000000 00:00 0                          [stack]
7fffabd3b000-7fffabd3d000 r--p 00000000 00:00 0                          [vvar]
7fffabd3d000-7fffabd3f000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

Comment 4 Karel Zak 2016-12-15 11:13:49 UTC
Hmm... is it possible to reproduce the issue without debugger? 

It seems that in reality the opportunity for the race condition is really small. 

I'll try to improve the code in upstream repository, but I have doubts we need to make RHEL update for such issues (especially if this is not reported by customer).

WONTFIX from my point of view.

Comment 5 Paulo Andrade 2016-12-15 11:48:02 UTC
The issue was a "random" coredump from a customer.
It would be very hard to reproduce without a debugger.
Need to press 'q' in a very loaded server, then press ^C
in the interval it releases "Line" and calls "_exit".

This should be trivial to fix, but no need to any kind
of fast update.

Comment 6 Karel Zak 2016-12-15 13:50:16 UTC
I see. You're right _exit() may freeze the process for pretty long time in some cases, so impatient user will have plenty of time to press ^C and send the signal.

Fixed by upstream commit 0ed2a954714992938b35893b70197090a61b3b2e.

Thanks Paulo!

Comment 11 errata-xmlrpc 2017-08-01 21:41:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2186