Bug 1926548

Currently, crash will use the system readline library to build the packages. I tried to compile the crash-7.2.9-4.fc34.x86_64 on Fedora 33 with the 'fedpkg local', which never reproduced this issue(crash failed). But, when I tried to compile the crash-7.2.9-4.fc34.x86_64 on Fedora 34, it always reproduces this issue.

# rpm -qa|grep readline
readline-8.0-5.fc33.x86_64
compat-readline5-5.2-37.fc33.x86_64
readline-devel-8.0-5.fc33.x86_64

# rpm -qa|grep readline
readline-8.1-2.fc34.x86_64
readline-devel-8.1-2.fc34.x86_64

In addition, I tried to compile the crash on Fedora 34(do not use the system readline library) with the readline library in gdb, that can not be reproduced on Fedora 34.

Seems that it could be related to the readline library on Fedora 34:

...
crash[512573]: segfault at 0 ip 00007fe147806eb3 sp 00007ffcf5c023a0 error 4 in libreadline.so.8.1[7fe1477ed000+2e000]
...

Also added Siteshwar Vashisht(readline maintainer) to cc list. Thanks.

Hi, Siteshwar

Can you help to confirm this issue? Thank you in advance.

This bug appears to have been reported against 'rawhide' during the Fedora 34 development cycle.
Changing version to 34.

I am having trouble loading kernel debug symbols in `crash` on my f34 system:

[root@fedora 127.0.0.1-2021-02-16-10:49:32]# crash /usr/lib/debug/lib/modules/5.11.0-0.rc7.149.fc34.x86_64/vmlinux vmcore

crash 7.2.9-4.fc34
Copyright (C) 2002-2020  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

crash: /usr/lib/debug/lib/modules/5.11.0-0.rc7.149.fc34.x86_64/vmlinux: no debugging data available

[root@fedora 127.0.0.1-2021-02-16-10:49:32]# rpm -q kernel
kernel-5.11.0-0.rc7.149.fc34.x86_64
[root@fedora 127.0.0.1-2021-02-16-10:49:32]# rpm -qa | grep -i kernel | grep -i debuginfo
kernel-debuginfo-common-x86_64-5.11.0-0.rc7.149.fc34.x86_64
kernel-debuginfo-5.11.0-0.rc7.149.fc34.x86_64



It would be easier for me to debug this if you can provide me readline coredump from your test system.

Thank you for the response, Siteshwar.

I tried to install fedora34 in beaker machine, but the installation always fails as below:

...
[  312.037882] dracut-initqueue[855]: Warning: /lib/dracut/hooks/initqueue/finished/nm.sh: "[ -f /tmp/nm.done ]"
[  312.049879] dracut-initqueue[855]: Warning: /lib/dracut/hooks/initqueue/finished/wait_for_settle.sh: "[ -f /tmp/settle.done ]"
[  312.063071] dracut-initqueue[855]: Warning: dracut-initqueue: starting timeout scripts
[  312.065952] dracut-initqueue[855]: Warning: Could not boot.
         Starting Dracut Emergency Shell...
Warning: /dev/root does not exist

Generating "/run/initramfs/rdsosreport.txt"


Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.


Press Enter for maintenance
(or press Control-D to continue): 
...


Currently, I can not login the system(fc34) to reproduce this issue, can you share your debug machine? I could copy the rhel8's vmlinux and vmcore to your debug machine, and then it can also reproduce this issue on your debug machine(Fedora34), which is irrelevant to the specific vmcore and vmlinux.

Thanks.

Hi, I moved this to readline for the time being, please feel free to assign back if it is not a readline bug finally.

TLDR: I think this may be an issue with GDB and I would like to hear opinion from GDB maintainer about it. Here is my debugging sesssion:

#######################################
I started by setting a breakpoint on `init_line_structures` function:

Breakpoint 1 (init_line_structures) pending.
(gdb) r
Starting program: /usr/bin/crash 

crash 7.2.9-4.el9
Copyright (C) 2002-2020  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
[Detaching after vfork from child process 16039]
[Detaching after vfork from child process 16040]
[Detaching after vfork from child process 16042]
[Detaching after vfork from child process 16043]
[Detaching after vfork from child process 16044]
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

WARNING: kernel relocated [618MB]: patching 118215 gdb minimal_symbol values

please wait... (patching 118215 gdb minimal_symbol values) [Detaching after vfork from child process 16049]
      KERNEL: /usr/lib/debug/lib/modules/5.11.0-1.el9.x86_64/vmlinux   
    DUMPFILE: /proc/kcore
        CPUS: 24
        DATE: Mon Mar  1 04:59:18 EST 2021
      UPTIME: 01:35:20
LOAD AVERAGE: 0.22, 0.06, 0.02
       TASKS: 356
    NODENAME: dell-prt5600-01.ml3.eng.bos.redhat.com
     RELEASE: 5.11.0-1.el9.x86_64
     VERSION: #1 SMP Wed Feb 17 15:51:31 EST 2021
     MACHINE: x86_64  (2893 Mhz)
      MEMORY: 24 GB
         PID: 16035
     COMMAND: "crash"
        TASK: ffff955202e19840  [THREAD_INFO: ffff955202e19840]
         CPU: 1
       STATE: TASK_RUNNING (ACTIVE)


Breakpoint 1, init_line_structures (minsize=minsize@entry=0) at ../display.c:661
661       if (invisible_line == 0)      /* initialize it */


Breakpoint 1, init_line_structures (minsize=minsize@entry=0) at ../display.c:661
661       if (invisible_line == 0)      /* initialize it */
(gdb) p invisible_line
No symbol "invisible_line" in current context.
(gdb) p line_state_invisible->line
$1 = 0x0

663           if (line_size > minsize)
(gdb) p line_size 
$2 = 0
(gdb) p minsize
$3 = 0


(gdb) s
realloc_line (minsize=0) at ../display.c:627
627       if (minsize < minimum_size)
(gdb) n

#######################################
`_rl_screenwidth` variable is set to `INT_MAX` value:

629       if (minsize <= _rl_screenwidth)       /* XXX - for gdb */
(gdb) p _rl_screenwidth
$5 = 2147483647


#######################################
Adding a 1 to it causes `minsize` value to be negative which later creates problem with initializing line_structures:

(gdb) n
630         minsize = _rl_screenwidth + 1;
(gdb) n
631       if (line_size >= minsize)
(gdb) p minsize
$6 = -2147483648


631       if (line_size >= minsize)
(gdb) p minsize
$6 = -2147483648
(gdb) p lines_size
No symbol "lines_size" in current context.
(gdb) p line_size
$7 = 0
(gdb) p line_size >= minsize
$8 = 1


#######################################
Then I set a breakpoint on `_rl_screenwidth` variable to find why it's value is set to INT_MAX:

(gdb) watch _rl_screenwidth
No symbol "_rl_screenwidth" in current context.
(gdb) break main
Breakpoint 1 at 0xf6f70: file main.c, line 80.
(gdb) r
Starting program: /usr/bin/crash 

Breakpoint 1, main (argc=1, argv=0x7fffffffdfd8) at main.c:80
80      {
(gdb) watch _rl_screenwidth
Hardware watchpoint 2: _rl_screenwidth
(gdb) c
Continuing.

crash 7.2.9-4.el9
Copyright (C) 2002-2020  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
[Detaching after vfork from child process 16325]
[Detaching after vfork from child process 16326]
[Detaching after vfork from child process 16328]
[Detaching after vfork from child process 16329]

Hardware watchpoint 2: _rl_screenwidth

Old value = 0
New value = 236
_rl_get_screen_size (tty=tty@entry=0, ignore_env=ignore_env@entry=0) at ../terminal.c:273
273           _rl_screenheight = wr;
(gdb) p wr
$1 = 56



Hardware watchpoint 2: _rl_screenwidth

Old value = 0
New value = 236
_rl_get_screen_size (tty=tty@entry=0, ignore_env=ignore_env@entry=0) at ../terminal.c:273
273           _rl_screenheight = wr;
(gdb) p wr
$1 = 56
(gdb) c
Continuing.

Hardware watchpoint 2: _rl_screenwidth

Old value = 236
New value = 0
rl_reset_terminal (terminal_name=terminal_name@entry=0x0) at ../terminal.c:656
656       _rl_init_terminal_io (terminal_name);
(gdb) l
651        has changed. */
652     int
653     rl_reset_terminal (const char *terminal_name)
654     {
655       _rl_screenwidth = _rl_screenheight = 0;
656       _rl_init_terminal_io (terminal_name);
657       return 0;
658     }
659
660     /* A function for the use of tputs () */


gdb) bt
#0  rl_reset_terminal (terminal_name=terminal_name@entry=0x0) at ../terminal.c:656
#1  0x00005555559a227f in init_page_info () at utils.c:1752
#2  0x00005555559a5291 in initialize_utils () at utils.c:2762
#3  0x00005555559a0a37 in gdb_init (argv0=0x5555565951f0 "/usr/bin/crash") at top.c:1723
#4  0x00005555558d8e20 in captured_main (data=data@entry=0x7fffffffde40) at main.c:740
#5  0x00005555558d70aa in catch_errors (func=func@entry=0x5555558d8b70 <captured_main>, func_args=func_args@entry=0x7fffffffde40, errstring=errstring@entry=0x555555b2d187 "", mask=mask@entry=6) at exceptions.c:557
#6  0x00005555558d9d95 in gdb_main (args=0x7fffffffde40) at main.c:1079
#7  gdb_main_entry (argc=<optimized out>, argv=<optimized out>) at main.c:1099
#8  0x000055555564b8e7 in main (argc=1, argv=0x7fffffffdfd8) at main.c:720


(gdb) c
Continuing.

Hardware watchpoint 2: _rl_screenwidth

Old value = 0
New value = 236
_rl_get_screen_size (tty=tty@entry=0, ignore_env=ignore_env@entry=0) at ../terminal.c:273
273           _rl_screenheight = wr;



Continuing.
[Detaching after vfork from child process 16358]
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

Hardware watchpoint 2: _rl_screenwidth

#######################################
Value of `_rl_screenwidth` is set to INT_MAX  due to gdb not able to correctly set `chars_per_line` (it's value is set to UINT_MAX):

Old value = 236
New value = 2147483647
_rl_set_screen_size (rows=2147483647, cols=2147483647) at ../terminal.c:347
347           if (_rl_term_autowrap == 0)


(gdb) frame 2
#2  0x00005555559a23d8 in set_screen_size () at utils.c:1831
1831      rl_set_screen_size (rows, cols);
(gdb) l
1826
1827      if (cols <= 0)
1828        cols = INT_MAX;
1829
1830      /* Update Readline's idea of the terminal size.  */
1831      rl_set_screen_size (rows, cols);
1832    }
1833
1834    /* Reinitialize WRAP_BUFFER according to the current value of
1835       CHARS_PER_LINE.  */



Breakpoint 1, 0x00005555559a23b8 in set_screen_size () at utils.c:1821
1821      int rows = lines_per_page;
(gdb) n
1822      int cols = chars_per_line;

(gdb) p chars_per_line
$3 = 4294967295

(gdb) p cols
$7 = -1

1827      if (cols <= 0)
1828        cols = INT_MAX;


(gdb) p cols
$8 = 2147483647



Basically it seems that gdb is not able to calculate number of characters that should be set in a line.


I am reassinging this bug to GDB for now. Feel free to reassign if the gdb maintainer think that it's a readline bug.

The crash has the same code on Fedora33 and Fedora34, and this issue is only reproduced on Fedora 34, not in Fedora 33. I'm curious about how the 'chars_per_line' is correctly handled on Fedora 33?

BTW: I saw some differences in comment#1 and comment#7. Thanks.

The current gdb release for Fedora 34 is gdb-10.1-6.fc34.x86_64.  When I do "gdb --version", I see:

GNU gdb (GDB) Fedora 10.1-6.fc34
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

In the output provided in Comment 9, I see:

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

I did a little checking and it seems that the "crash" package includes its own version of GDB: a REALLY, REALLY old version.  (Note the year in the copyright notice: 2013.)  I would not be at all surprised if that really old version of GDB did not play nicely with newer versions of readline.

It seems to me that this is a problem with that really old version of gdb in the "crash" package.  Therefore, I'm reassigning the component to "crash".

Thank you, Siteshwar and Kevin.

I have made a scratch build for Fedora 34 and tested on x86_64, it works well.

https://koji.fedoraproject.org/koji/taskinfo?taskID=63054967

I will update the fix for Fedora 34 and rawhide later.

The following patch(v2) can fix this issue in Fedora 34(later), but upstream doesn't have this issue, because the option "--with-system-readline" is not enabled on upstream crash utility. And upstream doesn't want to have this changes.

In view of this, as we discussed, I will remove the Fedora Only patch "use_system_readline_v3.patch" in order to keep the same with upstream(use the built-in static readline library that comes with GDB). If there is a good reason to convince that upstream accepts the option "--with-system-readline", we can easily pick it up again.



[PATCH v2] GDB: prevent overflow in the rl_set_screen_size()

Currently, crash uses the system readline library to build the packages,
and the readline in system has been upgraded from 8.0 to 8.1, the gdb has
an old version, which can not play with the newer readline.

GDB calls rl_set_screen_size() in readline and may pass the INT_MAX to
the rl_set_screen_size(), however, the rl_set_screen_size() internally
multiplies the number of rows and columns, which causes a signed integer
overflow.

To avoid this issue, let's reduce the "infinite" rows and columns before
calling the rl_set_screen_size().

Signed-off-by: Lianbo Jiang <lijiang>
---
 crash.spec                                    |  2 +
 ...t-overflow-in-the-rl_set_screen_size.patch | 60 +++++++++++++++++++
 2 files changed, 62 insertions(+)
 create mode 100644 gdb-prevent-overflow-in-the-rl_set_screen_size.patch

diff --git a/crash.spec b/crash.spec
index 8252b7baf7e6..7721e26eba79 100644
--- a/crash.spec
+++ b/crash.spec
@@ -34,6 +34,7 @@ Patch13: 0010-Fix-dev-d-option-on-Linux-5.11-rc1-and-later-kernels.patch
 Patch14: 0011-Fix-kmem-v-option-on-Linux-5.11-rc1-and-later-kernel.patch
 Patch15: 0012-mod-Show-the-base-address-of-module.patch
 Patch16: 0013-xen-increase-__PHYSICAL_MASK_SHIFT_XEN-to-52.patch
+Patch17: gdb-prevent-overflow-in-the-rl_set_screen_size.patch
 
 %description
 The core analysis suite is a self-contained tool that can be used to
@@ -70,6 +71,7 @@ offered by Mission Critical Linux, or the LKCD kernel patch.
 %patch14 -p1
 %patch15 -p1
 %patch16 -p1
+%patch17 -p1
 
 %build
 # This package has an internal copy of GDB which has broken configure code for
diff --git a/gdb-prevent-overflow-in-the-rl_set_screen_size.patch b/gdb-prevent-overflow-in-the-rl_set_screen_size.patch
new file mode 100644
index 000000000000..6fc2341d1625
--- /dev/null
+++ b/gdb-prevent-overflow-in-the-rl_set_screen_size.patch
@@ -0,0 +1,60 @@
+From 2f35469929ba91fd0404f1b09b32a299cac48ccf Mon Sep 17 00:00:00 2001
+From: Lianbo Jiang <lijiang>
+Date: Mon, 8 Mar 2021 13:59:21 +0800
+Subject: [PATCH] gdb: prevent overflow in the rl_set_screen_size()
+
+Signed-off-by: Lianbo Jiang <lijiang>
+---
+ gdb-7.6.patch | 38 ++++++++++++++++++++++++++++++++++++++
+ 1 file changed, 38 insertions(+)
+
+diff --git a/gdb-7.6.patch b/gdb-7.6.patch
+index f64b55fe547a..468ff2d02abc 100644
+--- a/gdb-7.6.patch
++++ b/gdb-7.6.patch
+@@ -2500,4 +2500,42 @@ diff -up gdb-7.6/opcodes/configure.orig gdb-7.6/opcodes/configure
+ +struct target_desc *tdesc_aarch64;
+  #include "features/aarch64.c"
+  #include "features/aarch64-without-fpu.c"
++
++--- gdb-7.6/gdb/utils.c.orig
+++++ gdb-7.6/gdb/utils.c
++@@ -1821,11 +1821,30 @@ set_screen_size (void)
++   int rows = lines_per_page;
++   int cols = chars_per_line;
++
++-  if (rows <= 0)
++-    rows = INT_MAX;
+++  /* If we get 0 or negative ROWS or COLS, treat as "infinite" size.
+++     A negative number can be seen here with the "set width/height"
+++     commands and either:
++
++-  if (cols <= 0)
++-    cols = INT_MAX;
+++     - the user specified "unlimited", which maps to UINT_MAX, or
+++     - the user specified some number between INT_MAX and UINT_MAX.
+++
+++     Cap "infinity" to approximately sqrt(INT_MAX) so that we don't
+++     overflow in rl_set_screen_size, which multiplies rows and columns
+++     to compute the number of characters on the screen.  */
+++
+++  const int sqrt_int_max = INT_MAX >> (sizeof (int) * 8 / 2);
+++
+++  if (rows <= 0 || rows > sqrt_int_max)
+++    {
+++      rows = sqrt_int_max;
+++      lines_per_page = UINT_MAX;
+++    }
+++
+++  if (cols <= 0 || cols > sqrt_int_max)
+++    {
+++      cols = sqrt_int_max;
+++      chars_per_line = UINT_MAX;
+++    }
++
++   /* Update Readline's idea of the terminal size.  */
++   rl_set_screen_size (rows, cols);
+  
+-- 
+2.17.1
+
-- 
2.17.1

FEDORA-2021-cc248f7717 has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-cc248f7717

FEDORA-2021-cc248f7717 has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-cc248f7717`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-cc248f7717

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2021-cc248f7717 has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.