RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1778558 - nftables segfaults at _gmpz_cmp function
Summary: nftables segfaults at _gmpz_cmp function
Keywords:
Status: CLOSED DUPLICATE of bug 1774742
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: nftables
Version: 8.1
Hardware: x86_64
OS: Linux
low
low
Target Milestone: rc
: 8.1
Assignee: Phil Sutter
QA Contact: qe-baseos-daemons
URL:
Whiteboard:
Depends On:
Blocks: 1774742
TreeView+ depends on / blocked
 
Reported: 2019-12-02 03:04 UTC by suresh kumar
Modified: 2023-09-07 21:10 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-01-17 16:10:09 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-32035 0 None None None 2023-09-07 21:10:53 UTC

Description suresh kumar 2019-12-02 03:04:37 UTC
Description of problem:
While adding nft rules, the gmp qsort function segfaults:


Version-Release number of selected component (if applicable):

nftables-0.9.0-14.el8.x86_64


How reproducible:

Always

Steps to Reproduce:

1. Create a rule like below:

# cat /etc/nftables/nft-rules.nft 
#!/usr/sbin/nft -f
# clear all prior state
flush ruleset
include "/etc/nftables/nft-rules/*"


# cat /etc/nftables/nft-rules/test.nft 
#!/usr/sbin/nft -f
add table inet filter { 
  chain input {
     ip saddr {192.168.1.0/25, 10.10.0.0/16} tcp dport ssh accept
  } 
  chain output {
     # allow all outgoing ssh trafic
     tcp dport ssh accept 
     tcp dport telnet accept 
  }
}


and call the program twice:



Actual results:
# nft flush ruleset
# nft -e -f /etc/nftables/nft-rules.nft 
add table inet filter
add chain inet filter input
add chain inet filter output
add rule inet filter input ip saddr { 10.10.0.0/16, 192.168.1.0/25 } tcp dport ssh accept
add rule inet filter output tcp dport ssh accept
add rule inet filter output tcp dport telnet accept
# 
# nft -e -f /etc/nftables/nft-rules.nft 
add table inet filter
add chain inet filter input
add chain inet filter output
Segmentation fault (core dumped) <-----------------



Expected results:

No segfault


Additional info:

1. This issue is observed only when rule files are called through "include" option.  If all these rules are added in main file and called directly, then no segfault occurs.

2. If we directly flush the ruleset, next call to 'nft -e -f /etc/nftables/nft-rules.nft' works.

3. Though it segfaults at qsort, the rules are added in background    <<<<



Below is my analysis of this issue:
====================================

(gdb) run -e -f nft-rules.nft 
Starting program: /usr/sbin/nft -e -f nft-rules.nft
add table inet filter
add chain inet filter input
add chain inet filter output

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff6e11978 in __gmpz_cmp (u=<optimized out>, v=v@entry=0x55555575f9a0) at cmp.c:52
52        MPN_CMP (cmp, up, vp, asize);
(gdb) bt
#0  0x00007ffff6e11978 in __gmpz_cmp (u=<optimized out>, v=v@entry=0x55555575f9a0) at cmp.c:52
#1  0x00007ffff7b74fba in expr_value_cmp (p1=p1@entry=0x555555761320, p2=p2@entry=0x555555761330) at segtree.c:803
#2  0x00007ffff6a6c325 in msort_with_tmp (p=p@entry=0x7fffffffc8b0, b=b@entry=0x555555761320, n=n@entry=3) at msort.c:83
#3  0x00007ffff6a6c081 in msort_with_tmp (n=3, b=0x555555761320, p=0x7fffffffc8b0) at msort.c:53
#4  msort_with_tmp (p=p@entry=0x7fffffffc8b0, b=b@entry=0x555555761320, n=n@entry=7) at msort.c:53
#5  0x00007ffff6a6c596 in msort_with_tmp (n=7, b=0x555555761320, p=0x7fffffffc8b0) at msort.c:297
#6  __GI___qsort_r (b=b@entry=0x555555761320, n=n@entry=7, s=s@entry=8, cmp=cmp@entry=0x7ffff7b74f80 <expr_value_cmp>, arg=arg@entry=0x0) at msort.c:297
#7  0x00007ffff6a6c6bc in __GI_qsort (b=b@entry=0x555555761320, n=n@entry=7, s=s@entry=8, cmp=cmp@entry=0x7ffff7b74f80 <expr_value_cmp>) at msort.c:308
#8  0x00007ffff7b76306 in interval_map_decompose (set=0x55555575f560) at segtree.c:837
#9  0x00007ffff7b7378f in nlr_for_each_set (nlr=nlr@entry=0x5555557631e0, cb=cb@entry=0x7ffff7b732d0 <rule_map_decompose_cb>, cache=0x55555575aab0, data=0x0) at monitor.c:193
#10 0x00007ffff7b7438f in netlink_events_rule_cb (monh=0x7fffffffcbf0, type=6, nlh=<optimized out>) at monitor.c:516
#11 netlink_events_cb (nlh=<optimized out>, data=data@entry=0x7fffffffcbf0) at monitor.c:880
#12 0x00007ffff7b74978 in netlink_echo_callback (nlh=nlh@entry=0x7fffffffcc90, data=data@entry=0x7fffffffde50) at monitor.c:911
#13 0x00007ffff792e53e in __mnl_cb_run (cb_ctl_array_len=0, cb_ctl_array=0x0, data=0x7fffffffde50, cb_data=0x7ffff7b74900 <netlink_echo_callback>, portid=0, seq=0, numbytes=<optimized out>, buf=0x7fffffffcc90)
    at callback.c:78
#14 mnl_cb_run (buf=buf@entry=0x7fffffffcc90, numbytes=<optimized out>, seq=seq@entry=0, portid=portid@entry=0, cb_data=0x7ffff7b74900 <netlink_echo_callback>, data=data@entry=0x7fffffffde50) at callback.c:162
#15 0x00007ffff7b78c72 in mnl_batch_talk (ctx=ctx@entry=0x7fffffffde50, err_list=err_list@entry=0x7fffffffde40) at mnl.c:273
#16 0x00007ffff7b6c1a9 in netlink_batch_send (ctx=ctx@entry=0x7fffffffde50, err_list=err_list@entry=0x7fffffffde40) at netlink.c:1658
#17 0x00007ffff7b7b025 in nft_netlink (nft=nft@entry=0x55555575aa20, cmds=cmds@entry=0x7fffffffdef0, msgs=msgs@entry=0x7fffffffdee0, nf_sock=<optimized out>) at libnftables.c:59
#18 0x00007ffff7b7b891 in nft_run_cmd_from_filename (nft=0x55555575aa20, filename=<optimized out>) at libnftables.c:497
#19 0x0000555555555fde in main (argc=4, argv=0x7fffffffe078) at main.c:284


(gdb) info registers
rax            0x5554              21844
rbx            0x55555575f940      93824994376000
rcx            0x0                 0
rdx            0x0                 0
rsi            0x19                25 <-----------------------------------
rdi            0x10                16 <-----------------------------------
rbp            0x55555575fb40      0x55555575fb40
rsp            0x7fffffffc738      0x7fffffffc738
r8             0x5555              21845
r9             0x555555766ce0      93824994405600
r10            0x55555575a010      93824994353168
r11            0x20                32
r12            0x55555575f880      93824994375808
r13            0x555555761320      93824994382624
r14            0x8                 8
r15            0x7fffffffc848      140737488341064
rip            0x7ffff6e11978      0x7ffff6e11978 <__gmpz_cmp+56>
eflags         0x10217             [ CF PF AF IF RF ]
cs             0x33                51
ss             0x2b                43
ds             0x0                 0
es             0x0                 0
fs             0x0                 0
gs             0x0                 0
(gdb) disas
Dump of assembler code for function __gmpz_cmp:
   0x00007ffff6e11940 <+0>:     endbr64
   0x00007ffff6e11944 <+4>:     movslq 0x4(%rdi),%r8
   0x00007ffff6e11948 <+8>:     movslq 0x4(%rsi),%rax
   0x00007ffff6e1194c <+12>:    mov    %r8,%rcx
   0x00007ffff6e1194f <+15>:    sub    %rax,%rcx
   0x00007ffff6e11952 <+18>:    mov    %rcx,%rax
   0x00007ffff6e11955 <+21>:    jne    0x7ffff6e1199b <__gmpz_cmp+91>
   0x00007ffff6e11957 <+23>:    mov    %r8,%rdx
   0x00007ffff6e1195a <+26>:    mov    0x8(%rdi),%rdi
   0x00007ffff6e1195e <+30>:    mov    0x8(%rsi),%rsi
   0x00007ffff6e11962 <+34>:    sar    $0x3f,%rdx
   0x00007ffff6e11966 <+38>:    mov    %rdx,%rax
   0x00007ffff6e11969 <+41>:    xor    %r8,%rax
   0x00007ffff6e1196c <+44>:    sub    %rdx,%rax
   0x00007ffff6e1196f <+47>:    jmp    0x7ffff6e11985 <__gmpz_cmp+69>
   0x00007ffff6e11971 <+49>:    nopl   0x0(%rax)
=> 0x00007ffff6e11978 <+56>:    mov    (%rdi,%rax,8),%rdx <----------------------- segfaults


Breakpoint 3, interval_map_decompose (set=0x55555575bf50) at segtree.c:815
815     {
(gdb) c
Continuing.

Breakpoint 3, interval_map_decompose (set=0x55555575f560) at segtree.c:815
815     {
(gdb)
Continuing.
add table inet filter
add chain inet filter input
add chain inet filter output

Breakpoint 3, interval_map_decompose (set=0x55555575f560) at segtree.c:815
815     {
(gdb) list-print set->expressions
list@0x55555575f5c0: {next = 0x55555575fb40, prev = 0x5555557644c0}
node@0x55555575fb40: {next = 0x55555575f880, prev = 0x55555575f5c0} #1
node@0x55555575f880: {next = 0x555555763eb0, prev = 0x55555575fb40} #2
node@0x555555763eb0: {next = 0x555555764100, prev = 0x55555575f880} #3
node@0x555555764100: {next = 0x555555764240, prev = 0x555555763eb0} #4
node@0x555555764240: {next = 0x555555764380, prev = 0x555555764100} #5
node@0x555555764380: {next = 0x5555557644c0, prev = 0x555555764240} #6
node@0x5555557644c0: {next = 0x55555575f5c0, prev = 0x555555764380} #7

(gdb) p *(void * const *)0x55555575fb40
$45 = (void * const) 0x55555575f880
(gdb) p *(void * const *)0x55555575f880
$46 = (void * const) 0x555555763eb0
(gdb) p *(void * const *)0x555555763eb0
$47 = (void * const) 0x555555764100
(gdb) p *(void * const *)0x555555764100
$48 = (void * const) 0x555555764240
(gdb) p *(void * const *)0x555555764240
$49 = (void * const) 0x555555764380
(gdb) p *(void * const *)0x555555764380
$50 = (void * const) 0x5555557644c0
(gdb) p *(void * const *)0x5555557644c0
$51 = (void * const) 0x55555575f5c0
(gdb) p *(void * const *)0x55555575f5c0
$52 = (void * const) 0x55555575fb40

(gdb) p (*(struct expr *)0x55555575f880)->key
$53 = (struct expr *) 0x55555575f940
(gdb) p (*(struct expr *)0x555555763eb0)->key
$54 = (struct expr *) 0x555555761bc0
(gdb) p (*(struct expr *)0x555555764100)->key
$55 = (struct expr *) 0x555555764060
(gdb) p (*(struct expr *)0x555555764240)->key
$56 = (struct expr *) 0x5555557641a0
(gdb) p (*(struct expr *)0x555555764380)->key
$57 = (struct expr *) 0x5555557642e0
(gdb) p (*(struct expr *)0x5555557644c0)->key
$58 = (struct expr *) 0x555555764420
(gdb) p (*(struct expr *)0x55555575f5c0)->key
$59 = (struct expr *) 0x0
(gdb) p (*(struct expr *)0x55555575fb40)->key
$60 = (struct expr *) 0x55555575fc00

(gdb) p (*(struct expr *)0x55555575f940)->value
$61 = {{_mp_alloc = 1433794368, _mp_size = 21845, _mp_d = 0x19}} <------------------------- bad pointer
(gdb) p (*(struct expr *)0x555555761bc0)->value
$62 = {{_mp_alloc = 1, _mp_size = 0, _mp_d = 0x555555763270}}
(gdb) p (*(struct expr *)0x555555764060)->value
$63 = {{_mp_alloc = 1, _mp_size = 1, _mp_d = 0x55555575f300}}
(gdb) p (*(struct expr *)0x5555557641a0)->value
$64 = {{_mp_alloc = 1, _mp_size = 1, _mp_d = 0x555555764040}}
(gdb) p (*(struct expr *)0x5555557642e0)->value
$65 = {{_mp_alloc = 1, _mp_size = 1, _mp_d = 0x555555763100}}
(gdb) p (*(struct expr *)0x555555764420)->value
$66 = {{_mp_alloc = 1, _mp_size = 1, _mp_d = 0x555555763120}}
(gdb) p (*(struct expr *)0x0)->value
Cannot access memory at address 0x60
(gdb) p (*(struct expr *)0x55555575fc00)->value
$67 = {{_mp_alloc = 1433795040, _mp_size = 21845, _mp_d = 0x10}} <------------------------- bad pointer.


and crashed at __GMPN_CMP after a call to 'qsort'.
//gmp-h.in

2066 /* Compare {xp,size} and {yp,size}, setting "result" to positive, zero or
2067    negative.  size==0 is allowed.  On random data usually only one limb will
2068    need to be examined to get a result, so it's worth having it inline.  */
2069 #define __GMPN_CMP(result, xp, yp, size)                                \
2070   do {                                                                  \
2071     mp_size_t  __gmp_i;                                                 \
2072     mp_limb_t  __gmp_x, __gmp_y;                                        \
2073                                                                         \
2074     /* ASSERT ((size) >= 0); */                                         \
2075                                                                         \
2076     (result) = 0;                                                       \
2077     __gmp_i = (size);                                                   \
2078     while (--__gmp_i >= 0)                                              \
2079       {                                                                 \
2080         __gmp_x = (xp)[__gmp_i];                                        \ <---pointer/array/vector access.
2081         __gmp_y = (yp)[__gmp_i];                                        \ <---pointer/array/vector access.
2082         if (__gmp_x != __gmp_y)                                         \
2083           {                                                             \
2084             /* Cannot use __gmp_x - __gmp_y, may overflow an "int" */   \
2085             (result) = (__gmp_x > __gmp_y ? 1 : -1);                    \
2086             break;                                                      \
2087           }                                                             \
2088       }                                                                 \
2089   } while (0)

This bad pointer is added after below cache_update call


475 int nft_run_cmd_from_filename(struct nft_ctx *nft, const char *filename)
476 {
477         struct cmd *cmd, *next;
478         LIST_HEAD(msgs);
479         LIST_HEAD(cmds);
480         int rc;
481
482         rc = cache_update(nft, CMD_INVALID, &msgs); <----------------------------------


Breakpoint 2, nft_run_cmd_from_filename (nft=0x55555575aa20, filename=0x7fffffffe397 "nft-rules.nft") at libnftables.c:476
476     {
(gdb) n
478             LIST_HEAD(msgs);
(gdb)
479             LIST_HEAD(cmds);
(gdb)
482             rc = cache_update(nft, CMD_INVALID, &msgs);
(gdb) s
cache_update (nft=nft@entry=0x55555575aa20, cmd=cmd@entry=CMD_INVALID, msgs=msgs@entry=0x7fffffffdee0) at rule.c:172
172     {
(gdb) n
175             struct netlink_ctx ctx = {
(gdb)
193             ret = cache_init(&ctx, cmd);
(gdb)
184             ctx.seqnum = cache->seqnum++;
(gdb)
185             genid = netlink_genid_get(&ctx);
(gdb)
186             if (cache->genid && cache_needs_more(cache->cmd, cmd))
(gdb)
193             ret = cache_init(&ctx, cmd);
(gdb) s
cache_init (cmd=CMD_INVALID, ctx=0x7fffffffdcf0) at rule.c:193
193             ret = cache_init(&ctx, cmd);
(gdb) n
144             ret = cache_init_tables(ctx, &handle, &ctx->nft->cache);
(gdb)
147             ret = cache_init_objects(ctx, cmd);
(gdb)
193             ret = cache_init(&ctx, cmd);


(gdb) list-print cache->list
list@0x55555575aab8: {next = 0x55555575b360, prev = 0x55555575b360}
node@0x55555575b360: {next = 0x55555575aab8, prev = 0x55555575aab8} #1

(gdb) list-print '(*(struct table *)0x55555575b360)->sets'
list@0x55555575b528: {next = 0x55555575b770, prev = 0x55555575b770}
node@0x55555575b770: {next = 0x55555575b528, prev = 0x55555575b528} #1

(gdb)  p (*(struct set *)0x55555575b770)->init
$36 = (struct expr *) 0x55555575bf50

(gdb) list-print '(*((struct expr *) 0x55555575bf50))->expressions'
list@0x55555575bfb0: {next = 0x55555575c530, prev = 0x55555575c270}
node@0x55555575c530: {next = 0x55555575c270, prev = 0x55555575bfb0} #1
node@0x55555575c270: {next = 0x55555575bfb0, prev = 0x55555575c530} #2

(gdb) p *(void * const *)0x55555575c530
$37 = (void * const) 0x55555575c270
(gdb) p *(void * const *)0x55555575c270
$38 = (void * const) 0x55555575bfb0
(gdb) p *(void * const *)0x55555575bfb0
$39 = (void * const) 0x55555575c530

(gdb) p (*(struct expr *)0x55555575c270)->key
$40 = (struct expr *) 0x55555575c330
(gdb) p (*(struct expr *)0x55555575bfb0)->key
$41 = (struct expr *) 0x0
(gdb) p (*(struct expr *)0x55555575c530)->key
$42 = (struct expr *) 0x55555575c5f0

(gdb) p (*(struct expr *)0x55555575c330)->value
$43 = {{_mp_alloc = 1433780528, _mp_size = 21845, _mp_d = 0x19}}   <----------------
(gdb) p (*(struct expr *)0x0)->value
Cannot access memory at address 0x60
(gdb) p (*(struct expr *)0x55555575c5f0)->value
$44 = {{_mp_alloc = 1433781200, _mp_size = 21845, _mp_d = 0x10}}


+++
545 #define PTR(x) ((x)->_mp_d)

 63 typedef struct
 64 {
 65   int _mp_alloc;                /* Number of *limbs* allocated and pointed
 66                                    to by the _mp_d field.  */
 67   int _mp_size;                 /* abs(_mp_size) is the number of limbs the
 68                                    last field points to.  If _mp_size is
 69                                    negative this is a negative number.  */
 70   mp_limb_t *_mp_d;             /* Pointer to the limbs.  */ <-------------------------------
 71 } __mpz_struct;
+++



It looks like both 0x19 and 0x10 are  subnetmask values from command string:

    {192.168.1.0/25, 10.10.0.0/16}

Comment 1 suresh kumar 2019-12-02 08:18:18 UTC
The bad pointers are added after the call to function:

                 interval_map_decompose(set->init);

Call path:
   --> cache_init_objects(ctx, cmd) ---> netlink_list_setelems ---> netlink_list_setelems -->interval_map_decompose(set->init);

Comment 2 Phil Sutter 2019-12-06 17:01:41 UTC
Hi Suresh,

Thanks for detailed analysis. The above testcase works for me with nftables-0.9.3-4.el8.x86_64, could you please verify?
Does customer require z-stream backport?

Thanks, Phil

Comment 7 Phil Sutter 2020-01-10 21:20:51 UTC
Hmm. I am not sure why I couldn't reproduce it in RHEL-8.2, but backporting the following commit to nftables-0.9.0-14.el8 fixes the problem for me:

commit 5d57fa3e99bb9f2044e236d4ddb7d874cfefe1dd
Author: Phil Sutter <phil>
Date:   Thu Jan 9 13:34:20 2020 +0100

    monitor: Do not decompose non-anonymous sets
    
    They have been decomposed already, trying to do that again causes a
    segfault. This is a similar fix as in commit 8ecb885589591 ("src:
    restore --echo with anonymous sets").
    
    Signed-off-by: Phil Sutter <phil>
    Acked-by: Pablo Neira Ayuso <pablo>

Comment 9 Phil Sutter 2020-01-13 16:15:04 UTC
Required follow-up:

commit ddbacd70d061eb1b6808f501969809bfb5d03001
Author: Phil Sutter <phil>
Date:   Mon Jan 13 14:53:24 2020 +0100

    monitor: Fix output for ranges in anonymous sets
    
    Previous fix for named interval sets was simply wrong: Instead of
    limiting decomposing to anonymous interval sets, it effectively disabled
    it entirely.
    
    Since code needs to check for both interval and anonymous bits
    separately, introduce set_is_interval() helper to keep the code
    readable.
    
    Also extend test case to assert ranges in anonymous sets are correctly
    printed by echo or monitor modes. Without this fix, range boundaries are
    printed as individual set elements.
    
    Fixes: 5d57fa3e99bb9 ("monitor: Do not decompose non-anonymous sets")
    Signed-off-by: Phil Sutter <phil>
    Reviewed-by: Pablo Neira Ayuso <pablo>

Comment 11 Phil Sutter 2020-01-17 16:10:09 UTC
Closing this ticket as a duplicate of bug 1774742 on who's behalf the same patches are backported into RHEL8.2.y. In order to adhere to z-stream process I will use the latter ticket to request z-stream backport to cover the case here.

*** This bug has been marked as a duplicate of bug 1774742 ***


Note You need to log in before you can comment on or make changes to this bug.