Bug 1641246

Summary: dhcpd(6) dies at irregular intervals with stacktrace
Product: [Fedora] Fedora Reporter: Kim Bisgaard <kim-rh>
Component: dhcpAssignee: Pavel Zhukov <pzhukov>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 30CC: agross, brandonk98, extras-orphan, jpopelka, marc.c.dionne, marek.gresko, mrehak, pemensik, pzhukov, thozza
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: dhcp-4.3.6-33.fc30 dhcp-4.3.6-31.fc29 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1704672 (view as bug list) Environment:
Last Closed: 2019-05-03 00:58:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1704672, 1708641    

Description Kim Bisgaard 2018-10-20 11:42:35 UTC
Description of problem:
dhcpd(6) dies at irregular intervals - started approx 3 months ago

Version-Release number of selected component (if applicable):
dhcp-server-4.3.6-21.fc28.x86_64



Actual results:
okt 08 10:53:59 XX dhcpd[2582]: Confirm message from fe80::482c:9953:3d6c:fc12 port 546, transaction ID 0x2D616B00
okt 08 10:53:59 XX dhcpd[2582]: Sending Reply to fe80::482c:9953:3d6c:fc12 port 546
okt 08 10:53:59 XX dhcpd[2582]: Solicit message from fe80::482c:9953:3d6c:fc12 port 546, transaction ID 0x208F5C00
okt 08 10:53:59 XX dhcpd[2582]: Advertise NA: address fd56:81d2:347f::f6 to client with duid 00:01:00:01:1e:47:bc:0d:2c:60:0c:f7:1f:1>
okt 08 10:53:59 XX dhcpd[2582]: Sending Advertise to fe80::482c:9953:3d6c:fc12 port 546
okt 08 10:54:00 XX dhcpd[2582]: Request message from fe80::482c:9953:3d6c:fc12 port 546, transaction ID 0x208F5C00
okt 08 10:54:00 XX dhcpd[2582]: Reply NA: address fd56:81d2:347f::f6 to client with duid 00:01:00:01:1e:47:bc:0d:2c:60:0c:f7:1f:1d ia> ../../../lib/isc/heap.c:253: REQUIRE(idx >= 1 && idx <= heap->last) failed, back trace
okt 08 10:54:00 XX dhcpd[2582]: #0 0x7fe614463cdb in ??
okt 08 10:54:00 XX dhcpd[2582]: #1 0x7fe614463c20 in ??
okt 08 10:54:00 XX dhcpd[2582]: #2 0x7fe61446afc2 in ??
okt 08 10:54:00 XX dhcpd[2582]: #3 0x5584acd4a821 in ??
okt 08 10:54:00 XX dhcpd[2582]: #4 0x5584acd453fe in ??
okt 08 10:54:00 XX dhcpd[2582]: #5 0x5584acd47d3c in ??
okt 08 10:54:00 XX dhcpd[2582]: #6 0x5584acd490e9 in ??
okt 08 10:54:00 XX dhcpd[2582]: #7 0x5584acd49d56 in ??
okt 08 10:54:00 XX dhcpd[2582]: #8 0x5584acd6bd28 in ??
okt 08 10:54:00 XX dhcpd[2582]: #9 0x5584acd5ab23 in ??
okt 08 10:54:00 XX dhcpd[2582]: #10 0x7fe6157b7b86 in ??
okt 08 10:54:00 XX dhcpd[2582]: #11 0x7fe614498e6f in ??
okt 08 10:54:00 XX dhcpd[2582]: #12 0x7fe6144881b2 in ??
okt 08 10:54:00 XX dhcpd[2582]: #13 0x7fe61448d0a0 in ??
okt 08 10:54:00 XX dhcpd[2582]: #14 0x7fe61448d565 in ??
okt 08 10:54:00 XX dhcpd[2582]: #15 0x5584acd5cd66 in ??
okt 08 10:54:00 XX dhcpd[2582]: #16 0x5584acd09886 in ??
okt 08 10:54:00 XX dhcpd[2582]: #17 0x7fe61301b11b in ??
okt 08 10:54:00 XX dhcpd[2582]: #18 0x5584acd0a17a in ??



Additional info:
Stack trace varies in depth.
I have tried to install debuginfo packages but the stacktrace seems the same

Comment 1 Marek Greško 2018-10-23 17:43:04 UTC
I observe nearly similar behaviour.

dhcpd6.service: Main process exited, code=killed, status=6/ABRT
dhcpd6.service: Failed with result 'signal'.

I suspect there is some problem with parsing of the leases file since it begun while connecting first computer with dynamically assigned address to the network.

Comment 2 Kim Bisgaard 2018-10-27 07:50:52 UTC
This is what happens in the leases file. First i notice an ABRT, then I list the file:
==============
# authoring-byte-order entry is generated, DO NOT DELETE
authoring-byte-order little-endian;

server-duid "\000\001\000\001\036\274|\017\010`ns4\270";

ia-na "\014`,\002\000\001\000\001\036G\274\015,`\014\367\037\035" {
  cltt 2 2018/10/23 12:41:25;
  iaaddr fd56:81d2:347f::f6 {
    binding state active;
    preferred-life 27000;
    max-life 43200;
    ends 3 2018/10/24 00:41:25;
  }
}

server-duid "\000\001\000\001\036\274|\017\010`ns4\270";

ia-na "\014`,\002\000\001\000\001\036G\274\015,`\014\367\037\035" {
  cltt 2 2018/10/23 12:41:25;
  iaaddr fd56:81d2:347f::f6 {
    binding state expired;
    preferred-life 27000;
    max-life 43200;
    ends 3 2018/10/24 00:41:25;
  }
}
==============

Then I restart dhcpd6.service - which immedialy ABRTs again. Looking i leases file:
==============
# The format of this file is documented in the dhcpd.leases(5) manual page.
# This lease file was written by isc-dhcp-4.3.6

# authoring-byte-order entry is generated, DO NOT DELETE
authoring-byte-order little-endian;

server-duid "\000\001\000\001\036\274|\017\010`ns4\270";

server-duid "\000\001\000\001\036\274|\017\010`ns4\270";
==============
I then restart again - normal sevice.

Comment 3 Marc Dionne 2018-11-01 15:14:23 UTC
I'm seeing this as well.  The server crashes the first time that it gets a request, after initially giving out a prefix delegation

coredumptl reports the crash occurs here:

Stack trace of thread 3995:
                #0  0x00007fd5514bdeab raise (libc.so.6)
                #1  0x00007fd5514a85b9 abort (libc.so.6)
                #2  0x00007fd5528dcc65 isc_assertion_failed (libisc-export.so.169)
                #3  0x00007fd5528e3f0a isc_heap_delete (libisc-export.so.169)
                #4  0x000000000045a06d cleanup_old_expired (dhcpd)
                #5  0x000000000045a30f lease_timeout_support (dhcpd)
                #6  0x000000000046e80f isclib_timer_callback (dhcpd)
                #7  0x00007fd552901322 isc__taskmgr_dispatch (libisc-export.so.169)
                #8  0x00007fd552906210 evloop (libisc-export.so.169)
                #9  0x00007fd5529066d5 isc__app_ctxrun (libisc-export.so.169)
                #10 0x000000000046e71e dispatch (dhcpd)
                #11 0x00000000004084d2 main (dhcpd)
                #12 0x00007fd5514aa11b __libc_start_main (libc.so.6)
                #13 0x0000000000406bea _start (dhcpd)

In my case the server is f29, running dhcp-server-4.3.6-28.fc29.x86_64.

Comment 4 Marek Greško 2019-01-03 21:16:13 UTC
Would not it be better to change component to dhcp instead of dhcpv6? It seems when set to dhcpv6 it is orphaned and will never get assigned.

Comment 5 David Cantrell 2019-01-04 20:32:42 UTC
(In reply to Marek Greško from comment #4)
> Would not it be better to change component to dhcp instead of dhcpv6? It
> seems when set to dhcpv6 it is orphaned and will never get assigned.

Yes.  dhcpv6 has been deprecated for many years.  It should not even be a component you can select in Bugzilla, IMHO.

Comment 6 Marek Greško 2019-03-08 11:44:11 UTC
The bug is still orphaned. Is somebody going to look at it?

Comment 7 Pavel Zhukov 2019-03-08 15:38:05 UTC
*** Bug 1686816 has been marked as a duplicate of this bug. ***

Comment 8 Marek Greško 2019-03-26 16:07:29 UTC
Could someone rise severity and priority on this case?

Comment 9 Fedora Update System 2019-04-30 13:11:02 UTC
dhcp-4.3.6-23.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2019-c82d274716

Comment 10 Fedora Update System 2019-04-30 13:12:50 UTC
dhcp-4.3.6-30.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-1918730c03

Comment 11 Fedora Update System 2019-04-30 13:34:34 UTC
dhcp-4.3.6-33.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-59d507e0c7

Comment 12 Fedora Update System 2019-05-01 01:44:11 UTC
dhcp-4.3.6-33.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-59d507e0c7

Comment 13 Fedora Update System 2019-05-01 02:28:21 UTC
dhcp-4.3.6-23.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-c82d274716

Comment 14 Fedora Update System 2019-05-01 03:21:35 UTC
dhcp-4.3.6-30.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-1918730c03

Comment 15 Pavel Zhukov 2019-05-02 10:22:26 UTC
Hello Marek,

Would you mind testing the fix/updated package?

Comment 16 Ben Cotton 2019-05-02 19:07:03 UTC
This message is a reminder that Fedora 28 is nearing its end of life.
On 2019-May-28 Fedora will stop maintaining and issuing updates for
Fedora 28. It is Fedora's policy to close all bug reports from releases
that are no longer maintained. At that time this bug will be closed as
EOL if it remains open with a Fedora 'version' of '28'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 28 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 17 Fedora Update System 2019-05-03 00:58:55 UTC
dhcp-4.3.6-33.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Comment 18 Fedora Update System 2019-05-04 20:43:18 UTC
bind-9.11.6-1.P1.fc29 bind-dyndb-ldap-11.1-15.fc29 dhcp-4.3.6-31.fc29 dnsperf-2.2.1-4.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-0370ad8310

Comment 19 Fedora Update System 2019-05-05 01:50:42 UTC
bind-9.11.6-1.P1.fc29, bind-dyndb-ldap-11.1-15.fc29, dhcp-4.3.6-31.fc29, dnsperf-2.2.1-4.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-0370ad8310

Comment 20 Fedora Update System 2019-05-06 14:51:45 UTC
bind-9.11.6-2.P1.fc29 bind-dyndb-ldap-11.1-15.fc29 dhcp-4.3.6-31.fc29 dnsperf-2.2.1-4.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-0370ad8310

Comment 21 Fedora Update System 2019-05-07 14:13:37 UTC
bind-9.11.6-2.P1.fc29, bind-dyndb-ldap-11.1-15.fc29, dhcp-4.3.6-31.fc29, dnsperf-2.2.1-4.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-0370ad8310

Comment 22 Pavel Zhukov 2019-05-10 13:49:07 UTC
*** Bug 1708642 has been marked as a duplicate of this bug. ***

Comment 23 Fedora Update System 2019-06-05 02:01:07 UTC
bind-9.11.6-2.P1.fc29, bind-dyndb-ldap-11.1-15.fc29, dhcp-4.3.6-31.fc29, dnsperf-2.2.1-4.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.