Please provide more details, what do you mean by records not getting removed. What (timeout) error do you see?

If you mean that reservation entries remains in dhcpd.leases that is how this works in ISC DHCP, it's append only file which is rebased every hour or during service restart. Reservation does not actually exist as new entry with "deleted" is appended.

The entries are not being cleaned up every hour.  We have seen entries exist for days after being marked for deletion.

Curtis, I still need more information, I do not understand what kind of entries do exist, you haven't confirmed to me. I am only guessing that this might be around dhcpd.leases. If this is the case (please confirm), than this is expected behavior. The entries do really deleted correctly, you can check this via Capsule DHCP API or omapi.

To check if an entry is present or not do this:

curl -ks --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem --cacert /etc/foreman/proxy_ca.pem https://$(hostname):9090/dhcp | json_reformat

Example output is here: https://access.redhat.com/solutions/2988341

That way you can really see what reservations and leases are really present.

For more info, read this: https://linux.die.net/man/5/dhcpd.leases

Format

Lease descriptions are stored in a format that is parsed by the same recursive descent parser used to read the dhcpd.conf(5) and dhclient.conf(5) files. Lease files can contain lease declarations, and also group and subgroup declarations, host declarations and failover state declarations. Group, subgroup and host declarations are used to record objects created using the OMAPI protocol.

The lease file is a log-structured file - whenever a lease changes, the contents of that lease are written to the end of the file. This means that it is entirely possible and quite reasonable for there to be two or more declarations of the same lease in the lease file at the same time. In that case, the instance of that particular lease that appears last in the file is the one that is in effect.

Group, subgroup and host declarations in the lease file are handled in the same manner, except that if any of these objects are deleted, a rubout is written to the lease file. This is just the same declaration, with { deleted; } in the scope of the declaration. When the lease file is rewritten, any such rubouts that can be eliminated are eliminated. It is possible to delete a declaration in the dhcpd.conf file; in this case, the rubout can never be eliminated from the dhcpd.leases file.

Lukas,

I'm out on PTO for the rest of the week.  I won't be back at the customer site until Tuesday at the earliest to collect the additional data.  I will attempt to run the curl commands as suggested once I am back on site.

In the meantime, I will attempt to better explain the issue.  We have a small DHCP pool used for our provisioning process.  As machines are provisioned, host entries are created in satellite and corresponding DHCP leases appear in the lease file.  Upon retirement, which includes removal of the host record in satellite, the records are sometimes marked for deletion, but not always.  In addition to that, the records remain in the lease file and do not get removed with the 1 hour rebase.  Often times, entries persist a service restart and we are having to manually clean up this file.  This is causing issues as even if a record gets marked for deletion, the host record and VM are removed, it is still throwing IP conflicts based on the lease file.

I hope this helps somewhat.  Like I mentioned previously, I will attempt to gather more specifics next week when I'm back at the client site.

Hey, this indeed make things a bit cleaner.

First, check if the host you are deleting does have Subnet associated and that Subnet has DHCP Capsule assigned. In case a host or a subnet is edited and DHCP Capsule flag is unset, DHCP orchestration will no longer be executed.

Also check if IP or MAC addresses were not deleted before host deletion, DHCP orchestration will only perform if host and NIC are both "managed", the NIC has still "provision" flag, there is a hostname attached to that host and OS is present.

Is the host that misbehaves an image-based provisioned VM or cloud intance? We are tracking a bug when DHCP reservation is not being deleted in that case: http://projects.theforeman.org/issues/9056 but it looks like this one is API only.

There was an upstream bug (regression) when DHCP smart-proxy service was misparting deleted record and these were showing as existing, but this was fixed in 6.2.0 (GA) release. http://projects.theforeman.org/issues/12465

So please reproduce this and then check via CURL command if the record is still present in the Satellite API. I will try to reproduce the mentioned issue on libvirt.

Please confirm Satellite version 6.2.8.

That means Capsule DHCP service encountered an error while processing this request and it returns some non-JSON output. Remove the "json_reformat" part and also investigate proxy.log which will contain root cause of the problem.

For the record, I tried to reproduce this behavior on Satellite 6.2.9 with libvirt image based VM and the DHCP record got deleted just fine. This is output of leases and DHCP API after I deleted VM called test1.nested.lan:

[root@zzzap ~]# curl -ks --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem --cacert /etc/foreman/proxy_ca.pem https://$(hostname):9090/dhcp/192.168.171.0 | json_reformat
{
    "reservations": [
    ],
    "leases": [
    ]
}

[root@zzzap ~]# cat /var/lib/dhcpd/dhcpd.leases
# The format of this file is documented in the dhcpd.leases(5) manual page.
# This lease file was written by isc-dhcp-4.2.5

host test1.nested.lan {
  dynamic;
  hardware ethernet 52:54:00:26:ee:bd;
  fixed-address 192.168.171.119;
        supersede server.filename = "pxelinux.0";
        supersede host-name = "test1.nested.lan";
}
server-duid "\000\001\000\001 \244\254\276RT\304\236\023\005";

host test1.nested.lan {
  dynamic;
  deleted;
}

This is all expected, reservation was deleted in the leases file and is not available in the DHCP API output.

Lukas, Ash, and Curt,

I have seen this "lexical error: invalid char in json text.could not read client cert fro" error before. It was a time issue between the two systems. The clocks were skewed between my DNS server and my Satellite server. And when the clocks are off you can't do any of the DNS operations. I would suggest checking that.