Bug 1403352

Summary:	FreeIPA server install fails (and existing servers probably fail to start) due to changes in 'dyndb' feature on merge to upstream BIND
Product:	[Fedora] Fedora	Reporter:	Adam Williamson <awilliam>
Component:	freeipa	Assignee:	Tomas Krizek <tkrizek>
Status:	CLOSED ERRATA	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	26	CC:	abokovoy, gmarr, ipa-maint, jcholast, jhrozek, jonstanley, jpazdziora, mbasti, pspacek, pvoborni, rcritten, robatino, sgallagh, ssorce, tkrizek
Target Milestone:	---	Keywords:	Reopened
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	freeipa-4.4.3-5.fc26	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-09-05 22:35:16 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1404409, 1410433, 1432149, 1452866
Bug Blocks:

Description Adam Williamson 2016-12-09 19:14:29 UTC

Prior to the release of BIND 9.11, Fedora and RHEL carried a downstream patch implementing a feature called 'dyndb' or 'dynamic-db'. For e.g., here's F25's patch:

http://pkgs.fedoraproject.org/cgit/rpms/bind.git/tree/bind-9.10-dyndb.patch?h=f25

FreeIPA sets up this feature (I don't know what it's for, or what it does, but I know we use it) when you do a default ipa-server-install deployment.

In BIND 9.11, the feature was merged into upstream, but with significant changes. pspacek says "The API accepted upstream is totally incompatible with the old API we used".

BIND 9.11 has landed in Fedora Rawhide and the downstream patch has been dropped, so Fedora Rawhide is now using "The API accepted upstream" rather than "the old API we used". However, freeipa doesn't appear to have been adapted to this.

Our openQA test that just does a pretty default FreeIPA server deployment (via rolekit) fails:

https://openqa.fedoraproject.org/tests/50976

if you check the logs (which can be downloaded at https://openqa.fedoraproject.org/tests/50976/file/role_deploy_domain_controller-var_log.tar.gz ), this is because (actually this log extract is from a couple of days ago, but it's the same failure):

Dec 07 11:57:26 ipa001.domain.local systemd[1]: Starting Berkeley Internet Name Domain (DNS) with native PKCS#11...
Dec 07 11:57:26 ipa001.domain.local bash[9698]: /etc/named.conf:46: unknown option 'dynamic-db'
Dec 07 11:57:26 ipa001.domain.local systemd[1]: named-pkcs11.service: Control process exited, code=exited status=1

As part of the changes to the feature when merged upstream, the config file directive name was changed from 'dynamic-db' to 'dyndb'. Apparently this isn't the only change, though. It sounds like ipa-server-install will need changing to make the named.conf modifications in the new format, and we will also need to migrate existing named.conf to the new format when BIND is upgraded from a < 9.11 build with the Fedora/RHEL downstream patches to >= 9.11 with the upstream implementation...

Proposing as a Fedora 26 Alpha blocker, per criterion "Release-blocking roles and the supported role configuration interfaces must meet the core functional Role Definition Requirements to the extent that supported roles can be successfully deployed, started, stopped, brought to a working configuration, and queried." - https://fedoraproject.org/wiki/Fedora_25_Alpha_Release_Criteria#Role_functional_requirements (I didn't copy the criteria for F26 yet) - this bug makes it impossible to deploy the domain controller role, which is a release-blocking role.

Comment 1 Alexander Bokovoy 2016-12-09 19:53:58 UTC

At this point, unless Petr and other DNS maintainers can provide a plan for migrating dyndb backend to the new API, we will have to back out the use of BIND 9.11.

Comment 2 Petr Spacek 2016-12-12 06:53:54 UTC

The plugin is ready for quite some time, the missing part is just FreeIPA-generated named.conf.

Let's fix this ASAP.

Comment 3 Tomas Krizek 2016-12-13 18:51:05 UTC

While attempting to fix new installations of IPA in rawhide, I encountered bug 1404409. It prevents named to start, because it fails to connect to LDAP.

Comment 4 Petr Vobornik 2016-12-15 15:34:05 UTC

Upstream ticket:
https://fedorahosted.org/freeipa/ticket/6565

Comment 5 Geoffrey Marr 2017-01-09 18:22:24 UTC

Discussed during the 2017-01-09 blocker review meeting: [1]

The decision to classify this bug as an "AcceptedBlocker" was made as it violates the following criteria:

"Release-blocking roles and the supported role configuration interfaces must meet the core functional Role Definition Requirements to the extent that supported roles can be successfully deployed, started, stopped, brought to a working configuration, and queried" (domain controller is a blocking role)

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2017-01-09/f26-blocker-review.2017-01-09-17.00.txt

Comment 6 Martin Bašti 2017-02-09 15:35:50 UTC

Fixed upstream
master:
https://fedorahosted.org/freeipa/changeset/c26dd805bdb020b12346d8cb66638883c1f46b9e
https://fedorahosted.org/freeipa/changeset/e8a2abd548b594e6f22f38445ee32bcaa7f27303
https://fedorahosted.org/freeipa/changeset/5de7065fe5769e5c3d90205b0ecc963d96f4db58

Comment 7 Martin Bašti 2017-02-10 11:46:58 UTC

Fixed upstream
master:
https://fedorahosted.org/freeipa/changeset/52582ae9284b80c22a272f0793f0cddfb761f6dc
https://fedorahosted.org/freeipa/changeset/2f4442fff52090bad95a9b1f4f078e4d9acc8069

Comment 8 Martin Bašti 2017-02-14 14:31:19 UTC

master:
https://fedorahosted.org/freeipa/changeset/6cb7bca68486a5ae4be6f93c1acacb7b9890ba9a

Comment 9 Fedora End Of Life 2017-02-28 10:45:42 UTC

This bug appears to have been reported against 'rawhide' during the Fedora 26 development cycle.
Changing version to '26'.

Comment 10 Adam Williamson 2017-03-02 23:40:32 UTC

According to the package changelog, this should be fixed since 4.4.3-5:

* Wed Feb 15 2017 Tomas Krizek <tkrizek> - 4.4.3-5
- Fixes #1403352 - bind-dyndb-ldap: support new named.conf API in BIND 9.11
- Fixes #1412739 - ipa-kdb: support DAL version 6.1

but we've had various other issues since then which prevent verification. Setting to ON_QA; I'm hoping the next compose should have the certmonger and system-python bugs fixed and we can see if this one is fixed.

Comment 11 Adam Williamson 2017-03-03 22:42:20 UTC

This does indeed look to be fixed; in recent openQA tests, FreeIPA server deployment finally works again. Client enrolment is failing, so it looks like there's still a blocker bug, but we're at least moving forward...

Comment 12 Fedora Update System 2017-03-13 15:19:24 UTC

bind-dyndb-ldap-11.1-2.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-f6f66523b8

Comment 13 Tomas Krizek 2017-03-13 15:26:37 UTC

I've submitted a build that has two minor fixes related to this bug. It correctly converts named.conf of existing FreeIPA installations and also bumps the required version of BIND.

Comment 14 Fedora Update System 2017-03-14 03:21:56 UTC

bind-dyndb-ldap-11.1-2.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-f6f66523b8

Comment 15 Adam Williamson 2017-03-15 16:51:26 UTC

Then I'll move this to Beta blocker, as upgrades block Beta, not Alpha.

Comment 16 Geoffrey Marr 2017-03-20 21:01:50 UTC

Discussed during the 2017-03-20 blocker review meeting: [1]

The decision was made to classify this bug as an AcceptedBlocker (Beta) as it violates the following criteria:

"For each one of the release-blocking package sets, it must be possible to successfully complete a direct upgrade from fully updated installations of the last two stable Fedora releases with that..."

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2017-03-20/f26-blocker-review.2017-03-20-16.06.txt

Comment 17 Fedora Update System 2017-04-01 17:01:16 UTC

bind-dyndb-ldap-11.1-2.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Comment 18 Adam Williamson 2017-04-04 19:53:53 UTC

So, should upgrades of FreeIPA servers to F26 be working now so far as anyone knows? tkrizek, is anything still outstanding here? I guess I can try and set up a test for it.

Comment 20 Tomas Krizek 2017-04-05 07:26:33 UTC

FreeIPA upgrade with DNS should work with bind-dyndb-ldap-11.1-2.fc26

When bind-dyndb-ldap is upgraded, it should pull in the new version of bind that has the new API (>9.11.0-6.P2) and transform /etc/named.conf to conform with the new API.

As of now, I'm not aware of any bugs that would be blocking FreeIPA server upgrade.

Comment 24 Adam Williamson 2017-05-01 23:06:19 UTC

Stephen, I think you were gonna test an upgrade to F26 from an existing FreeIPA install, right? Did you do that? I forget.

Comment 25 Stephen Gallagher 2017-05-09 20:41:59 UTC

(In reply to Adam Williamson from comment #24)
> Stephen, I think you were gonna test an upgrade to F26 from an existing
> FreeIPA install, right? Did you do that? I forget.

I have not done so and I unfortunately don't see myself having the time in the next week before Beta Freeze.

Comment 26 Adam Williamson 2017-05-09 20:51:25 UTC

OK. I'll try and get to doing it myself, then.

Comment 27 Adam Williamson 2017-05-19 18:47:23 UTC

So I've been working on getting openQA to test FreeIPA upgrade scenarios, and I *think* I've got it to the point where the test is valid and it's finding problems. I'm not sure yet if the problems are related to this bug; I'll file a new one and leave this open a bit longer while we figure out if they're related.

Comment 28 Adam Williamson 2017-05-19 19:02:09 UTC

https://bugzilla.redhat.com/show_bug.cgi?id=1452866 is the new bug, logs are available there.

Comment 29 Adam Williamson 2017-05-23 16:46:46 UTC

So ths bug effectively depends on 1452866 now, for the upgrade case; FreeIPA upgrade process currently doesn't work properly at all. I'm gonna drop the blocker metadata from this bug, though, as we decided in review of 1452866 that FreeIPA upgrades in general don't block Beta, so the Alpha/Beta-blocking aspect of this is fixed already. If this turns out to be a problem on upgrade after the general upgrade bug is fixed, we can propose it as a Final blocker.

Comment 30 Adam Williamson 2017-09-05 22:35:16 UTC

I'm pretty sure this turned out to be fine in testing.