Looks relatively easy to fix by extending https://pagure.io/freeipa/blob/master/f/install/oddjob/etc/oddjobd.conf.d/oddjobd-ipa-trust.conf to pass more than one argument to the helper. Oddjobd itself treats anything passed to the helper on the command line as arguments up to the specified value. So if we add --server name option that would be two arguments in addition to the forest name. I think it makes sense to just set arguments number high enough (10-20) to allow future option extensibility.

Then we can change IPA helper https://pagure.io/freeipa/blob/master/f/install/oddjob/com.redhat.idm.trust-fetch-domains.in (it has hardcoded 1 argument) and IPA callers to pass the options.

IPA callers: 
 - dbus helper is https://pagure.io/freeipa/blob/master/f/ipaserver/plugins/trust.py#_421 (only accepts one argument for the forest name)
 - actual call from trust_add.execute() is https://pagure.io/freeipa/blob/master/f/ipaserver/plugins/trust.py#_776
 - actual call from trust_fetch_domains.execute() is https://pagure.io/freeipa/blob/master/f/ipaserver/plugins/trust.py#_1782

Also, passing existing discovered server should be preferred too. If we established trust with a specific server, we most likely should be using that for the helper runs as well because it would allow us to avoid possible replication delays on AD side and do not run discovery of AD DCs again and again.

Clonet to upstream: https://pagure.io/freeipa/issue/7895

Pull request: https://github.com/freeipa/freeipa/pull/2965

Fixed upstream
master:
https://pagure.io/freeipa/c/de4a9875d410c68ae4f9602b70c753a11034b31b

Fixed upstream
ipa-4-7:
https://pagure.io/freeipa/c/12bc3fd16370788edbf1b66e9aa08fd307cf9e7f

Fixed upstream
ipa-4-6:
https://pagure.io/freeipa/c/916d4f3cd0bf26e96f696903bd6fb19693437712

Luc, Alexander: is it true that to stably reproduce the bug we need 

1) set SRV records for AD domain to point to some unreachable address
2) execute "ipa trust-add --server HOST" where HOST is a reachable AD instance?

'ipa trust-add --server HOST' was already supporting use of external server. 'ipa trust-fetch-domains' was missing it. So if you add

3) execute 'ipa trust-fetch-domains ad.domain' without --server and with --server option when SRV records point to unreachable addresses

then that would be a test case.

As per my examination, the bug is not fixed.

# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.7 Beta (Maipo)

# rpm -q ipa-server
ipa-server-4.6.5-9.el7.x86_64

In my setup AD server has address 192.168.121.10

My steps:
1. install ipa server: ipa-server-install -U -p Secret123 -a Secret123 -n testrelm.test -r TESTRELM.TEST --hostname master2.testrelm.test --setup-dns --setup-adtrust --auto-forwarders
2. install adtrust: ipa-adtrust-install -U --enable-compat --netbios-name MASTER2 -a Secret123 --add-sids
3. instead of adding dnz zone forwarder to AD domain I create zone file for AD domain with all records copied from Windows machine, replacing ip address with unreachable one:
cat > /etc/named.ad.test <<EOF
\$ORIGIN ad.test.
\$TTL 86400

@  IN A 192.168.254.254
   IN NS ad1.ad.test.
   IN SOA ad1.ad.test. hostmaster.ad.test. 39 900 600 86400 3600
_msdcs IN NS ad1.ad.test.
_gc._tcp.Default-First-Site-Name._sites IN SRV 0 100 3268 ad1.ad.test.
_kerberos._tcp.Default-First-Site-Name._sites IN SRV 0 100 88 ad1.ad.test.
_ldap._tcp.Default-First-Site-Name._sites IN SRV 0 100 389 ad1.ad.test.
_gc._tcp IN SRV 0 100 3268 ad1.ad.test.
_kerberos._tcp IN SRV 0 100 88 ad1.ad.test.
_kpasswd._tcp IN SRV 0 100 464 ad1.ad.test.
_ldap._tcp IN SRV 0 100 389 ad1.ad.test.
_kerberos._udp IN SRV 0 100 88 ad1.ad.test.
_kpasswd._udp IN SRV 0 100 464 ad1.ad.test.
ad1 IN A 192.168.254.254
DomainDnsZones IN A 192.168.254.254
_ldap._tcp.Default-First-Site-Name._sites.DomainDnsZones IN SRV 0 100 389 ad1.ad.test.
_ldap._tcp.DomainDnsZones IN SRV 0 100 389 ad1.ad.test.
ForestDnsZones IN A 192.168.254.254
_ldap._tcp.Default-First-Site-Name._sites.ForestDnsZones IN SRV 0 100 389 ad1.ad.test.
_ldap._tcp.ForestDnsZones IN SRV 0 100 389 ad1.ad.test.
EOF

cat >> /etc/named.conf <<EOF
zone "ad.test" {
  type master;
  file "/etc/named.ad.test";
};
EOF

systemctl restart named-pkcs11

4. Check that trust can not be established without --server option:
echo Secret123 | ipa trust-add ad.test  --admin Administrator --password
ipa: ERROR: CIFS server communication error: code "3221225653", message "{Device Timeout} The specified I/O operation on %hs was not completed before the time-out period expired." (both may be "None")

5. try to establish trust with --server option (FAIL, takes several minutes to complete):
echo Secret123 | ipa trust-add ad.test  --admin Administrator --password --server 192.168.121.10

ipa: ERROR: error on server 'master2.testrelm.test': Fetching domains from trusted forest failed. See details in the error_log

6. ipa trust-fetch-domains ad.test (expected to fail)
ipa: ERROR: error on server 'master2.testrelm.test': Fetching domains from trusted forest failed. See details in the error_log

7. ipa trust-fetch-domains ad.test --server 192.168.121.10 (FAIL)
ipa: ERROR: error on server 'master2.testrelm.test': Fetching domains from trusted forest failed. See details in the error_log


I get same output with ipa version not containing the fix: ipa-server-4.6.5-3.el7.x86_64

I also checked that when there is no AD zone in BIND config and no forwarder, the steps 5-7 are successful in both version.
and the step  4 expectedly fails with 
ipa: ERROR: Cannot find specified domain or server name

Alexander, can you check that verification steps are correct?

I need apache logs (error_log), after you set 

[global]
debug=True

in /etc/ipa/server.conf and restarted httpd, then retried.

error_log will have output from the oddjob helper.

Created attachment 1584813 [details]
apache error log

Added apache error log collected during execution of step 5

Thanks. Looks like we need to investigate the oddjob helper directly.

1. add 'log level = 100' to /usr/share/ipa/smb.conf.empty
2. run as root
  # oddjob_request -i com.redhat.idm.trust.fetch_domains \
                   -s com.redhat.idm.trust -o / \
                   com.redhat.idm.trust.fetch_domains ad.test

there will be a lot of output which I need to see.

There was a mistake in zone file which produced totally unusable setup. The proper zone file should be:

cat > /etc/named.ad.test <<EOF
\$ORIGIN ad.test.
\$TTL 86400

@  IN A 192.168.121.10
   IN NS ad1.ad.test.
   IN SOA ad1.ad.test. hostmaster.ad.test. 39 900 600 86400 3600
_msdcs IN NS ad1.ad.test.
_gc._tcp.Default-First-Site-Name._sites IN SRV 0 100 3268 unreachable.ad.test.
_kerberos._tcp.Default-First-Site-Name._sites IN SRV 0 100 88 unreachable.ad.test.
_ldap._tcp.Default-First-Site-Name._sites IN SRV 0 100 389 unreachable.ad.test.
_gc._tcp IN SRV 0 100 3268 unreachable.ad.test.
_kerberos._tcp IN SRV 0 100 88 unreachable.ad.test.
_kpasswd._tcp IN SRV 0 100 464 unreachable.ad.test.
_ldap._tcp IN SRV 0 100 389 unreachable.ad.test.
_kerberos._udp IN SRV 0 100 88 unreachable.ad.test.
_kpasswd._udp IN SRV 0 100 464 unreachable.ad.test.
ad1 IN A 192.168.121.10
unreachable IN A 192.168.254.254
DomainDnsZones IN A 192.168.121.10
_ldap._tcp.Default-First-Site-Name._sites.DomainDnsZones IN SRV 0 100 389 unreachable.ad.test.
_ldap._tcp.DomainDnsZones IN SRV 0 100 389 unreachable.ad.test.
ForestDnsZones IN A 192.168.121.10
_ldap._tcp.Default-First-Site-Name._sites.ForestDnsZones IN SRV 0 100 389 unreachable.ad.test.
_ldap._tcp.ForestDnsZones IN SRV 0 100 389 unreachable.ad.test.
EOF

But result of test is still the same.

Since we are authenticating against AD DC before talking to it (by using trusted domain object's credentials), we need to override krb5.conf configuration in case --server option is specified. I now have a code that does this and it passes the test on Sergey's test VM. I'll submit a pull request shortly.

I submitted https://github.com/freeipa/freeipa/pull/3328 to fix the issue

Fixed upstream
master:
https://pagure.io/freeipa/c/6c9fcccfbcbb78b49b65c12bf05b2647f1522609

Fixed upstream
ipa-4-6:
https://pagure.io/freeipa/c/076cb69f00db734a378a74f50f5b4b74321c0fa4

Fixed upstream
ipa-4-7:
https://pagure.io/freeipa/c/2dcf15b990e374cf616e1a55c112fe34779006cf

Checked version from repo rhel-7.7-candidate

# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.7 Beta (Maipo)

# rpm -q ipa-server
ipa-server-4.6.5-11.el7.x86_64


Steps 4-7 from https://bugzilla.redhat.com/show_bug.cgi?id=1686302#c11:

[root@master1 vagrant]# echo Secret123 | ipa trust-add ad.test  --admin Administrator --password
ipa: ERROR: CIFS server communication error: code "3221225653", message "{Device Timeout} The specified I/O operation on %hs was not completed before the time-out period expired." (both may be "None")

[root@master1 vagrant]# echo Secret123 | ipa trust-add ad.test  --admin Administrator --password --server ad1.ad.test
------------------------------------------------
Added Active Directory trust for realm "ad.test"
------------------------------------------------
  Realm name: ad.test
  Domain NetBIOS name: AD
  Domain Security Identifier: S-1-5-21-3370645280-1493649052-842853331
  Trust direction: Trusting forest
  Trust type: Active Directory domain
  Trust status: Established and verified

[root@master1 vagrant]# ipa trust-fetch-domains ad.test
ipa: ERROR: error on server 'master1.testrelm.test': Fetching domains from trusted forest failed. See details in the error_log

[root@master1 vagrant]# ipa trust-fetch-domains ad.test --server ad1.ad.test
----------------------------------------------------------------------------------------
List of trust domains successfully refreshed. Use trustdomain-find command to list them.
----------------------------------------------------------------------------------------
----------------------------
Number of entries returned 0
----------------------------


The bug is fixed, but still need to verify version in compose (nightly build) when it is ready.

Also verified for nightly build.

Test automation created: https://github.com/freeipa/freeipa/blob/843f57abe431bcf493e0bcce8ef07255be986435/ipatests/test_integration/test_trust.py#L539

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2241

I am very sorry. 

This is a wrong post. Removing need info flag.

Additional test fixes:

Fixed upstream
ipa-4-8:
https://pagure.io/freeipa/c/814b47e85c87bc3c80c91ebd0aa9085ac06b521e
ipa-4-7:
https://pagure.io/freeipa/c/db1b9fc428d8932f4fa9da79501ff2be8e4e7cc2