1379858 – [RFE] better debugging for ipa-replica-conncheck

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1379858 - [RFE] better debugging for ipa-replica-conncheck

Summary: [RFE] better debugging for ipa-replica-conncheck

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	ipa
Sub Component:
Version:	7.3
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	---
Assignee:	IPA Maintainers
QA Contact:	Scott Poore
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2016-09-27 21:37 UTC by Scott Poore
Modified:	2017-08-01 09:42 UTC (History)
CC List:	5 users (show)
Fixed In Version:	ipa-4.5.0-1.el7
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-08-01 09:42:02 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2017:2304	0	normal	SHIPPED_LIVE	ipa bug fix and enhancement update	2017-08-01 12:41:35 UTC

Description Scott Poore 2016-09-27 21:37:34 UTC

Description of problem:

ipa-replica-conncheck doesn't give a lot of detailed information in the log other than pass, fail, and maybe a traceback.  We need the option to generate more verbose output for debugging ipa-replica-conncheck failures like in bug #1379029 .  

Some of the additional information that might help debug conncheck issues:

- Hostnames and IPs being used to confirm DNS resolution

- Replica port listeners status to confirm they are running before the master to replica check.

- Maybe also a generic master to replica ping to confirm connectivity in general in that direction

Version-Release number of selected component (if applicable):
4.4.0-12

How reproducible:
always

Steps to Reproduce:
1.  ipa-replica-conncheck --debug

or:

1.  ipa-server-install on master
2.  ipa-replica-install --debug (would run conncheck with --debug as well)

Actual results:
Currently does nothing.

Expected results:
Would enable ipa-p

Additional info:

Comment 2 Petr Vobornik 2016-10-07 12:47:51 UTC

Upstream ticket:
https://fedorahosted.org/freeipa/ticket/6387

Comment 3 Tomas Krizek 2017-02-08 14:38:18 UTC

The logging was improved as a part of the replica conncheck refactoring.

Fixed upstream

master:

https://git.fedorahosted.org/cgit/freeipa.git/commit/?id=de981d348efed6dc58b2e355e65244853f06ebc1
https://git.fedorahosted.org/cgit/freeipa.git/commit/?id=af0ba661889c2e2c9a35d4cff9681c2abab73649
https://git.fedorahosted.org/cgit/freeipa.git/commit/?id=a24cd01304aaef77b66d0e178585c9ec8bbce9b5

Comment 6 Scott Poore 2017-03-31 02:06:16 UTC

Following is the conncheck after running with --debug in ipa-replica-install:

[root@rhel7-1 log]# cat ipareplica-conncheck.log 
2017-03-31T01:38:38Z DEBUG /usr/sbin/ipa-replica-conncheck was invoked with options: {'realm': 'EXAMPLE.COM', 'log_to_file': True, 'hostname': 'rhel7-1.example.com', 'quiet': False, 'kdc': None, 'replica': None, 'master': 'rhel6-1.example.com', 'auto_master_check': True, 'debug': False, 'ca_cert_file': '/tmp/tmpADo9H3ipa/realm_info/ca.crt', 'check_ca': True, 'principal': 'admin'}
2017-03-31T01:38:38Z DEBUG missing options might be asked for interactively later

2017-03-31T01:38:38Z DEBUG IPA version 4.5.0-4.el7
2017-03-31T01:38:38Z INFO Check connection from replica to remote master 'rhel6-1.example.com':
2017-03-31T01:38:38Z INFO    Directory Service: Unsecure port (389): OK
2017-03-31T01:38:38Z INFO    Directory Service: Secure port (636): OK
2017-03-31T01:38:38Z INFO    Kerberos KDC: TCP (88): OK
2017-03-31T01:38:38Z INFO    Kerberos Kpasswd: TCP (464): OK
2017-03-31T01:38:38Z INFO    HTTP Server: Unsecure port (80): OK
2017-03-31T01:38:38Z INFO    HTTP Server: Secure port (443): OK
2017-03-31T01:38:38Z INFO    PKI-CA: Directory Service port (7389): OK
2017-03-31T01:38:38Z INFO 
The following list of ports use UDP protocoland would need to be
checked manually:
2017-03-31T01:38:38Z INFO    Kerberos KDC: UDP (88): SKIPPED
2017-03-31T01:38:38Z INFO    Kerberos Kpasswd: UDP (464): SKIPPED
2017-03-31T01:38:38Z INFO 
Connection from replica to master is OK.
2017-03-31T01:38:38Z INFO Start listening on required ports for remote master check
2017-03-31T01:38:38Z DEBUG Starting listening thread.
2017-03-31T01:38:38Z DEBUG 389 tcp: Started listening
2017-03-31T01:38:38Z DEBUG 636 tcp: Started listening
2017-03-31T01:38:38Z DEBUG 88 tcp: Started listening
2017-03-31T01:38:38Z DEBUG 88 udp: Started listening
2017-03-31T01:38:38Z DEBUG 464 tcp: Started listening
2017-03-31T01:38:38Z DEBUG 464 udp: Started listening
2017-03-31T01:38:38Z DEBUG 80 tcp: Started listening
2017-03-31T01:38:38Z DEBUG 443 tcp: Started listening
2017-03-31T01:38:38Z DEBUG 7389 tcp: Started listening
2017-03-31T01:38:38Z DEBUG Ports opened, notify original thread
2017-03-31T01:38:38Z INFO Get credentials to log in to remote master
2017-03-31T01:38:38Z DEBUG Writing temporary Kerberos configuration to /tmp/tmpPPoQtS:
#File created by ipa-replica-conncheck

[libdefaults]
  default_realm = EXAMPLE.COM
  dns_lookup_realm = false
  dns_lookup_kdc = true
  rdns = false
  ticket_lifetime = 24h
  forwardable = true
  udp_preference_limit = 0


[realms]
  EXAMPLE.COM = {
    kdc = rhel6-1.example.com:88
    master_kdc = rhel6-1.example.com:88
    admin_server = rhel6-1.example.com:749

  }


[appdefaults]
  pam = {
    debug = false
    ticket_lifetime = 36000
    renew_lifetime = 36000
    forwardable = true
    krb4_convert = false

  }


2017-03-31T01:38:38Z DEBUG Starting external process
2017-03-31T01:38:38Z DEBUG args=/usr/bin/kinit admin
2017-03-31T01:38:38Z DEBUG Process finished, return code=0
2017-03-31T01:38:38Z DEBUG stdout=Password for admin: 

2017-03-31T01:38:38Z DEBUG stderr=
2017-03-31T01:38:38Z DEBUG Starting external process
2017-03-31T01:38:38Z DEBUG args=/usr/bin/kvno host/rhel6-1.example.com
2017-03-31T01:38:38Z DEBUG Process finished, return code=0
2017-03-31T01:38:38Z DEBUG stdout=host/rhel6-1.example.com: kvno = 2

2017-03-31T01:38:38Z DEBUG stderr=
2017-03-31T01:38:38Z INFO Check RPC connection to remote master
2017-03-31T01:38:38Z DEBUG Starting external process
2017-03-31T01:38:38Z DEBUG args=/usr/bin/certutil -d /tmp/tmppUnzmG -N -f /tmp/tmppUnzmG/pwdfile.txt -f /tmp/tmppUnzmG/pwdfile.txt
2017-03-31T01:38:38Z DEBUG Process finished, return code=0
2017-03-31T01:38:38Z DEBUG stdout=
2017-03-31T01:38:38Z DEBUG stderr=
2017-03-31T01:38:38Z DEBUG Starting external process
2017-03-31T01:38:38Z DEBUG args=/usr/bin/certutil -d /tmp/tmppUnzmG -A -n CN=Certificate Authority,O=EXAMPLE.COM -t C,, -f /tmp/tmppUnzmG/pwdfile.txt
2017-03-31T01:38:38Z DEBUG Process finished, return code=0
2017-03-31T01:38:38Z DEBUG stdout=
2017-03-31T01:38:38Z DEBUG stderr=
2017-03-31T01:38:38Z INFO trying https://rhel6-1.example.com/ipa/json
2017-03-31T01:38:38Z DEBUG Created connection context.rpcclient_61590032
2017-03-31T01:38:38Z INFO Forwarding 'schema' to json server 'https://rhel6-1.example.com/ipa/json'
2017-03-31T01:38:38Z DEBUG New HTTP connection (rhel6-1.example.com)
2017-03-31T01:38:38Z DEBUG HTTP connection destroyed (rhel6-1.example.com)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 677, in single_request
    self.get_auth_info()
  File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 629, in get_auth_info
    self._handle_exception(e, service=service)
  File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 586, in _handle_exception
    raise errors.CCacheError()
CCacheError: did not receive Kerberos credentials
2017-03-31T01:38:38Z DEBUG Destroyed connection context.rpcclient_61590032
2017-03-31T01:38:38Z INFO Retrying using SSH...
2017-03-31T01:38:38Z INFO Check SSH connection to remote master
2017-03-31T01:38:38Z DEBUG Starting external process
2017-03-31T01:38:38Z DEBUG args=/bin/ssh -v -o StrictHostKeychecking=no -o UserKnownHostsFile=/tmp/tmp8DGT5D -o GSSAPIAuthentication=yes -o User=admin rhel6-1.example.com echo OK
2017-03-31T01:38:43Z DEBUG Process finished, return code=0
2017-03-31T01:38:43Z DEBUG stdout=OK

2017-03-31T01:38:43Z DEBUG stderr=OpenSSH_7.4p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug1: Connecting to rhel6-1.example.com [192.168.122.61] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000000
debug1: Authenticating to rhel6-1.example.com:22 as 'admin'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC: umac-64 compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: umac-64 compression: none
debug1: kex: diffie-hellman-group-exchange-sha256 need=16 dh_need=16
debug1: kex: diffie-hellman-group-exchange-sha256 need=16 dh_need=16
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<3072<8192) sent
debug1: got SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: got SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: ssh-rsa SHA256:bxzGIsbZ5RFowZmVtRGfNOxcOWVzrSxgwxXT2ULwSCY
Warning: Permanently added 'rhel6-1.example.com,192.168.122.61' (RSA) to the list of known hosts.
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug1: Next authentication method: gssapi-with-mic
debug1: Authentication succeeded (gssapi-with-mic).
Authenticated to rhel6-1.example.com ([192.168.122.61]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions
debug1: Entering interactive session.
debug1: pledge: network
debug1: Sending environment.
debug1: Sending command: echo OK
Could not chdir to home directory /home/admin: No such file or directory
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 1 clearing O_NONBLOCK
debug1: fd 2 clearing O_NONBLOCK
Transferred: sent 2992, received 2880 bytes, in 0.2 seconds
Bytes per second: sent 18526.4, received 17832.9
debug1: Exit status 0

2017-03-31T01:38:43Z INFO Execute check on remote master
2017-03-31T01:38:43Z DEBUG Starting external process
2017-03-31T01:38:43Z DEBUG args=/bin/ssh -o StrictHostKeychecking=no -o UserKnownHostsFile=/tmp/tmp8Jsf86 -o GSSAPIAuthentication=yes -o User=admin rhel6-1.example.com /usr/sbin/ipa-replica-conncheck --replica rhel7-1.example.com
2017-03-31T01:38:43Z DEBUG 389 tcp: Responded to ::ffff:192.168.122.61
2017-03-31T01:38:43Z DEBUG 636 tcp: Responded to ::ffff:192.168.122.61
2017-03-31T01:38:43Z DEBUG 88 tcp: Responded to ::ffff:192.168.122.61
2017-03-31T01:38:43Z DEBUG 88 udp: Responded to ::ffff:192.168.122.61
2017-03-31T01:38:43Z DEBUG 464 tcp: Responded to ::ffff:192.168.122.61
2017-03-31T01:38:43Z DEBUG 464 udp: Responded to ::ffff:192.168.122.61
2017-03-31T01:38:43Z DEBUG 80 tcp: Responded to ::ffff:192.168.122.61
2017-03-31T01:38:43Z DEBUG 443 tcp: Responded to ::ffff:192.168.122.61
2017-03-31T01:38:43Z DEBUG Process finished, return code=0
2017-03-31T01:38:43Z DEBUG stdout=Check connection from master to remote replica 'rhel7-1.example.com':
   Directory Service: Unsecure port (389): OK
   Directory Service: Secure port (636): OK
   Kerberos KDC: TCP (88): OK
   Kerberos KDC: UDP (88): OK
   Kerberos Kpasswd: TCP (464): OK
   Kerberos Kpasswd: UDP (464): OK
   HTTP Server: Unsecure port (80): OK
   HTTP Server: Secure port (443): OK

Connection from master to replica is OK.

2017-03-31T01:38:43Z DEBUG stderr=Warning: Permanently added 'rhel6-1.example.com,192.168.122.61' (RSA) to the list of known hosts.
Could not chdir to home directory /home/admin: No such file or directory

2017-03-31T01:38:43Z INFO Check connection from master to remote replica 'rhel7-1.example.com':
   Directory Service: Unsecure port (389): OK
   Directory Service: Secure port (636): OK
   Kerberos KDC: TCP (88): OK
   Kerberos KDC: UDP (88): OK
   Kerberos Kpasswd: TCP (464): OK
   Kerberos Kpasswd: UDP (464): OK
   HTTP Server: Unsecure port (80): OK
   HTTP Server: Secure port (443): OK

Connection from master to replica is OK.

2017-03-31T01:38:43Z DEBUG Stopping listening thread.
2017-03-31T01:38:44Z DEBUG 389 tcp: Stopped listening
2017-03-31T01:38:44Z DEBUG 636 tcp: Stopped listening
2017-03-31T01:38:44Z DEBUG 88 tcp: Stopped listening
2017-03-31T01:38:44Z DEBUG 88 udp: Stopped listening
2017-03-31T01:38:44Z DEBUG 464 tcp: Stopped listening
2017-03-31T01:38:44Z DEBUG 464 udp: Stopped listening
2017-03-31T01:38:44Z DEBUG 80 tcp: Stopped listening
2017-03-31T01:38:44Z DEBUG 443 tcp: Stopped listening
2017-03-31T01:38:44Z DEBUG 7389 tcp: Stopped listening

Comment 7 Scott Poore 2017-03-31 02:07:48 UTC

I checked a manual run of ipa-replica-conncheck though both with and without debug set and I can't see much of a difference.

Should there be a difference in logging when debug is enabled?

Thanks,
Scott

Comment 8 Tomas Krizek 2017-03-31 08:05:58 UTC

Here are the updated links to the commits from comment 3 (they were broken due to migration):

master:
https://pagure.io/freeipa/c/de981d348efed6dc58b2e355e65244853f06ebc1
https://pagure.io/freeipa/c/af0ba661889c2e2c9a35d4cff9681c2abab73649
https://pagure.io/freeipa/c/a24cd01304aaef77b66d0e178585c9ec8bbce9b5

The following improvements to logging were made:
- messages that used to only appear on-screen are now also logged to ipareplica-conncheck.log
- when ipa-replica-conncheck is run in --master mode, there is more information about success/failure to bind on specific ports, e.g.:

    WARNING 636 tcp: Failed to bind
    DEBUG 443 tcp: Started listening
    ...
    DEBUG 464 tcp: Stopped listening

- if replica conncheck fails to verify connectivity, information about the specific IP address is displayed (warnings for udp, errors for tcp)

    WARNING Failed to connect to port 88 udp on 1234:4567:abcd::1
    WARNING Failed to connect to port 88 udp on 10.0.0.1
    INFO    Kerberos KDC: UDP (88): WARNING

    ERROR Failed to connect to port 443 tcp on 1234:4567:abcd::1
    ERROR Failed to connect to port 443 tcp on 10.0.0.1
    INFO    HTTP Server: Secure port (443): FAILED

To answer your question, there should not be much of a difference when the replica conncheck succeeds. The major difference in this case is that all displayed messages are also logged in ipareplica-conncheck.log. When the replica conncheck fails, the extra messages should help to track down the issue.

Comment 9 Scott Poore 2017-03-31 13:47:52 UTC

It almost seems like it's always running in debug mode now regardless of using the flag.  To test, I just shutdown httpd on the IPA master.  Below when I diff the two logs, I don't see much besides timestamp that differs.  So, does this show that it's always in debug mode?  Or am I missing something?

Thanks

[root@rhel7-1 ~]# /usr/sbin/ipa-replica-conncheck --master rhel6-1.example.com --auto-master-check --realm EXAMPLE.COM --hostname rhel7-1.example.com --principal admin --password Secret123 --check-ca --ca-cert-file /root/ca.crt 
Check connection from replica to remote master 'rhel6-1.example.com':
   Directory Service: Unsecure port (389): OK
   Directory Service: Secure port (636): OK
   Kerberos KDC: TCP (88): OK
   Kerberos Kpasswd: TCP (464): OK
Failed to connect to port 80 tcp on 192.168.122.61
   HTTP Server: Unsecure port (80): FAILED
Failed to connect to port 443 tcp on 192.168.122.61
   HTTP Server: Secure port (443): FAILED
   PKI-CA: Directory Service port (7389): OK
ERROR: Port check failed! Inaccessible port(s): 80 (TCP), 443 (TCP)


[root@rhel7-1 ~]# cp /var/log/ipareplica-conncheck.log /var/log/ipareplica-conncheck.log.without_debug 
cp: overwrite ‘/var/log/ipareplica-conncheck.log.without_debug’? y


[root@rhel7-1 ~]# /usr/sbin/ipa-replica-conncheck --master rhel6-1.example.com --auto-master-check --realm EXAMPLE.COM --hostname rhel7-1.example.com --principal admin --password Secret123 --check-ca --ca-cert-file /root/ca.crt --debug
/usr/sbin/ipa-replica-conncheck was invoked with options: {'realm': 'EXAMPLE.COM', 'log_to_file': True, 'hostname': 'rhel7-1.example.com', 'quiet': False, 'kdc': None, 'replica': None, 'master': 'rhel6-1.example.com', 'auto_master_check': True, 'debug': True, 'ca_cert_file': '/root/ca.crt', 'check_ca': True, 'principal': 'admin'}
missing options might be asked for interactively later

IPA version 4.5.0-4.el7
Check connection from replica to remote master 'rhel6-1.example.com':
   Directory Service: Unsecure port (389): OK
   Directory Service: Secure port (636): OK
   Kerberos KDC: TCP (88): OK
   Kerberos Kpasswd: TCP (464): OK
Failed to connect to port 80 tcp on 192.168.122.61
   HTTP Server: Unsecure port (80): FAILED
Failed to connect to port 443 tcp on 192.168.122.61
   HTTP Server: Secure port (443): FAILED
   PKI-CA: Directory Service port (7389): OK
ERROR: Port check failed! Inaccessible port(s): 80 (TCP), 443 (TCP)


[root@rhel7-1 ~]# cp /var/log/ipareplica-conncheck.log /var/log/ipareplica-conncheck.log.with_debug cp: overwrite ‘/var/log/ipareplica-conncheck.log.with_debug’? y


[root@rhel7-1 ~]# diff /var/log/ipareplica-conncheck.log.without_debug /var/log/ipareplica-conncheck.log.with_debug
1,2c1,2
< 2017-03-31T13:42:26Z DEBUG /usr/sbin/ipa-replica-conncheck was invoked with options: {'realm': 'EXAMPLE.COM', 'log_to_file': True, 'hostname': 'rhel7-1.example.com', 'quiet': False, 'kdc': None, 'replica': None, 'master': 'rhel6-1.example.com', 'auto_master_check': True, 'debug': False, 'ca_cert_file': '/root/ca.crt', 'check_ca': True, 'principal': 'admin'}
< 2017-03-31T13:42:26Z DEBUG missing options might be asked for interactively later
---
> 2017-03-31T13:42:39Z DEBUG /usr/sbin/ipa-replica-conncheck was invoked with options: {'realm': 'EXAMPLE.COM', 'log_to_file': True, 'hostname': 'rhel7-1.example.com', 'quiet': False, 'kdc': None, 'replica': None, 'master': 'rhel6-1.example.com', 'auto_master_check': True, 'debug': True, 'ca_cert_file': '/root/ca.crt', 'check_ca': True, 'principal': 'admin'}
> 2017-03-31T13:42:39Z DEBUG missing options might be asked for interactively later
4,15c4,15
< 2017-03-31T13:42:26Z DEBUG IPA version 4.5.0-4.el7
< 2017-03-31T13:42:26Z INFO Check connection from replica to remote master 'rhel6-1.example.com':
< 2017-03-31T13:42:26Z INFO    Directory Service: Unsecure port (389): OK
< 2017-03-31T13:42:26Z INFO    Directory Service: Secure port (636): OK
< 2017-03-31T13:42:26Z INFO    Kerberos KDC: TCP (88): OK
< 2017-03-31T13:42:26Z INFO    Kerberos Kpasswd: TCP (464): OK
< 2017-03-31T13:42:26Z ERROR Failed to connect to port 80 tcp on 192.168.122.61
< 2017-03-31T13:42:26Z INFO    HTTP Server: Unsecure port (80): FAILED
< 2017-03-31T13:42:26Z ERROR Failed to connect to port 443 tcp on 192.168.122.61
< 2017-03-31T13:42:26Z INFO    HTTP Server: Secure port (443): FAILED
< 2017-03-31T13:42:26Z INFO    PKI-CA: Directory Service port (7389): OK
< 2017-03-31T13:42:26Z ERROR ERROR: Port check failed! Inaccessible port(s): 80 (TCP), 443 (TCP)
---
> 2017-03-31T13:42:39Z DEBUG IPA version 4.5.0-4.el7
> 2017-03-31T13:42:39Z INFO Check connection from replica to remote master 'rhel6-1.example.com':
> 2017-03-31T13:42:39Z INFO    Directory Service: Unsecure port (389): OK
> 2017-03-31T13:42:39Z INFO    Directory Service: Secure port (636): OK
> 2017-03-31T13:42:39Z INFO    Kerberos KDC: TCP (88): OK
> 2017-03-31T13:42:39Z INFO    Kerberos Kpasswd: TCP (464): OK
> 2017-03-31T13:42:39Z ERROR Failed to connect to port 80 tcp on 192.168.122.61
> 2017-03-31T13:42:39Z INFO    HTTP Server: Unsecure port (80): FAILED
> 2017-03-31T13:42:39Z ERROR Failed to connect to port 443 tcp on 192.168.122.61
> 2017-03-31T13:42:39Z INFO    HTTP Server: Secure port (443): FAILED
> 2017-03-31T13:42:39Z INFO    PKI-CA: Directory Service port (7389): OK
> 2017-03-31T13:42:39Z ERROR ERROR: Port check failed! Inaccessible port(s): 80 (TCP), 443 (TCP)

Comment 10 Tomas Krizek 2017-04-03 07:17:14 UTC

ipa-replica-conncheck now logs the DEBUG level and above to file even without the --debug option, similarly to installer scripts. This behavior has changed, but it is intended. I see no conflict with the man page that says:

 -d, --debug
              Print debugging information

The difference when running with --debug is that extra debug information may be printed directly to the console output, such as:

$ /usr/sbin/ipa-replica-conncheck --master vm1.example.com --debug
ipa         : DEBUG    /usr/sbin/ipa-replica-conncheck was invoked with options: {'realm': None, 'log_to_file': True, 'hostname': None, 'quiet': False, 'kdc': None, 'replica': None, 'master': 'vm1.example.com', 'auto_master_check': False, 'debug': True, 'ca_cert_file': None, 'check_ca': False, 'principal': None}
ipa         : DEBUG    missing options might be asked for interactively later

ipa         : DEBUG    IPA version 4.4.3-2.fc25
Check connection from replica to remote master 'vm1.example.com':
   Directory Service: Unsecure port (389): OK
   Directory Service: Secure port (636): OK
   Kerberos KDC: TCP (88): OK
   Kerberos Kpasswd: TCP (464): OK
   HTTP Server: Unsecure port (80): OK
   HTTP Server: Secure port (443): OK

The following list of ports use UDP protocol and would need to be
checked manually:
   Kerberos KDC: UDP (88): SKIPPED
   Kerberos Kpasswd: UDP (464): SKIPPED

Connection from replica to master is OK.
Start listening on required ports for remote master check
ipa         : DEBUG    Start listening on port 389 (Directory Service: Unsecure port)
ipa         : DEBUG    Start listening on port 636 (Directory Service: Secure port)
ipa         : DEBUG    Start listening on port 88 (Kerberos KDC: TCP)
ipa         : DEBUG    Start listening on port 88 (Kerberos KDC: UDP)
ipa         : DEBUG    Start listening on port 464 (Kerberos Kpasswd: TCP)
ipa         : DEBUG    Start listening on port 464 (Kerberos Kpasswd: UDP)
ipa         : DEBUG    Start listening on port 80 (HTTP Server: Unsecure port)
ipa         : DEBUG    Start listening on port 443 (HTTP Server: Secure port)
Listeners are started. Use CTRL+C to terminate the listening part after the test.

Please run the following command on remote master:
/usr/sbin/ipa-replica-conncheck --replica vm2.example.com

Comment 11 Scott Poore 2017-04-03 13:07:07 UTC

Ok, thanks.  

Verfied.

Version ::

ipa-server-4.5.0-4.el7.x86_64


Results ::

See comment #6 for a full log listing and below for a comparison of output from command line with and without:

[root@rhel7-1 ~]# ipa-replica-conncheck --master rhel6-1.example.com 
Check connection from replica to remote master 'rhel6-1.example.com':
   Directory Service: Unsecure port (389): OK
   Directory Service: Secure port (636): OK
   Kerberos KDC: TCP (88): OK
   Kerberos Kpasswd: TCP (464): OK
   HTTP Server: Unsecure port (80): OK
   HTTP Server: Secure port (443): OK

The following list of ports use UDP protocoland would need to be
checked manually:
   Kerberos KDC: UDP (88): SKIPPED
   Kerberos Kpasswd: UDP (464): SKIPPED

Connection from replica to master is OK.
Start listening on required ports for remote master check
Listeners are started. Use CTRL+C to terminate the listening part after the test.

Please run the following command on remote master:
/usr/sbin/ipa-replica-conncheck --replica rhel7-1.example.com
^C
Cleaning up...


[root@rhel7-1 ~]# ipa-replica-conncheck --master rhel6-1.example.com --debug
/usr/sbin/ipa-replica-conncheck was invoked with options: {'realm': None, 'log_to_file': True, 'hostname': None, 'quiet': False, 'kdc': None, 'replica': None, 'master': 'rhel6-1.example.com', 'auto_master_check': False, 'debug': True, 'ca_cert_file': None, 'check_ca': False, 'principal': None}
missing options might be asked for interactively later

IPA version 4.5.0-4.el7
Check connection from replica to remote master 'rhel6-1.example.com':
   Directory Service: Unsecure port (389): OK
   Directory Service: Secure port (636): OK
   Kerberos KDC: TCP (88): OK
   Kerberos Kpasswd: TCP (464): OK
   HTTP Server: Unsecure port (80): OK
   HTTP Server: Secure port (443): OK

The following list of ports use UDP protocoland would need to be
checked manually:
   Kerberos KDC: UDP (88): SKIPPED
   Kerberos Kpasswd: UDP (464): SKIPPED

Connection from replica to master is OK.
Start listening on required ports for remote master check
Starting listening thread.
389 tcp: Started listening
636 tcp: Started listening
88 tcp: Started listening
88 udp: Started listening
464 tcp: Started listening
464 udp: Started listening
80 tcp: Started listening
443 tcp: Started listening
Ports opened, notify original thread
Listeners are started. Use CTRL+C to terminate the listening part after the test.

Please run the following command on remote master:
/usr/sbin/ipa-replica-conncheck --replica rhel7-1.example.com
^C
Cleaning up...
Stopping listening thread.
389 tcp: Stopped listening
636 tcp: Stopped listening
88 tcp: Stopped listening
88 udp: Stopped listening
464 tcp: Stopped listening
464 udp: Stopped listening
80 tcp: Stopped listening
443 tcp: Stopped listening

Comment 12 Martin Kosek 2017-05-26 09:40:42 UTC

Please note that Red Hat officially released public RHEL-7.4 Beta this week, as announced here:
https://www.redhat.com/en/about/blog/red-hat-enterprise-linux-74-beta-now-available

The new RHEL-7.4 release includes a lot of new IdM functionality, including this RFE. Highlights can be found in RHEL-7.4 Release Notes, especially in the Authentication & Interoperability chapter:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7-Beta/html/7.4_Release_Notes/new_features_authentication_and_interoperability.html

IdM Engineering team would like to encourage everyone interested in this new functionality (and especially customers or community members requesting it) to try Beta and provide us with your feedback!

Comment 13 errata-xmlrpc 2017-08-01 09:42:02 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2304

Note You need to log in before you can comment on or make changes to this bug.