Bug 1373427 - Clock skew makes SSSD return System Error
Summary: Clock skew makes SSSD return System Error
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: sssd
Version: 7.3
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Varun Mylaraiah
QA Contact: Varun Mylaraiah
URL:
Whiteboard:
Keywords:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-09-06 09:14 UTC by Patrik Kis
Modified: 2018-02-22 22:15 UTC (History)
10 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2017-08-01 08:58:07 UTC


Attachments (Terms of Use)
krb5_child log (58.43 KB, text/plain)
2017-06-07 16:17 UTC, Varun Mylaraiah
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:2294 normal SHIPPED_LIVE sssd bug fix and enhancement update 2017-08-01 12:39:55 UTC

Description Patrik Kis 2016-09-06 09:14:53 UTC
Description of problem:
Sometime login via ssh as remote (IPA) user fails with the following error. The issue appears after machine reboot.

 [sssd[krb5_child[11125]]][11125]: Error constructing AP-REQ armor: Ticket not yet valid
 [sssd[krb5_child[11125]]][11125]: Error constructing AP-REQ armor: Ticket not yet valid
 sshd[11123]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=localhost user=amy@ipa.baseos.qe
 sshd[11123]: pam_sss(sshd:auth): received for user amy@ipa.baseos.qe: 4 (System error)
 sshd[11121]: error: PAM: Authentication failure for amy@ipa.baseos.qe from localhost
 sshd[11126]: pam_sepermit(sshd:auth): Parsing config file: /etc/security/sepermit.conf
 sshd[11126]: pam_sepermit(sshd:auth): Enforcing mode, access will be allowed on match
 sshd[11126]: pam_sepermit(sshd:auth): sepermit_match returned: -1
 sshd[11121]: Connection closed by 127.0.0.1 [preauth]

The issue does not appear all the time and all test machines.
I'm not sure if it is sssd or other component. Please help to investigate.

Version-Release number of selected component (if applicable):
sssd-1.14.0-35.el7.x86_64
pam-1.1.8-18.el7.x86_64
package krb5 is not installed
realmd-0.16.1-8.el7.x86_64

How reproducible:
~50% chance on test machines

Steps to Reproduce:
1. join to IPA via realmd
2. ssh works
3. reboot
4. ssh does not work

Actual results:


Expected results:


Additional info:

Comment 4 Sumit Bose 2016-09-06 09:22:52 UTC
(In reply to Patrik Kis from comment #0)
> Description of problem:
> Sometime login via ssh as remote (IPA) user fails with the following error.
> The issue appears after machine reboot.
> 
>  [sssd[krb5_child[11125]]][11125]: Error constructing AP-REQ armor: Ticket
> not yet valid
>  [sssd[krb5_child[11125]]][11125]: Error constructing AP-REQ armor: Ticket
> not yet valid

This sounds like times on client and server differ.

Please note that even if recent MIT Kerberos versions can handled clock skews with plain kinit well SSSD might still need synchronized clocks because it does more than plain kinit. The error above indicates that the error happens while setting up the FAST tunnel which includes a service ticket request. Due to the different times on the client and the server the service ticket is 'not yet valid' in the clients view and cannot be used, hence the request failed.

HTH

bye,
Sumit

Comment 5 Jakub Hrozek 2016-09-06 09:45:19 UTC
Right:
[root@sheep-66 ~]# date
Tue Sep  6 09:38:22 CEST 2016

I can't login to the IPA server, but at least on the client the time is off from the 'real world' time.

The only part that smells like a bug to me is returning System Error on time skew. We already convert some error codes like KRB5_KDCREP_SKEW to ERR_NETWORK_IO, I think we should do the same with KRB5KRB_AP_ERR_TKT_EXPIRED and KRB5KRB_AP_ERR_TKT_NYV.

That way, the user could at least authenticate offline, right now, we just kick him out.

Comment 6 Patrik Kis 2016-09-06 09:56:23 UTC
It is indeed because the time skew, how i did not noticed that! Thank you booth for help and sorry for the noise.

Feel free to close this as notabug or keep it if you want to update the error message.

Comment 7 Jakub Hrozek 2016-09-06 10:22:23 UTC
Upstream ticket:
https://fedorahosted.org/sssd/ticket/3174

Comment 8 Lukas Slebodnik 2016-09-13 13:29:29 UTC
master:
* d3348f49260998880bb7cd3b2fb72d562b1b7a64

Comment 10 Varun Mylaraiah 2017-06-07 16:17 UTC
Created attachment 1285837 [details]
krb5_child log

Verified

# rpm -qa sssd pam realmd
pam-1.1.8-18.el7.x86_64
sssd-1.15.2-43.el7.x86_64
realmd-0.16.1-9.el7.x86_64


[root@vm-idm-023 ~]# ssh -l testuser01 vm-idm-004.testrelm.test 
Password: 
Last failed login: Wed Jun  7 21:39:47 IST 2017 from 10.65.206.157 on ssh:notty
There was 1 failed login attempt since the last successful login.
Last login: Wed Jun  7 20:44:27 2017 from 10.65.206.157
**  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **
                 This System is reserved by mvarun@redhat.com.

 To return this system early. You can run the command: return2beaker.sh
  Ensure you have your logs off the system before returning to Beaker

 To extend your reservation time. You can run the command:
  extendtesttime.sh
 This is an interactive script. You will be prompted for how many
  hours you would like to extend the reservation.

 You should verify the watchdog was updated succesfully after
  you extend your reservation.
  https://beaker.engineering.redhat.com/recipes/3920506

 For ssh, kvm, serial and power control operations please look here:
  https://beaker.engineering.redhat.com/view/vm-idm-004.testrelm.test

 For the default root password, see:
  https://beaker.engineering.redhat.com/prefs/

      Beaker Test information:
                         HOSTNAME=vm-idm-004.testrelm.test
                            JOBID=1892925
                         RECIPEID=3920506
                    RESULT_SERVER=[::1]:7083
                           DISTRO=RHEL-7.4-20170606.n.0
                     ARCHITECTURE=x86_64

      Job Whiteboard: IPA :: RHEL 7.4 :: Only Master and Client :: quickinstall :: with 99 hrs reserved :: ipa12

      Recipe Whiteboard: CLIENT1
**  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **
Could not chdir to home directory /home/testuser01: No such file or directory
-sh-4.2$ 
-sh-4.2$ 
-sh-4.2$ id 
uid=1063200001(testuser01) gid=1063200001(testuser01) groups=1063200001(testuser01) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
-sh-4.2$ klist
Ticket cache: KEYRING:persistent:1063200001:krb_ccache_pi5zCeb
Default principal: testuser01@TESTRELM.TEST

Valid starting       Expires              Service principal
2017-06-07T21:39:56  2017-06-08T21:39:56  krbtgt/TESTRELM.TEST@TESTRELM.TEST
-sh-4.2$ logout
Connection to vm-idm-004.testrelm.test closed.

[root@vm-idm-023 ~]# reboot


[root@vm-idm-023 ~]# ssh -l testuser01 vm-idm-004.testrelm.test 
Password: 
Last login: Wed Jun  7 21:39:57 2017 from 10.65.206.157
**  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **
                 This System is reserved by mvarun@redhat.com.

 To return this system early. You can run the command: return2beaker.sh
  Ensure you have your logs off the system before returning to Beaker

 To extend your reservation time. You can run the command:
  extendtesttime.sh
 This is an interactive script. You will be prompted for how many
  hours you would like to extend the reservation.

 You should verify the watchdog was updated succesfully after
  you extend your reservation.
  https://beaker.engineering.redhat.com/recipes/3920506

 For ssh, kvm, serial and power control operations please look here:
  https://beaker.engineering.redhat.com/view/vm-idm-004.testrelm.test

 For the default root password, see:
  https://beaker.engineering.redhat.com/prefs/

      Beaker Test information:
                         HOSTNAME=vm-idm-004.testrelm.test
                            JOBID=1892925
                         RECIPEID=3920506
                    RESULT_SERVER=[::1]:7083
                           DISTRO=RHEL-7.4-20170606.n.0
                     ARCHITECTURE=x86_64

      Job Whiteboard: IPA :: RHEL 7.4 :: Only Master and Client :: quickinstall :: with 99 hrs reserved :: ipa12

      Recipe Whiteboard: CLIENT1
**  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **
Could not chdir to home directory /home/testuser01: No such file or directory
-sh-4.2$ klist
Ticket cache: KEYRING:persistent:1063200001:krb_ccache_pi5zCeb
Default principal: testuser01@TESTRELM.TEST

Valid starting       Expires              Service principal
2017-06-07T21:43:22  2017-06-08T21:43:22  krbtgt/TESTRELM.TEST@TESTRELM.TEST
-sh-4.2$ 



On Master
[root@vm-idm-023 ~]# date
Wed Jun  7 20:43:13 IST 2017
 
[root@vm-idm-023 ~]# ssh -l testuser01 vm-idm-004.testrelm.test -v
OpenSSH_7.4p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 62: Applying options for *
debug1: Executing proxy command: exec /usr/bin/sss_ssh_knownhostsproxy -p 22 vm-idm-004.testrelm.test
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type 2
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: permanently_drop_suid: 0
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to vm-idm-004.testrelm.test:22 as 'testuser01'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:8YAhNTtzXNCvWQRawAgiNd9tAW/sfkNQCAxfiigiwu4
debug1: Host 'vm-idm-004.testrelm.test' is known and matches the ECDSA host key.
debug1: Found key in /var/lib/sss/pubconf/known_hosts:3
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:0)
 
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:0)
 
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug1: Offering DSA public key: /root/.ssh/id_dsa
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Trying private key: /root/.ssh/id_ed25519
debug1: Next authentication method: keyboard-interactive
Password:
debug1: Authentication succeeded (keyboard-interactive).
Authenticated to vm-idm-004.testrelm.test (via proxy).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: proc
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Sending environment.
debug1: Sending env LANG = en_IN.UTF-8
Last failed login: Wed Jun  7 20:30:48 IST 2017 from 10.65.206.157 on ssh:notty
There were 7 failed login attempts since the last successful login.
**  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **
                 This System is reserved by mvarun@redhat.com.
 
 To return this system early. You can run the command: return2beaker.sh
  Ensure you have your logs off the system before returning to Beaker
 
 To extend your reservation time. You can run the command:
  extendtesttime.sh
 This is an interactive script. You will be prompted for how many
  hours you would like to extend the reservation.
 
 You should verify the watchdog was updated succesfully after
  you extend your reservation.
  https://beaker.engineering.redhat.com/recipes/3920506
 
 For ssh, kvm, serial and power control operations please look here:
  https://beaker.engineering.redhat.com/view/vm-idm-004.testrelm.test
 
 For the default root password, see:
  https://beaker.engineering.redhat.com/prefs/
 
      Beaker Test information:
                         HOSTNAME=vm-idm-004.testrelm.test
                            JOBID=1892925
                         RECIPEID=3920506
                    RESULT_SERVER=[::1]:7083
                           DISTRO=RHEL-7.4-20170606.n.0
                     ARCHITECTURE=x86_64
 
      Job Whiteboard: IPA :: RHEL 7.4 :: Only Master and Client :: quickinstall :: with 99 hrs reserved :: ipa12
 
      Recipe Whiteboard: CLIENT1
**  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **
Could not chdir to home directory /home/testuser01: No such file or directory
 
 
-sh-4.2$ klist
Ticket cache: KEYRING:persistent:1063200001:krb_ccache_pi5zCeb
Default principal: testuser01@TESTRELM.TEST
 
Valid starting       Expires              Service principal
2017-06-07T20:43:40  2017-06-08T20:43:40  krbtgt/TESTRELM.TEST@TESTRELM.TEST
 
-sh-4.2$ date
Wed Jun  7 20:43:56 IST 2017
 
-sh-4.2$ logoutdebug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
 
debug1: channel 0: free: client-session, nchannels 1
Connection to vm-idm-004.testrelm.test closed.
Transferred: sent 3632, received 5084 bytes, in 18.4 seconds
Bytes per second: sent 196.9, received 275.6
debug1: Exit status 0
 

Modifying the time into the past on Master

[root@vm-idm-023 ~]#
[root@vm-idm-023 ~]#
[root@vm-idm-023 ~]# date +%T -s "18:09:16"
18:09:16
 
 
[root@vm-idm-023 ~]#
[root@vm-idm-023 ~]# ssh -l testuser01 vm-idm-004.testrelm.test -v
OpenSSH_7.4p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 62: Applying options for *
debug1: Executing proxy command: exec /usr/bin/sss_ssh_knownhostsproxy -p 22 vm-idm-004.testrelm.test
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type 2
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: permanently_drop_suid: 0
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to vm-idm-004.testrelm.test:22 as 'testuser01'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:8YAhNTtzXNCvWQRawAgiNd9tAW/sfkNQCAxfiigiwu4
debug1: Host 'vm-idm-004.testrelm.test' is known and matches the ECDSA host key.
debug1: Found key in /var/lib/sss/pubconf/known_hosts:3
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:0)
 
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:0)
 
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug1: Offering DSA public key: /root/.ssh/id_dsa
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Trying private key: /root/.ssh/id_ed25519
debug1: Next authentication method: keyboard-interactive
Password:
debug1: Authentication succeeded (keyboard-interactive).
Authenticated to vm-idm-004.testrelm.test (via proxy).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: proc
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Sending environment.
debug1: Sending env LANG = en_IN.UTF-8
Last login: Wed Jun  7 20:43:42 2017 from 10.65.206.157
**  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **
                 This System is reserved by mvarun@redhat.com.
 
 To return this system early. You can run the command: return2beaker.sh
  Ensure you have your logs off the system before returning to Beaker
 
 To extend your reservation time. You can run the command:
  extendtesttime.sh
 This is an interactive script. You will be prompted for how many
  hours you would like to extend the reservation.
 
 You should verify the watchdog was updated succesfully after
  you extend your reservation.
  https://beaker.engineering.redhat.com/recipes/3920506
 
 For ssh, kvm, serial and power control operations please look here:
  https://beaker.engineering.redhat.com/view/vm-idm-004.testrelm.test
 
 For the default root password, see:
  https://beaker.engineering.redhat.com/prefs/
 
      Beaker Test information:
                         HOSTNAME=vm-idm-004.testrelm.test
                            JOBID=1892925
                         RECIPEID=3920506
                    RESULT_SERVER=[::1]:7083
                           DISTRO=RHEL-7.4-20170606.n.0
                     ARCHITECTURE=x86_64
 
      Job Whiteboard: IPA :: RHEL 7.4 :: Only Master and Client :: quickinstall :: with 99 hrs reserved :: ipa12
 
      Recipe Whiteboard: CLIENT1
**  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **
Could not chdir to home directory /home/testuser01: No such file or directory
 
 
-sh-4.2$ klist
Ticket cache: KEYRING:persistent:1063200001:krb_ccache_pi5zCeb
Default principal: testuser01@TESTRELM.TEST
 
Valid starting       Expires              Service principal
2017-06-07T20:43:40  2017-06-08T20:43:40  krbtgt/TESTRELM.TEST@TESTRELM.TEST
 
 
-sh-4.2$ date
Wed Jun  7 20:44:40 IST 2017
 
 
-sh-4.2$
-sh-4.2$ logoutdebug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
 
debug1: channel 0: free: client-session, nchannels 1
Connection to vm-idm-004.testrelm.test closed.
Transferred: sent 3692, received 5096 bytes, in 21.5 seconds
Bytes per second: sent 171.7, received 237.0
debug1: Exit status 0
 
[root@vm-idm-023 ~]# date
Wed Jun  7 18:09:55 IST 2017
[root@vm-idm-023 ~]#

Comment 11 errata-xmlrpc 2017-08-01 08:58:07 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:2294


Note You need to log in before you can comment on or make changes to this bug.