Bug 1373427

Summary: Clock skew makes SSSD return System Error
Product: Red Hat Enterprise Linux 7 Reporter: Patrik Kis <pkis>
Component: sssdAssignee: Varun Mylaraiah <mvarun>
Status: CLOSED ERRATA QA Contact: Varun Mylaraiah <mvarun>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.3CC: cobrown, grajaiya, jhrozek, ksiddiqu, lslebodn, mkosek, mzidek, pbrezina, sbose, sssd-qe
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: sssd-1.15.0-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-01 08:58:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
krb5_child log none

Description Patrik Kis 2016-09-06 09:14:53 UTC 
Description of problem:
Sometime login via ssh as remote (IPA) user fails with the following error. The issue appears after machine reboot.

 [sssd[krb5_child[11125]]][11125]: Error constructing AP-REQ armor: Ticket not yet valid
 [sssd[krb5_child[11125]]][11125]: Error constructing AP-REQ armor: Ticket not yet valid
 sshd[11123]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=localhost user=amy.qe
 sshd[11123]: pam_sss(sshd:auth): received for user amy.qe: 4 (System error)
 sshd[11121]: error: PAM: Authentication failure for amy.qe from localhost
 sshd[11126]: pam_sepermit(sshd:auth): Parsing config file: /etc/security/sepermit.conf
 sshd[11126]: pam_sepermit(sshd:auth): Enforcing mode, access will be allowed on match
 sshd[11126]: pam_sepermit(sshd:auth): sepermit_match returned: -1
 sshd[11121]: Connection closed by 127.0.0.1 [preauth]

The issue does not appear all the time and all test machines.
I'm not sure if it is sssd or other component. Please help to investigate.

Version-Release number of selected component (if applicable):
sssd-1.14.0-35.el7.x86_64
pam-1.1.8-18.el7.x86_64
package krb5 is not installed
realmd-0.16.1-8.el7.x86_64

How reproducible:
~50% chance on test machines

Steps to Reproduce:
1. join to IPA via realmd
2. ssh works
3. reboot
4. ssh does not work

Actual results:


Expected results:


Additional info:
Comment 4 Sumit Bose 2016-09-06 09:22:52 UTC 
(In reply to Patrik Kis from comment #0)
> Description of problem:
> Sometime login via ssh as remote (IPA) user fails with the following error.
> The issue appears after machine reboot.
> 
>  [sssd[krb5_child[11125]]][11125]: Error constructing AP-REQ armor: Ticket
> not yet valid
>  [sssd[krb5_child[11125]]][11125]: Error constructing AP-REQ armor: Ticket
> not yet valid

This sounds like times on client and server differ.

Please note that even if recent MIT Kerberos versions can handled clock skews with plain kinit well SSSD might still need synchronized clocks because it does more than plain kinit. The error above indicates that the error happens while setting up the FAST tunnel which includes a service ticket request. Due to the different times on the client and the server the service ticket is 'not yet valid' in the clients view and cannot be used, hence the request failed.

HTH

bye,
Sumit
Comment 5 Jakub Hrozek 2016-09-06 09:45:19 UTC 
Right:
[root@sheep-66 ~]# date
Tue Sep  6 09:38:22 CEST 2016

I can't login to the IPA server, but at least on the client the time is off from the 'real world' time.

The only part that smells like a bug to me is returning System Error on time skew. We already convert some error codes like KRB5_KDCREP_SKEW to ERR_NETWORK_IO, I think we should do the same with KRB5KRB_AP_ERR_TKT_EXPIRED and KRB5KRB_AP_ERR_TKT_NYV.

That way, the user could at least authenticate offline, right now, we just kick him out.
Comment 6 Patrik Kis 2016-09-06 09:56:23 UTC 
It is indeed because the time skew, how i did not noticed that! Thank you booth for help and sorry for the noise.

Feel free to close this as notabug or keep it if you want to update the error message.
Comment 7 Jakub Hrozek 2016-09-06 10:22:23 UTC 
Upstream ticket:
https://fedorahosted.org/sssd/ticket/3174
Comment 8 Lukas Slebodnik 2016-09-13 13:29:29 UTC 
master:
* d3348f49260998880bb7cd3b2fb72d562b1b7a64
Comment 10 Varun Mylaraiah 2017-06-07 16:17:25 UTC 
Created attachment 1285837 [details]
krb5_child log

Verified

# rpm -qa sssd pam realmd
pam-1.1.8-18.el7.x86_64
sssd-1.15.2-43.el7.x86_64
realmd-0.16.1-9.el7.x86_64


[root@vm-idm-023 ~]# ssh -l testuser01 vm-idm-004.testrelm.test 
Password: 
Last failed login: Wed Jun  7 21:39:47 IST 2017 from 10.65.206.157 on ssh:notty
There was 1 failed login attempt since the last successful login.
Last login: Wed Jun  7 20:44:27 2017 from 10.65.206.157
**  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **
                 This System is reserved by mvarun.

 To return this system early. You can run the command: return2beaker.sh
  Ensure you have your logs off the system before returning to Beaker

 To extend your reservation time. You can run the command:
  extendtesttime.sh
 This is an interactive script. You will be prompted for how many
  hours you would like to extend the reservation.

 You should verify the watchdog was updated succesfully after
  you extend your reservation.
  https://beaker.engineering.redhat.com/recipes/3920506

 For ssh, kvm, serial and power control operations please look here:
  https://beaker.engineering.redhat.com/view/vm-idm-004.testrelm.test

 For the default root password, see:
  https://beaker.engineering.redhat.com/prefs/

      Beaker Test information:
                         HOSTNAME=vm-idm-004.testrelm.test
                            JOBID=1892925
                         RECIPEID=3920506
                    RESULT_SERVER=[::1]:7083
                           DISTRO=RHEL-7.4-20170606.n.0
                     ARCHITECTURE=x86_64

      Job Whiteboard: IPA :: RHEL 7.4 :: Only Master and Client :: quickinstall :: with 99 hrs reserved :: ipa12

      Recipe Whiteboard: CLIENT1
**  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **
Could not chdir to home directory /home/testuser01: No such file or directory
-sh-4.2$ 
-sh-4.2$ 
-sh-4.2$ id 
uid=1063200001(testuser01) gid=1063200001(testuser01) groups=1063200001(testuser01) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
-sh-4.2$ klist
Ticket cache: KEYRING:persistent:1063200001:krb_ccache_pi5zCeb
Default principal: testuser01

Valid starting       Expires              Service principal
2017-06-07T21:39:56  2017-06-08T21:39:56  krbtgt/TESTRELM.TEST
-sh-4.2$ logout
Connection to vm-idm-004.testrelm.test closed.

[root@vm-idm-023 ~]# reboot


[root@vm-idm-023 ~]# ssh -l testuser01 vm-idm-004.testrelm.test 
Password: 
Last login: Wed Jun  7 21:39:57 2017 from 10.65.206.157
**  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **
                 This System is reserved by mvarun.

 To return this system early. You can run the command: return2beaker.sh
  Ensure you have your logs off the system before returning to Beaker

 To extend your reservation time. You can run the command:
  extendtesttime.sh
 This is an interactive script. You will be prompted for how many
  hours you would like to extend the reservation.

 You should verify the watchdog was updated succesfully after
  you extend your reservation.
  https://beaker.engineering.redhat.com/recipes/3920506

 For ssh, kvm, serial and power control operations please look here:
  https://beaker.engineering.redhat.com/view/vm-idm-004.testrelm.test

 For the default root password, see:
  https://beaker.engineering.redhat.com/prefs/

      Beaker Test information:
                         HOSTNAME=vm-idm-004.testrelm.test
                            JOBID=1892925
                         RECIPEID=3920506
                    RESULT_SERVER=[::1]:7083
                           DISTRO=RHEL-7.4-20170606.n.0
                     ARCHITECTURE=x86_64

      Job Whiteboard: IPA :: RHEL 7.4 :: Only Master and Client :: quickinstall :: with 99 hrs reserved :: ipa12

      Recipe Whiteboard: CLIENT1
**  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **
Could not chdir to home directory /home/testuser01: No such file or directory
-sh-4.2$ klist
Ticket cache: KEYRING:persistent:1063200001:krb_ccache_pi5zCeb
Default principal: testuser01

Valid starting       Expires              Service principal
2017-06-07T21:43:22  2017-06-08T21:43:22  krbtgt/TESTRELM.TEST
-sh-4.2$ 



On Master
[root@vm-idm-023 ~]# date
Wed Jun  7 20:43:13 IST 2017
 
[root@vm-idm-023 ~]# ssh -l testuser01 vm-idm-004.testrelm.test -v
OpenSSH_7.4p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 62: Applying options for *
debug1: Executing proxy command: exec /usr/bin/sss_ssh_knownhostsproxy -p 22 vm-idm-004.testrelm.test
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type 2
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: permanently_drop_suid: 0
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to vm-idm-004.testrelm.test:22 as 'testuser01'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305 MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305 MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:8YAhNTtzXNCvWQRawAgiNd9tAW/sfkNQCAxfiigiwu4
debug1: Host 'vm-idm-004.testrelm.test' is known and matches the ECDSA host key.
debug1: Found key in /var/lib/sss/pubconf/known_hosts:3
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:0)
 
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:0)
 
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug1: Offering DSA public key: /root/.ssh/id_dsa
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Trying private key: /root/.ssh/id_ed25519
debug1: Next authentication method: keyboard-interactive
Password:
debug1: Authentication succeeded (keyboard-interactive).
Authenticated to vm-idm-004.testrelm.test (via proxy).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions
debug1: Entering interactive session.
debug1: pledge: proc
debug1: client_input_global_request: rtype hostkeys-00 want_reply 0
debug1: Sending environment.
debug1: Sending env LANG = en_IN.UTF-8
Last failed login: Wed Jun  7 20:30:48 IST 2017 from 10.65.206.157 on ssh:notty
There were 7 failed login attempts since the last successful login.
**  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **
                 This System is reserved by mvarun.
 
 To return this system early. You can run the command: return2beaker.sh
  Ensure you have your logs off the system before returning to Beaker
 
 To extend your reservation time. You can run the command:
  extendtesttime.sh
 This is an interactive script. You will be prompted for how many
  hours you would like to extend the reservation.
 
 You should verify the watchdog was updated succesfully after
  you extend your reservation.
  https://beaker.engineering.redhat.com/recipes/3920506
 
 For ssh, kvm, serial and power control operations please look here:
  https://beaker.engineering.redhat.com/view/vm-idm-004.testrelm.test
 
 For the default root password, see:
  https://beaker.engineering.redhat.com/prefs/
 
      Beaker Test information:
                         HOSTNAME=vm-idm-004.testrelm.test
                            JOBID=1892925
                         RECIPEID=3920506
                    RESULT_SERVER=[::1]:7083
                           DISTRO=RHEL-7.4-20170606.n.0
                     ARCHITECTURE=x86_64
 
      Job Whiteboard: IPA :: RHEL 7.4 :: Only Master and Client :: quickinstall :: with 99 hrs reserved :: ipa12
 
      Recipe Whiteboard: CLIENT1
**  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **
Could not chdir to home directory /home/testuser01: No such file or directory
 
 
-sh-4.2$ klist
Ticket cache: KEYRING:persistent:1063200001:krb_ccache_pi5zCeb
Default principal: testuser01
 
Valid starting       Expires              Service principal
2017-06-07T20:43:40  2017-06-08T20:43:40  krbtgt/TESTRELM.TEST
 
-sh-4.2$ date
Wed Jun  7 20:43:56 IST 2017
 
-sh-4.2$ logoutdebug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow reply 0
 
debug1: channel 0: free: client-session, nchannels 1
Connection to vm-idm-004.testrelm.test closed.
Transferred: sent 3632, received 5084 bytes, in 18.4 seconds
Bytes per second: sent 196.9, received 275.6
debug1: Exit status 0
 

Modifying the time into the past on Master

[root@vm-idm-023 ~]#
[root@vm-idm-023 ~]#
[root@vm-idm-023 ~]# date +%T -s "18:09:16"
18:09:16
 
 
[root@vm-idm-023 ~]#
[root@vm-idm-023 ~]# ssh -l testuser01 vm-idm-004.testrelm.test -v
OpenSSH_7.4p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 62: Applying options for *
debug1: Executing proxy command: exec /usr/bin/sss_ssh_knownhostsproxy -p 22 vm-idm-004.testrelm.test
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type 2
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: permanently_drop_suid: 0
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to vm-idm-004.testrelm.test:22 as 'testuser01'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305 MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305 MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:8YAhNTtzXNCvWQRawAgiNd9tAW/sfkNQCAxfiigiwu4
debug1: Host 'vm-idm-004.testrelm.test' is known and matches the ECDSA host key.
debug1: Found key in /var/lib/sss/pubconf/known_hosts:3
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:0)
 
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:0)
 
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug1: Offering DSA public key: /root/.ssh/id_dsa
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Trying private key: /root/.ssh/id_ed25519
debug1: Next authentication method: keyboard-interactive
Password:
debug1: Authentication succeeded (keyboard-interactive).
Authenticated to vm-idm-004.testrelm.test (via proxy).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions
debug1: Entering interactive session.
debug1: pledge: proc
debug1: client_input_global_request: rtype hostkeys-00 want_reply 0
debug1: Sending environment.
debug1: Sending env LANG = en_IN.UTF-8
Last login: Wed Jun  7 20:43:42 2017 from 10.65.206.157
**  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **
                 This System is reserved by mvarun.
 
 To return this system early. You can run the command: return2beaker.sh
  Ensure you have your logs off the system before returning to Beaker
 
 To extend your reservation time. You can run the command:
  extendtesttime.sh
 This is an interactive script. You will be prompted for how many
  hours you would like to extend the reservation.
 
 You should verify the watchdog was updated succesfully after
  you extend your reservation.
  https://beaker.engineering.redhat.com/recipes/3920506
 
 For ssh, kvm, serial and power control operations please look here:
  https://beaker.engineering.redhat.com/view/vm-idm-004.testrelm.test
 
 For the default root password, see:
  https://beaker.engineering.redhat.com/prefs/
 
      Beaker Test information:
                         HOSTNAME=vm-idm-004.testrelm.test
                            JOBID=1892925
                         RECIPEID=3920506
                    RESULT_SERVER=[::1]:7083
                           DISTRO=RHEL-7.4-20170606.n.0
                     ARCHITECTURE=x86_64
 
      Job Whiteboard: IPA :: RHEL 7.4 :: Only Master and Client :: quickinstall :: with 99 hrs reserved :: ipa12
 
      Recipe Whiteboard: CLIENT1
**  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **  **
Could not chdir to home directory /home/testuser01: No such file or directory
 
 
-sh-4.2$ klist
Ticket cache: KEYRING:persistent:1063200001:krb_ccache_pi5zCeb
Default principal: testuser01
 
Valid starting       Expires              Service principal
2017-06-07T20:43:40  2017-06-08T20:43:40  krbtgt/TESTRELM.TEST
 
 
-sh-4.2$ date
Wed Jun  7 20:44:40 IST 2017
 
 
-sh-4.2$
-sh-4.2$ logoutdebug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow reply 0
 
debug1: channel 0: free: client-session, nchannels 1
Connection to vm-idm-004.testrelm.test closed.
Transferred: sent 3692, received 5096 bytes, in 21.5 seconds
Bytes per second: sent 171.7, received 237.0
debug1: Exit status 0
 
[root@vm-idm-023 ~]# date
Wed Jun  7 18:09:55 IST 2017
[root@vm-idm-023 ~]#
Comment 11 errata-xmlrpc 2017-08-01 08:58:07 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:2294