Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Created attachment 1413666[details]
datanode log from the server
Description of problem:
Hadoop Datanode service fails with error attached in hadoop-hdfs-datanode.log.
Below errors were seen in Kerberos log:
Mar 27 14:48:17 pts00433-vm38.persistent.co.in krb5kdc[8737](info): TGS_REQ (1 etypes {16}) 10.77.67.132: PROCESS_TGS: authtime 0, dn/pts00433-vm38.persistent.co.in for nn/pts00433-vm38.persistent.co.in, Ticket expired
Mar 27 14:48:55 pts00433-vm38.persistent.co.in krb5kdc[8737](info): TGS_REQ (4 etypes {18 17 16 23}) 10.77.67.132: PROCESS_TGS: authtime 0, nn/pts00433-vm38.persistent.co.in for nn/pts00433-vm38.persistent.co.in, Ticket expired
# krb5-config --version
Kerberos 5 release 1.15.1
# uname -a
Linux pts00433-vm38.persistent.co.in 3.10.0-830.el7.ppc64le #1 SMP Mon Jan 15 12:26:57 EST 2018 ppc64le ppc64le ppc64le GNU/Linux
# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.5 Beta (Maipo)
Version-Release number of selected component (if applicable):
# yum list installed | grep krb
krb5-devel.ppc64le 1.15.1-18.el7 installed
krb5-libs.ppc64le 1.15.1-18.el7 @anaconda/7.5
krb5-pkinit.ppc64le 1.15.1-18.el7 installed
krb5-server.ppc64le 1.15.1-18.el7 installed
krb5-workstation.ppc64le 1.15.1-18.el7 installed
How reproducible:
Errors logged multiple times in Datanode service log and even for other services log eg: Ambari Infra Solr, Namenode, etc.
Steps to Reproduce:
1.Install HDP2.6.4 with Ambari2.6.1
2.Enable Kerberos security configuration on Ambari.
3.Start Datanode service.
4.Verify Datanode log for Exception
Actual results:
Below error in service log:
2018-03-27 14:46:44,739 WARN ipc.Client (Client.java:run(711)) - Couldn't setup connection for dn/pts00433-vm38.persistent.co.in to pts00433-vm38.persistent.co.in/10.77.67.132:8020
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Ticket expired (32) - PROCESS_TGS)]
Expected results:
No exception shown for Kerberos with Datanode or any service in HDP.
Additional info:
The errors are not seen when Ambari on RHEL7.5 is pointed to Kerberos server running on RHEL7.3.
Note: difference is krb build versions on each platform ie: 1.15.1-8.el7 and 1.15.1-18.el7 on RHEL7.3 and RHEL7.5 respectively.
With source compiled krb5 versions 1.15.1 & 1.15.2 on Power RHEL-7.5, the issue was not reproducible for us. These are maintenance releases from the community. We are not able to find exact source for krb5 build version 1.15.1-8 or 1.15.1-18.
Comment 5Hanns-Joachim Uhl
2018-03-29 16:07:19 UTC
(In reply to Robbie Harwood from comment #4)
> Hi, I'm aware of this issue and planned to fix it with rhel-7.5 GA -
> krb5-1.15.1-19.
>
.
Hello Red Hat / Robbie or Joe,
... with RHEL7.5 being closed is there already a 7.5.z zstream bugzilla
open for this issue ...?
If yes, can you please authorize us for that bugzilla ...?
Please advise ...
Thanks in advance for your support.
Hello,
We can not request a Z stream until we have a fix that has been approved for RHEL 7.6.
Robbie,
Would you be able to confirm is there is a RHEL 7.5 BZ for comment 4?
Thank You
Joe Kachuck
Created attachment 1413666 [details] datanode log from the server Description of problem: Hadoop Datanode service fails with error attached in hadoop-hdfs-datanode.log. Below errors were seen in Kerberos log: Mar 27 14:48:17 pts00433-vm38.persistent.co.in krb5kdc[8737](info): TGS_REQ (1 etypes {16}) 10.77.67.132: PROCESS_TGS: authtime 0, dn/pts00433-vm38.persistent.co.in for nn/pts00433-vm38.persistent.co.in, Ticket expired Mar 27 14:48:55 pts00433-vm38.persistent.co.in krb5kdc[8737](info): TGS_REQ (4 etypes {18 17 16 23}) 10.77.67.132: PROCESS_TGS: authtime 0, nn/pts00433-vm38.persistent.co.in for nn/pts00433-vm38.persistent.co.in, Ticket expired # krb5-config --version Kerberos 5 release 1.15.1 # uname -a Linux pts00433-vm38.persistent.co.in 3.10.0-830.el7.ppc64le #1 SMP Mon Jan 15 12:26:57 EST 2018 ppc64le ppc64le ppc64le GNU/Linux # cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.5 Beta (Maipo) Version-Release number of selected component (if applicable): # yum list installed | grep krb krb5-devel.ppc64le 1.15.1-18.el7 installed krb5-libs.ppc64le 1.15.1-18.el7 @anaconda/7.5 krb5-pkinit.ppc64le 1.15.1-18.el7 installed krb5-server.ppc64le 1.15.1-18.el7 installed krb5-workstation.ppc64le 1.15.1-18.el7 installed How reproducible: Errors logged multiple times in Datanode service log and even for other services log eg: Ambari Infra Solr, Namenode, etc. Steps to Reproduce: 1.Install HDP2.6.4 with Ambari2.6.1 2.Enable Kerberos security configuration on Ambari. 3.Start Datanode service. 4.Verify Datanode log for Exception Actual results: Below error in service log: 2018-03-27 14:46:44,739 WARN ipc.Client (Client.java:run(711)) - Couldn't setup connection for dn/pts00433-vm38.persistent.co.in to pts00433-vm38.persistent.co.in/10.77.67.132:8020 javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Ticket expired (32) - PROCESS_TGS)] Expected results: No exception shown for Kerberos with Datanode or any service in HDP. Additional info: