Bug 1814585

Summary:	"gnutls_record_recv: An unexpected TLS packet was received" when trying to connect to vsftpd configured on RHEL7
Product:	Red Hat Enterprise Linux 7	Reporter:	Fazlin <fazlin.ahamed>
Component:	vsftpd	Assignee:	aegorenk
Status:	CLOSED WONTFIX	QA Contact:	qe-baseos-daemons
Severity:	high	Docs Contact:
Priority:	medium
Version:	7.7	CC:	aegorenk
Target Milestone:	rc
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-12-01 14:03:41 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Fazlin 2020-03-18 10:30:56 UTC

Description of problem:

After configuring "vsftpd" on RHEL7, trying to connect to this FTPS server using "lftp" on RHEL6 fails with following error:

gnutls_record_recv: An unexpected TLS packet was received

The same error is seen when trying to connect using "gnutls-cli" tool also.

Version-Release number of selected component (if applicable):

# lftp -v
LFTP | Version 4.0.9 | Copyright (c) 1996-2010 Alexander V. Lukyanov

# rpm -q gnutls
gnutls-2.12.23-22.el6.x86_64

How reproducible:

Run "lftp" to connect to the vsftpd server configured on RHEL7. The "lftp" works when trying to connect to vsftpd configured on RHEL6.

Steps to Reproduce:
1. Configure vsftpd on RHEL7 with SSL enabled to allow TLS1.2 only.

   ssl_tlsv1_1=NO
   ssl_tlsv1_2=YES
   ssl_tlsv1=NO
   ssl_sslv2=NO
   ssl_sslv3=NO

2. On a RHEL6 box, run "lftp".
3. "lftp" fails.

Actual results:

"lftp" fails with following error:

---> USER anonymous
<--- 331 Please specify the password.
---> PASS xxxx@<hostname>
**** gnutls_record_recv: An unexpected TLS packet was received.
---- Closing control socket

Expected results:

The same "lftp" command when connecting to vsftpd configured on RHEL6:

---> USER anonymous
<--- 331 Please specify the password.                
---> PASS xxxx@<hostname>
<--- 230 Login successful.                                 
---> PWD
<--- 257 "/"                                               
---> PBSZ 0
<--- 200 PBSZ set to 0.                                        
---> CWD /incoming/
<--- 250 Directory successfully changed.                       


Additional info:

Comment 2 Daiki Ueno 2020-03-18 13:09:27 UTC

Thank you for the report, but I can't reproduce the issue.

On a RHEL-7 machine I set up vsftp with:
  ssl_enable=YES
  ssl_tlsv1_1=NO
  ssl_tlsv1_2=YES
  ssl_tlsv1=NO
  ssl_sslv2=NO
  ssl_sslv3=NO

  rsa_cert_file=/etc/vsftpd/gnutls/doc/credentials/x509/cert-rsa.pem
  rsa_private_key_file=/etc/vsftpd/gnutls/doc/credentials/x509/key-rsa.pem

  implicit_ssl=YES
  listen_address=0.0.0.0
  listen_port=9900

using vsftpd-3.0.2-27.el7.x86_64.

On a RHEL-6 machine I could connect to the service with lftp:
  [root@ci-vm-10-0-138-165 ~]# lftp -d ftps://10.0.138.200:9900
  lftp 10.0.138.200:~> set ssl:ca-file "ca.pem"
  lftp 10.0.138.200:~> set ssl:check-hostname false
  lftp 10.0.138.200:~> user test test
  lftp test.138.200:~> ls
  ---- Connecting to 10.0.138.200 (10.0.138.200) port 9900
  Certificate: CN=GnuTLS Test Server (RSA certificate)
   Issued by: CN=GnuTLS Test CA
    Trusted
  <--- 220 (vsFTPd 3.0.2)
  ---> FEAT
  <--- 211-Features:
  <---  AUTH TLS
  <---  EPRT
  <---  EPSV
  <---  MDTM
  <---  PASV
  <---  PBSZ
  <---  PROT
  <---  REST STREAM
  <---  SIZE
  <---  TVFS
  <---  UTF8
  <--- 211 End
  ---> OPTS UTF8 ON
  <--- 200 Always in UTF8 mode.
  ---> USER test
  <--- 331 Please specify the password.
  ---> PASS test
  <--- 230 Login successful.       
  ---> PWD
  <--- 257 "/home/test"
  ---> PBSZ 0
  <--- 200 PBSZ set to 0.
  ---> PROT P
  <--- 200 PROT now Private.
  ---> PASV
  <--- 227 Entering Passive Mode (10,0,138,200,149,170).
  ---- Connecting data socket to (10.0.138.200) port 38314
  ---- Data connection established
  ---> LIST
  <--- 150 Here comes the directory listing.
  Certificate: CN=GnuTLS Test Server (RSA certificate)
   Issued by: CN=GnuTLS Test CA
    Trusted
  ---- Got EOF on data connection 
  ---- Closing data socket
  <--- 226 Directory send OK.
  lftp test.138.200:~> 
  [root@ci-vm-10-0-138-165 ~]# rpm -qa lftp gnutls
  gnutls-2.12.23-22.el6.x86_64
  lftp-4.0.9-15.el6.x86_64

Would it be possible to provide the complete server configuration?

Comment 3 Fazlin 2020-03-18 13:17:52 UTC

Here is my vsftpd conf file. Please note i am using anonymous transfer.

write_enable=YES
local_umask=022
anon_upload_enable=YES
anon_mkdir_write_enable=YES
dirmessage_enable=YES
xferlog_enable=YES
connect_from_port_20=YES
chown_uploads=YES
chown_username=cfgdb
xferlog_file=/var/opt/ftp/logs/xferlog-ssl
xferlog_std_format=NO
ascii_upload_enable=NO
ascii_download_enable=NO
ftpd_banner=secure FTP service
listen=YES
pam_service_name=vsftpd-ssl
userlist_enable=YES
tcp_wrappers=YES
ftp_username=cfgdb
anon_root=/var/opt/ftp/pub/secure
dual_log_enable=YES
hide_file={incoming,other,.*}
hide_ids=YES
log_ftp_protocol=YES
vsftpd_log_file=/var/opt/ftp/logs/vsftpd.log
ftp_data_port=8020
listen_port=8021
ssl_enable=YES
debug_ssl=YES
allow_anon_ssl=YES
anon_umask=022
force_anon_data_ssl=YES
force_anon_logins_ssl=YES
force_local_data_ssl=YES
force_local_logins_ssl=YES
pasv_enable=YES
pasv_min_port=58000
pasv_max_port=58999
ssl_tlsv1_1=NO
ssl_tlsv1_2=YES
ssl_tlsv1=NO
ssl_sslv2=NO
ssl_sslv3=NO
require_ssl_reuse=NO
ssl_ciphers=DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-DSS-AES128-GCM-SHA256:DHE-DSS-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-CHACHA20-POLY1305:DHE-RSA-CHACHA20-POLY1305:AES256-SHA256
rsa_cert_file=/etc/vsftpd/vsftpd.pem

Comment 4 Fazlin 2020-03-18 15:20:25 UTC

Below is the "lftp" script that i used:

set net:max-retries 3
set net:reconnect-interval-base 30
set net:reconnect-interval-max 180
set net:reconnect-interval-multiplier 2
set net:timeout 120
set cmd:trace yes
set ssl:check-hostname no
set ssl:verify-certificate no
open -u anonymous,XXXXXX ftp://<server>:8021
quote pwd
bye

Comment 5 Daiki Ueno 2020-03-18 16:57:07 UTC

Have you checked that this works even on RHEL-7 machine? I had to add modifications to both files to make it work.
- in vsftpd.conf, "rsa_private_key_file" and "implicit_ssl" are missing. also "ftp_username" seems to shadow "anon_root" setting, so I had to set necessary permissions on /home/cfgdb, and also had to ensure permissions for log files
- in lftp script, the protocol should be "ftps" not "ftp"

With those changes in effect, I managed to get the file list as below.

[root@ci-vm-10-0-137-99 ~]# lftp -f lftp.txt -d
+ set ssl:check-hostname no
+ set ssl:verify-certificate no
+ open -u anonymous,XXXXXX ftps://10.0.138.195:9900
---- Resolving host address...
---- 1 address found: 10.0.138.195
+ quote pwd
---- Connecting to 10.0.138.195 (10.0.138.195) port 9900
Certificate: CN=GnuTLS Test Server (RSA certificate)
 Issued by: CN=GnuTLS Test CA
WARNING: Certificate verification: Not trusted
<--- 220 secure FTP service       
---> FEAT
<--- 211-Features:
<---  AUTH TLS
<---  EPRT
<---  EPSV
<---  MDTM
<---  PASV
<---  PBSZ
<---  PROT
<---  REST STREAM
<---  SIZE
<---  TVFS
<---  UTF8
<--- 211 End
---> OPTS UTF8 ON
<--- 200 Always in UTF8 mode.
---> USER anonymous
<--- 331 Please specify the password.
---> PASS XXXXXX
<--- 230 Login successful.
---> PWD
<--- 257 "/"
---> PBSZ 0
<--- 200 PBSZ set to 0.
---> TYPE I
<--- 200 Switching to Binary mode.
---> pwd
257 "/"
+ bye
---> QUIT
---- Closing control socket
[root@ci-vm-10-0-137-99 ~]# rpm -qa lftp
lftp-4.0.9-15.el6.x86_64

Comment 6 Fazlin 2020-03-19 09:44:20 UTC

Thanks for the details. Yes the same issue exists from RHEL7 "lftp" also.

- In vsftpd.conf, added "implicit_ssl=YES", "listen_address=0.0.0.0" and "rsa_private_key_file" and restarted vsftpd.
- In "lftp", used "ftps" instead of "ftp"

But the same error persists.

And here is the concerned portion from the vsftpd log file:

Thu Mar 19 05:21:39 2020 [pid 44829] CONNECT: Client "169.172.87.61"
Thu Mar 19 05:21:39 2020 [pid 44829] DEBUG: Client "169.172.87.61", "SSL version: TLSv1/SSLv3, SSL cipher: AES256-SHA256, not reused, no cert"
Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61", "220 SCpier secure FTP service"
Thu Mar 19 05:21:39 2020 [pid 44829] FTP command: Client "169.172.87.61", "FEAT"
Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61", "211-Features:"
Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61", " AUTH TLS??"
Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61", " EPRT??"
Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61", " EPSV??"
Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61", " MDTM??"
Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61", " PASV??"
Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61", " PBSZ??"
Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61", " PROT??"
Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61", " REST STREAM??"
Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61", " SIZE??"
Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61", " TVFS??"
Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61", " UTF8??"
Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61", "211 End"
Thu Mar 19 05:21:39 2020 [pid 44829] FTP command: Client "169.172.87.61", "OPTS UTF8 ON"
Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61", "200 Always in UTF8 mode."
Thu Mar 19 05:21:39 2020 [pid 44829] FTP command: Client "169.172.87.61", "USER anonymous"
Thu Mar 19 05:21:39 2020 [pid 44829] [anonymous] FTP response: Client "169.172.87.61", "331 Please specify the password."
Thu Mar 19 05:21:39 2020 [pid 44829] [anonymous] FTP command: Client "169.172.87.61", "PASS <password>"
Thu Mar 19 05:21:39 2020 [pid 44828] [cfgdb] OK LOGIN: Client "169.172.87.61", anon password "xxxx@<hostname>"

But the "vsftpd" logs on RHEL6 seems to be able to communicate further after "OK LOGIN":

Thu Mar 19 00:14:22 2020 [pid 21173] [cfgdb] OK LOGIN: Client "168.72.17.208", anon password "root@sd-0bfd-cbc0"
Thu Mar 19 00:14:22 2020 [pid 21175] [cfgdb] FTP response: Client "168.72.17.208", "230 Login successful."
Thu Mar 19 00:14:22 2020 [pid 21175] [cfgdb] FTP command: Client "168.72.17.208", "PWD"
Thu Mar 19 00:14:22 2020 [pid 21175] [cfgdb] FTP response: Client "168.72.17.208", "257 "/""
Thu Mar 19 00:14:22 2020 [pid 21175] [cfgdb] FTP command: Client "168.72.17.208", "PBSZ 0"
Thu Mar 19 00:14:22 2020 [pid 21175] [cfgdb] FTP response: Client "168.72.17.208", "200 PBSZ set to 0."
Thu Mar 19 00:14:22 2020 [pid 21175] [cfgdb] FTP command: Client "168.72.17.208", "CWD /incoming/"
Thu Mar 19 00:14:22 2020 [pid 21175] [cfgdb] FTP response: Client "168.72.17.208", "250 Directory successfully changed."

Comment 7 Fazlin 2020-03-19 14:48:00 UTC

(In reply to Fazlin from comment #6)
> Thanks for the details. Yes the same issue exists from RHEL7 "lftp" also.
> 
> - In vsftpd.conf, added "implicit_ssl=YES", "listen_address=0.0.0.0" and
> "rsa_private_key_file" and restarted vsftpd.
> - In "lftp", used "ftps" instead of "ftp"
> 
> But the same error persists.
> 
> And here is the concerned portion from the vsftpd log file:
> 
> Thu Mar 19 05:21:39 2020 [pid 44829] CONNECT: Client "169.172.87.61"
> Thu Mar 19 05:21:39 2020 [pid 44829] DEBUG: Client "169.172.87.61", "SSL
> version: TLSv1/SSLv3, SSL cipher: AES256-SHA256, not reused, no cert"
> Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61",
> "220 SCpier secure FTP service"
> Thu Mar 19 05:21:39 2020 [pid 44829] FTP command: Client "169.172.87.61",
> "FEAT"
> Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61",
> "211-Features:"
> Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61", "
> AUTH TLS??"
> Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61", "
> EPRT??"
> Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61", "
> EPSV??"
> Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61", "
> MDTM??"
> Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61", "
> PASV??"
> Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61", "
> PBSZ??"
> Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61", "
> PROT??"
> Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61", "
> REST STREAM??"
> Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61", "
> SIZE??"
> Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61", "
> TVFS??"
> Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61", "
> UTF8??"
> Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61",
> "211 End"
> Thu Mar 19 05:21:39 2020 [pid 44829] FTP command: Client "169.172.87.61",
> "OPTS UTF8 ON"
> Thu Mar 19 05:21:39 2020 [pid 44829] FTP response: Client "169.172.87.61",
> "200 Always in UTF8 mode."
> Thu Mar 19 05:21:39 2020 [pid 44829] FTP command: Client "169.172.87.61",
> "USER anonymous"
> Thu Mar 19 05:21:39 2020 [pid 44829] [anonymous] FTP response: Client
> "169.172.87.61", "331 Please specify the password."
> Thu Mar 19 05:21:39 2020 [pid 44829] [anonymous] FTP command: Client
> "169.172.87.61", "PASS <password>"
> Thu Mar 19 05:21:39 2020 [pid 44828] [cfgdb] OK LOGIN: Client
> "169.172.87.61", anon password "xxxx@<hostname>"
> 
> But the "vsftpd" logs on RHEL6 seems to be able to communicate further after
> "OK LOGIN":
> 
> Thu Mar 19 00:14:22 2020 [pid 21173] [cfgdb] OK LOGIN: Client
> "168.72.17.208", anon password "root@sd-0bfd-cbc0"
> Thu Mar 19 00:14:22 2020 [pid 21175] [cfgdb] FTP response: Client
> "168.72.17.208", "230 Login successful."
> Thu Mar 19 00:14:22 2020 [pid 21175] [cfgdb] FTP command: Client
> "168.72.17.208", "PWD"
> Thu Mar 19 00:14:22 2020 [pid 21175] [cfgdb] FTP response: Client
> "168.72.17.208", "257 "/""
> Thu Mar 19 00:14:22 2020 [pid 21175] [cfgdb] FTP command: Client
> "168.72.17.208", "PBSZ 0"
> Thu Mar 19 00:14:22 2020 [pid 21175] [cfgdb] FTP response: Client
> "168.72.17.208", "200 PBSZ set to 0."
> Thu Mar 19 00:14:22 2020 [pid 21175] [cfgdb] FTP command: Client
> "168.72.17.208", "CWD /incoming/"
> Thu Mar 19 00:14:22 2020 [pid 21175] [cfgdb] FTP response: Client
> "168.72.17.208", "250 Directory successfully changed."

Don't know if this matters:

Everytime i restart vsftpd, i see below message in "journalctl -xe" output:

-- Unit vsftpd.service has finished starting up.
-- 
-- The start-up result is done.
Mar 19 10:45:21 testhost.net polkitd[791]: Unregistered Authentication Agent for unix-process:28469:1506130 (system bus name :1.145, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)

Comment 8 Daiki Ueno 2020-03-19 16:38:16 UTC

(In reply to Fazlin from comment #6)
> Thanks for the details. Yes the same issue exists from RHEL7 "lftp" also.
> 
> - In vsftpd.conf, added "implicit_ssl=YES", "listen_address=0.0.0.0" and
> "rsa_private_key_file" and restarted vsftpd.
> - In "lftp", used "ftps" instead of "ftp"
> 
> But the same error persists.
> 
> And here is the concerned portion from the vsftpd log file:
[...]
> But the "vsftpd" logs on RHEL6 seems to be able to communicate further after
> "OK LOGIN":
[...]

Are you sure that both settings are equivalent? I suspect that there is some configuration error (e.g., permission mismatch, as I fixed to make it work) and vsftpd is terminating the connection instead of writing it to the log for security reasons. In any case, I would say it's not a problem in GnuTLS.

I'm reassigning this bug to RHEL-7 vsftpd; probably its maintainers should have more insights.

Comment 10 Fazlin 2020-03-20 08:08:57 UTC

(In reply to Daiki Ueno from comment #8)
> (In reply to Fazlin from comment #6)
> > Thanks for the details. Yes the same issue exists from RHEL7 "lftp" also.
> > 
> > - In vsftpd.conf, added "implicit_ssl=YES", "listen_address=0.0.0.0" and
> > "rsa_private_key_file" and restarted vsftpd.
> > - In "lftp", used "ftps" instead of "ftp"
> > 
> > But the same error persists.
> > 
> > And here is the concerned portion from the vsftpd log file:
> [...]
> > But the "vsftpd" logs on RHEL6 seems to be able to communicate further after
> > "OK LOGIN":
> [...]
> 
> Are you sure that both settings are equivalent? I suspect that there is some
> configuration error (e.g., permission mismatch, as I fixed to make it work)
> and vsftpd is terminating the connection instead of writing it to the log
> for security reasons. In any case, I would say it's not a problem in GnuTLS.
> 
> I'm reassigning this bug to RHEL-7 vsftpd; probably its maintainers should
> have more insights.

Thanks Daiki!!

I cross-checked the RHEL6 & RHEL7 configuration and there is no difference in "vsftpd" settings. Only things that were suggested in this bugzilla are new in RHEL7.

I could see 2 differences in the RHEL6 and RHEL7 setup:
1. RHEL6 does not have SElinux. But i disabled SElinux on RHEL7 but issue still persists.
2. "polkitd" does not run on RHEL6. And as i have shown above, there seems to be some kind of warning/error on RHEL7 related to "polkitd".

Comment 11 Ondřej Lysoněk 2020-03-23 17:41:53 UTC

Hi,

can you run vsftpd using 'strace -f' and post the output? You can limit the set of syscalls to write().

I suspect that the problem is that vsftpd calls its die() function somewhere. It's used for sending error messages to the socket. The message is sent in plaintext, though, which breaks the TLS session.

Comment 12 Fazlin 2020-03-24 07:45:39 UTC

Thanks for suggesting "strace". I was able to identify the issue finally.

I had not created the home directory for the vsftpd user and "strace" output showed failure to change directory (relevant strace output pasted below).

[pid 25421] write(1, "500 OOPS: ", 10)  = 10
[pid 25421] write(1, "cannot change directory:/home/cf"..., 35) = 35
[pid 25421] write(1, "\r\n", 2)         = 2

The questions now are:

1. Why is "vsftpd" looking for home directory of the user and why it is trying to switch to that directory?
2. Is it possible to make "vsftpd" work without having the home directory created?

Comment 15 aegorenk 2020-12-01 14:03:41 UTC

Hi Fazlin, I was able to reproduce the behavior and the strace output as provided by you (cannot change directory:/home/...).
Also, I think with your initial configuration the output of strace would be different, which means there were different problems with the configuration.

However, RHEL-7 is already in Maintenance Support 2 phase, which means that only Critical impact Security Advisories and selected Urgent Priority Bug Fix Advisories may be addressed. Please see https://access.redhat.com/support/policy/updates/errata#Maintenance_Support_2_Phase for further information.

Since this bug does not meet the criteria, we'll close it as WONTFIX. Feel free to discuss this Bug with (Product Management)/Support Representative, if this is a critical issue for the customer/for you. Please provide business justification in such case.

Thank you for taking the time to report this issue to us. We appreciate the feedback and use reports such as this one to guide our efforts at improving our products. That being said, this bug tracking system is not a mechanism for requesting support, and we are not able to guarantee the timeliness or suitability of a resolution.

If this issue is critical or in any way time sensitive, please raise a ticket through the regular Red Hat support channels to ensure it receives the proper attention and prioritization to assure a timely resolution.

For information on how to contact the Red Hat production support team, please visit:
https://access.redhat.com/support

Thanks,
Artem