Bug 1732077

Description of problem:

LFTP from RHEL 7 fails to one of the AWS S3 location  - 

RHEL7 Summary:
---- Resolving host address...
---- 4 addresses found: 52.85.213.20, 52.85.213.136, 52.85.213.76, 52.85.213.39
---- Connecting to yum.datadoghq.com (52.85.213.20) port 443
---- Sending request...
---> HEAD /stable/6/x86_64/ HTTP/1.1
---> Host: yum.datadoghq.com
---> User-Agent: lftp/4.4.8
---> Accept: */*
---> Connection: keep-alive
--->
Certificate: OU=Domain Control Validated,OU=PositiveSSL Wildcard,CN=*.datadoghq.com
 Issued by:        C=GB,ST=Greater Manchester,L=Salford,O=COMODO CA Limited,CN=COMODO RSA Domain Validation Secure Server CA
 Checking against: C=GB,ST=Greater Manchester,L=Salford,O=COMODO CA Limited,CN=COMODO RSA Domain Validation Secure Server CA
  Trusted
Certificate: C=GB,ST=Greater Manchester,L=Salford,O=COMODO CA Limited,CN=COMODO RSA Domain Validation Secure Server CA
 Issued by:        C=GB,ST=Greater Manchester,L=Salford,O=COMODO CA Limited,CN=COMODO RSA Certification Authority
 Checking against: C=GB,ST=Greater Manchester,L=Salford,O=COMODO CA Limited,CN=COMODO RSA Certification Authority
  Trusted
Certificate: C=GB,ST=Greater Manchester,L=Salford,O=COMODO CA Limited,CN=COMODO RSA Certification Authority
 Issued by: C=SE,O=AddTrust AB,OU=AddTrust External TTP Network,CN=AddTrust External CA Root
  Trusted
<--- HTTP/1.1 404 Not Found
<--- Connection: keep-alive
<--- x-amz-error-code: NoSuchKey
<--- x-amz-error-message: The specified key does not exist.
<--- x-amz-error-detail-Key: stable/6/x86_64/index.html
<--- Date: Mon, 22 Jul 2019 13:26:35 GMT
<--- Server: AmazonS3
<--- X-Cache: Error from cloudfront
<--- Via: 1.1 9e0d30e9e48ffdef404c36841b7cb7e5.cloudfront.net (CloudFront)
<--- X-Amz-Cf-Pop: DFW3
<--- X-Amz-Cf-Id: kBeARH1o6fm0gUI_xUv_GEldesmKBDgdE1oBn48Yz9N7gaWZuBREdQ==
<---
mirror: Access failed: 404 Not Found (/stable/6/x86_64)
---- Closing HTTP connection

Version-Release number of selected component (if applicable):

latest version

How reproducible:

Always

Steps to Reproduce:

[root@shaurya23091c ~]# lftp -e "debug 13" https://yum.datadoghq.com/stable/6/x86_64
cd: Access failed: 404 Not Found (/stable/6/x86_64)                       
lftp yum.datadoghq.com:/> ls
FileCopy(0x1258a40) enters state INITIAL
FileCopy(0x1258a40) enters state DO_COPY
---- Sending request...
---> GET / HTTP/1.1
---> Host: yum.datadoghq.com
---> User-Agent: lftp/4.4.8
---> Accept: */*
---> Connection: keep-alive
---> 
<--- HTTP/1.1 200 OK                 
<--- Content-Type: text/html
<--- Content-Length: 313
<--- Connection: keep-alive
<--- Date: Mon, 24 Jun 2019 13:42:41 GMT
<--- Last-Modified: Thu, 27 Aug 2015 18:49:24 GMT
<--- x-amz-version-id: fCREzyXEk8_dxBgZ7DnoouUVvNHKfWpI
<--- ETag: "1426d3be19451b5a37f9b130a9e5afd5"
<--- Server: AmazonS3
<--- X-Cache: RefreshHit from cloudfront
<--- Via: 1.1 e38902d67e98c06c59b2b9295ce6ef05.cloudfront.net (CloudFront)
<--- X-Amz-Cf-Pop: DUB2-C1
<--- X-Amz-Cf-Id: VZXSPDvwFX1-ROCWq5G9OLcLjxrWJn5O-rlJZ1tLqVmziIQ32Ixg-A==
<--- 
---- Receiving body...
0:0 translated to pair 0:0 (0,0)
0 translated to pair 0:0 (0,0)
0:0 translated to pair 0:0 (0,0)
0 translated to pair 0:0 (0,0)
---- Received all
Found tag script, link_target=https://ajax.googleapis.com/ajax/libs/jquery/1.9.0/jquery.min.js
Found tag script, link_target=/list.js
-rw-r--r--  --  list.js
copy: get hit eof
copy: waiting for put confirmation
FileCopy(0x1258a40) enters state CONFIRM_WAIT
copy: put confirmed store
FileCopy(0x1258a40) enters state GET_DONE_WAIT
copy: get is finished - all done
FileCopy(0x1258a40) enters state ALL_DONE
lftp yum.datadoghq.com:/> 

Actual results:

404 Not found

Expected results:
Expected LFTP Result: Mirror Packages

Additional info: