Description of problem:

sendmail use macro MAXHOSTNAMELEN as limit for host name length. It's
defined as '64' in /usr/include/asm/param.h.

MAXHOSTNAMELEN should be used against the dot('.') separated label
length only, but in some parts it be used against FQDN length. FQDN
max length is defined '255' by the RFC1035.

Therefore some problems occur.

Version-Release number of selected component (if applicable):

I checked the sendmail-8.13.1-3.3.el4 for RHEL4.7.  But it may be same
for RHEL5.3, I guess.

How reproducible:

always

Steps to Reproduce:

I confirmed 2 cases.

case 1)

1. setup DNS for 2 hosts, one's name length is less than 64
   (1.0.168...com), the other is more(128.128.128...com).

   # host 1.0.168.192.long.longlong.longlonglong.veryverylong.example.com
   1.0.168.192.long.longlong.longlonglong.veryverylong.example.com has address 127.0.0.64
   # host 128.128.168.192.long.longlong.longlonglong.veryverylong.example.com
   128.128.168.192.long.longlong.longlonglong.veryverylong.example.com has address 127.0.0.64

2. configure /etc/mail/sendmail.cf for dnsbl.

   # DNS based IP address spam list long.longlong.longlonglong.veryverylong.example.com
   R$*                     $: $&{client_addr}
   R$-.$-.$-.$-            $: <?> $(dnsbl $4.$3.$2.$1.long.longlong.longlonglong.veryverylong.example.com. $: OK $)
   R<?>OK                  $: OKSOFAR
   R<?>$+<TMP>             $: TMPOK
   R<?>$+                  $#error $@ 5.7.1 $:  "550 Mail from " $&{client_addr} " blocked using Email Reputation." 

3. test these addresses.

   # echo -e ".D{client_addr}192.168.0.1\ncheck_rcpt root@localhost" | sendmail -bt | tail
   A                  input: < 192 . 168 > < ? > < + Connect > < >
   A                  input: < 192 > < ? > < + Connect > < >
   A                returns: < ? > < >
   A                returns: < ? > < >
   A                returns: < ? > < >
   A                returns: < ? > < >
   Basic_check_rela returns: $# error $@ 5 . 7 . 1 $: "550 Mail from " 192 . 168 . 0 . 1 " blocked using Email Reputation."
   checkrelay       returns: $# error $@ 5 . 7 . 1 $: "550 Mail from " 192 . 168 . 0 . 1 " blocked using Email Reputation."
   check_rcpt       returns: $# error $@ 5 . 7 . 1 $: "550 Mail from " 192 . 168 . 0 . 1 " blocked using Email Reputation."
   > 
   # echo -e ".D{client_addr}192.168.128.128\ncheck_rcpt root@localhost" | sendmail -bt | tail
   A                  input: < 192 . 168 > < ? > < + Connect > < >
   A                  input: < 192 > < ? > < + Connect > < >
   A                returns: < ? > < >
   A                returns: < ? > < >
   A                returns: < ? > < >
   A                returns: < ? > < >
   Basic_check_rela returns: OKSOFAR
   checkrelay       returns: OKSOFAR
   check_rcpt       returns: < ? > root < @ localhost . example . com >

case 2)

1. setup DNS for 2 CNAME. One's name length is less than 64
   (long.verylong...com), the other is more(longhost.verylong...com).

   # host long.verylong.veryverylong.veryveryverylong.toolong.example.com
   long.verylong.veryverylong.veryveryverylong.toolong.example.com is an alias for server.example.com.
   server.example.com has address 192.168.0.1
   # host longhost.verylong.veryverylong.veryveryverylong.toolong.example.com
   longhost.verylong.veryverylong.veryveryverylong.toolong.example.com is an alias for server.example.com.
   server.example.com has address 192.168.0.1

2. test with '/canon'

   # echo '/canon long.verylong.veryverylong.veryveryverylong.toolong.example.com' | sendmail -bt 
   ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
   Enter <ruleset> <address>
   > getcanonname(long.verylong.veryverylong.veryveryverylong.toolong.example.com) returns server.example.com
   # echo '/canon longhost.verylong.veryverylong.veryveryverylong.toolong.example.com' | sendmail -bt 
   ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
   Enter <ruleset> <address>
   > Name too long
  
Actual results:

case 1) Error doesn't occur by the host '192.168.128.128...'.
case 2) The host 'longhost...' isn't resolved.

Expected results:

case 1) Error occurs by both hosts.
case 2) Both hosts are resolved.

Additional info:

case 1)
sendmail/sm_resolve.c::parse_dns_reply()
----------------------------------------------------------------------
 171         char host[MAXHOSTNAMELEN];
	:
 185         status = dn_expand(data, data + len, p, host, sizeof host);
 186         if (status < 0)
 187         {
 188                 dns_free_data(r);
 189                 return NULL;
----------------------------------------------------------------------

dn_expand() return '-1', if expanded host name length is more than
MAXHOSTNAMELEN as 64. Then expected error don't occur.

case 2)
./sendmail/main.c::testmodeline()
----------------------------------------------------------------------
4125       else if (sm_strcasecmp(&line[1], "canon") == 0)
4126       {
4127               char host[MAXHOSTNAMELEN];
	:
4135               else if (sm_strlcpy(host, p, sizeof host) >= sizeof host)
4136               {
4137                       (void) sm_io_fprintf(smioout, SM_TIME_DEFAULT,
4138                                            "Name too long\n");
4139                       return;
----------------------------------------------------------------------

If /canon hostname length is more than MAXHOSTNAMELEN, it occurs "Name
too long".

case X)
MAXHOSTNAMELEN is used other parts, too. I don't check these, but they
might also have problems.