SMTP STARTTLS in sendmail

SMTP STARTTLS in sendmail/Secure Switch

Last Update 2003-11-30

[Terminology] [Compilation] [Configuration] [Operation] [Misc]

Introduction

Sendmail Secure Switch/ sendmail 8.11/ 8.12 support SMTP STARTTLS as defined in RFC 2487 which is based on TLS. This document describes the necessary steps to use this feature.

Terminology

TLS can provide authentication (identification of the communication partner), privacy/confidentiality (communication is not intercepted or eavesdropped), and integrity (message has not been modified). TLS uses different algorithms for encryption, signing, message authentication etc. An introduction to these can also be found in books about cryptography and in RFC 2246 and its list of references. Some important terms used here are:

• CA: certificate authority,
• cert: a certificate (signed/issued by a CA),
• private key: the private key belonging to a cert.

For those familiar with public key encryption (and signing), a cert can be considered a signed public key (with associated data to identify an entity).

Initial Setup

If sendmail should use STARTTLS as server and as client, you have to (names starting with conf refer to m4 variable names used in a .mc file):

1. Install one or more CA certs in confCACERT. Notice: do not list too many root CAs in that file, otherwise the TLS handshake may fail; e.g.,

   error:14094417:SSL routines:SSL3_READ_BYTES:
   sslv3 alert illegal parameter:s3_pkt.c:964:SSL alert number 47
   

   You should probably put only the CA cert into that file that signed your own cert(s).
2. Install zero or more CA certs in confCACERT_PATH with (symbolic) links of their hashes pointing to them:

   C=FileName_of_CA_Certificate
   ln -s $C `openssl x509 -noout -hash < $C`.0
   

   (or sslc instead of openssl)

   These CA certificates are required to successfully authenticate another entity. The signature of the certificate presented by the other side is checked against these CAs. If one of them issued the certificate, the authentication is considered successful. Moreover, during the TLS handshake, the DNs of these CA certificates are sent to the client so it can properly select a certificate that is signed by one of those CAs.

3. Get a cert (make sure the CN is the fully qualified name of your host) and install it as confSERVER_CERT and the private key as confSERVER_KEY (make sure the file is only readable by root or the trusted user). For simplicity, use the same filenames for confCLIENT_CERT and confCLIENT_KEY, respectively.
   Note: the private key must not be encrypted. This is required for unattended startup of sendmail. Otherwise someone/something would have to enter the passphrase each time sendmail is started as server or client.
4. If you run sendmail 8.11 or later and your OS doesn't have /dev/urandom(4), then you need to setup a source to seed the pseudo random number generator. For Solaris 7 and 8, you may have a look at a kernel module for /dev/random (please check yourself whether it is ok to use!), or see whether Sun has a package called SUNWski for your OS. It is strongly advised to use at least EGD (Entropy Gathering Daemon) and compile sendmail with the flag EGD, and point confRAND_FILE to the socket used by EGD (use `egd:' as prefix!). If neither /dev/urandom(4) nor EGD are available, you have to make sure that useful random data is available all the time in confRAND_FILE (use `file:' as prefix). If the file hasn't been modified in the last 10 minutes before it is supposed to be used by sendmail the content is considered obsolete. In this case, the PRNG for TLS is only seeded with other random data if the DontBlameSendmail option InsufficientEntropy is set. This is almost always not sufficient for security.

You can create your own CA: Lutz Jänicke, Gregory Neil Shapiro, and Martin Ouwehand (for Apache, but most of the steps can be used for an MTA too) wrote explanations how to do it.

Extract from a .mc file:

define(`confCACERT_PATH', `/etc/mail/certs')dnl
define(`confCACERT', `/etc/mail/certs/CAcert.pem')dnl
define(`confSERVER_CERT', `/etc/mail/certs/MYcert.pem')dnl
define(`confSERVER_KEY', `/etc/mail/certs/MYkey.pem')dnl
define(`confCLIENT_CERT', `/etc/mail/certs/MYcert.pem')dnl
define(`confCLIENT_KEY', `/etc/mail/certs/MYkey.pem')dnl

Initial Test

Start the sendmail daemon, connect to it and see whether it comes up with

250-STARTTLS

in the EHLO response:

% telnet localhost 25
Trying 127.0.0.1...
Connected to localhost
Escape character is '^]'.
220 local.sendmail.org ESMTP Sendmail Sendmail 8.12.0/8.12.0; Sun, 30 Sep 2001 10:47:28 -0700 (PDT)
ehlo localhost
250-local.sendmail.org Hello localhost [127.0.0.1], pleased to meet you
250-ENHANCEDSTATUSCODES
250-DSN
250-STARTTLS
250 HELP
quit

If it doesn't, check your logfile whether any security problems are listed (unsafe files). If this doesn't reveal any problems, increase the LogLevel to 14 and try again.

Operation

STARTTLS can be used to allow relaying based on certificates, and to restrict incoming or outgoing connections. For this purpose, several rulesets are available which require some new macros and the access map.

New Macros

New macros for SMTP STARTTLS are

{cert_issuer}

holds the DN of the CA (the cert issuer).

{cert_subject}

holds the DN of the cert (called the cert subject).

{tls_version}

the TLS/SSL version used for the connection: TLSv1, SSLv3, SSLv2.

{cipher}

the cipher suite used for the connection, e.g., EDH-DSS-DES-CBC3-SHA, EDH-RSA-DES-CBC-SHA, DES-CBC-MD5, DES-CBC3-SHA, RC2-CBC-MD5, RC4-MD5

{cipher_bits}

the key length (in bits) of the symmetric encryption algorithm used for the connection.

{verify}

holds the result of the verification of the presented cert. Possible values are:

OK

verification succeeded

NO

no cert presented

FAIL

cert presented but could not be verified, e.g., CA is missing.

NONE

STARTTLS has not been performed

TEMP

temporary error occurred, e.g., the other side replied with 454.

PROTOCOL

protocol error occurred

SOFTWARE

problems during the handshake at the TLS level. In this case the connection will be dropped.

{server_name}

the name of the server of the current outgoing SMTP connection.

{server_addr}

the address of the server of the current outgoing SMTP connection.

Relaying

SMTP STARTTLS can allow relaying for senders who have successfully authenticated themselves. This is done in the ruleset RelayAuth. If the verification of the cert failed ({verify} != OK), relaying is subject to the usual rules. Otherwise the DN of the issuer is looked up in the access map using the tag CERTISSUER. If the resulting value is RELAY, relaying is allowed. If it is SUBJECT, the DN of the CERT subject is looked up next in the access map. using the tag CERTSUBJECT. If the value is RELAY, relaying is allowed.

To make things a bit more flexible (or complicated), the values for {cert_issuer} and {cert_subject} can be optionally modified by regular expressions defined in the m4 variables _CERT_REGEX_ISSUER_ and _CERT_REGEX_SUBJECT_, respectively. To avoid problems with those macros in rulesets and map lookups, they are modified as follows: each non-printable character and the characters '<', '>', '(', ')', '"', '+' are replaced by their HEX value with a leading '+'. For example:
/C=US/ST=California/O=endmail.org/OU=private/CN=Darth Mail (Cert)/Email=darth+cert@endmail.org
is encoded as:
/C=US/ST=California/O=endmail.org/OU=private/CN=Darth+20Mail+20+28Cert+29/Email=darth+2Bcert@endmail.org

To allow relaying for everyone who can present a cert signed by /C=US/ST=California/O=endmail.org/OU=private/CN=Darth+20Mail+20+28Cert+29/Email=darth+2Bcert@endmail.org simply use:

CERTIssuer:/C=US/ST=California/O=endmail.org/OU=private/CN=Darth+20Mail+20+28Cert+29/Email=darth+2Bcert@endmail.org		RELAY

To allow relaying only for a subset of machines that have a cert signed by /C=US/ST=California/O=endmail.org/OU=private/CN=Darth+20Mail+20+28Cert+29/Email=darth+2Bcert@endmail.org use:

CERTIssuer:/C=US/ST=California/O=endmail.org/OU=private/CN=Darth+20Mail+20+28Cert+29/Email=darth+2Bcert@endmail.org		SUBJECT
CERTSubject:/C=US/ST=California/O=endmail.org/OU=private/CN=DeathStar/Email=deathstar@endmail.org		RELAY

Of course it is also possible to write a simple rulesets that allows relaying for everyone who can present a cert that can be verified, e.g.,

LOCAL_RULESETS
SLocal_check_rcpt
R$*		$: $&{verify}
ROK		$# OK

Allowing Connections

The rulesets tls_server and tls_client are used to decide whether an SMTP connection is accepted (or should continue).

tls_server is called when sendmail acts as client after a STARTTLS command (should) have been issued. The parameter is the value of {verify}.

tls_client is called when sendmail acts as server, after a STARTTLS command has been issued, and from check_mail. The parameter is the value of {verify} and STARTTLS or MAIL, respectively.

Both rulesets behave the same. If no access map is in use, the connection will be accepted unless {verify} is SOFTWARE, in which case the connection is always aborted. Otherwise, {client_name}/{server_name} is looked up in the access map using the tag TLS_Srv (or TLS_Clt), which is done with the ruleset LookUpDomain. If no entry is found, {client_addr}/{server_addr} is looked up in the access map (same tag, ruleset LookUpAddr). If this doesn't result in an entry either, just the tag is looked up in the access map (included the trailing :). Values for those entries in the access map should be:

VERIFY

verification must have succeeded

VERIFY:bits

verification must have succeeded and {cipher_bits} must be greater than or equal bits.

ENCR:bits

{cipher_bits} must be greater than or equal bits.

The RHS can optionally be prefixed by TEMP+ or PERM+ to select a temporary or permanent error. The default is a temporary error code (403 4.7.0) unless the macro TLS_PERM_ERR is set during generation of the .cf file.

If a certain level of encryption is required, then it might also be possible that this level is provided by TLS from a SASL algorithm, e.g., DIGEST-MD5. The number of bits used for the key of the symmetric cipher (here: 3DES) is stored in the macro {auth_ssf}.

Example: e-mail send to secure.example.com should only use an encrypted connection. e-mail received from hosts within the laptop.example.com domain should only be accepted if they have been authenticated.

TLS_Srv:secure.example.com	ENCR:112
TLS_Clt:laptop.example.com	PERM+VERIFY:112

sendmail 8.12 additionally offers TLS_Rcpt to enable per-recipient restrictions.

Received: Header

The Received: header reveals whether STARTTLS has been used. It contains an extra line:

(using ${tls_version} with cipher ${cipher} (${cipher_bits} bits) verified ${verify})

Notice: sendmail 8.12 has extended or slightly change some of this. For example, the STARTTLS related part of the Received: header looks now like this:

(version=${tls_version} cipher=${cipher} bit=${cipher_bits}  verify=${verify})

Testing

If you have setup all necessary certificates, you can test a STARTTLS connection by sending an e-mail to bouncer@test.smtp.org, see test.smtp.org for details.

Advantages and Limits of TLS

STARTTLS has the following advantages:

• authentication: client and server of a SMTP connection can be identified.
• privacy/confidentiality: the transmission of an e-mail between a client and server utilizingSTARTTLS can not be read and retranslated into plaintext provided a sufficiently secure ciphersuite has been negotiated.
• integrity: the plaintext of an e-mail between a client and server utilizing STARTTLS can not be modified by an adversary provided a sufficiently secure ciphersuite has been negotiated.

All of these advantages are provided transparently by MTAs without interaction of users. They do not need to have special software installed in their MUAs which additionally must be compatible with the software of the recipient. This is also the reason for several limits:

• It does not provide end-to-end encryption, since a user can usually not control the whole transmission. This is in contrast to the use of TLS for http: here the user's client (a WWW browser) connects directly to the server that provides the data. e-mail can be transferred via multiple hops of which the sender can control at most the first.
• It does not provide message authentication, unless the e-mail has been sent directly from the client's (STARTTLS-capable) MUA to the recipients MTA which must record the client's certificate. Even then the message might be faked during local delivery.

Summary: to get end-to-end privacy, integrity and authentication, end-user software like PGP or S/MIME must be used. This requires at least some knowledge of such software and responsible use by end-users.

Misc

Interoperability

Sendmail Secure Switch/ sendmail 8.11 and later has been successfully tested against MTAs running postfix+STARTTLS patch, qmail+STARTTLS patch, PMDF V5.2-32, and zmailer.

Interoperability Problems

Sendmail Secure Switch/ sendmail 8.11 and later can't talk with some MTAs/MUAs running

• CommuniGate Pro 3.2.4 (and 3.2.3): if this MTA is not correctly configured, it offers STARTTLS but then fails during handshake.
• CommuniGate Pro 3.3betaX: this MTA speaks SSLv3 but not TLSv1. (2000-05-19): This was a bug in older versions of CommuniGate Pro 3.3 which is solved in Beta7 (?).
• InterChange v3.61.01. This MTA seems to understand only SSLv3, but not TLSv1 or the default mode that OpenSSL uses. However, v3.61.06 seems to interact just fine with sendmail.
• Some Lotus Domino versions simply drop the connection when STARTTLS is tried.
• sendmail can't talk with MTAs that use TLS implementations which use RSARef, if key length greater than 1024bits(?) are used. Please don't use RSARef since there are fully functional RSA implementations available.
• Eudora seems to have a problem with TLS, see this article from the openssl mailing list:

  Message-Id: <m18E2D5-000QoJC@epsilon>
  Date: Tue, 19 Nov 2002 07:50:11 +0100 (CET)
  From: (Bodo Moeller)
  Subject: Re: Error using Eudora, Sendmail and SSL
  
  Claus Assmann  wrote
  >> Ed Kasky  wrote
  
  >>> 27781:error:140890E9:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:tls peer did
  >>> not respond with certificate list:s3_srvr.c:1638:
  
  > AFAIK the client can respond with an empty list (RFC 2246, section
  > 7.4.6).  This is what sendmail (using OpenSSL) does when it acts
  > as client and no client cert is available.
  > 
  > The question is whether Eudora is broken and should respond with
  > an empty list too.
  
  It is broken, but the solution to Ed's problem is to simply disable
  TLS 1.0 support in Eudora.  (SSL 3.0 is specified differently, and the
  Eudora behaviour of not sending a certificate list when no appropriate
  certificate is available is correct for SSL 3.0.)
  

M$ Exchange can't send e-mail to Sendmail Secure Switch/ sendmail if STARTTLS is used. M$ knows about the problem and will provide a fix: ... and a QFE fix is now available from PSS. The fix will also be included in the next Windows 2000 service pack after SP2 and is already in Windows Xp 2002 beta.

Do not try STARTTLS with some servers

A configuration option to turn off STARTTLS in sendmail (as client) is implemented in the 8.12, it is not available in 8.11.

STARTTLS Debugging by Hand

A patch has been posted to the OpenSSL mailing list which contains a program similar to s_client to connect to MTAs for testing STARTTLS. Starting with OpenSSL 0.9.7, you can use
openssl s_client -starttls smtp -connect host:port ...
directly.

Credits

The implementation of STARTTLS in sendmail was influenced by the example programs of OpenSSL and the work of Lutz Jänicke.

References

An introduction to TLS by Stephen Thomas: "SSL and TLS Essentials", ISBN 0-471-38354-6. A basic introduction is Applied Cryptography. A lot of details are available in Menezes, van Oorschot and Vanstone: "Handbook of Applied Cryptography", ISBN: 0-8493-8523-7

If you prefer french instead of english, please read the instructions from Jean-Philippe Donnio.