Warning: Please read the configuration section below before you use the RPM packages which you can download from here. Ignore this warning and you will be an open relay. You have been warned.


This article explains how to configure Sendmail MTA to use secure SMTP connections with STARTTLS on Red Hat Linux 7.3.

Red Hat 7.3 comes with Sendmail 8.11 and all updates keep that version number. STARTTLS is disabled at the compilation time. If you want to use STARTTLS without upgrading the operating system, then you need to compile Sendmail with STARTTLS enabled. There is no way around.

I could have used the source RPM and compiled the 8.11 with STARTTLS support, but rumors say that crypto support in version 8.12 is a lot better, so I decided to use that version. Since RH 7.3 update area does not offer this version, I used the one for RH 8.0. Note that you cannot simply get the RPM for RH 8.0 and install it. It needs a newer glibc package and updating that could easily break everything else. Instead, I took the source RPM of Sendmail 8.12 from RH 8.0 and built it to work with RH 7.3

There are other resources about this topic which also helped me doing this, but none offered a pre-built RPM to get, install and be done with it. To correct this situation, I do offer my RPM for download here. What follows is the explanation of what I have done to build it.


Backup your data. Backup your entire server. Best, try this out on a spare machine where you can afford to loose it all. Don't trust the coincidence, it does not exist.

Once again, backup your data. Backup the Sendmail configuration files, backup everything. I mean it, just the way I said it, backup everything.

Updating The System

You probably won't need to update anything to successfuly build Sendmail 8.12 RPM on your RH 7.3 system. However, I haven't tried it without updates, so I don't really know. Also, if you will download and use my RPM, then you should have at least the same versions of the packages Sendmail depends on. This is usually just check-and-skip, because your system probably is up-to-date allready.

If you must upgrade something, I recommend shutting down as many daemons as you can before upgrading. I don't know what happens when you change libraries running daemons depend on. I had just the sshd running while I upgraded the packages.

The packages Sendmail depends on which I have updated or installed before making the RPM are:

  • rpm-build, version 4.0.4-7x.18
  • gdbm and gdbm-devel version 1.8.0-14
  • hesiod and hesiod-devel version 3.0.2-18
  • krb5-libs and krb5-devel version 1.2.4-11
  • openldap and openldap-devel version 2.0.27-2.7.3
  • openssl and openssl-devel version 0.9.6b-32.7
  • pam and pam-devel version 0.75-46.7.3
  • cyrus-sasl and cyrus-sasl-devel version 1.5.24-25

If you haven't installed one of the packages above, install it. They are needed, all of them. If you have them installed, then make sure that version numbers are at least those which you see above. To see currently installed version, type

# rpm -q <packagename>

All of these packages are either a part of the stock RH 7.3, or can be found among the RH 7.3 updates. You can get them in the usual places and install them in the usual way, for example, go to Red Hat's site and locate a download mirror nearest to you, which has the distribution and updates. If these packages need other packages to be installed or updated beforehand, do it.

Installing Sendmail Source

You will need the source RPM of Sendmail 8.12. This you can get from the stock RH 8.0, or better, check if the RH 8.0 updates contain a newer one. The one I have used does come from the RH 8.0 updates and can be downloaded here.

Install the Sendmail's source RPM. Assuming you have the same version as I do, then type

# rpm -i sendmail-8.12.8-9.80.src.rpm

and if that goes good, the source will end up in the /usr/src/redhat tree. If things go wrong, make sure you have rpm-build package installed before doing this.

Building And Installing

The Sendmail package from RH 8.0 which you just have installed has STARTTLS allready activated, so we don't need to worry about that. But we must make sure that it builds on RH 7.3. Fortunately, Sendmail source RPM allready knows how it can be built on RH 7.3 if you just tell it to do so.

Edit the file /usr/src/redhat/SPECS/sendmail.spec and change the variable 'errata' to be '73' instead of '80':

# cd /usr/src/redhat/SPECS
# cp sendmail.spec sendmail.spec.old
# cat sendmail.spec.old \
  | sed "s/%define *errata *80/%define errata 73/" \
  > sendmail.spec

With that done, build the Sendmail RPMs:

# cd /usr/src/redhat/SPECS
# rpm -ba sendmail.spec

The last command will take a while. Once it finishes, you will have a set of Sendmail RPMs piled neatly beneath /usr/src/redhat/RPMS. If it fails, check the output, perhaps you are missing a package or two needed to build Sendmail. The RPMs I have built can be downloaded here.

Now you can install the new Sendmail. Remember to shut down the allready running one before you do so. Install at least sendmail and sendmail-cf RPMs.

Configuring Sendmail

Whatever you did to your /etc/mail/ file with the old Sendmail in charge is still there. RPM has installed the new file as /etc/mail/ I suggest you use this new file as a basis and merge your modifications from the old file. You must restart sendmail whenever you modify this file.

Now you must ensure that the file /etc/mail/ contains things which are required for the proper operation of STARTTLS. The first thing the server needs are certificates. You tell Sendmail where they are with the following instructions. Change the paths as necessary.


If you wish your server to use a certificate when it does STARTTLS to another server, then add the following lines as well.


Now and here, I assume that you allready have a key and a certificate for Sendmail. How to get or make keys and certificates is beyond the scope of this article.

The next step is enabling SMTP AUTH if you wish for authenticated users to be able to relay. this involves telling Sendmail which authentication mechanisms you like to trust. This is done with the following instructions:


The first line lists allowed authenification mechanisms, the second lists the trusted ones. Those in the first line are allowed to be performed, but only those users who authenticate using one of the mechanisms in the second line will be able to relay.

My advice is, keep the two lists identical. Clients only look at which auth mechanisms are being offered, not which are trusted. For example, assume your server offers LOGIN and PLAIN, but trusts only PLAIN. The client might just use LOGIN and the authentication will fail, the client won't retry with PLAIN even if it can. Sense or not, this is what I have observed and it is best to offer only what you also trust.

Basically, LOGIN and PLAIN make you operable with all clients. The most commonly used ones like Mozilla Mail and Outlook don't support anything else. However, they can can both do STARTTLS and thus make these auth mechanisms secure.

Warning: Please do not put EXTERNAL auth mechanism in the list of the trusted ones, despite it allready being suggested in the mc file. Remove it from the list if it is there. Read the section about the EXTERNAL auth mechanism further in this article for details.

Verifying The Installation

Now you can check if everything went right. Let's see if Sendmail advertises its new STARTTLS capability. For that, we'll open a telnet connection to its SMTP port. We'll issue an EHLO command to see the response. Finally, we will close the connection.

# telnet localhost 25
Connected to localhost.
Escape character is '^]'.
220 ESMTP Sendmail 8.12.8/8.12.8; Tue, 21 Oct 2003 19:06:59 +0200
-> EHLO localhost Hello localhost.localdomain [], pleased to meet you
250 HELP
221 2.0.0 closing connection
Connection closed by foreign host.

Your output may vary, but the line with STARTTLS should be there. If it isn't, then either STARTTLS hasn't been compiled in at all, or the configuration is wrong. Here is how you can check if the STARTTLS support has been compiled in:

# sendmail -d0.1 -bv | grep -o STARTTLS

I got one line of output saying STARTTLS. If you get no output, then the compilation went wrong. If there appears a line containing the STARTTLS keyword, then recheck your configuration.

And if it works properly, well then, get yourself a deserved mug of beer and enjoy your new secure sendmail.

EXTERNAL Authentication Mechanism

I have observed that if EXTERNAL is trusted, anyone who can set up a TLS session can also relay, regardless wether her certificate can be verified.

On my system, with EXTERNAL trusted, Sendmail checks if an user can relay based on the distinguished name of the certificate, not on the fact wether the verification succeeded. Putting CERTISSUER entries in the access map doesn't seem to have any effect. The fact that the certificate verification failed is logged, but relaying is allowed anyway. This means, anyone who can make a bogus certificate with OpenSSL can also relay.

This can very well be a problem with this particular build, or the lack of additional configuration, but this is exactly the state with the default configuration. If you know what to do to change it, go ahead, but if you leave it as it is and trust EXTERNAL, you are an open relay.

And yes, this means that you cannot use this configuration to verify user's identity based on her certificate and the proof that she has the corresponding secret key. You must use other mechanisms to authenticate, such as LOGIN or PLAIN, and use the TLS session just to prevent passwords to go over the wire as clear text. If you know how to fix this and prevent relaying unless the client certificate can be verified, I would love to know about it. What the Sendmail documentation recommends, namely

R$*    $: $&{verify}
ROK    $# OK

does not work here, Sendmail ignores it completely and does as if it weren't there. The logs look something like this, with omitted entries enclosed in angle brackets:

[...] STARTTLS=server, [...] verify=FAIL, [...]
[...] AUTH=server, authid=[DN of the cert] mech=EXTERNAL, bits=0
[...] from=[local address]
[...] to=[non-local address], [...] stat=Sent (Message accepted.)

As you can see, the server relayed to a remote address and the certificate verification clearly failed. The cert had the same DN as one of the good certs from the access map, but it was a self-signed fake. However, if the client doesn't use STARTTLS, relaying is denied unless the client supplies valid credentials via SMTP AUTH.

To sum it up, do not trust EXTERNAL mechanism with this configuration. You have been warned.


Here are few links to related, or identical, information on the net, listed in no particular order.

Updating The System
Installing Sendmail Source
Building And Installing
Configuring Sendmail
Verifying The Installation
EXTERNAL Authentication Mechanism
Download RPMs:
Source and binary RPMs (HTTP)
Source and binary RPMs (FTP)
© 2000-2007 Igor Zlatković. All rights reserved. Last modification: Mon, 10 Sep 2007 21:12:33 GMT+02:00
Comments? Questions? Please tell us about it.