I spent some time this afternoon doing compatibility testing with Courier's TLS_PROTOCOL settings, with both OpenSSL and GnuTLS libraries.  The tables that follow detail the results that I observed.  GnuTLS got somewhat less testing than OpenSSL.  If someone else wants to test GnuTLS against sendmail, that would be quite useful.

The first series of tests were against Courier 0.58.0, compiled with OpenSSL support, to determine what other MTAs could successfully establish TLS connections.  Sendmail, courierd (openssl, unless noted with gnutls), and openssl's s_client were tested.  All of the senders were running on Fedora 8.  Sendmail was using its default configuration.  In both of the courierd client test configurations, only TLS_PROTOCOL had been changed from its default.

The second series of tests were against Courier 0.58.0, compiled with GnuTLS support.  The senders in that series of tests were running on CentOS 5.

Courier (OpenSSL)

	esmtpd: TLS1*	esmtpd: SSL2	esmtpd: SSL3	esmtpd: SSL23	esmtpd: SSL23 TLS_CIPHER_LIST="SSLv3:TLSv1:!SSLv2:HIGH:!LOW:!MEDIUM:!EXP:!NULL@STRENGTH"
sendmail (f8)	no	yes	no	yes	yes
courierd: SSL3*	no	no	yes	yes	yes
courierd: SSL2	no	yes	no	yes	no
courierd: SSL23	no	yes	no	yes	yes
courierd: TLS1	yes	no	no	yes	yes
s_client: -ssl2	no	yes	no	yes	no
s_client: -ssl3	no	no	yes	yes	yes
s_client: -tls1	yes	no	no	yes	yes
s_client: -no_ssl2	yes	no	yes	yes	yes
s_client: -no_ssl3	no	yes	no	yes	yes
s_client: -no_tls1	no	yes	no	yes	yes
courierd(gnutls): SSL3	no	no	yes	yes
courierd(gnutls): TLS1	yes	no	no	yes
courierd(gnutls): TLS1_1	no	no	no	no
courierd(gnutls): TLS1_1:TLS1:SSL3	yes	no	yes	yes

Courier (GnuTLS)

	esmtpd: SSL3	esmtpd: TLS1*	esmtpd: TLS1_1	esmtpd: TLS1_1:TLS1:SSL3
courierd: SSL3*	yes	no	no	yes
courierd: SSL2	no	no	no	no
courierd: SSL23	yes	yes	no	yes
courierd: TLS1	no	yes	no	yes
s_client: -ssl2	no	no	no	no
s_client: -ssl3	yes	no	no	yes
s_client: -tls1	no	yes	no	yes
s_client: -no_ssl2	yes	yes	no	yes
s_client: -no_ssl3	no	yes	no	yes
s_client: -no_tls1	yes	no	no	yes

Several of the results are notable:

* The biggest and most important:  As was pointed out previously by another list member, Courier's esmtpd default setting is TLS_PROTOCOL=TLS1.  Courier's courierd default setting is SSL3.  They are not interoperable.  In its default configuration, one installation of Courier is not able to send email to another.

* I'm not sure what SSL settings Sendmail uses by default.  It behaves identically to courier when courierd uses SSL2 and also SSL23.  Courier doesn't accept mail from sendmail by default, either.

* SSL23 used in courierd won't allow it to connect to a courier server using SSL3 in esmtpd.  Weird.

* More generally, any of SSL2, SSL3, and TLS1 settings in courierd will only connect to an esmtpd that uses either the identical setting, or SSL23.

* The most interoperable client was openssl's "s_client" when using the -no_ssl tls protocol setting.

* There is no setting for courierd (with openssl) that will successfully connect to either TLS1 or SSL3 servers, as s_client will.

* GnuTLS's TLS1_1 setting doesn't work with anything tested.  It probably only works with GnuTLS, and an identical setting.

Based on those results, I'm personally inclined to believe that the TLS_PROTOCOL setting should be eliminated.  When openssl is used, all of the server components of courier should behave as they do with SSL23, and the default TLS_CIPHER_LIST should disable all of SSL2's ciphers.  When GnuTLS is used, TLS_PROTOCOL should probably behave as it does with SSL3:TLS1:TLS1_1.

courierd's default setting should behave like "s_client -no_ssl2" does, when using openssl.  It should probably use SSL3:TLS1:TLS1_1 when using GnuTLS.