Bug #7437
closedMail Report package 3.1 removed support for STARTTLS
0%
Description
I had pfSense configured to send mail reports via FastMail on port 587 with STARTTLS.
All worked great until recent package upgrade when suddenly I stopped receiving daily status emails.
Initial check showed that test mail from Notification configuration works fine, but sending reports fails.
Finally got some time to troubleshoot.
Adding $mail->SMTPDebug = 2; to mail_reports.inc reports "SMTP Error: Could not authenticate."
But I can see that STARTTLS still works on another install with package version 3.0_1.
Comparing mail_reports.inc between versions revealed that package no longer sets $mail->SMTPSecure = "tls"; when "Enable STARTTLS" checkbox is checked in Notification configuration.
Luckily, I can switch to SSL on port 465 for now, but I would appreciate if this can be fixed - current situation is very misleading because system notifications still work properly.
Files
Updated by Kill Bill over 7 years ago
It was not removed, it's supposed to be used automatically when you tick SSL and the mailserver is advertising STARTTLS. If it doesn't work, then there's some bug with phpmailer or the mailserver behaves in a non-standard way.
Updated by Jim Pingle over 7 years ago
- Status changed from New to Rejected
It was changed because phpmailer changed. It detects STARTTLS support automatically. If it can't, then the server isn't properly advertising STARTTLS support or phpmailer isn't seeing it.
Updated by Dmitry Gromov over 7 years ago
Hi!
Well, that is kind of strange way to treat the issue, let's not jump to conclusions that fast.
I had a bit more time to look at this today and I can confirm that this is a bug in mail_reports.inc
First - I tested with openssl - all perfect. Then I tested with Gmail - same issue.
Finally, I looked up "Opportunistic TLS" (https://github.com/PHPMailer/PHPMailer/wiki/Troubleshooting#opportunistic-tls) - it clearly states that automatic STARTTLS will work by default, unless it is disabled with:
$mail->SMTPAutoTLS = false;
And that is exactly what version 3.1 does - it disables handling of STARTTLS if STARTTLS checkbox is set (else part):
if (isset($config['notifications']['smtp']['ssl'])) { $mail->SMTPSecure = "ssl"; $mail->SMTPAutoTLS = true; } else { $mail->SMTPSecure = ""; $mail->SMTPAutoTLS = false; }
Obviously, this change was not tested properly.
This can be fixed by removing SMTPAutoTLS from both parts of if statement or, much better - having another check box to toggle "Opportunistic TLS".
So please - reopen this bug.
Updated by Kill Bill over 7 years ago
Dmitry Gromov wrote:
And that is exactly what version 3.1 does - it disables handling of STARTTLS if STARTTLS checkbox is set (else part):
Erm, no, that's not what it does. It disables STARTTLS when you uncheck "Enable SMTP over SSL/TLS". Just don't do that.
Updated by Dmitry Gromov over 7 years ago
Are we looking at different codebases?
There are two checkboxes on /usr/local/www/system_advanced_notifications.php - "Enable SMTP over SSL/TLS" and "Enable STARTTLS"
First one sets $config['notifications']['smtp'] to 'ssl', second one sets it to 'tls'
In the if statement from mail_reports.inc I quoted above only 'ssl' value is handled. If it is set to 'tls', else part executes, effectively disabling STARTTLS.
The old code in 3.01 handled both and worked properly:
if ((isset($config['notifications']['smtp']['ssl']) && $config['notifications']['smtp']['ssl'] != "unchecked") || $config['notifications']['smtp']['ssl'] == "checked") $mail->SMTPSecure = "ssl"; if ((isset($config['notifications']['smtp']['tls']) && $config['notifications']['smtp']['tls'] != "unchecked") || $config['notifications']['smtp']['tls'] == "checked") $mail->SMTPSecure = "tls";
Updated by Kill Bill over 7 years ago
I must be speaking Chinese. Tick the "Enable SMTP over SSL/TLS" and it will work. Simple. (The "Enable STARTTLS" thing does not even exist any more on 2.4.)
Updated by Dmitry Gromov over 7 years ago
I am glad it works for you in 2.4, but last I checked 2.3.3-RELEASE-p1 is the current release and it does NOT work there.
There is a difference between SMTPS (Enable SMTP over SSL/TLS) which is usually used on port 465 and STARTTLS (usually on 25/587).
You can't simply autodetect between them.
SMTPS on port 465 works, o course, but so did STARTTLS on port 587 until this change - and it is easy to test against Gmail SMTP server in pfSensse 2.3.3
Updated by Kill Bill over 7 years ago
It works just fine on 2.3.3 when you tick the checkbox that you stubbornly refuse to tick for god knows what reason. Have a nice day.
Updated by Dmitry Gromov over 7 years ago
Hi!
It looks like you do not understand the difference between SMTPS and STARTTLS.
If I check "Enable SMTP over SSL/TLS" AND change port to 465 - it will work with SMTPS, I do not dispute that.
However, STARTTLS over port 587 DOES NOT WORK. And it cannot work - because php mailer's property SMTPSecure explicitly specifies which protocol to use - if you set it to 'ssl' it will only use SMTPS and will not use STARTTLS.
I would suggest that you understand the difference between SMTPS and STARTTLS and how they are configured in php mailer and how and when they are used.
Once again - there is no issue with SMTPS - it works fine with Google or FastMail on port 465.
The issue (as per subject) is with STARTTLS - it does NOT work for this package version only after the recent update.
STARTTLS works perfectly fine for pfSense notifications.
Updated by Kill Bill over 7 years ago
I'm telling you what to tick so that you have the mail reports working with STARTTLS without any changes needed in the package or anywhere else, but it's apparently hopeless.
Updated by Dmitry Gromov over 7 years ago
- File smtp-err1.jpg smtp-err1.jpg added
Of course, I tested it - see attached screenshot, in this case notifications cease to works as well as reports.
Only these combinations work for notifications:- "Enable SMTP over SSL/TLS" port 465
- "Enable STARTTLS " port 587
Same is working on another system with mailreports 3.01
New mailreports 3.1 only works when I use "Enable SMTP over SSL/TLS" port 465 - but that is not what I need, because it is SMTPS.
Using "Enable SMTP over SSL/TLS" on port 587 fails because there is no SMTPS on that port, so it will not connect.
Adding $mail->SMTPDebug = 3; to /mail_reports.inc and running mail_reports_generate.php produces this error:
2017-03-31 12:44:29 Connection: opening to ssl://smtp.fastmail.com:587, timeout=300, options=array ( ) 2017-03-31 12:44:29 Connection: Failed to connect to server. Error number 2. "Error notice: stream_socket_client(): SSL operation failed with code 1. OpenSSL Error messages: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol 2017-03-31 12:44:29 Connection: Failed to connect to server. Error number 2. "Error notice: stream_socket_client(): Failed to enable crypto 2017-03-31 12:44:29 Connection: Failed to connect to server. Error number 2. "Error notice: stream_socket_client(): unable to connect to ssl://smtp.fastmail.com:587 (Unknown error) 2017-03-31 12:44:29 SMTP ERROR: Failed to connect to server: (0) 2017-03-31 12:44:29 SMTP connect() failed. https://github.com/PHPMailer/PHPMailer/wiki/Troubleshooting Mailer Error: SMTP connect() failed. https://github.com/PHPMailer/PHPMailer/wiki/Troubleshooting
Setting "Enable STARTTLS " on port 587 and successfully testing notifications, then running mail_reports_generate.php with $mail->SMTPDebug = 3; produces this output on console:
2017-03-29 17:11:00 Connection: opening to smtp.fastmail.com:587, timeout=300, options=array ( ) 2017-03-29 17:11:01 Connection: opened 2017-03-29 17:11:01 SERVER -> CLIENT: 220 smtp.fastmail.com ESMTP ready 2017-03-29 17:11:01 CLIENT -> SERVER: EHLO mysmtp.lan 2017-03-29 17:11:01 SERVER -> CLIENT: 250-smtp.fastmail.com 250-PIPELINING 250-SIZE 71000000 250-ENHANCEDSTATUSCODES 250-8BITMIME 250 STARTTLS 2017-03-29 17:11:01 SMTP Error: Could not authenticate. 2017-03-29 17:11:01 CLIENT -> SERVER: QUIT 2017-03-29 17:11:01 SERVER -> CLIENT: 221 2.0.0 Bye 2017-03-29 17:11:01 Connection: closed 2017-03-29 17:11:01 SMTP connect() failed. https://github.com/PHPMailer/PHPMailer/wiki/Troubleshooting
Here is the same with previous version of mailreports:
2017-03-29 17:19:08 Connection: opening to smtp.fastmail.com:587, timeout=300, options=array ( ) 2017-03-29 17:19:08 Connection: opened 2017-03-29 17:19:08 SERVER -> CLIENT: 220 smtp.fastmail.com ESMTP ready 2017-03-29 17:19:08 CLIENT -> SERVER: EHLO mysmtp.lan 2017-03-29 17:19:08 SERVER -> CLIENT: 250-smtp.fastmail.com 250-PIPELINING 250-SIZE 71000000 250-ENHANCEDSTATUSCODES 250-8BITMIME 250 STARTTLS 2017-03-29 17:19:08 CLIENT -> SERVER: STARTTLS 2017-03-29 17:19:08 SERVER -> CLIENT: 220 2.0.0 Start TLS 2017-03-29 17:19:08 CLIENT -> SERVER: EHLO mysmtp.lan 2017-03-29 17:19:08 SERVER -> CLIENT: 250-smtp.fastmail.com 250-PIPELINING 250-SIZE 71000000 250-ENHANCEDSTATUSCODES 250-8BITMIME 250-AUTH PLAIN LOGIN 250 AUTH=PLAIN LOGIN 2017-03-29 17:19:08 CLIENT -> SERVER: AUTH LOGIN .............. successful submit follows ..............
I do not know what other proof you need...
Updated by Dmitry Gromov over 7 years ago
- File smtp-err1.jpg smtp-err1.jpg added
Updated by Dmitry Gromov over 7 years ago
Is it possible delete or replace attachment here?
Seems like I can edit message, but not delete or replace attached files...
Updated by Dmitry Gromov over 7 years ago
I just spent some time, installed pfSense 2.4 in Hyper-V and tested.
Yes, there is no STARTTLS setting anymore, which might be OK but not recommended from compatibility point of view - using "Opportunistic TLS" works in most cases, but I bet there are instances which require explicit setting - see "Authentication failures" in "Troubleshooting PHPMailer Problems" as mentioned earlier.
HOWEVER - and this is the entire point of this bug report - mailreports package fails when port is set to 587 and automatic STARTTLS is required, but package explicitly disables it.
Please, re-open and assign this bug report properly, so it can be fixed.
Updated by Joshua Sign almost 6 years ago
In my opinion, it is better to set SMTPAutoTLS to TRUE because :
1 - if both actors (client and server) can use STARTTLS, the TLS will be initiated,
BUT if we specifiy explicitly SMTPAutoTLS to false, even if server can operate with TLS our client (pfsense) will never initiate TLS connection, and the server could only accept TLS.
It can result in an error as Dmitry describe.
2 - after reading the phpmailer class, it is suggested to let the default SMTPAutoTLS value to true "_even if `SMTPSecure` is not set to 'tls'_":
/** * Whether to enable TLS encryption automatically *if a server supports it, * even if `SMTPSecure` is not set to 'tls'.* * Be aware that in PHP >= 5.6 this requires that the server's certificates are valid. * @var boolean */ public $SMTPAutoTLS = true;
And all of this, whatever the port you use.
Pull request created : https://github.com/pfsense/FreeBSD-ports/pull/600