pfSense » pfSense Packages

It's known and we're already working on it.

Additionally, already reported in #3585

Not exactly. The problem in packages is distinct from the one in base. The base firmware update won't fix package and vice versa because the packages use PBIs which would include their own copy of the OpenSSL library rather than using the one from the base system. It's different enough to keep separate tickets for now.

At this point, I would vastly prefer to see OpenSSL kicked to the curb as unceremoniously as possible in favor of PolarSSL. OpenVPN has suppported PolarSSL for quite some time now, has a vastly better track record, and has actually been audited.

"actually been audited", "has a vastly better track record"? Uh, no. OpenSSL has had a lot more eyes on it than PolarSSL. It's actually been FIPS 140 validated. It's had tons of eyes on it outside of that. PolarSSL has had plenty of vulnerabilities in its much shorter existence including some very serious ones. Crypto's hard, every implementation has screwed it up at some point. There isn't a viable alternative to OpenSSL that's "vastly better" in any regard.

Revised time estimate? Says "1 hour" up top, which strikes me as overly optimistic.

Thanks,
-danny

nothing says "1 hour", it takes 4-5 times that long just to build a release, much less actually test it and push it out. Release and packages are all rebuilding now, it'll be out "soon".

oh that. 1 hour, hah! I wish. We've burned easily 20+ man hours in the last day on this.

Chris Buechler wrote:

"actually been audited", "has a vastly better track record"? Uh, no. OpenSSL has had a lot more eyes on it than PolarSSL. It's actually been FIPS 140 validated. It's had tons of eyes on it outside of that. PolarSSL has had plenty of vulnerabilities in its much shorter existence including some very serious ones. Crypto's hard, every implementation has screwed it up at some point. There isn't a viable alternative to OpenSSL that's "vastly better" in any regard.

One would presume governments know a thing or two about security. If OpenSSL has "more eyes on it" (which does nothing for security) and was not demonstrably worse, why did they immediately select PolarSSL instead? And if it didn't undergo a thorough technical review, how do you think it got NLNCSA Level 2 certification?
https://openvpn.fox-it.com/
https://www.aivd.nl/onderwerpen/infobeveiliging

Nor is OpenSSL FIPS-140 validated in any fashion - only OpenSSL FIPS Object Module. Yes, they are distinct. Unless you're installing from a CD, OpenSSL is not and does not contain code which has passed FIPS-140-2. It is an separately maintained tree which is API compatible and must be built using very, very specific instructions to maintain validation.
The version of OpenSSL being used is not FIPS-140-2 nor can it be, unless somebody's paying NIST about $50K any time even the compile flags change.
http://www.openssl.org/docs/fips/fipsnotes.html

As for comparisons of shipped-broken code: the numbers speak for themselves.
OpenSSL CVSS Scores (Descending) - http://www.cvedetails.com/vulnerability-list.php?vendor_id=217&product_id=&version_id=&page=1&hasexp=0&opdos=0&opec=0&opov=0&opcsrf=0&opgpriv=0&opsqli=0&opxss=0&opdirt=0&opmemc=0&ophttprs=0&opbyp=0&opfileinc=0&opginf=0&cvssscoremin=0&cvssscoremax=0&year=0&month=0&cweid=0&order=3&trc=85&sha=d709ee3c0dc47c3827b5990023842398148d082b
PolarSSL CVSS Scores (Descending) - http://www.cvedetails.com/vulnerability-list.php?vendor_id=12001&product_id=&version_id=&page=1&hasexp=0&opdos=0&opec=0&opov=0&opcsrf=0&opgpriv=0&opsqli=0&opxss=0&opdirt=0&opmemc=0&ophttprs=0&opbyp=0&opfileinc=0&opginf=0&cvssscoremin=0&cvssscoremax=0&year=0&month=0&cweid=0&order=3&trc=6&sha=fdc273640aa0937b178c8806950e0887354a1b6f

Also of note, the only remote exec bug in PolarSSL would not have affected pfSense, because it only impacted very old versions. Ones that wouldn't have been in the ports tree since 2012.

So what, exactly, is the technical or security argument for not going to PolarSSL with OpenVPN? Because I'm seeing a whole lot of reasons to do exactly that.

Please note that haproxy-devel seems to ship its own instance own instance of openssl, so will need to be reviewed as well:

--
2.1-RELEASE (amd64)
built on Wed Sep 11 18:17:48 EDT 2013
FreeBSD 8.3-RELEASE-p11

haproxy-devel 1.5-dev22 pkg v 0.7
--

$ find / -type f -name 'openssl'
/usr/bin/openssl
/usr/local/bin/openssl
/usr/pbi/haproxy-devel-amd64/bin/openssl

$ find / -type f -name 'openssl' -exec \{\} version \;
OpenSSL 0.9.8y 5 Feb 2013
OpenSSL 1.0.1e 11 Feb 2013
WARNING: can't open config file: /usr/pbi/haproxy-devel-amd64/openssl/openssl.cnf
OpenSSL 1.0.1f 6 Jan 2014

Phil Jaenke wrote:

Lot's of PolarSSL stuff and about how awesome it is ....

Don't think a bug report is the right place to have a discussion about the pros and cons ... take it to the forums.

Thanks!

KR,

Grimeton.

Is there any ETA on new release? A realistic one, not 1 hour then 10+ :)

I need to patch but I'd rather wait and patch once rather than twice...

Thanks all

I don't know more than you, but once Chris and the US wakes up, by the look of the above. ie sometime on Apr 9: US time, unless testing throws up issues?

When will haproxy-devel be available as a separate update? This would solve my problem.

Guys, can someone fix the CRLs in 2.1.1 before releasing 2.1.2? A LOT of people will want/need to revoke certificates, but it does not work.

https://forum.pfsense.org/index.php?topic=74935.msg408977#msg408977

Any update with this?
It's pretty critical...

Agreed with Sam. I've had to make the call to disable our VPN. The natives are getting restless. This is a security appliance distribution, right?

The release will be done when its done.

I release involves some 80+ variants all of which have to be built.
Builds have to be tested and upgraded.

Justin Foreman wrote:

Agreed with Sam. I've had to make the call to disable our VPN. The natives are getting restless. This is a security appliance distribution, right?

And you registered only today to tell us that?

Note: This bug is for Heartbleed in packages, and many (if not all) of those have already been updated and bumped so this one can probably be set to Feedback status.

For the base OS, see #3585

in any case, yes, this bug had to be fixed.

and while we were in there, the ECDSA bug had to be fixed (note that it came out yesterday).

and fixing CRLs was important too.

And it takes hours to build all the various bits of a pfSense release.
Then we have to test it.
Then, if it works, we sign, deploy, and announce.

Jim Thompson wrote:

Justin Foreman wrote:

Agreed with Sam. I've had to make the call to disable our VPN. The natives are getting restless. This is a security appliance distribution, right?

And you registered only today to tell us that?

I understand that this bug is not your fault. The immensity of this vulnerability is mind-boggling. That said, a little bit of PR can go a long way. A news bulletin, a tweet, an occasional Redmine update, anything letting people know what's going on. If I missed it somewhere, I apologize, but I didn't see anything. A lot of people depend on pfSense. An ETA would be huge, even if it was Geordie La Forge style.

I've been using pfSense in the enterprise since 2005 and m0n0wall prior to that. I love your distribution. I wasn't trying to be a jerk. I was just wondering what was going on.

I appreciate your work.

Jim Thompson wrote:

And you registered only today to tell us that?

Hi Jim,
I do not think comments like this reflect well upon PFSense.

Your opinion is noted.

This is a critical bug that should be taken seriously and I would have expected PFSense to have been patched by now.

As Justin said, I too appreciate the work the PFSense team(s) do, however one of PFSense's main selling points is it's reputation for being secure, and being left
vulnerable for this long damages that reputation.

Heartbeat was announced 07 April 2014 https://www.openssl.org/news/secadv_20140407.txt There was actually a potential release of pfSense 2.1.2 (http://lists.pfsense.org/pipermail/list/2014-April/005958.html), but it was pulled before actual release (the testing images were up for no more than 15 minutes, in the middle of the night.)

This premature release was pulled because we had just been made aware (via the FreeBSD SAs) of a different bug that left OpenSSL open to a side-channel attack. http://www.freebsd.org/security/advisories/FreeBSD-SA-14:06.openssl.asc

There is a lot that goes into making a pfSense release, especially this one. As Jeremy pointed out, there are some 80 variants that all get built. A lot of the team were working extended hours, attempting to ensure that these bugs were both fixed (as well as a couple others, such as one which would have kept people from issuing a CRL, revoking any previously-generated certificates.

Yes, I reacted when someone created an account just to flog us at 7pm. Nor do I withdraw same. Anything that negatively affects morale while we're still in the trenches is suspect.

pfSense 2.1.2 was released, with relatively minor issues (there is an issue which can affect auto-config backup) earlier today.

Apparently many in the community think that making a release is as simple as typing 'make' and copying the result onto a web server. Such is not even close to reality.

Two months ago, there weren't even a lot of scripts to help automate the process of making a pfSense release. After personally observing the process during both the 2.0.3 and 2.1 releases, I directed some internal effort into both automating some of the process (where possible) and documenting the entire process. The first time these scripts and documentation were used was during the 2.1.1 release (which, I will remind, was only 6 days ago.)

I agree that three days is too long to issue a release, and we will be re-architecting how pfSense is built and released in order to overcome some of the ... shortcomings of both the build and release process.

We did attempt keep people advised of the status, both here in redmine, and on the list.

To everyone involved, is there anything we can do to assist with getting this released? Rather keen to get this patched on all our pfsense systems (not to mention the pressure we're under to do so)

Since others are pointing me to this ticket to keep tabs I thought it best to comment with something useful.

If you are waiting on the new 2.1.2 build for the OpenSSL patch take a moment to consider the PFSense Gold Membership: https://portal.pfsense.org/gold-subscription.php

FWIW, haproxy-devel package seems to have been updated a few hours ago and bumped to 1.5-dev22 pkg v 0.8.
I did the upgrade on our two pfSense frontends which went flawlessly, and now the heartbleed test passes and reports it's SAFE!
If it helps anyone in the same case.

Now, for other included packages like OpenVPN for instance, I guess we have to wait for the pfSense team to wrap this up... without us being helpful in anyway even though we're willing to help.

Genuine and naive questions:
- I've tried to look throughout the available documentation and yet failed to find a documented build procedure for pfSense. If it's impossible for the community to build their own based on sources, how opensource pfSense really is?
- Even if it proves useful as customer as well as for pfSense in the long run to get to buy the "Gold Membership" mentioned above or even official support, I doubt it would have change anything to the current situation, would it?
- Trying to think positive here, how could we improve pfSense to be easier to update, thus a lot quicker to be able to respond to critical security threats like this?

Anyway, thanks for the hard work probably still underway.

Not really the place for a long off-topic discussion in a bug. I'll support Lane's suggestion to sign up for pfSense Gold Membership. The ESF guys are hard at work building and checking the bucketloads of various installation images that get produced. They need to eat and drink while they work.
On Frederic's comments - there are threads in the Development forum discussing all the issues surrounding the taking off-line of the pfSense-tools repo, and a bunch of banter in the dev mail list archive. Look in those places and add your input.

Agreed. Not the best place.

Will look at the development forum.

Frederic MEYER wrote:

FWIW, haproxy-devel package seems to have been updated a few hours ago and bumped to 1.5-dev22 pkg v 0.8.
I did the upgrade on our two pfSense frontends which went flawlessly, and now the heartbleed test passes and reports it's SAFE!

heartbleed tests differ in their ability. I can confirm your observation but not your conclusion.

Although haproxy-devel has been updated to 1.5-dev22 pkg v 0.8, on closer inspection your statement looks too good to be true because haproxy does not seem to be linked to OpenSSL 1.0.1g, nor to be patched based on an older release.

/usr/pbi/haproxy-devel-amd64/sbin/haproxy -version -v
HA-Proxy version 1.5-dev22-1a34d57 2014/02/03
Copyright 2000-2014 Willy Tarreau <w@1wt.eu>

ldd /usr/pbi/haproxy-devel-amd64/sbin/haproxy 
sbin/haproxy:
    libcrypt.so.5 => /lib/libcrypt.so.5 (0x800727000)
    libz.so.5 => /lib/libz.so.5 (0x800847000)
    libssl.so.8 => /usr/local/lib/libssl.so.8 (0x80095c000)
    libcrypto.so.8 => /usr/local/lib/libcrypto.so.8 (0x800ac2000)
    libthr.so.3 => /lib/libthr.so.3 (0x800d95000)
    libc.so.7 => /lib/libc.so.7 (0x800eae000)

strings /usr/local/lib/libssl.so.8 | grep OpenSSL
OpenSSLDie
SSLv2 part of OpenSSL 1.0.1e 11 Feb 2013
SSLv3 part of OpenSSL 1.0.1e 11 Feb 2013
TLSv1 part of OpenSSL 1.0.1e 11 Feb 2013
DTLSv1 part of OpenSSL 1.0.1e 11 Feb 2013
OpenSSL 1.0.1e 11 Feb 2013

strings /usr/pbi/haproxy-devel-amd64/lib/libssl.so.8 | grep OpenSSL
OpenSSLDie
SSLv2 part of OpenSSL 1.0.1f 6 Jan 2014
SSLv3 part of OpenSSL 1.0.1f 6 Jan 2014
TLSv1 part of OpenSSL 1.0.1f 6 Jan 2014
DTLSv1 part of OpenSSL 1.0.1f 6 Jan 2014
OpenSSL 1.0.1f 6 Jan 2014

strings /usr/lib/libssl.so| grep OpenSSL
OpenSSLDie
TLSv1 part of OpenSSL 0.9.8y 5 Feb 2013
OpenSSL 0.9.8y 5 Feb 2013
SSLv2 part of OpenSSL 0.9.8y 5 Feb 2013
DTLSv1 part of OpenSSL 0.9.8y 5 Feb 2013
SSLv3 part of OpenSSL 0.9.8y 5 Feb 2013

PBI packages are run using wrappers such that they see the libraries present inside of their own PBI dir. Checking with ldd without ldd also being run through such a wrapper would report incorrect results from the perspective of haproxy.

Example:

lrwxr-xr-x  1 root  wheel  42 Apr 10 11:00 /usr/local/sbin/haproxy -> /usr/pbi/haproxy-devel-amd64/.sbin/haproxy

.sbin/haproxy != sbin/haproxy

If the test shows haproxy as not vulnerable while running, it's accurate.

Correct.

And, as Jeremy said,

"Please note that haproxy-devel seems to ship its own instance own instance of openssl, so will need to be reviewed as well:"

And with the updated 0.8 package:

/usr/pbi/haproxy-devel-amd64/bin/openssl version
WARNING: can't open config file: /usr/pbi/haproxy-devel-amd64/openssl/openssl.cnf
OpenSSL 1.0.1g 7 Apr 2014

Finally,

strings /usr/pbi/haproxy-devel-amd64/lib/libssl.so.8 | grep OpenSSL
OpenSSLDie
SSLv2 part of OpenSSL 1.0.1g 7 Apr 2014
SSLv3 part of OpenSSL 1.0.1g 7 Apr 2014
TLSv1 part of OpenSSL 1.0.1g 7 Apr 2014
DTLSv1 part of OpenSSL 1.0.1g 7 Apr 2014
OpenSSL 1.0.1g 7 Apr 2014

Have you really update the package?

What is wrong in that output? 1.0.1g is the updated/fixed/correct version.

That's my point!
So I don't understand David's output even though he claims to have updated his system.

Frederic MEYER wrote:

That's my point!
So I don't understand David's output even though he claims to have updated his system.

Did I find another bug?

System->Package Manager->Installed Packages-> 1.5-dev22 pkg v 0.8
Clicked on Re-install

https://xxxxx/pkg_mgr_install.php?mode=reinstallpkg&pkg=haproxy-devel

Loading package instructions...
Custom commands...
Executing custom_php_install_command()...HAProxy, running haproxy_custom_php_install_command()
HAProxy, conf_mount_rw
HAProxy, create '/usr/local/etc/rc.d/haproxy.sh'
HAProxy, update configuration
HAProxy, conf_mount_ro
HAProxy, starting haproxy (if previously enabled)
HAProxy, running haproxy_custom_php_install_command() DONE
done.
Menu items... done.
Services... done.
Writing configuration... done.

Package reinstalled.

strings /usr/pbi/haproxy-devel-amd64/lib/libssl.so.8 | grep OpenSSL
OpenSSLDie
SSLv2 part of OpenSSL 1.0.1f 6 Jan 2014
SSLv3 part of OpenSSL 1.0.1f 6 Jan 2014
TLSv1 part of OpenSSL 1.0.1f 6 Jan 2014
DTLSv1 part of OpenSSL 1.0.1f 6 Jan 2014
OpenSSL 1.0.1f 6 Jan 2014

No, but you may have to uninstall and reinstall the package if the version number on the PBI file itself did not change.

Don't get me wrong, but "if the version number on the PBI file itself did not change" is just something that never should happen (and definitely NOT in cases like this), even though messing with stuff without version bump seems like commonplace practice with pfSense. Simply STOP doing this. You change a package, you bump the version, no matter how trivial the change is, as soon as it affects the installed files. Period.

We're looking into a way to do that but the version numbers are controlled by the FreeBSD port versions and not directly by us. Unless the FreeBSD port gets bumped, then we'd have to maintain our own copies of the port with our own custom version numbers and so on. That is, unless there's another mechanism in the PBI build process that lets us set an additional number to signify a change.

The version number of the pfSense packages did change.

Frederic MEYER wrote:

That's my point!
So I don't understand David's output even though he claims to have updated his system.

Updating haproxy-devel on PFSense 2.1-RELEASE and rebooting is not enough. When I update the firmware to 2.1.1-RELEASE after upgrading haproxy-devel, then the OpenSSL is reported as 1.0.1g:

[2.1.1-RELEASE][xxx@xxx]/root(1): strings /usr/pbi/haproxy-devel-amd64/lib/libssl.so.8 | grep OpenSSL
OpenSSLDie
SSLv2 part of OpenSSL 1.0.1g 7 Apr 2014
SSLv3 part of OpenSSL 1.0.1g 7 Apr 2014
TLSv1 part of OpenSSL 1.0.1g 7 Apr 2014
DTLSv1 part of OpenSSL 1.0.1g 7 Apr 2014
OpenSSL 1.0.1g 7 Apr 2014

These CVE-2014-0160 tests are passing now:

http://filippo.io/Heartbleed
http://possible.lv/tools/hb

This ticket can be CLOSED as VERIFIED by reporter

I am on 2.1 and did not upgrade to 2.1.1 (obviously waiting for 2.1.2 now...).
Nor did I have to reboot to see the proper version numbers and to see the CVE-2014-0160 test pass. I have not actually.
I only had to restart haproxy which it did as part as the package upgrading process.
Weird.

Frederic MEYER wrote:

I am on 2.1 and did not upgrade to 2.1.1 (obviously waiting for 2.1.2 now...).
Nor did I have to reboot to see the proper version numbers and to see the CVE-2014-0160 test pass. I have not actually.
I only had to restart haproxy which it did as part as the package upgrading process.
Weird.

Weird indeed, i'm stuck again with the procedure to upgrade firmware too as it now says: "Unable to check for updates." but http://updates.pfsense.org/_updaters/amd64/ has had a refresh at April 10 17:55

Chris Buechler wrote:

fixed.

PFSense 2.1.2 fixes CVE-2014-0160.