Project

General

Profile

Actions

Feature #6546

closed

pfSense should support logging to e.g. ELK stacks

Added by Bruce Simpson almost 8 years ago. Updated over 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
Start date:
06/27/2016
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
Release Notes:

Description

pfSense logging is based around the FreeBSD base system's syslogd logging daemon.

This can be tricky to integrate into a distributed system e.g. ELK, Graylog, Splunk etc.

We're specifically looking at using ELK here (Gardenia). We've found the least painful way to get an Ubuntu server logging into ELK was to use Elastic's 'filebeat' tool.

This is basically a log crawler written in Go. However, it lacks support for pfSense's native CLOG format. Whilst the low-level details of this are something I've already started working on (i.e. teach filebeat to crawl CLOG, by hacking Go) it would still need to be integrated into the GUI somehow, perhaps as a package.

Actions #1

Updated by Nathan Stocks over 7 years ago

Have you made any progress on this? I am trying to get filebeat set up on pfsense right now and ran into CLOG. :-(

Actions #2

Updated by Bruce Simpson over 7 years ago

Not as such, but we've raised CCLAs with ESF so we can get onto it.

Also, the relatively large binary footprint of filebeat itself (12M binary, statically linked Go!) might not be appropriate for pfSense.

We're trying to knit a Python application into the same infrastructure, and went down an alley with the now-deprecated Lumberjack this afternoon.

The relatively low-level, official Python Elasticsearch client library [[https://elasticsearch-py.readthedocs.io/en/master/]] is probably the way to go there.

However, there is no official C API. The main ingredients are: JSON, GZIP encoding, and RESTful POST. Rolling something from scratch may actually be easier (and more appropriate for the pfSense platform).

Actions #3

Updated by Bruce Simpson over 7 years ago

We have a candidate library. [[https://github.com/QHedgeTech/cpp-elasticsearch]] requires only libcurl, already in base from strongswan. Needs porting -- currently unpackaged, targets OS X and Linux.

Actions #4

Updated by Bruce Simpson over 7 years ago

We have Python pushing log records to ELK as a rough prototype. The code is dog simple, and should be relatively easy to adapt to a local, cut-down log scraper on e.g. pfSense in C/C++.

Actions #5

Updated by Bruce Simpson over 7 years ago

The Python prototype we have internally is now plug-and-play as filebeat and topbeat are. It uses the bulk posting API, and pre-populates the index template for Kibana to just pick up the feed.

We'll look into adapting this for embedded use in C/C++ on pfSense.

Actions #6

Updated by Bruce Simpson over 7 years ago

I now have that Python code wrapped up as a class, for use with Python logging. A current limitation is that logging requests from urllib3, requests, or elasticsearch modules themselves can cause recursion into this class; it shouldn't be directly used as a root-level logger instance, UNLESS you specifically disable logging from those channels while the ELKLogger is doing its thing.

We've noticed Python getting pulled in as we update to pfSense 2.3.2. Whilst doing it from Python would only be ideal for native Python apps, it is still likely to have way lower overhead than doing it through filebeat.

Maybe we can use a Syslog handler in Python as an input channel, and effectively just run a pfSense syslog-ELK relay in a single python process using my code.

Actions #7

Updated by Bruce Simpson over 7 years ago

Parsing syslog input is indeed dog simple in Python with asyncore. I should be able to just glue this together to my new ELKHandler support code for the Python logging framework.

If it's possible to use the Python pulled into pfSense 2.3.2 to support this, deployment might be as simple as a simple package, without requiring additional binaries.
Actions #8

Updated by Jim Pingle over 4 years ago

  • Status changed from New to Closed

This will be moot on 2.5.0: See #8350

Actions

Also available in: Atom PDF