pfSense

Have you made any progress on this? I am trying to get filebeat set up on pfsense right now and ran into CLOG. :-(

Not as such, but we've raised CCLAs with ESF so we can get onto it.

Also, the relatively large binary footprint of filebeat itself (12M binary, statically linked Go!) might not be appropriate for pfSense.

We're trying to knit a Python application into the same infrastructure, and went down an alley with the now-deprecated Lumberjack this afternoon.

The relatively low-level, official Python Elasticsearch client library [[https://elasticsearch-py.readthedocs.io/en/master/]] is probably the way to go there.

However, there is no official C API. The main ingredients are: JSON, GZIP encoding, and RESTful POST. Rolling something from scratch may actually be easier (and more appropriate for the pfSense platform).

We have a candidate library. [[https://github.com/QHedgeTech/cpp-elasticsearch]] requires only libcurl, already in base from strongswan. Needs porting -- currently unpackaged, targets OS X and Linux.

We have Python pushing log records to ELK as a rough prototype. The code is dog simple, and should be relatively easy to adapt to a local, cut-down log scraper on e.g. pfSense in C/C++.

The Python prototype we have internally is now plug-and-play as filebeat and topbeat are. It uses the bulk posting API, and pre-populates the index template for Kibana to just pick up the feed.

We'll look into adapting this for embedded use in C/C++ on pfSense.

I now have that Python code wrapped up as a class, for use with Python logging. A current limitation is that logging requests from urllib3, requests, or elasticsearch modules themselves can cause recursion into this class; it shouldn't be directly used as a root-level logger instance, UNLESS you specifically disable logging from those channels while the ELKLogger is doing its thing.

We've noticed Python getting pulled in as we update to pfSense 2.3.2. Whilst doing it from Python would only be ideal for native Python apps, it is still likely to have way lower overhead than doing it through filebeat.

Maybe we can use a Syslog handler in Python as an input channel, and effectively just run a pfSense syslog-ELK relay in a single python process using my code.

Parsing syslog input is indeed dog simple in Python with asyncore. I should be able to just glue this together to my new ELKHandler support code for the Python logging framework.

If it's possible to use the Python pulled into pfSense 2.3.2 to support this, deployment might be as simple as a simple package, without requiring additional binaries.