Bug #13444
openzabbix_proxy : cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
0%
Description
Hi
I frequently come across this issue when trying to investigate why a Zabbix agent isn't communicating successfully with our Zabbix server.
When I navigate to https://pfsense-ip-address/status_logs_packages.php?pkg=Zabbix%20Proxy%205.0 I'm presented with the following (truncated) logs
Jul 15 03:09:00 queeg500 newsyslog[90148]: logfile turned over due to size>500K zabbix_proxy [78631]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [82116]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied *** Above lines repeated 50+ times *** Jul 15 03:09:00 queeg500 newsyslog[90148]: logfile turned over due to size>500K ... ...
Logging appears to have stopped ~40 days ago.
Restarting the Zabbix proxy service (via https://pfsense-ip-address/status_services.php#) gets logging working again, however its a pain because I generally speaking I wanted to see the logs for the past to investigate the problem I'm dealing with at that specific time.
I suspect the issue is related to log rotation and file permissions based on the Permission denied error and that newsyslog is mentioned before and after the logging stops working.
Today, before I restart the service I checked who owned the log file...
[2.6.0-RELEASE][root@pfsense-ip-address]/root: ls -l /var/log/zabbix-proxy/ total 106 -rw------- 1 root wheel 80 Jul 15 03:09 zabbix_proxy.log -rw------- 1 root wheel 29744 Jul 15 03:09 zabbix_proxy.log.0.bz2 -rw------- 1 root wheel 33193 Jun 6 13:47 zabbix_proxy.log.1.bz2 -rw------- 1 root wheel 34871 May 4 09:48 zabbix_proxy.log.2.bz2
After I restarted the service I checked again...
[2.6.0-RELEASE][root@fsense-ip-address]/root: ls -l /var/log/zabbix-proxy/ total 110 -rw------- 1 zabbix zabbix 3218 Aug 25 13:42 zabbix_proxy.log -rw------- 1 zabbix zabbix 29744 Jul 15 03:09 zabbix_proxy.log.0.bz2 -rw------- 1 zabbix zabbix 33193 Jun 6 13:47 zabbix_proxy.log.1.bz2 -rw------- 1 zabbix zabbix 34871 May 4 09:48 zabbix_proxy.log.2.bz2
Investigating further I found the contents of `/var/etc/newsyslog.conf.d/zabbix_proxy.log.conf` does indeed set the owner to root
# Automatically generated for package Zabbix Proxy 5.0. Do not edit. /var/log/zabbix-proxy/zabbix_proxy.log root:wheel 600 7 500 * JC
I'll try and remember to check tomorrow but I suspect the files will be owned by root again after the (presumably) daily log rotation occurs.
I haven't made any customizations to the pfsense box. The only other plugins installed are- open-vm-tools v10.1.0_5,1
- openvpn-client-export v1.6_4
- zabbix-agent5 v1.0.4_12
- zabbix-proxy5 v1.0.4_12
I compared `/var/etc/newsyslog.conf.d/zabbix_ agentd .log.conf` with `/var/etc/newsyslog.conf.d/zabbix_ proxy .log.conf`, both set the owners to root
I then checked the ownership of the agent's log files, to my surprize they're owned by Zabbix. I have not restarted the Zabbix agent service today
ls -l /var/log/zabbix-agent/ total 5 -rw-rw-r-- 1 zabbix zabbix 11450 Aug 15 11:49 zabbix_agentd.log
Updated by Kris Phillips over 1 year ago
Hello Steve,
Which version of the Zabbix package are you seeing this behavior? There are several.
Updated by Kris Phillips over 1 year ago
- Status changed from New to Incomplete
Marking as Incomplete until additional details, per previous request, is provided.
Updated by Juraj Lutter over 1 year ago
I see the same behavior, independently from Zabbix versions. The problem indeed, is in newsyslog
config for the services.
These erroneous files are not leftovers after previous versions or something similar. I even did
pkg install -f pfSense-pkg-zabbix-proxy5
to no avail.
Updated by Steve Scotter over 1 year ago
Apologies for the delay Kris. Zabbix had been behaving itself for quite some time (or rather I hadn't noticed it was playing up) until recently (based on the logs, on the Dec 12th, some ten days ago).
In my environment I'm running the Zabbix 5.0 LTS edition. The pfsense package details were in the original description however, I've now formatted it a little better for you. Here are all the packages installed on my system
Pakcage Name | Category | Version |
---|---|---|
Open-VM-Tools | emulators | 10.1.0_5,1 |
openvpn-client-export | security | 1.6_4 |
Service_Watchdog | sysutils | 1.8.7_1 |
zabbix-agent5 | net-mgmt | 1.0.4_12 |
zabbix-proxy5 | net-mgmt | 1.0.4_12 |
I'm aware there is an update from 1.0.4_12 to 1.0.5 which I'm now about to install it.
Recent log entries
Dec 12 20:14:00 queeg500 newsyslog[3283]: logfile turned over due to size>500K zabbix_proxy [12769]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [16820]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [16655]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [10636]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [11324]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [11803]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [10961]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [10498]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [10616]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [15441]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [15396]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [15279]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [12769]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [16820]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [11324]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [11803]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [10616]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [16655]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [10961]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [16600]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [16102]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [16277]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [11636]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [15562]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [14473]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [13732]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [14126]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [11155]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [15853]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [14012]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [12178]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [15134]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [16102]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [12273]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [14473]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [14126]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [15562]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [13732]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [11155]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [15853]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [16277]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [14012]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [12178]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [15134]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [11636]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [15441]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [12273]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [16600]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied zabbix_proxy [15396]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied Dec 12 20:14:00 queeg500 newsyslog[3283]: logfile turned over due to size>500K 10636:20221212:201313.437 cannot send list of active checks to "172.20.223.4": host [prowler] not monitored 12453:20221212:201129.424 housekeeper [deleted 243661 records in 0.365225 sec, idle for 1 hour(s)] 12453:20221212:201129.049 executing housekeeper
I think it's as simple as changing the contents of /var/etc/newsyslog.conf.d/zabbix_proxy.log.conf
from
/var/log/zabbix-proxy/zabbix_proxy.log root:wheel 600 7 500 * JC
to
/var/log/zabbix-proxy/zabbix_proxy.log zabbix:zabbix 600 7 500 * JC
I think the reason I can go a long time without seeing the problem is because the logs only rotated once it reaches 500K.. not daily like I falsey imagined it was.
I also believe it'll be worth repeating for /var/etc/newsyslog.conf.d/zabbix_agentd.log.conf
Updated by Steve Scotter over 1 year ago
I've updated to 1.0.5
and checked the contents of /var/etc/newsyslog.conf.d/zabbix_proxy.log.conf
, it's still setting the owner:group to root:wheel, so the updated hasn't resolved the issue
In all fairness, the package update to 1.0.5 was likely more around updating Zabbix Proxy software and not addressing this issue.
In order to speed up troubleshooting I have added DebugLevel=4
under Advanced Parameters in order to fill the log up much quicker than it normally would.
As soon as it hit 500K it was rotated and the ownership changed from zabbix:zabbix to root:wheel. I think this confirms my proposed solution above.
[2.6.0-RELEASE][root@queeg500.iman.ptptech.co.uk]/root: ls -l /var/log/zabbix-proxy/ total 334 -rw------- 1 root wheel 80 Dec 22 15:36 zabbix_proxy.log -rw------- 1 root wheel 121308 Dec 22 15:36 zabbix_proxy.log.0.bz2 -rw-rw-r-- 1 zabbix zabbix 1120914 Dec 22 15:45 zabbix_proxy.log.old
I don't know if it's important but the properties on the files are set to 0664
upon (re)starting the Zabbix proxy service, but newsyslog is setting it to 0600
Cheers
Steve