Project

General

Profile

Actions

Bug #13444

open

zabbix_proxy : cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied

Added by Steve Scotter over 1 year ago. Updated about 1 year ago.

Status:
Incomplete
Priority:
Normal
Assignee:
-
Category:
Zabbix
Target version:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Plus Target Version:
Affected Version:
Affected Plus Version:
Affected Architecture:

Description

Hi

I frequently come across this issue when trying to investigate why a Zabbix agent isn't communicating successfully with our Zabbix server.

When I navigate to https://pfsense-ip-address/status_logs_packages.php?pkg=Zabbix%20Proxy%205.0 I'm presented with the following (truncated) logs

Jul 15 03:09:00 queeg500 newsyslog[90148]: logfile turned over due to size>500K
zabbix_proxy [78631]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [82116]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
*** Above lines repeated 50+ times ***
Jul 15 03:09:00 queeg500 newsyslog[90148]: logfile turned over due to size>500K
...
...

Logging appears to have stopped ~40 days ago.

Restarting the Zabbix proxy service (via https://pfsense-ip-address/status_services.php#) gets logging working again, however its a pain because I generally speaking I wanted to see the logs for the past to investigate the problem I'm dealing with at that specific time.

I suspect the issue is related to log rotation and file permissions based on the Permission denied error and that newsyslog is mentioned before and after the logging stops working.

Today, before I restart the service I checked who owned the log file...

[2.6.0-RELEASE][root@pfsense-ip-address]/root: ls -l /var/log/zabbix-proxy/
total 106
-rw-------  1 root  wheel     80 Jul 15 03:09 zabbix_proxy.log
-rw-------  1 root  wheel  29744 Jul 15 03:09 zabbix_proxy.log.0.bz2
-rw-------  1 root  wheel  33193 Jun  6 13:47 zabbix_proxy.log.1.bz2
-rw-------  1 root  wheel  34871 May  4 09:48 zabbix_proxy.log.2.bz2

After I restarted the service I checked again...

[2.6.0-RELEASE][root@fsense-ip-address]/root: ls -l /var/log/zabbix-proxy/
total 110
-rw-------  1 zabbix  zabbix   3218 Aug 25 13:42 zabbix_proxy.log
-rw-------  1 zabbix  zabbix  29744 Jul 15 03:09 zabbix_proxy.log.0.bz2
-rw-------  1 zabbix  zabbix  33193 Jun  6 13:47 zabbix_proxy.log.1.bz2
-rw-------  1 zabbix  zabbix  34871 May  4 09:48 zabbix_proxy.log.2.bz2

Investigating further I found the contents of `/var/etc/newsyslog.conf.d/zabbix_proxy.log.conf` does indeed set the owner to root

# Automatically generated for package Zabbix Proxy 5.0. Do not edit.
/var/log/zabbix-proxy/zabbix_proxy.log          root:wheel      600     7       500     *       JC

I'll try and remember to check tomorrow but I suspect the files will be owned by root again after the (presumably) daily log rotation occurs.

I haven't made any customizations to the pfsense box. The only other plugins installed are
  • open-vm-tools v10.1.0_5,1
  • openvpn-client-export v1.6_4
  • zabbix-agent5 v1.0.4_12
  • zabbix-proxy5 v1.0.4_12

I compared `/var/etc/newsyslog.conf.d/zabbix_ agentd .log.conf` with `/var/etc/newsyslog.conf.d/zabbix_ proxy .log.conf`, both set the owners to root

I then checked the ownership of the agent's log files, to my surprize they're owned by Zabbix. I have not restarted the Zabbix agent service today

ls -l /var/log/zabbix-agent/
total 5
-rw-rw-r--  1 zabbix  zabbix  11450 Aug 15 11:49 zabbix_agentd.log
Actions #1

Updated by Kris Phillips over 1 year ago

Hello Steve,

Which version of the Zabbix package are you seeing this behavior? There are several.

Actions #2

Updated by Kris Phillips over 1 year ago

  • Status changed from New to Incomplete

Marking as Incomplete until additional details, per previous request, is provided.

Actions #3

Updated by Juraj Lutter over 1 year ago

I see the same behavior, independently from Zabbix versions. The problem indeed, is in newsyslog config for the services.
These erroneous files are not leftovers after previous versions or something similar. I even did

pkg install -f pfSense-pkg-zabbix-proxy5

to no avail.

Actions #4

Updated by Steve Scotter about 1 year ago

Apologies for the delay Kris. Zabbix had been behaving itself for quite some time (or rather I hadn't noticed it was playing up) until recently (based on the logs, on the Dec 12th, some ten days ago).

In my environment I'm running the Zabbix 5.0 LTS edition. The pfsense package details were in the original description however, I've now formatted it a little better for you. Here are all the packages installed on my system

Pakcage Name Category Version
Open-VM-Tools emulators 10.1.0_5,1
openvpn-client-export security 1.6_4
Service_Watchdog sysutils 1.8.7_1
zabbix-agent5 net-mgmt 1.0.4_12
zabbix-proxy5 net-mgmt 1.0.4_12

I'm aware there is an update from 1.0.4_12 to 1.0.5 which I'm now about to install it.

Recent log entries

Dec 12 20:14:00 queeg500 newsyslog[3283]: logfile turned over due to size>500K
zabbix_proxy [12769]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [16820]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [16655]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [10636]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [11324]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [11803]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [10961]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [10498]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [10616]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [15441]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [15396]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [15279]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [12769]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [16820]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [11324]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [11803]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [10616]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [16655]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [10961]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [16600]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [16102]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [16277]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [11636]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [15562]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [14473]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [13732]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [14126]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [11155]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [15853]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [14012]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [12178]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [15134]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [16102]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [12273]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [14473]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [14126]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [15562]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [13732]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [11155]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [15853]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [16277]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [14012]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [12178]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [15134]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [11636]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [15441]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [12273]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [16600]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
zabbix_proxy [15396]: cannot open "/var/log/zabbix-proxy/zabbix_proxy.log": [13] Permission denied
Dec 12 20:14:00 queeg500 newsyslog[3283]: logfile turned over due to size>500K
10636:20221212:201313.437 cannot send list of active checks to "172.20.223.4": host [prowler] not monitored
12453:20221212:201129.424 housekeeper [deleted 243661 records in 0.365225 sec, idle for 1 hour(s)]
12453:20221212:201129.049 executing housekeeper

I think it's as simple as changing the contents of /var/etc/newsyslog.conf.d/zabbix_proxy.log.conf from

/var/log/zabbix-proxy/zabbix_proxy.log          root:wheel      600     7       500     *       JC

to

/var/log/zabbix-proxy/zabbix_proxy.log          zabbix:zabbix      600     7       500     *       JC

I think the reason I can go a long time without seeing the problem is because the logs only rotated once it reaches 500K.. not daily like I falsey imagined it was.

I also believe it'll be worth repeating for /var/etc/newsyslog.conf.d/zabbix_agentd.log.conf

Actions #5

Updated by Steve Scotter about 1 year ago

I've updated to 1.0.5 and checked the contents of /var/etc/newsyslog.conf.d/zabbix_proxy.log.conf, it's still setting the owner:group to root:wheel, so the updated hasn't resolved the issue

In all fairness, the package update to 1.0.5 was likely more around updating Zabbix Proxy software and not addressing this issue.

In order to speed up troubleshooting I have added DebugLevel=4 under Advanced Parameters in order to fill the log up much quicker than it normally would.

As soon as it hit 500K it was rotated and the ownership changed from zabbix:zabbix to root:wheel. I think this confirms my proposed solution above.

[2.6.0-RELEASE][root@queeg500.iman.ptptech.co.uk]/root: ls -l /var/log/zabbix-proxy/
total 334
-rw-------  1 root    wheel        80 Dec 22 15:36 zabbix_proxy.log
-rw-------  1 root    wheel    121308 Dec 22 15:36 zabbix_proxy.log.0.bz2
-rw-rw-r--  1 zabbix  zabbix  1120914 Dec 22 15:45 zabbix_proxy.log.old

I don't know if it's important but the properties on the files are set to 0664 upon (re)starting the Zabbix proxy service, but newsyslog is setting it to 0600

Cheers

Steve

Actions

Also available in: Atom PDF