Project

General

Profile

Bug #11466

PHP exit with sig 11 on SG-3100

Added by Marcos Mendoza about 2 months ago. Updated 5 days ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
PHP Interpreter
Target version:
-
Start date:
02/19/2021
Due date:
% Done:

0%

Estimated time:
Affected Plus Version:
21.02
Affected Architecture:
SG-3100
Release Notes:
Default

Description

After installing Snort and starting the service on an interface, fails to start and the following is reported on the system logs:

Feb 18 19:21:06 hostname snort[5625]: Commencing packet processing (pid=5625)
Feb 18 19:21:13 hostname kernel: pid 5625 (snort), jid 0, uid 0: exited on signal 10

on another SG3100 when trying to replicate the issue, it shows:

Feb 20 00:30:30     php     64883     [Snort] Updating rules configuration for: WAN ...
Feb 20 00:30:30     php     64883     [Snort] Enabling any flowbit-required rules for: WAN...
Feb 20 00:30:30     php     64883     [Snort] WARNING: Flowbit resolution not done - no rules in /usr/local/etc/snort/rules/ ...
Feb 20 00:30:30     php     64883     [Snort] Building new sid-msg.map file for WAN...
Feb 20 00:30:32     kernel         pid 64883 (php), jid 0, uid 0: exited on signal 11 (core dumped)

Further discussion here:
https://forum.netgate.com/topic/161050/snort-won-t-start-after-upgrade-to-21-02-on-sg-3100/5

This is possibly related to #11444
https://redmine.pfsense.org/issues/11444

History

#1 Updated by Michael Spears about 2 months ago

Marcos Mendoza wrote:

After installing Snort and starting the service on an interface, fails to start and the following is reported on the system logs:
[...]

on another SG3100 when trying to replicate the issue, it shows:
[...]

Further discussion here:
https://forum.netgate.com/topic/161050/snort-won-t-start-after-upgrade-to-21-02-on-sg-3100/5

This is possibly related to #11444
https://redmine.pfsense.org/issues/11444

I was able to reproduce on a 3100.

#2 Updated by Bill Meeks about 2 months ago

The Signal 10 error occurs when an executable attempts to access a memory address on a non-word aligned boundary in ARM hardware. This issue has occurred previously in Netgate's ARM-based appliances (specifically the SG-3100 and SG-1000). The previous fix was to patch the configure script for the Snort binary in the ARM ports tree to always enable a debug build. This suppresses optimization by the llvm compiler. It is the compiler's optimization choices relative to picking "register load from memory" instructions that do not provide auto-fixup of non-aligned access. Thus the Signal 10 error is thrown and the running process (Snort) aborted.

Check the patch files included for the Snort binary in the ARM Ports tree and verify this file is present: patch-pfSense-ARM317.diff

#3 Updated by Scott Long about 2 months ago

I don't think that this is related to https://redmine.pfsense.org/issues/11444.

#4 Updated by Scott Long about 2 months ago

  • Target version set to 2.6.0

#5 Updated by Bill Meeks about 2 months ago

Scott Long wrote:

I don't think that this is related to https://redmine.pfsense.org/issues/11444.

I agree. The Signal 10 problem is not related to Issue #11444.

The Snort binary's C source code is littered with pointer casts, some of which lead to non-aligned memory access. Finding and fixing all of these is a tall order. As I mentioned in a reply to a thread on the pfSense Forum, there is not much appetite upstream to invest the necessary time and energy to find and fix the wayward pointer casts. This is mostly due to the fact Snort use on ARM hardware is rare. As I also mentioned in the forum thread, Intel CPUs will auto-fixup the non-aligned access performed by the wayward pointer casting. I debugged this issue rather extensively a couple of years back with an SG-3100 and idenfied the exact two ARM binary instructions that cause the problem. They are LDM and STM. Their auto-fixup equivalents (albeit a littler slower to execute because of the fixup) are LDR and STR. When the llvm compiler is in optimizing mode, it will choose to use the LDM/STM instructions for speed. However, in some places in the Snort binary C source code, that leads to non-aligned access violations. And where this happens can appear to be random because it takes a particular sequence of data to trigger the action that produces the non-aligned access. When optimization is turned off, the compiler chooses the LDR/STR instructions and thus the same errant C source code will now work. It still produces non-aligned access violations, but instead of triggering the interrupt to halt execution, the CPU will fixup the access by converting it into a series of single-byte accesses that work on non-word boundary addresses.

It's worth noting that Snort is not the only legacy C source code binary that suffers from this problem on ARM hardware. Because Intel hardware has historically always just "fixed it for you", the bad pointer casting has continued to exist. It really would be nice if the llvm compiler had a switch to make sure the LDM/STM instructions could be excluded even when optimizing ARM CPU binary code. From my research into ARM architecture, these are the only two instructions that will never auto-fixup a non-aligned memory access.

Finally, if memory serves me correctly, there is a control register in the ARM CPU that has a bit flag for turning auto-fixup of non-aligned access "on" and "off". At the time I was troubleshooting, the state was toggled to "on", so auto-fixup was enabled. But even with that flag set, those two LDM/STM instructions still will NOT perform auto-fixup by design. But all of the other instructions will perform auto-fixup with that flag set.

#6 Updated by Marcos Mendoza about 2 months ago

The ARM patch for snort is still there:
https://github.com/pfsense/FreeBSD-ports/blob/devel/security/snort/files/patch-pfSense-ARM317.diff

I tried it on another SG-3100 and got:

Feb 22 20:01:19     php-fpm     96482     [Snort] Updating rules configuration for: ...
Feb 22 20:01:20     php-fpm     96482     [Snort] Enabling any flowbit-required rules for: ...
Feb 22 20:01:20     php-fpm     96482     [Snort] Enabling any flowbit-required rules for: ...
Feb 22 20:01:20     php-fpm     96482     [Snort] Building new sid-msg.map file for ...
Feb 22 20:04:17     nginx         2021/02/22 20:04:17 [error] 69690#100091: *842 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.137.1, server: , request: "POST /snort/snort_rulesets.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "192.168.137.141", referrer: "https://192.168.137.141/snort/snort_rulesets.php?id=0" 
Feb 22 20:04:25     kernel         pid 96482 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)

Given the info so far and some further reproducable testing, the signal 11 exit is a different issue hence warrants a different bug report. I have not been able to reproduce the signal 10 exit yet. I will update the ticket accordingly.

#7 Updated by Marcos Mendoza about 2 months ago

  • Subject changed from Snort exit with sig 10 and sig 11 on SG-3100 to Snort exit with sig 10 on SG-3100

#8 Updated by Bill Meeks about 2 months ago

Marcos Mendoza wrote:

The ARM patch for snort is still there:
https://github.com/pfsense/FreeBSD-ports/blob/devel/security/snort/files/patch-pfSense-ARM317.diff

I tried it on another SG-3100 and got:
[...]

Given the info so far and some further reproducable testing, the signal 11 exit is a different issue hence warrants a different bug report. I have not been able to reproduce the signal 10 exit yet. I will update the ticket accordingly.

The logged error messages are quite unusual. There should be an interface name shown after the colon in each line. Was the interface name sanitized before posting the log snippet? The normal log entry will say something like this (assuming it was the LAN interface):

Feb 22 20:01:19     php-fpm     96482     [Snort] Updating rules configuration for: LAN(em1) ...

If the interface name was not scrubbed from the log entry before posting, that would be something to investigate (why it's missing) and might be a clue to the underlying issue. Also, unless I'm am misunderstanding the log sequence, it appears that php-fpm is what died with Signal 11, not the Snort binary.

#9 Updated by Marcos Mendoza about 2 months ago

They were not scrubbed. Here are the steps to reproduce it (was not able to reproduce on a x86 system).

Only Snort installed:
  1. Enable ET Open rules
  2. Force rules update
  3. Go to "Services / Snort / Interfaces"
  4. Click "Add" then switch to "WAN Categories" without saving
  5. Tab is now named "None Categories"
  6. Select a category then click "Save"
Result
  • GUI times out with HTTP 502 error; must refresh
  • Crash seen on system log "pid 19650 (php), jid 0, uid 0: exited on signal 11 (core dumped) "

Only Snort installed:
  1. Enable ET Open rules
  2. Force rules update
  3. Go to "Services / Suricata / Interfaces"
  4. Click "Add", WAN interface is selected, click Save
  5. Go to "WAN Categories"
  6. Select a category then click "Save"
Result:
  • GUI responds correctly
  • No crash in system logs.
  • If the start button is manually clicked on the "Snort / Interfaces" tab, same signal 11 crash appears.

And here's some weird behavior I noticed that maybe as the dev you can answer:

Suricata AND Snort installed (same steps as above):
  1. Enable ET Open rules
  2. Force rules update
  3. Go to "Services / Snort / Interfaces"
  4. Click "Add" then switch to "WAN Categories" without saving
  5. Tab is now named "None Categories"
  6. Select a category then click "Save"
Result:
  • GUI responds correctly
  • No crash in system logs.
  • If the start button is manually clicked on the "Snort / Interfaces" tab, same signal 11 crash appears.

Lastly, I re-did the same tests (possibly with some variation in order) over a dozen times. I ran into some behavior that I have not been able to reproduce consistently:
  • Suricata experienced the same signal 11 crashes following similar steps. Someone else managed to reproduce it as well.
  • Snort started successfully
  • Snort crashed with signal 10:
    Feb 23 00:40:20     php-fpm     15106     Starting Snort on WAN(mvneta2) per user request...
    Feb 23 00:40:20     php     21226     [Snort] Updating rules configuration for: WAN ...
    Feb 23 00:40:22     php     21226     [Snort] Enabling any flowbit-required rules for: WAN...
    Feb 23 00:40:22     php     21226     [Snort] Enabling any flowbit-required rules for: WAN...
    Feb 23 00:40:22     php     21226     [Snort] Building new sid-msg.map file for WAN...
    Feb 23 00:40:22     php     21226     [Snort] Snort START for WAN(mvneta2)...
    Feb 23 00:40:23     kernel         mvneta2: promiscuous mode enabled
    Feb 23 00:40:25     kernel         pid 25778 (snort), jid 0, uid 0: exited on signal 10
    Feb 23 00:40:25     kernel         mvneta2: promiscuous mode disabled 
    

Hopefully this is enough to give a clue as to what's going on.

#10 Updated by Bill Meeks about 2 months ago

So to make sure I understand, this only happens on an SG-3100 and you can't reproduce on x86 hardware.

The first test case, adding an interface and NOT saving the change but then clicking the CATEGORIES tab (and probably any other tab) results in a GUI timeout and Signal 11 crash in php-fpm, is likely due to an unexpected condition. All of the tabs (except INTERFACE SETTINGS, when adding a new interface) expect a valid interface to be passed to them either via query string (via $_GET) or as a $_POST parameter. So the fix there is likely just additional validation to prevent a user from doing that sequence of steps as it is an undefined scenario. There would never be a reason to attempt to view CATEGORIES or any other Snort parameters for a non-existent interface (meaning one that was not saved).

The second case, adding an interface and saving it before leaving the INTERFACE SETTINGS tab, is stranger. It only triggers the Signal 11 from php-fpm when attempting a manual start. Don't really know what's happening there. The manual start icon, when clicked, launches a background process to start Snort on the interface and then uses a jQuery postback loop to monitor the status of the startup. When it detects the correct PID file in /var/run, then it updates the icon from "starting" to "running". So apparently the above is not happening as designed on the SG-3100 for some reason.

Your final case with both Snort and Suricata installed is very strange. Are you 100% positive of those steps and that result? I ask because there is zero interaction between Snort and Suricata in the GUI. All of their files are in separate sub-directories. The only thing they share is access to the snort2c pf table when Legacy Mode Blocking is used.

Can you give the version number of the Snort package you are using on the SG-3100? Is it the same as the one in the x86 CE package repository? Is the Snort binary 2.9.17?

And I thought of something else that might help. Can you grep the dmesg log (or elsewhere) to see if php-fpm is logging any additional information about its crash? Specifically I'm hoping it will log which library (or plugin) it is executing when the crash happens. The fact this does not happen on x86 hardware makes me wonder if there is actually a problem in one of the libraries for PHP, and Snort happens it to tickle it just right so it triggers.

#11 Updated by Marcos Mendoza about 2 months ago

The behavior with both Snort and Suricata installed was definitely strange and didn't make sense to me. I did a fresh install and spent some more time testing to try narrowing it down.

The errors that can trigger are:
  • kernel         pid 56473 (*php*), jid 0, uid 0: exited on signal 11 (core dumped)
    
  • kernel         pid 48675 (*php-fpm*), jid 0, uid 0: exited on signal 11 (core dumped)
    
  • kernel         pid 4696 (*snort*), jid 0, uid 0: exited on signal 10
    
I can trigger the errors with ease, but what exactly triggers them... is something I hope this helps find:
  1. Go to "Services / Snort"
  2. Add interface; click Save
  3. Click on "Snort Interfaces" to return to interface list
  4. Edit interface, switch to categories
  5. Select rule, save (may trigger php-fpm error)
  6. Go to "Snort Interfaces" to return to interface list
  7. EITHER: Click on Start button (likely trigger php error; may start normally)
  8. OR: Go to "Status / Services" (likely trigger sig 10 snort error)
  9. Go to "Snort Interfaces", click Stop button
  10. Delete interface
  11. Repeat steps

When it starts normally, it says:

php     25214     [Snort] Snort START for WAN(mvneta2)...

Checking logs:
  • dmesg only shows the same signal 10/11 errors seen in system log
  • No luck trying to enable logging in /usr/local/etc/php-fpm.d/www.conf
  • /tmp/php_errors.txt also kept empty

Using:
snort-2.9.17
pfSense-pkg-snort-4.1.3_1

#12 Updated by Bill Meeks about 2 months ago

Thanks for the additional info. I will investigate further. The Signal 10 from the Snort binary I am not really surprised about. There are many places in the Snort binary's C source code where memory pointer casts between types are done, and those are likely the source of the Signal 10 bus errors (which are actually non-aligned memory access aborts). The frequency and repeatibility of the Signal 10 errors was influenced by the particular Snort configuration and traffic at the time. This is because the program execution steps are in many ways dependent on the traffic (data) being analyzed at a particular moment. Things like if-then tests can result in different execution paths where one path works and the other gives you a Signal 10.

The GUI issues are more perplexing. Really can't imagine how that is killing php-fpm.

I am rebuilding my package repository and test environment for the 2.5.0-RELEASE branch. That will take a few hours to complete, then I can look into this in earnest. I loaned my SG-3100 to our church for production use, so the only ARM appliance I currently have is an SG-1100. I will fire it up and see if the issue can be reproduced there, since it's also an ARM device (albeit a 64-bit one and not 32-bit like the SG-3100).

Bill

#13 Updated by Bill Meeks about 2 months ago

Marcos:

I'm running into difficulty updating my SG-1100 to the latest version. It is still on the 2.4.4 factory image. I am unable to get it to see the 2.4.5 branch, nor can I get it to see the new 21.02 branch. And it is also unable to see any packages (even on the 2.4.4 branch), so I can't install Snort.

Got an image file from Netgate Support to update the SG-1100 to 21.02-RELEASE.

Can we take this offline to an email thread? I assume you are a Netgate employee. You can contact me here "bill at themeeks dot net" (actual email obfuscated a bit to hinder web scraping).

Thanks,
Bill

#14 Updated by Bill Meeks about 2 months ago

Another Update

None of the conditions described in this bug report occur on an SG-1100 (64-bit ARM CPU), and neither do they occur on x86-64 hardware. My theory is the Signal 11 event for php-fpm is something related to 32-bit code. Might be some kind of integer overflow or something, and Snort's GUI PHP code happens to tickle it just right to trigger the event. We don't have any 32-bit x86 images to test this with, though. It would be interesting to see if these problems happen on 32-bit x86 platforms.

Since I can't reproduce this on the hardware I currently have to test with, I'm stalled with troubleshooting the issues any further.

I do have a fix for the essentially cosmetic issue of adding an interface, but not saving the change, and then browsing to another Snort GUI tab that expects a valid interface to be passed. The fixed code simply hides all of the Snort GUI tabs that expect a valid interface ID to be passed to them when adding a new interface. I will incorporate this fix into the next Snort package update.

I will be glad to help troubleshoot the SG-3100 issue remotely if we can work that out.

#15 Updated by Bill Meeks about 2 months ago

Update on this issue

The problem is somewhere within the PHP base function preg_match().

Here is a PHP code snippet that works fine. This is lifted from the actual Snort GUI code. Notice the last call to preg_match() is commentend out.

<?php

$rule1 = 'alert ( msg:"DECODE_NOT_IPV4_DGRAM"; sid:1; gid:116; rev:1; metadata:rule-type decode; classtype:protocol-command-decode;)';

$matches = array();

if (preg_match('/\bmsg\s*:\s*"(.+?)"\s*;/i', $rule1, $matches))
      $msg = trim($matches[1]);
if (preg_match('/\bsid\s*:\s*(\d+)\s*;/i', $rule1, $matches))
      $sid = trim($matches[1]);
if (preg_match('/\bgid\s*:\s*(\d+)\s*;/i', $rule1, $matches))
      $gid = trim($matches[1]);
if (preg_match('/\brev\s*:\s*([^\;]+)/i', $rule1, $matches))
      $rev = trim($matches[1]);
//if (preg_match('/\bclasstype\s*:\s*([^\;]+)/i', $rule1, $matches))
//      $classtype = trim($matches[1]);

print $msg . "\n";
print $sid . "\n";
print $gid . "\n";
print $rev . "\n";
print $classtype . "\n";

?>

The code snippet below will fail and trigger a Signal 11 core dump in PHP on the last call to preg_match(). Notice the comments are now removed for that last call.

<?php

$rule1 = 'alert ( msg:"DECODE_NOT_IPV4_DGRAM"; sid:1; gid:116; rev:1; metadata:rule-type decode; classtype:protocol-command-decode;)';

$matches = array();

if (preg_match('/\bmsg\s*:\s*"(.+?)"\s*;/i', $rule1, $matches))
      $msg = trim($matches[1]);
if (preg_match('/\bsid\s*:\s*(\d+)\s*;/i', $rule1, $matches))
      $sid = trim($matches[1]);
if (preg_match('/\bgid\s*:\s*(\d+)\s*;/i', $rule1, $matches))
      $gid = trim($matches[1]);
if (preg_match('/\brev\s*:\s*([^\;]+)/i', $rule1, $matches))
      $rev = trim($matches[1]);
if (preg_match('/\bclasstype\s*:\s*([^\;]+)/i', $rule1, $matches))
      $classtype = trim($matches[1]);

print $msg . "\n";
print $sid . "\n";
print $gid . "\n";
print $rev . "\n";
print $classtype . "\n";

?>

And even more curious, this code below will also run successfully. Notice all the calls to preg_match() except the last one are commented out this time. This indicates to me there is nothing wrong with the regex expression in that last function call. Rather it is something in PHP itself.

<?php

$rule1 = 'alert ( msg:"DECODE_NOT_IPV4_DGRAM"; sid:1; gid:116; rev:1; metadata:rule-type decode; classtype:protocol-command-decode;)';

$matches = array();

//if (preg_match('/\bmsg\s*:\s*"(.+?)"\s*;/i', $rule1, $matches))
//      $msg = trim($matches[1]);
//if (preg_match('/\bsid\s*:\s*(\d+)\s*;/i', $rule1, $matches))
//      $sid = trim($matches[1]);
//if (preg_match('/\bgid\s*:\s*(\d+)\s*;/i', $rule1, $matches))
//      $gid = trim($matches[1]);
//if (preg_match('/\brev\s*:\s*([^\;]+)/i', $rule1, $matches))
//      $rev = trim($matches[1]);
if (preg_match('/\bclasstype\s*:\s*([^\;]+)/i', $rule1, $matches))
      $classtype = trim($matches[1]);

print $msg . "\n";
print $sid . "\n";
print $gid . "\n";
print $rev . "\n";
print $classtype . "\n";

?>

This code snippet only fails on 32-bit ARM hardware. There are no issues with this exact same code on 64-bit hardware (SG-1100) nor on x86-64 hardware.

#16 Updated by Steve Yates about 2 months ago

Simply out of curiosity I did a quick search and found this "not a bug" from 2008: https://bugs.php.net/bug.php?id=45735 "When running a preg_match with a capturing subpattern against large input, php crashes with a Segmentation Fault"

#17 Updated by Bill Meeks about 2 months ago

Steve Yates wrote:

Simply out of curiosity I did a quick search and found this "not a bug" from 2008: https://bugs.php.net/bug.php?id=45735 "When running a preg_match with a capturing subpattern against large input, php crashes with a Segmentation Fault"

Thanks Steve for the info and link. Very interesting!

I was planning on spending some time today searching the web to see if any similar issues had been reported.

In the case of the Snort and Suricata GUI code, it's like this is some kind of cumulative thing. As I said, the calls work individually, and they all work sequentially so long as you comment out that last one. But the last one will work individually with the others commented out.

I'm not a regex guru for sure, but I will see about finding another pattern match string that might work without generating the segfault.

#18 Updated by Bill Meeks about 2 months ago

Another day of frustrating, but ultimately not too productive, testing leads me to conclude this is something with 32-bit PHP somewhere. Simply rearranging the sequence of preg_match() calls will result in all of them succeeding. And in the original order of calls, commenting out some will result in success as well.

At first I was thinking the regex grouping criteria was perhaps leading to backtrack recursion problems (especially considering the link posted above by Steve Yates). But if the regex was truly the root cause, you would expect the backtrack recursion crash to happen in the same regex all the time no matter when it was called (when it is called with the same data). The fact simply rearranging the sequence of calls stops the problem seems to eliminate the regex itself as the root cause. These preg_match() calls (6 of them) are designed to pull out specific pieces of a Snort or Suricata text rule. In the actual Snort GUI code they exist in a foreach() iteration that loops over each line in a rules file to process the single rule from that line. So in a given iteration, the preg_match() calls are all working from the same source string.

While I could fix the immediate issue (apparently) on the SG-3100 by simply rearranging the function calls, I am certainly not confident in that as a long-term solution. It is highly likely a slightly different set of circumstances could break it again.

#19 Updated by Marcos Mendoza about 2 months ago

  • Subject changed from Snort exit with sig 10 on SG-3100 to Snort exit with sig 11 on SG-3100
  • Affected Architecture SG-3100 added

#20 Updated by Jim Pingle about 1 month ago

Has anyone tried this on a 21.05 snapshot with PHP 7.4.16? The release notes for PHP 7.4.16 mention they fixed a segfault in SPL (Standard PHP Library), which could potentially be relevant. Worth trying at least.

Make sure you have an install image for 21.02-p1 on hand in case you have to downgrade.

#21 Updated by Marcos Mendoza about 1 month ago

Tested on:

21.05-DEVELOPMENT (arm)
built on Tue Mar 09 11:27:41 EST 2021
FreeBSD 12.2-STABLE

  1. Fresh install on SG-3100
  2. Create /tmp/test.php with example code from Bill above
  3. Ran "php /tmp/test.php"
    [21.05-DEVELOPMENT][root@pfSense.home.arpa]/root: php /tmp/test.php
    Segmentation fault (core dumped)
    

System log shows:

Mar 9 20:53:38     kernel         pid 35324 (php), jid 0, uid 0: exited on signal 11 (core dumped)

I did not install any package - all was done from console.

#22 Updated by Marcos Mendoza about 1 month ago

  • Project changed from pfSense Packages to pfSense Plus
  • Subject changed from Snort exit with sig 11 on SG-3100 to PHP exit with sig 11 on SG-3100
  • Category changed from Snort to PHP Interpreter
  • Target version deleted (2.6.0)
  • Affected Version deleted (2.5.0)
  • Affected Plus Version set to 21.02

Updating bug report to focus on PHP issue, given that the snort sig 10 issue is unlikely related, and this seems to affect more than just the snort package.

#23 Updated by Marcos Mendoza about 1 month ago

Likely related #11605 and #11551

#24 Updated by Bill Meeks 11 days ago

One of the issues identified in this ticket, the logging of "blank" interface names and the display of "Unknown" as the interface name in some GUI tabs when adding a new interface (or cloning an existing one) have been fixed in Pull Request 1058 here: https://github.com/pfsense/FreeBSD-ports/pull/1058.

#25 Updated by Renato Botelho 5 days ago

Bill Meeks wrote:

One of the issues identified in this ticket, the logging of "blank" interface names and the display of "Unknown" as the interface name in some GUI tabs when adding a new interface (or cloning an existing one) have been fixed in Pull Request 1058 here: https://github.com/pfsense/FreeBSD-ports/pull/1058.

PR merged to devel branches. Will be cherry-picked to stable after some tests

Also available in: Atom PDF