Project

General

Profile

Bug #7426

UDP packet drops

Added by Chris Macmahon 9 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
03/24/2017
Due date:
% Done:

100%

Affected Version:
2.4
Affected Architecture:
SG-1000

Description

When doing an iperf test outside of pfsense there is a strange packet loss at the start of the test.

UDP packets drop
TCP packets go through.

Was able to confirm it's isolated to an SG1000, on 2.4 SG-2220 no issues were present in this test, but we were able to reproduce on SG1000.

See ticket 15982

sg1k-udp-64b-test.png - sg1k 64 byte pkts test result (57.3 KB) Constantine Kormashev, 10/18/2017 01:37 AM

History

#1 Updated by Constantine Kormashev 8 months ago

I made some tests with simple DNS answer/reply and noticed problem with states overload (250 clients and 250 servers which is 62500 states in total vs 49000 by default) and UDP drops. Just increased states to 500000 and there was not UDP drops until huge UDP load (10000 answer/reply in second)

#2 Updated by Luiz Souza 4 months ago

  • Target version changed from 2.4.0 to 2.4.1

#3 Updated by Constantine Kormashev about 2 months ago

The reason of UDP drop is packet processing slowdown which happens on ARM devices (1k, 3100). I observed ~2-7% for different packet size and one does not change if bandwidth is changed (provided that packet size is not changed). It does not affect data transmission in whole, but UDP tests show drop because UDP traffic generator knows nothing about delay.
Picture below demonstrates this, blue line is number of RX packets one is the same every time ~24.5kpps which is maximum performance for sg1k on 64byte frames.

sg1k 64 byte pkts test result

And no drops for DNS request/answer (~5000pps)

--------------- 
port : 0 
------------
 opackets                                 : 112499 
 obytes                                   : 8662423 
 ipackets                                 : 112502 
 ibytes                                   : 10462483 
 Tx :       1.31 Mbps  
port : 1 
------------
 opackets                                 : 112499 
 obytes                                   : 10462407 
 ipackets                                 : 112506 
 ibytes                                   : 8662871 

#4 Updated by Constantine Kormashev about 2 months ago

An important note

The issue can be found on:
  • sg1k for ordinary data transmission
  • 3100 for VPN data transmission when CPU is highly loaded
  • hardly ever on intel devices, I saw it several times for VPN data transmission with hight CPU load and huge number of packets (usually small/random)

#5 Updated by Jim Pingle about 2 months ago

  • Target version changed from 2.4.1 to 2.4.2

#6 Updated by Jim Pingle about 2 months ago

  • Project changed from pfSense Packages to pfSense
  • Target version changed from 2.4.2 to 2.4.3

#7 Updated by Luiz Souza about 2 months ago

  • Status changed from New to Feedback
  • % Done changed from 0 to 100

Fixed. Add nmbclusters set to 500000 on SG-1000 and 1000000 on SG-3100.

#8 Updated by James Dekker about 1 month ago

Tested iperf3 in UDP mode between SG-1000 and SG-2440 (and SG-3100), could not reproduce the bad behavior. Saw a problem on one device, but it appears the issue there was the client's nic (not the SG-1000 itself). Tested with pfSense-netgate-uFW-recover-2.4.2-DEVELOPMENT-armv6-20171104-1805

#9 Updated by Jim Pingle about 1 month ago

  • Status changed from Feedback to Resolved
  • Target version changed from 2.4.3 to 2.4.2

Also available in: Atom PDF