Fortigate Firewall Performance Troubleshooting

CPU and Memory Usage

get system performance status gives a rough overview over the system status.

get system performance status

Single processes

diag sys top shows the detail of every single process. To debug CPU problems, the ideal tool

diag sys top 1 30

Run Time:  44 days, 10 hours and 20 minutes
0U, 0N, 0S, 99I, 0WA, 0HI, 1SI, 0ST; 1867T, 1236F
          cw_acd      150      S       0.9     1.4
          hasync      133      S <     0.9     0.5
         src-vis    23593      S       0.9     0.4
          newcli    12969      R       0.9     0.2
         miglogd      114      S       0.0     1.6
         cmdbsvr       93      S       0.0     1.5
         miglogd      177      S       0.0     1.5
         miglogd      176      S       0.0     1.5
         pyfcgid     2504      S       0.0     1.1
       forticron      123      S       0.0     0.9
          httpsd      116      S       0.0     0.8
          httpsd    19863      S       0.0     0.8
          httpsd     2683      S       0.0     0.8
         pyfcgid     2508      S       0.0     0.7
         pyfcgid     2506      S       0.0     0.7
         pyfcgid     2507      S       0.0     0.7
         updated      222      S       0.0     0.5

Refresh every 1 second, 30 processes displayed.

Top CPU usage on top: Shift-P

top memory usage on top: Shift-M

The columns show process name, process ID, status, % CPU usage, % memory usage.

Process status: S = Sleeping, R = Running, D = Do not Disturb, Z = Zombie. D and Z are not killable. D can happen rarely and shortly. Z must not appear.

Complete subsystems

diag sys top-summary

diag sys top-summary shows a summary to the complete subsystem, shared memory included.

diag sys top-summary got a problem in 5.6.3.

Update: diag sys top-summary wars removed in 6.4.

Kill processes

Processes with the status “S” or “R” can be killed. There is a watchdog running on the FortiGate wich launches the process again, if it is killed. DANGER! Killing processes can result in a malfunction of your device and interrupt your production environment. Use diag sys kill only, if you know exactly what you do.

diag sys kill 9 <process-id>

A good friend, who needs to be restarted from time to time is the IPS engine. The IPS engine sometimes consumes all available memory. Instead of rebooting the device or killing the processes, you can do

diag test appl ipsmonitor 99

With this command you do a clean restart of the IPS subsystem.

A log is available on the FortiGate. There you can see how the processes were terminated.

diag debug crashlog read

In case of a clean termination it looks like:

61: 2018-01-15 08:47:29 the killed daemon is /bin/pyfcgid: status=0x0
62: 2018-01-16 22:50:05 the killed daemon is /bin/hatalk: status=0x0

Crashed or killed processes look like:

68: 2018-01-25 10:59:06 <00136> firmware FortiGate-80E v5.6.3,build1547b1547,171204 (GA) (Release)
69: 2018-01-25 10:59:06 <00136> application src-vis
70: 2018-01-25 10:59:06 <00136> *** signal 11 (Segmentation fault) received ***
71: 2018-01-25 10:59:06 <00136> Register dump:
72: 2018-01-25 10:59:06 <00136> R0: 04b2d300   R1: 5ec6a240   R2: 00000014    R3: 00000000
73: 2018-01-25 10:59:06 <00136> R4: 01932260   R5: 00000000   R6: 5ec6a368    R7: 5ec6a254
74: 2018-01-25 10:59:06 <00136> R8: 5ec6a354   R9: 01994184  R10: 04b8d660    FP: 01994188
75: 2018-01-25 10:59:06 <00136> IP: 00000000   SP: 5ec6a220   LR: 00153aa3    PC: 001538ea
76: 2018-01-25 10:59:06 <00136> CPSR: 000e0030   Addr: 00000000
77: 2018-01-25 10:59:06 <00136> Trap: 0000000e   Error: 00000017   OldMask: 00000000
78: 2018-01-25 10:59:06 <00136> Backtrace:
79: 2018-01-25 10:59:06 <00136> [0x001538ea] => /bin/src-vis  
80: 2018-01-25 10:59:06 <00136> [0x00153aa2] => /bin/src-vis  
81: 2018-01-25 10:59:06 <00136> [0x00155a12] => /bin/src-vis  
82: 2018-01-25 10:59:06 <00136> [0x0014e400] => /bin/src-vis  
83: 2018-01-25 10:59:06 <00136> [0x00150a92] => /bin/src-vis  
84: 2018-01-25 10:59:06 <00136> [0x0014f696] => /bin/src-vis  
85: 2018-01-25 10:59:06 <00136> [0x0016cf56] => /bin/src-vis  
86: 2018-01-25 10:59:06 <00136> [0x00c65d40] => /bin/src-vis

Conserve Mode

If your FortiGate uses to much memory, it ends in conserve mode. Conserve Mode disables the execution of security profiles.

diag hardware sysinfo conserve

Using this command, you can get the thresholds of your machine and you can see if your device is in conserve mode or not.

Tags: ,