Fortigate Firewall Traffic Processing Troubleshooting

Traffic Processing

Application Debugging

diag debug application 

shows what happens during the execution of a process. Detailed information is shown. Debug level is a bit mask.

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
diag debug application [application name] [debug level]
diag debug application [application name] [debug level]
diag debug application [application name] [debug level]

Debug level: -1 or 255 displays everything (normally). 

It is possible to show date and time:

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
diag debug console timestamp enable
diag debug console timestamp enable
diag debug console timestamp enable

Start or stop to display the output of the information.

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
diag debug enable
diag debug disable
diag debug enable diag debug disable
diag debug enable
diag debug disable

Reset all debugging to 0

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
diag debug reset
diag debug reset
diag debug reset

 

diag test application 

this command prints settings and status of processes. Additionally it is possible to modify these settings or to change the behaviour.

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
diag test application [application name] [test]
diag test application [application name] [test]
diag test application [application name] [test]

Using Test Level “0” prints usually a help page.

Example with ipsmonitor:

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
diag test application ipsmonitor 0
diag test application ipsmonitor 0
diag test application ipsmonitor 0

will show

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
2018-03-10 08:09:00
IPS Engine Test Usage:
2018-03-10 08:09:00 1: Display IPS engine information
2018-03-10 08:09:00 2: Toggle IPS engine enable/disable status
2018-03-10 08:09:00 3: Display restart log
2018-03-10 08:09:00 4: Clear restart log
2018-03-10 08:09:00 5: Toggle bypass status
2018-03-10 08:09:00 6: Submit attack characteristics now
2018-03-10 08:09:00 10: IPS queue length
2018-03-10 08:09:00 11: Clear IPS queue length
2018-03-10 08:09:00 12: IPS L7 socket statistics
2018-03-10 08:09:00 13: IPS session list
2018-03-10 08:09:00 14: IPS NTurbo statistics
2018-03-10 08:09:00 15: IPSA statistics
2018-03-10 08:09:00 16: Display device identification cache
2018-03-10 08:09:00 17: Clear device identification cache
2018-03-10 08:09:00 18: Display session info cache
2018-03-10 08:09:00 19: Clear session info cache
2018-03-10 08:09:00 21: Reload FSA malicious URL database
2018-03-10 08:09:00 22: Reload whitelist URL database
2018-03-10 08:09:00 24: Display Flow AV statistics
2018-03-10 08:09:00 25: Reset Flow AV statistics
2018-03-10 08:09:00 96: Toggle IPS engines watchdog timer
2018-03-10 08:09:00 97: Start all IPS engines
2018-03-10 08:09:00 98: Stop all IPS engines
2018-03-10 08:09:00 99: Restart all IPS engines and monitor
2018-03-10 08:09:00 IPS Engine Test Usage: 2018-03-10 08:09:00 1: Display IPS engine information 2018-03-10 08:09:00 2: Toggle IPS engine enable/disable status 2018-03-10 08:09:00 3: Display restart log 2018-03-10 08:09:00 4: Clear restart log 2018-03-10 08:09:00 5: Toggle bypass status 2018-03-10 08:09:00 6: Submit attack characteristics now 2018-03-10 08:09:00 10: IPS queue length 2018-03-10 08:09:00 11: Clear IPS queue length 2018-03-10 08:09:00 12: IPS L7 socket statistics 2018-03-10 08:09:00 13: IPS session list 2018-03-10 08:09:00 14: IPS NTurbo statistics 2018-03-10 08:09:00 15: IPSA statistics 2018-03-10 08:09:00 16: Display device identification cache 2018-03-10 08:09:00 17: Clear device identification cache 2018-03-10 08:09:00 18: Display session info cache 2018-03-10 08:09:00 19: Clear session info cache 2018-03-10 08:09:00 21: Reload FSA malicious URL database 2018-03-10 08:09:00 22: Reload whitelist URL database 2018-03-10 08:09:00 24: Display Flow AV statistics 2018-03-10 08:09:00 25: Reset Flow AV statistics 2018-03-10 08:09:00 96: Toggle IPS engines watchdog timer 2018-03-10 08:09:00 97: Start all IPS engines 2018-03-10 08:09:00 98: Stop all IPS engines 2018-03-10 08:09:00 99: Restart all IPS engines and monitor
2018-03-10 08:09:00 
IPS Engine Test Usage:

2018-03-10 08:09:00     1: Display IPS engine information
2018-03-10 08:09:00     2: Toggle IPS engine enable/disable status
2018-03-10 08:09:00     3: Display restart log
2018-03-10 08:09:00     4: Clear restart log
2018-03-10 08:09:00     5: Toggle bypass status
2018-03-10 08:09:00     6: Submit attack characteristics now
2018-03-10 08:09:00    10: IPS queue length
2018-03-10 08:09:00    11: Clear IPS queue length
2018-03-10 08:09:00    12: IPS L7 socket statistics
2018-03-10 08:09:00    13: IPS session list
2018-03-10 08:09:00    14: IPS NTurbo statistics
2018-03-10 08:09:00    15: IPSA statistics
2018-03-10 08:09:00    16: Display device identification cache
2018-03-10 08:09:00    17: Clear device identification cache
2018-03-10 08:09:00    18: Display session info cache
2018-03-10 08:09:00    19: Clear session info cache
2018-03-10 08:09:00    21: Reload FSA malicious URL database
2018-03-10 08:09:00    22: Reload whitelist URL database
2018-03-10 08:09:00    24: Display Flow AV statistics
2018-03-10 08:09:00    25: Reset Flow AV statistics
2018-03-10 08:09:00    96: Toggle IPS engines watchdog timer
2018-03-10 08:09:00    97: Start all IPS engines
2018-03-10 08:09:00    98: Stop all IPS engines
2018-03-10 08:09:00    99: Restart all IPS engines and monitor

Careful: Using diag test application can have severe impacts. Use it first on a test system, if you do not know what you do.

Packet Sniffer

Every FortiGate got a built in sniffer

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
diag sniffer packet [interface] '[filter]' [verbose level] [count] [tsformat]
diag sniffer packet [interface] '[filter]' [verbose level] [count] [tsformat]
diag sniffer packet [interface] '[filter]' [verbose level] [count] [tsformat]

Interface: ‘any’ or the name of an interface

Filter: corresponds with the ⇒libcap format like tcpdump.

Verbose level:

1packet header
2packet header, IP data
3packet header, IP data, ethernet header
4interface, direction, packet header
5interface, direction, packet header, IP data
6interface, direction, packet header, IP data, ethernet header

 

Count: 0 – unlimited packets or a number of packets

Tsformat: a or l, GMT or localtime. If tsformat is omitted, number of seconds since program start will be shown.

Careful: If a policy is accelerated, only the first and the last packet is shown. It is possible to disable hardware acceleration in a firewall policy.

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
config firewall policy
edit <id>
set auto-asic-offload disable
next
end
config firewall policy edit <id> set auto-asic-offload disable next end
config firewall policy
  edit <id>
    set auto-asic-offload disable
  next
end

Please reactivate acceleration again after your troubleshooting session!

Flow Trace

The sniffer shows you the packets at the interface. diag debug flow shows you, what happens in between. With the flow trace you can find out what exactly blocks the traffic.

Clear possible filters from a previous session

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
diag debug flow filter clear
diag debug flow filter clear
diag debug flow filter clear

Limit the traffic to specific filters

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
diag debug flow filter [filter]
diag debug flow filter [filter]
diag debug flow filter [filter]

Show the function name

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
diag debug flow show function-name enable
diag debug flow show function-name enable
diag debug flow show function-name enable

Start the output on the terminal

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
diag debug enable
diag debug enable
diag debug enable

Start the trace

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
daig debug flow trace start [packet count]
daig debug flow trace start [packet count]
daig debug flow trace start [packet count]

Packet count: 0 – unlimited number of packets, or a specific number of packets.

Stop the trace

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
diag debug flow trace stop
diag debug flow trace stop
diag debug flow trace stop

 

Session Table Troubleshooting

Clear possible filters

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
diag sys session filter clear
diag sys session filter clear
diag sys session filter clear

Limit the output to specific sessions

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
diag sys session filter [filter]
diag sys session filter [filter]
diag sys session filter [filter]

The filter entries are cumulated. You can set multiple filters like address, port, interface.

Show the filters set

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
diag sys session filter
diag sys session filter
diag sys session filter

Show the actual sessions limited by the filter

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
diag sys session list [expect]
diag sys session list [expect]
diag sys session list [expect]

expect will show you temporary sessions created by a session helper.

Clear the sessions limited by the filter

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
diag sys session clear
diag sys session clear
diag sys session clear

Only the sessions that match the filter will be cleared. If no filter is specified, all sessions will be cleared.

Get some session statistics

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
diag sys session stat
diag sys session stat
diag sys session stat

 

The command “diag sys session list” uses the interface indices instead of the names. Using

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
diagnose netlink interface list
diagnose netlink interface list
diagnose netlink interface list

the name and the index of the interface is shown.

 

You can reset the counters of all or of specific firewall polices on the  CLI

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
diag firewall iprope clear 00100004 [<id>]
diag firewall iprope clear 00100004 [<id>]
diag firewall iprope clear 00100004 [<id>]