In one of my work environments we use Fortigate firewalls. With a complex rule-set, including multiple VDOMs, there are times where we need to figure out why some traffic (source) is not reaching its destination.

We had such a case recently, and I wrote this up for documentation, sharing is caring :). I’ve changed the IPs, vlans, vlinks, and VDOMs involved to obfuscate the data, but it should still prove to be a good example for you. In this example I assume you only know how to SSH into your firewall and that you know which VDOM the source or destination is in which you want to troubleshoot…

Problem – Determine why 192.168.1.10, is unable to reach 172.16.1.10 on TCP 25 as Fortigate and server firewall rules are configured properly to allow this traffic.

Troubleshooting – Setup debug mode and then reproduce the issue.

=============================================================================
### Start an SSH session ###
=============================================================================

mycomputer:~ Gabe$ ssh gabe.beaver@noneofyourbiz.net
gabe.beaver@noneofyourbiz.net’s password:

=============================================================================
### Here I show you the options – most always going to config vdom ###
### Notice these section match the web UI in separation ###
=============================================================================

FORTIGATE01 #
config Configure object.
get Get dynamic and system information.
show Show configuration.
exit Exit the CLI.

FORTIGATE01 # config
global config global
vdom config vdom

FORTIGATE01 # config vdom

=============================================================================
### Next you need to specify the specific VDOM you want to work with ###
### There is an offset root is ID:0; so VDOM0 is actually ID:1 ###
=============================================================================

FORTIGATE01 (vdom) # edit VDOM0
current vf=VDOM0:1

=============================================================================
### We want to start with a clear plate: ###
=============================================================================

FORTIGATE01 (VDOM0) # diag debug flow filter clear

FORTIGATE01 (VDOM0) # diag debug reset

FORTIGATE01 (VDOM0) # diag debug disable

=============================================================================
### Make sure to enable console trace message and function ###
=============================================================================

FORTIGATE01 (VDOM0) # diag debug flow show console enable
show trace messages on console

FORTIGATE01 (VDOM0) # diag debug flow show function-name enable
show function name

=============================================================================
### Now we enter the parameters to trace traffic, in this example I am ###
### Looking for traffic from my VPN PPP adapter, SMTP traffic into VDOM0 ###
### Where X is the number of lines, I usually say 100 ###
=============================================================================

FORTIGATE01 (VDOM0) # diag debug flow filter dport 25

FORTIGATE01 (VDOM0) # diag debug flow filter addr 192.168.1.10

FORTIGATE01 (VDOM0) # diag debug enable

FORTIGATE01 (VDOM0) # diag debug flow trace start X

=============================================================================
### Now begin your testing – you should see console data from test ###
### If you do not see data, you are too restrictive or data is not ###
### Reaching the VDOM or firewall, verify routes and test data ###
### Below is sample data from specific source, saddr 192.168.1.10:25 ###
=============================================================================

id=13 trace_id=4407 func=resolve_ip_tuple_fast line=4299 msg=”vd-VDOM0 received a packet(proto=6, 192.168.1.10:53023->172.16.1.10:25) from vlan20.”
id=13 trace_id=4407 func=init_ip_session_common line=4430 msg=”allocate a new session-c5e8064b”
id=13 trace_id=4407 func=vf_ip4_route_input line=1603 msg=”find a route: gw-10.20.1.1 via vlink11″
id=13 trace_id=4407 func=__iprope_tree_check line=534 msg=”use addr/intf hash, len=10″
id=13 trace_id=4407 func=fw_forward_handler line=664 msg=”Allowed by Policy-7:”
id=13 trace_id=4408 func=resolve_ip_tuple_fast line=4299 msg=”vd-root received a packet(proto=6, 192.168.1.10:53023->172.16.1.10:25) from vlink10.”
id=13 trace_id=4408 func=init_ip_session_common line=4430 msg=”allocate a new session-c5e8064c”
id=13 trace_id=4408 func=vf_ip4_route_input line=1603 msg=”find a route: gw-10.10.1.1 via port0″
id=13 trace_id=4408 func=__iprope_tree_check line=534 msg=”use addr/intf hash, len=4″
id=13 trace_id=4408 func=fw_forward_handler line=534 msg=”Denied by forward policy check”
id=13 trace_id=4409 func=resolve_ip_tuple_fast line=4299 msg=”vd-VDOM0 received a packet(proto=6, 192.168.1.10:53023->172.16.1.10:25) from vlan20.”
id=13 trace_id=4409 func=resolve_ip_tuple_fast line=4335 msg=”Find an existing session, id-c5e8064b, original direction”
id=13 trace_id=4409 func=ipv4_fast_cb line=50 msg=”enter fast path”
id=13 trace_id=4410 func=resolve_ip_tuple_fast line=4299 msg=”vd-root received a packet(proto=6, 192.168.1.10:53023->172.16.1.10:25) from vlink10.”
id=13 trace_id=4410 func=init_ip_session_common line=4430 msg=”allocate a new session-c5e818ae”
id=13 trace_id=4410 func=vf_ip4_route_input line=1603 msg=”find a route: gw-10.10.1.1 via port0″
id=13 trace_id=4410 func=__iprope_tree_check line=534 msg=”use addr/intf hash, len=4″
id=13 trace_id=4410 func=fw_forward_handler line=534 msg=”Denied by forward policy check”
id=13 trace_id=4411 func=resolve_ip_tuple_fast line=4299 msg=”vd-VDOM0 received a packet(proto=6, 192.168.1.10:53023->172.16.1.10:25) from vlan20.”
id=13 trace_id=4411 func=resolve_ip_tuple_fast line=4335 msg=”Find an existing session, id-c5e8064b, original direction”
id=13 trace_id=4411 func=ipv4_fast_cb line=50 msg=”enter fast path”
id=13 trace_id=4412 func=resolve_ip_tuple_fast line=4299 msg=”vd-root received a packet(proto=6, 192.168.1.10:53023->172.16.1.10:25) from vlink10.”
id=13 trace_id=4412 func=init_ip_session_common line=4430 msg=”allocate a new session-c5e831ff”
id=13 trace_id=4412 func=vf_ip4_route_input line=1603 msg=”find a route: gw-10.10.1.1 via port0″
id=13 trace_id=4412 func=__iprope_tree_check line=534 msg=”use addr/intf hash, len=4″
id=13 trace_id=4412 func=fw_forward_handler line=534 msg=”Denied by forward policy check”

=============================================================================
### When you finish testing, disable debugging and clear the filters ###
=============================================================================

FORTIGATE01 (VDOM0) # diag debug reset

FORTIGATE01 (VDOM0) # diag debug disable

FORTIGATE01 (VDOM0) # diag debug flow filter clear

FORTIGATE01 (VDOM0) # end

FORTIGATE01 #
config Configure object.
get Get dynamic and system information.
show Show configuration.
exit Exit the CLI.

FORTIGATE01 # exit
Connection to noneofyourbiz.net closed.

With the debug we were able to find “Denied by forward policy check”.

http://kb.fortinet.com/kb/documentLink.do?externalID=FD31702

In our case we worked with Fortinet support and the issue was a problem with the NAT configuration for the subnet. Support resolved it almost immediately thanks to our ability to provide them a debug.

Fortigate Firewall Debug
Tagged on:     
%d bloggers like this: