I’ve recently set up routing for two uplink providers. The advanced routing howto is a good guide for getting routing via several providers running. I’ve observed two points worth mentioning, though.

• When using NAT and several different internal/DMZ networks the source address of packets (in my experiments) doesn’t work reliably for determining the routing table as in the example in the advanced routing howto. It is easier to tag connections with a connmark and copy this mark to the individual packets of the connection. This looks like the following (I’m using the naming from the advanced routing howto):
  
iptables -t mangle -A PREROUTING -i $IF1 -p tcp
-mstate –state NEW -j CONNMARK –set-mark 1/1
iptables -t mangle -A PREROUTING -i $IF2 -p tcp
-mstate –state NEW -j CONNMARK –set-mark 2/2
# mark packets with connection mark
# to be usable in routing
iptables -t mangle -A PREROUTING -p tcp
-j CONNMARK –restore-mark –mask 0x0F

  Then we can reuse that mark to determine the routing table to use:
  
ip rule add fwmark 1 T1
ip rule add fwmark 2 T2

  This can also be reused for routing some services via one and other services via the other interface. Just apply the correct connmark to the connection.
• When using complex routing rules, source validation of the Linux Kernel can get into the way, thanks to Peter Holzer for pointing this out. There are special files in
  
/proc/sys/net/ipv4/conf/*/rp_filter

  one for each interface and some to apply global defaults. Some Linux distributions automagically set these all to “1″. I’m currently turning all of them to 0 and routing is working now. You know that you might have this problem when packets from an external interface are dropped after PREROUTING and are never seen in the FORWARD chain of iptables. The Linux-Kernel documentation in Documentation/networking/ip-sysctl.txt say this about rp_filter:

  rp_filter – BOOLEAN

  1 – do source validation by reversed path, as specified in RFC1812 Recommended option for single homed hosts and stub network routers. Could cause troubles for complicated (not loop free) networks running a slow unreliable protocol (sort of RIP), or using static routes.

  0 – No source validation.

  conf/all/rp_filter must also be set to TRUE to do source validation on the interface