How to overcome Asymmetric routing in ASA

Introduction

This document provides the basic procedures for identifying, understanding, and mitigating asymmetric routing issues in networks that are protected by the Cisco Adaptive Security Appliance (ASA).

The ASA protects the network by denying traffic unless it can inspect both sides of the flow. To allow the ASA to continue maintaining the security of the network, the correct and ideal solution for asymmetric routing is to resolve the problem that causes the asymmetry elsewhere in the network. However, this document is designed to help identify the problem and provide workarounds until the asymmetry can be fixed.

The term asymmetric routing refers to a packet or connection flow that takes different paths through the network in the forward and reverse directions. For example, a packet may leave the internal network destined for the Internet through ASA1, but the server's response to that packet may return from the Internet and reach the network through ASA2.

Because the ASA keeps track of the active connections passing through the firewall, packets that arrive and that are not part of an existing connection or explicitly permitted by an access list will be dropped when asymmetric routing occurs. In the example from the previous paragraph, the first packet leaving for the Internet through ASA1 will cause ASA1 to build an entry in its connection table. Once this entry is built, all return packets from the Internet that are specific to this connection will be allowed through until the connection is removed from the table. Since ASA2 never saw the 1st packet, no connection is built and thus it is not expecting the response packet from the Internet, which causes the packet to be dropped without an explicit rule to allow it. The following sections in this document will provide alternatives to prevent these packets from being dropped by the ASA.

Examples

Example A - Normal Flow

In order to understand what asymmetric routing looks like from a  diagnostic point of view, it is important to first understand how a  typical packet flow will look. Consider the following simple topology:

When the client wants to send HTTP traffic to the web server at www.cisco.com, it will start a TCP connection by sending its first packet to its default gateway, which is the ASA. The ASA will then route the packet out to the Internet so that it can eventually reach the web server. When the web server receives the request, it will send its response back toward the client, which will eventually arrive at the ASA. Since, having seen the initial request, the ASA knows about this connection, it will allow the response through and forward it back to the client.

Using the packet capture feature of the ASA, it is easy to see that the routing in this topology is symmetric. In the example below, packet #1 arrives on the inside interface of the ASA and leaves on the outside interface of the ASA. Likewise, the response packet, packet #2, returns on the outside interface and is forwarded on through the inside interface. Seeing bidirectional communication on both of the ASA's interfaces is a good sign that everything is working as expected.

ASA inside capture:

1: 11:21:22.385172 802.1Q vlan#301 P0 192.168.1.1.39035 > 198.133.219.25.80: S 1714606600:1714606600(0) win 5840 <mss 1460,sackOK,timestamp 2359218074 0,nop,wscale 7>
2: 11:21:22.461661 802.1Q vlan#301 P0 198.133.219.25.80 > 192.168.1.1.39035: S 830304317:830304317(0) ack 1714606601 win 16384 <mss 1380>

ASA outside capture:

1: 11:21:22.385706 802.1Q vlan#10 P0 192.168.1.1.39035 > 198.133.219.25.80: S 3731327602:3731327602(0) win 5840 <mss 1380,sackOK,timestamp 2359218074 0,nop,wscale 7>
2: 11:21:22.461615 802.1Q vlan#10 P0 198.133.219.25.80 > 192.168.1.1.39035: S 3873337058:3873337058(0) ack 3731327603 win 16384 <mss 1380>

Example B - Multiple Internet Paths

One common example of the asymmetric routing problem occurs when a network has more than one path to the Internet. Depending on the way routing is configured at the network edge or by the ISP, the initial packet may leave through one path but the response returns through a different path as shown in the example below.

ASA inside capture:

1: 11:31:18.914945 802.1Q vlan#301 P0 192.168.1.1.58458 > 198.133.219.25.80: S 2331481362:2331481362(0) win 5840 <mss 1460,sackOK,timestamp 2359730964 0,nop,wscale 7>

2: 11:31:18.991678 802.1Q vlan#301 P0 192.168.1.1.58458 > 198.133.219.25.80: . ack 712296789 win 5840

ASA outside capture:

1: 11:31:18.915434 802.1Q vlan#10 P0 192.168.1.1.58458 > 198.133.219.25.80: S 126872520:126872520(0) win 5840 <mss 1380,sackOK,timestamp 2359730964 0,nop,wscale 7>

Client capture:

10:45:58.734881 IP 192.168.1.1.58458 > 198.133.219.25.80: S 2331481362:2331481362(0) win 5840 <mss 1460,sackOK,timestamp 2359730964 0,nop,wscale 7>
10:45:51.787530 IP 198.133.219.25.80 > 192.168.1.1.58458: S 712296788:712296788(0) ack 126872521 win 16384 <mss 1380>

In this example, the client sends packet #1 to its default gateway, which is the ASA. It is routed through the Internet where it reaches the web server. When the web server responds, routing takes the reply back through the Internet where it arrives on RouterA, who forwards it to RouterB, who sends it directly to the client. Note that because the ASA randomizes the TCP sequence and acknowledgement numbers by default, the server will respond with a SYN ACK packet that has an ACK value of 126872521 (the andomized initial sequence number  from the SYN packet + 1). Because the sequence and acknowledgement numbers are now out of sync, the client will not be able to establish the connection because the client expects the ACK value to be 2331481363.

Example C - Active/Active Failover

Another issue with asymmetric routing can arise when 2 ASAs are configured in Active/Active failover mode.

In this example, ASA1 and ASA2 are connected together in an Active/Active failover mode. There are 2 contexts configured on each unit, ContextA and ContextB. ContextA is active on ASA1 and standby on ASA2, while ContextB is active on ASA2 and standby on ASA1.

When a packet leaves the client destined for www.cisco.com, it arrives at its default gateway of the inside interface of ContextA. ContextA then forwards it on toward the Internet where it reaches the server. Then, the server sends its reply back toward the client. When the response arrives at the Router, it is routed to the outside interface of ContextB, rather than ContextA where the original request came from. Because ContextB never saw the initial request and does not have an existing entry its connection table, the packet is dropped and thus the client is not able to establish a connection with the server. Packet captures on the firewall contexts will clearly illustrate the asymmetry, while the client will simply see no response from the server.

ContextA inside capture:

1: 12:10:44.671657 802.1Q vlan#301 P0 192.168.1.1.40400 > 198.133.219.25.80: S 518915629:518915629(0) win 5840 <mss 1460,sackOK,timestamp 2361776640 0,nop,wscale 7>

2: 12:10:44.671657 802.1Q vlan#301 P0 192.168.1.1.40400 > 198.133.219.25.80: S 518915629:518915629(0) win 5840 <mss 1460,sackOK,timestamp 2361776640 0,nop,wscale 7>

ContextA outside capture:

1: 12:10:44.672145 802.1Q vlan#10 P0 192.168.1.1.40400 > 198.133.219.25.80: S 654525205:654525205(0) win 5840 <mss 1380,sackOK,timestamp 2361776640 0,nop,wscale 7>

2: 12:10:44.672145 802.1Q vlan#10 P0 192.168.1.1.40400 > 198.133.219.25.80: S 654525205:654525205(0) win 5840 <mss 1380,sackOK,timestamp 2361776640 0,nop,wscale 7>

ContextB outside capture:

1: 12:10:44.747931 802.1Q vlan#10 P0 198.133.219.25.80 > 192.168.1.1.40400: S 807750905:807750905(0) ack 654525206 win 16384 <mss 1380>

2: 12:10:44.747931 802.1Q vlan#10 P0 198.133.219.25.80 > 
192.168.1.1.40400: S 807750905:807750905(0) ack 654525206 win 16384 
<mss 1380>

Client capture:

11:25:18.201065 IP 192.168.1.1.40400 > 198.133.219.25.80: S 518915629:518915629(0) win 5840 <mss 1460,sackOK,timestamp 2361776640 0,nop,wscale 7>
11:25:18.201065 IP 192.168.1.1.40400 > 198.133.219.25.80: S 518915629:518915629(0) win 5840 <mss 1460,sackOK,timestamp 2361776640 0,nop,wscale 7>

Diagnosing/Troubleshooting

Packet Captures

Packet captures on the ASA can help easily identify asymmetric routing issues. For example, if you see only one-way traffic (i.e. only client to server traffic), asymmetric routing may be present.

It is important to setup the captures on the ASA to be both bidirectional (i.e. client to server, and server to client) and simultaneous on both sides of the firewall (i.e. both the inside and outside interfaces). For more information on using the ASA's packet capture utility, please see the following document:

https://supportforums.cisco.com/docs/DOC-1222

Syslogs

Syslogs are another useful indication of asymmetric routing. For example, if asymmetric routing exists and the ASA only sees one side of a given connection, a successful connection will never be established and thus many packets will be denied for this reason. In this example, the ASA will log the following message at the informational level:

%ASA-6-106015: Deny TCP (no connection) from 198.133.219.25/80 to 192.168.1.1/40400 flags SYN ACK on interface outside

Mitigation

The following sections outline the available mitigation techniques when the ASA drops packets due to asymmetric routing.

Routing configuration

One of the best, and perhaps most obvious, solutions is to simply fix the routing configuration in the packet's path so that the ASA sees both sides of the connection. In Example B above, RouterA's routing table should send all packets destined for the client to the ASA's outside interface, rather than sending them to RouterB.

Unfortunately, this solution is not always available to network administrators. In some cases, the asymmetry could be caused by devices outside the administrator's control, such as routers managed by an ISP or 3rd party vendor. When this occurs, it is still usually best to contact the responsible organization and work with them to resolve the issue, but the remaining mitigation techniques on the ASA may also be used.

ASR Groups

For situations similar to Example C above, ASA's that are configured for Active/Active Stateful Failover can put their interfaces into ASR groups:

ASA/ContextA(config)# interface Vlan 101

ASA/ContextA(config-if)# nameif outside

ASA/ContextA(config-if)# asr-group 1

ASA/ContextB(config)# interface Vlan201

ASA/ContextB(config-if)# nameif outside

ASA/ContextB(config-if)# asr-group 1

When an interface that is not in an ASR group receives a packet that it does not have an established connection for, it will normally drop it. However, when an interface is in an ASR group, the ASA checks the connection information for all other interfaces in the same group. If the ASA finds another context that has connection information that would match the received packet, the Layer 2 header information is rewritten and the packet is redirected to the appropriate context for processing.

TCP State Bypass

Specifically for TCP-based connections, disabling stateful TCP checks can help mitigate asymmetric routing. When TCP state checks are disabled, the ASA can allow packets in a TCP connection even if the ASA didn't see the entire TCP 3-way handshake. This feature is called TCP State Bypass (introduced in ASA 8.2).

ASA(config)# access-list tcp_bypass extended permit tcp 192.168.1.0 255.255.255.0 any
ASA(config)# class-map tcp_bypass
ASA(config-cmap)# match access-list tcp_bypass
ASA(config-cmap)# policy-map tcp_bypass_policy
ASA(config-pmap)# class tcp_bypass
ASA(config-pmap-c)# set connection advanced-options tcp-state-bypass
ASA(config-pmap-c)# set connection timeout idle 0:10:00
ASA(config-pmap-c)# service-policy tcp_bypass_policy inside

For ASA's running software versions prior to 8.2, similar functionality can be enabled using the 'norandomseq nailed' option at the end of a static statement. For more information on this configuration, see the following document:

http://www.cisco.com/en/US/docs/security/asa/asa80/command/reference/s8.html#wp1414075

Please note that using TCP State Bypass (or the 'nailed' option) will also disable all TCP-based security checks and application inspection. Also, until the fix for enhancement CSCtk84429 is implemented, it is recommended that when TCP State Bypass is enabled an explicitly lower idle timeout is configured to prevent the ASA from reaching its resource limit in high connection environments. This is because the ASA no longer removes a TCP connection from its table based on a TCP FIN teardown (since TCP state checking no longer occurs). As a result, the ASA will have to wait until the global idle timer expires (1 hour by default) before the connection is removed.

Summary

The ASA protects the network by blocking traffic flows unless it is able to inspect the entire (both sides) of the connection. Using tools such as packet captures and syslogs, the ASA can also be a useful troubleshooting tool in identifying asymmetric routing problems. Although the ASA keeps the network secure by blocking this traffic, the above workarounds can be used to allow traffic through the firewall until the problem can be solved permanently.