Recently we observed a strange issue while building a site to site VPN tunnel between a Cisco ASA [9.1( 5) ] and Palo Alto Next Generation firewall.(PAN-OS 7.0.9) It was observed always phase 1 part of tunnel established successfully with peer however phase 2 failed to come up.
Always we were seeing issues with encapsulation, the packets sent were never encapsulated, however the packets received from remote peers were de capsulated, this means the ASA was not encrypting the data. The below logs demonstrates the error,
#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
#pkts decaps: 74, #pkts decrypt: 0, #pkts verify: 0
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts comp failed: 0, #pkts decomp failed: 0
#pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
#TFC rcvd: 0, #TFC sent: 0
#Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
#send errors: 0, #recv errors: 74
We did a through troubleshooting and we ensured the following ay both ends of the firewalls
- Ensure both the firewalls have an appropriate route for the interesting traffic / proxy id
- Ensured the ACL / Policies are matched
- Ensured NAT configuration is done properly as were using source based NATTing at both the end.
- Ensured proper debugging is done at both ends,
- Involved vendors to see what the issue was
Upon through troubleshooting it was discovered the ASA was hitting a bug CSCuo58411
This bug basically creates duplicate entries in tunnel manger and one could see from the below logs
IPSEC(crypto_map_check)-3: Checking crypto map outside_map 10: matched.
Jul 24 08:20:49 [IKE COMMON DEBUG]Duplicate entry already in Tunnel ManagerIPSEC(crypto_map_check)-3: Looking for crypto map matching 5-tuple: Prot=1, saddr=10.0.100.2, sport=45638, daddr=192.168.120.100, dport=45638
IPSEC(crypto_map_check)-3: Checking crypto map outside_map 10: matched.
Jul 24 08:20:50 [IKE COMMON DEBUG]Duplicate entry already in Tunnel Manager
The work around worked for us to fix this issue, was to upgrade the ASA from 9.1(5) to 9.5(3)6 . The other recommended work around to fix this issue is
Issue “debug menu ike-common 10” to remove the stale IKEv2 entries (this will delete all current IKEv2 connections)