- Enterasys Security Router User's Guide

Ensuring VPN Security with IPSec/IKE/GRE
XSR User’s Guide 14-3
Since IPSec is the standard security protocol, the XSR can establish IPSec connections with third-
node devices including routers as well as PCs. An IPSec tunnel basically acts as the network layer
protecting all data packets that pass through, regardless of the application or device.
The XSR makes it possible to control the type of traffic sent over a VPN by allowing you to define
group-based filters (Access Control Lists) which control IP address and protocol/port services
allowed through the tunnel. An IPSec-based VPN also permits you to define a list of specific
networks and applications to which traffic can be passed.
Central to IPSec is the concept of the Security Association (SA). A primary role of IKE is to
establish and maintain SAs by its use of the IP Authentication Header (AH) or Encapsulating
Security Payload (ESP). An SA is a uni-directional logical connection between two communicating
IP endpoints that applies security to the traffic carried by it using the AH or ESP features listed in
a transform-set (described below).
The endpoint of an SA can be an IP client (host) or IP security gateway. Providing security for the
more typical scenario of bi-directional communication between two endpoints requires the
establishment of two SAs (one in each direction). An SA is uniquely identified by the following:
• A 32-bit identifier of the connection
• The IP destination address
• A security protocol identifier (AH or ESP)
The IP Authentication Header (AH), defined in RFC-2402, checks for data integrity, data origin
authentication, and replay on IP packets using HMAC with MD5 (RFC-2403), or HMAC with
SHA-1 (RFC-2404).
The IP Encapsulating Security Payload (ESP), described in RFC-2406, performs confidentiality in
addition to integrity and authentication checks, but it does not check the integrity of the IP header.
As in AH, ESP uses HMAC with MD5 or SHA-1 authentication (RFC-2403/2404); privacy is
provided using DES-CBC (RFC-2405), 3DES or AES encryption.
Two types of modes are defined in IPSec, tunnel and transport. At the packet level, transport mode
leaves the original IP header intact and inserts AH or ESP headers after the original IP header as
shown in Figure 14-1 below.
Figure 14-1 Transport Mode Processing
Tunnel mode adds a new IP header and encapsulates the original IP packet as shown in
Figure 14-2.
Original packet IP
data
After processing
AH/ESP
Can be encrypted
IP
data