Set of Agreements
When two devices want to communicate securely, even though their connection may not be secure, the two devices need to establish a set of agreements according to IPSec that both of them abide by so that their secure communication can be sent and received without any trouble. The devices must first agree on a set of security protocols to use so that each device can communicate to each one another in a form which can be understood. Next, the encryption algorithm that is going to be used to encrypt the data must be agreed upon so that each device can decrypt the other’s communication. The device cannot decrypt the other’s encrypted communication without the exchange of the others device keys. Finally, the devices need to always adhere to the set of agreements established so that the secure communication continues without error.
There are two main “protocols” of IPSec which take on the task of encrypting the data and ensuring its security. These are referred to as IPSec Authentication Header, or AH, and the Encapsulating Security Payload, or ESP. They are pseudo-protocols in the sense that they rely on many other services to be able to function correctly. The AH and ESP “protocols” can be used together or individually based upon the requirements desired for the connection between two devices.
The Authentication Header gives authenticating services to IPSec as well as data integrity and protection against replay attacks. With AH, the receiver of a message is able to confirm sending the message says sent the message really is in fact the person who sent the message. It also protects against any insertion, deletion, or modification of the message contents by any unknown device or opponent. AH is able to provide these features by adding a cryptographic checksum to packets.
If one wishes for the data to be confidential, Encapsulated Security Payload takes on the task of encrypting the data so that it cannot be easily read by anyone who may be sniffing packets on the network. ESP normally uses CBC-mode encryption for data confidentially and without data integrity protection, the data is vulnerable to message modification attacks.
Modes of Operation for AH and ESP
AH and ESP utilized two modes of operation, tunnel and transport modes. VPNs mostly use tunnel mode to provide protection between the two devices, such as two servers. Transport mode is used to encrypt data within tunnel mode and it provides security for end-to-end communication. AH and ESP can each be configured to operate in tunnel or transport mode independently.
Other IPSec Services and Protocols
The Authentication Header and the Encapsulated Security Payload make use of other services and protocols to conform their generic structure into a more detailed form of an application. AH and ESP do not directly specify an encryption or hashing algorithm, so they may allow the flexibility of functioning with more than one choice. Two of the most widely used hashing algorithms for AH and ESP are Message Digest 5, or MD5, and Secure Hash Algorithm, or SHA-1.
The security agreements held between two devices, as mentioned earlier, must be stored in some structure so that the agreements can be accessed at any time and can be transmitted to other devices. The structures that hold the security agreements in IPSec are known as security associations and security policies. Since IPSec’s Encapsulated Security Payload encrypts the data with a secret key, it may send the confidential data to the other device. There needs to be some sort of mechanism though which exchanges keys between devices so that they may decrypt each other’s transmission. This feature in IPSec is known as the Internet Key Exchange, or IKE, which deals with the exchange of keys and security associations between devices so that the security agreements and key distribution are kept up to date.