Decrypting IPSec Protocols (ISAKMP and ESP) With Wireshark

Celal Dogan
4 min readOct 24, 2020

--

IPSec is a group of protocols that help us to encrypt traffic between two devices. Before transporting data between two devices, a tunnel is created with ISAKMP (Internet Security Association and Key Management Protocol) through which parameters for ESP (Encapsulating Security Payload) are negotiated. IPsec ISAKMP negotiations are made in two phases, Main Mode (Phase1) and Quick Mode (Phase2). Assuming that we would like to send data from Site2 to Site1 with IPSec encryption.

The network topology

The first thing we need to do is telling Site1 (Gateway) that we want to use ESP, aes256 and sha2–256 for encryption and authentication. Since we can’t send that very sensitive information over a clear text network connection, ISAKMP Main Mode kicks in for that purpose. With Main Mode, an encrypted tunnel is created, then in Quick Mode the parameters for ESP (aes256, sha2–256 …) are negotiated. Wireshark output shows the packets belong to Main Mode and Quick Mode as below.

ISAKMP Main and Quick Modes

We will decrypt ISAKMP phases (Main Mode and Quick Mode) as well as ESP in this writing.
Unfortunately with Cisco routers, we are not able to acquire information like encryption and authentication keys. For that purpose, Strongswan is a nice open source IPSec implementation which provides everything we need. As seen in the network topology, an IPSec tunnel is created between Strongswan and Cisco Router (Gateway). We will install Strongswan on Ubuntu with minimum configuration.

Step 1 Installing Strongswan

sudo apt install strongswan

Step 2 Configuring Strongswan

We configure Strongswan for ISAKMP and ESP parameters, adding directives into /etc/ipsec.conf file like below. Beside that we need to enable debugging as well. Since Initiator COOKIE (Initiator SPI) and encryption key are two pieces of information required for decrypting ISAKMP tunnel, we will extract them from debug logs.

Simple IPSec configuration

After basic configuration and enabling debugging, we need to set the log file path, configuring /etc/strongswan.conf like below. All debug logs, including ISAKMP encryption key will be stored in the charon.log file.

Path for log file

We will keep the configuration as simple as possible so we will use preshared authentication.
Preshared key is stored in a different file, we configure /etc/ipsec.secrets file like below.

Preshared key configuration

Step 3 Configuring networks and routes

We set some basic IP address assigning and routing configurations on Ubuntu client (Strongswan), considering the network topology.
sudo ifconfig ens41 192.168.10.1 netmask 255.255.255.0 up
sudo ifconfig ens39 192.168.30.1 netmask 255.255.255.0 up
sudo route add -net 1.1.1.1 netmask 255.255.255.255 gw 192.168.10.2
sudo route add -net 2.2.2.2 netmask 255.255.255.255 gw 192.168.30.2
sudo echo 1 > /proc/sys/net/ipv4/ip_forward

Step 4 Acquiring ICOOKIE (Initiator SPI) and the encryption key from log file (charon.log) for ISAKMP

First we restart the ipsec service with applying “sudo ipsec restart” command then filter logs like below.

Obtaining ICOOKIE
Obtaining encryption key

Since we got the information we need, now it is time to feed Wireshark with that information from Edit -> Preferences -> Protocols -> ISAKMP -> IKEv1 Decryption Table: as shown below.

Feeding Wireshark with ICOOKIE and encryption key

After clicking ok button, we will be able to see decrypted traffic and details of the packets. Before and after decryption of ISAKMP (Quick and Main Modes) output as below. We can see SA(security association), Diffie Helman exchange and so on.

Before decrypting ISAKMP
After decrypting ISAKMP

Step 5 Acquiring authentication, encryption keys and algorithms for ESP

Unlike ISAKMP ICOOKIE and encryption key, the keys used for ESP tunnel are not stored in the charon.log file. We will obtain that information with applying “ip xfrm state” command as below. SPI, authentication and encryption keys are different for each direction.

Obtaining encryption and authentication keys for ESP

Last but not least, we will provide that information to Wireshark through Edit -> Preferences -> Protocols -> ESP menu like below.

Feeding Wireshark with keys and algorithms

After filling the menu correctly, Wireshark will show us the payload in clear text. Wireshark output before and after decryption is shown below.

Before decrypting ESP packets
After decrypting ESP packets

References

IPSEC & IKE

How to decrypt IPSec Packets (ISAKMP and ESP) — Wireshark

Decrypt RDP Traffic with Wireshark and Frida

Wireshark: Decrypt SSL/TLS Practical Examples

The Internet Key Exchange (IKE)

How to Set Up IPsec-based VPN with Strongswan on Debian and Ubuntu

Test ikev2/rw-cert

--

--

No responses yet