Decrypting IPSec Protocols (ISAKMP and ESP) With Wireshark

4 min readOct 24, 2020

IPSec is a group of protocols that help us to encrypt traffic between two devices. Before transporting data between two devices, a tunnel is created with ISAKMP (Internet Security Association and Key Management Protocol) through which parameters for ESP (Encapsulating Security Payload) are negotiated. IPsec ISAKMP negotiations are made in two phases, Main Mode (Phase1) and Quick Mode (Phase2). Assuming that we would like to send data from Site2 to Site1 with IPSec encryption.

The first thing we need to do is telling Site1 (Gateway) that we want to use ESP, aes256 and sha2–256 for encryption and authentication. Since we can’t send that very sensitive information over a clear text network connection, ISAKMP Main Mode kicks in for that purpose. With Main Mode, an encrypted tunnel is created, then in Quick Mode the parameters for ESP (aes256, sha2–256 …) are negotiated. Wireshark output shows the packets belong to Main Mode and Quick Mode as below.

We will decrypt ISAKMP phases (Main Mode and Quick Mode) as well as ESP in this writing.
Unfortunately with Cisco routers, we are not able to acquire information like encryption and authentication keys. For that purpose, Strongswan is a nice open source IPSec implementation which provides everything we need. As seen in the network topology, an IPSec tunnel is created between Strongswan and Cisco Router (Gateway). We will install Strongswan on Ubuntu with minimum configuration.

Step 1 Installing Strongswan

sudo apt install strongswan

Step 2 Configuring Strongswan

We configure Strongswan for ISAKMP and ESP parameters, adding directives into /etc/ipsec.conf file like below. Beside that we need to enable debugging as well. Since Initiator COOKIE (Initiator SPI) and encryption key are two pieces of information required for decrypting ISAKMP tunnel, we will extract them from debug logs.

After basic configuration and enabling debugging, we need to set the log file path, configuring /etc/strongswan.conf like below. All debug logs, including ISAKMP encryption key will be stored in the charon.log file.

We will keep the configuration as simple as possible so we will use preshared authentication.
Preshared key is stored in a different file, we configure /etc/ipsec.secrets file like below.

Step 3 Configuring networks and routes

We set some basic IP address assigning and routing configurations on Ubuntu client (Strongswan), considering the network topology.
sudo ifconfig ens41 192.168.10.1 netmask 255.255.255.0 up
sudo ifconfig ens39 192.168.30.1 netmask 255.255.255.0 up
sudo route add -net 1.1.1.1 netmask 255.255.255.255 gw 192.168.10.2
sudo route add -net 2.2.2.2 netmask 255.255.255.255 gw 192.168.30.2
sudo echo 1 > /proc/sys/net/ipv4/ip_forward

Step 4 Acquiring ICOOKIE (Initiator SPI) and the encryption key from log file (charon.log) for ISAKMP

First we restart the ipsec service with applying “sudo ipsec restart” command then filter logs like below.

Obtaining encryption key

Since we got the information we need, now it is time to feed Wireshark with that information from Edit -> Preferences -> Protocols -> ISAKMP -> IKEv1 Decryption Table: as shown below.

Feeding Wireshark with ICOOKIE and encryption key

After clicking ok button, we will be able to see decrypted traffic and details of the packets. Before and after decryption of ISAKMP (Quick and Main Modes) output as below. We can see SA(security association), Diffie Helman exchange and so on.

Step 5 Acquiring authentication, encryption keys and algorithms for ESP

Unlike ISAKMP ICOOKIE and encryption key, the keys used for ESP tunnel are not stored in the charon.log file. We will obtain that information with applying “ip xfrm state” command as below. SPI, authentication and encryption keys are different for each direction.