Extracting Secrets from Packet Captures (A CMIYC2024 Story)

"Interest is the most important thing in life; happiness is temporary, but interest is continuous."

- Georgia O'Keeffe

The focus of this blog entry will be on tools and scripts to analyze packet captures. This is the result of falling down a rabbit hole when writing the previous tutorial on the CMIYC 2024 WIFI cracking challenge: [Link]. In that writeup I realized I hadn't been keeping up on the state of automated tooling to help extract secrets and interesting data from packet captures. So I asked for tips and suggestions on what I could use. And you all responded! This is another reason why these blog posts are really beneficial to me. I learn so much writing them, so thank you!

As a disclaimer, while I will be using the CMIYC2024 dataset to explore using some of these tools, these tools are not really suited for password cracking competitions. For short competitions, you are better off performing manual analysis of the data. As a spoiler, none of the tools I looked at identified many of the CMIYC2024 secrets out of the box. I needed to look through the packet captures myself to figure out those clues. Instead as you read through this blog entry, I'd like you to consider analyzing real world data which can take up Gigabytes of disk space. That's where these tools can be helpful.

Important Links, Tools, and References for this Post:\

To start things off, I wanted to create a control set of myself manually analyzing a packet capture file. That way I knew what to expect when I start trying to process the same data with automated tools. For this blog post I'm going to use CMIYC2024 contest's packet capture challenge since it includes different types of secrets, and quite honestly this provides a good incentive to spend more time trying to solve the other CMIYC2024 challenges as well.

To reiterate from my previous post on how I manually analyzed the data, I opened up the packet capture in Wireshark, decrypted the WPA1 encrypted traffic, and then manually walked through the different TCP streams using the Wireshark filters.

Note: There was UDP traffic as well, (which you can walk through in a similar fashion). Looking through the UDP traffic there wasn't anything that I saw that was interesting to this contest. It was mostly "plumbing" traffic like DHCP address leases. When doing this yourself though, make sure you don't forget to look at UDP traffic since there can often be some really interesting findings in it.

As far as content went, there were four main types of sessions that Korelogic included in the contest:

FTP Sessions (Covered in the previous blog post)

Interesting Secrets:

Username/password to log into the FTP server
Each session downloaded a passwords file containing the same three passwords.

Number of Sessions:

10 FTP protocol (Packet capture missed one of the FTP protocol sessions)
11 FTP Data

HTTP Sessions (Covered in the previous blog post)

Interesting Secrets:

HTTP Basic Auth: Username/password
Each session displayed a "password vault" which was just a plain HTML page containing the same four passwords

Number of Sessions:

Telnet

Interesting Secrets:

Username/password to log into the telnet server

Number of Sessions:

Email

Interesting Secrets:

This was a password reset e-mail for the *WORST* password reset service. So it included the original username/password combo and the new username/password combo as text inside the e-mail.

Number of Sessions:

I talked about most of these session types in the previous blog post, but I wanted to highlight the password reset e-mails that Korelogic created for this contest:

Getting a user's previous password as well as what the password reset service reset it to will probably be helpful for cracking other passwords during this competition! You may also notice that the new password matches the HTTP Basic Auth password highlighted in the previous blog post. So Korelogic is working to try and tell a consistent story here.

To actually save the data, I copy/pasted interesting fields from Wireshark into an Excel Workbook.

In data analysis there is a general idea/term called "Time to Excel" that asks how long it takes to import data into Excel and display it in a way you can understand it. The idea behind it is that Excel is the best general purpose data analysis tool out there. The competition isn't even close, (Kind of like how ThatOnePasswordWas40Passwords dominated the CMIYC2024 Street contest). I would not have graduated with my doctorate without Excel. So if you are looking to use automation or a different data analysis platform/tool you need to compare them to the "Time to Excel" and see if they save you any time/work. If they don't, then you are better off sticking with large Excel worksheets. For this effort here was my manual "Time to Excel" metrics:

Time spent on task:

Roughly an hour total to extract the data from 40 session and copy/paste them into Excel

This is a little more than a minute a session

Notes on time spent on task:

The first session type (FTP/Email/Telnet/HTTP) I encountered took more time than all the follow up sessions

Ignoring taking screenshots and blog related notes, it probably took me a couple of minutes for each new session type encounter to look through them.

There was a lot of duplicate data that I could have skipped if I was doing this during the contest. I probably could have completed the entire task in under 30 minutes if I was rushing.
The initial time analyzing a session is probably more representative of real life data as it isn't as scripted as the CMIYC2024 contest material. So my gut says 5 minutes per "interesting" session is a reasonable general estimate of time for manual analysis.

Putting it all together, here are all the "secrets" and interesting information I found manually going through the packet capture:

Stats on Total Secrets Found:

85 total secrets

4 HTTP Basic Auth
16 passwords in e-mail HTML code
10 FTP Logins
23 FTP Data (Text File)
8 Telnet Logins
14 Passwords found in E-mail Text (original and reset password)

The first thing we need to do when performing automated analysis of the traffic is to save the decrypted WPA1 packets. Up till now, I've been using Wireshark to display the decrypted traffic but the tools I'll be using may not have the ability to decrypt the traffic themselves. Therefore I need to export the plaintext packets and create a new packet capture file containing them.

Disclaimer: I'm going to skip a lot of the troubleshooting and debugging that I had to do for this next step and instead focus on the current "solution" I found that mostly works. One thing that this research has highlighted for me is how *weird* this use-case is. Normally you want to feed decrypted traffic directly to your analysis tools by forwarding it to another interface or loopback after it has decrypted by your wireless card. Basically you want your tools to work off a tap on the LAN. Likewise capturing WPA1 traffic and cracking the password often happens as its own step, after which you can connect to the network and start sniffing for secrets using the "LAN Tap" strategy. It's very rare to need to feed a packet capture of encrypted traffic through these tools. It just doesn't fall into many real life workflows outside of a contest, and the dodgy tool support for supporting this use-case reflects that. Lots of words to say that if you find yourself doing the below as part of a real world assessment, you are probably doing something wrong and should rethink your collection strategy.

Task 1) Strip off all the datalink layer content from the packet capture using the "Strip Headers" option in Wireshark.

Without going too deep into the weeds about networking 101 or side-tangents about how horrible the OSI network model is, what you really need to know is that WPA1 encryption happens at the data link layer. Higher level data like the TCP/IP information is encapsulated (and in the case of WPA1, encrypted) in a data link packet. This way your FTP server doesn't need to know anything about the fact that a client is connecting over a wireless network. What this means is that if we rip out all the datalink layer content and rewrite the TCP/IP data to a new "fake" plaintext datalink packet we can then save the fully decrypted packet capture to disk for analysis by other tools. Wireshark supports this using the "Strip Headers" feature.

The procedure to "Strip Headers" from a packet capture are as follows:

Load up the encrypted packet capture in Wireshark and decrypt it using the steps described in the previous blog post.

The traffic MUST be decrypted for the next step to work.
If step 3 returns NO packets, one possible cause is you didn't first decrypt the traffic in Wireshark

In Wireshark, click File->Strip Headers
In the pop-up dialog box, make sure you select "IP" in the drop down box and then click ok

I first tried selecting "Ethernet" and it didn't return any traffic. Also the fact that they reference "Ethernet" vs. the more generic "Data Link" is also weird.
Long story short, there is a significant degree of "magical incantations" vs. understanding what Wireshark is doing under the hood. But this seems to work, so I'd recommend treating it as a magical spell and following these steps.

The traffic should now be decrypted

Task 2) Save the "stripped down" packet capture to disk.

I would *STRONGLY* recommend saving it as the older/more-generic "wireshark/tcpdump" format with a .pcap file extension vs. the newer .pcap-ng file extension. While the newer file extension is nice and supports some nifty features, keeping it more generic is helpful when running it through non-Wireshark tools.

Side tangent on what NOT to do: A lot of the online documentation recommends that you use the Wireshark "Export PDUs to File" option. In fact, the links I found didn't even mention the "Strip Headers" option. While Export PDUs does save the traffic without any layer2 encryption, it saves that decrypted data in a format that only Wireshark can process. I spent hours troubleshooting different options to make the resulting PDUs play nice with other tools. This includes passing the PDU packet captures through tcpreplay tcprewrite, text2pcap, and various other tools trying to rebuild the data link layer encapsulation in a format non-Wireshark tools could process. I totally failed. Then I found out that strip headers does exactly what I needed it to do. So this is a note to you as well as my future self saying don't use "Export PDUs to File".

Google SEO: Unable to process unsupported DLT type: (null) (0xfc). If you reached this page looking for a solution to this error, please read above.

While stripping the data link layer is helpful for parsing the data using external tools, I want to take a moment about what we are losing. The first thing you'll notice when looking at the new packet capture is that all the 802.11 WIFI traffic is gone. This actually makes it a bit easier to manually find interesting traffic for contest related activities (you don't need to apply an additional filters), but we also no longer can snoop in on what Korelogic set their SSID up as. The other major change is that you'll notice the Data Frame now uses encapsulation type 7, "Raw IP". This is a fancy way of saying there is literally no Layer 2 data link layer encapsulation/frames in the packet. This goes beyond not having any MAC addresses. As the name implies, it truly is "Raw IP". Here is a screenshot of a packet displayed in Wireshark:

The bits on the wire (or in this case the packet capture) start at the IP layer. While this isn't a problem for some tools to display, this can cause problems with other tools (such as PCredz) that expect that data link layer to be present. I'll talk more in the tool section for how to handle that.

Finally we are at the point where we can start looking at automated tools to parse the packet capture file for secrets! For each of the tools I'm going to break it down into three different sub-sections: 1) How to install the tool, 2) Results of parsing the packet capture with the tools, and 3) Any closing thoughts about using the tool for other real-life activities.

DSNIFF

I'm sure there were tools to extract credentials from network taps before DSniff, but it was one of the cornerstone hacking tools back in the day. Usually DSniff would be paired with Ettercap to redirect traffic to your local system. Then while running a person in the middle attack against all switched traffic you would turn on DSniff to save any plaintext credentials people were using to a file. It was very simple to run, and quite effective against a lot of the earlier Internet protocols.

Installation:

DSniff hasn't been updated in over 14 years so you don't have to worry about missing out on features by installing the pre-built binary from your local package manager. So for example, installing it on Ubuntu I just did:

sudo apt install dsniff

Operation and Results:

To run DSniff against a saved packet capture, simply use the '-p' flag with the path to the packet capture to read in. For example:

dsniff -p decrypted_pcap.pcap

As far as the effectiveness, DSniff did an excellent job of picking out the FTP, Telnet, and HTTP authorization usernames + passwords, but missed picking out any secrets from the contents of the traffic (such as the e-mail, the passwords in the HTTP body, and the password file downloaded by FTP). To put it another way, DSniff works great to capture credentials that are part of a protocol, but it does not run any regular expressions to pull out secrets or interesting data from protocol's traffic after the initial authentication is complete.

Overall Review:

It warms my heart to see DSniff continue to work after all these years. Dsniff was a breeze to get up and running which certainly makes it attractive to someone looking to get a basic capability operational as quickly as possible. If you just want to catch low hanging fruit and basic misconfigurations, DSniff is still a respectable option.

Where the gaps start to show is DSniff's lack of support for extracting secrets from the content of anything besides Telnet sessions. Another major challenge is that Dsniff doesn't have a standardized format to save the content of different secret types. This makes it harder to pull these secrets out and create a input dictionary for password cracking, or an Excel spreadsheet for analysis. While you can save the results to disk vs. stdout, DSniff saves log files using a Berkley database format limiting your ability to manually open the save file up with a text editor (or script) for further processing.

Conclusion: DSniff works, but it feels like a 20 year old tool. 3.5 our of 5 stars.

PCredz

This tool was recommended to me by various people and seems to be the main credential dumper being used by red team members and penetration testers. It has a specific focus on credit card numbers which I'm sure makes it popular with the PCI enforcement crowd.

Installation:

The following was taken from the PCredz Linux installation instructions and validate on a WSL Ubuntu system.

Clone the PCredz repository to where you want to save/install it

git clone https://github.com/lgandx/PCredz.git

sudo apt install python3-pip
sudo apt-get install libpcap-dev
pip3 install Cython
pip3 install python-libpcap
If you are parsing a raw IP packet dump, you will then need to modify the PCredz code as described in the next section.

If I have time I may submit a merge request to help fix this problem for others

Operation and Results:

To run PCredz against a packet capture file you simply need to give it the -f option. For example:

python3 Pcredz -f decrypted_pcap.pcap

Unfortunately when I tried this with the decrypted packet capture it failed.

Luckily, PCredz is a python3 program, and that's something I can troubleshoot pretty easily! Looking through the code, I quickly found that it was expecting a Layer2 dataframe, and without that encapsulation it was failing. I quickly put together a very hacked together fix as shown below to force it to parse the data as an IPv4 packet.

This is why I'm a menace on development teams by the way. The good news though is this fix "worked" and I could now extract secrets from the contest packet capture file.

This is much better! Like DSniff, PCredz extracts the login information for FTP and HTTP sessions. It missed the Telnet sessions though along with e-mail, HTTP body, and FTP Data secrets. I was really surprised by the omission of Telnet credentials. Looking at the supported protocols though, (you can see that list when PCredz starts up) Telnet is not included.

Where PCredz does shine compared to DSniff is that it can use regular expressions to look for specific data inside the packets, and PCredz is written in Python so you can add your own regular expressions to it with only a few lines of code.

Overall Review:

PCredz is more of a developer tool than DSniff. It provides a lot of extensibility and the ability to tailor it to a particular problem that you are trying to solve. Since it's just Python3 code, you can tailor its detection logic as well as how it outputs the data it collects, making it a great starting point to develop a pipeline for more advanced analysis. It's not a one-stop shop though, and I'd probably direct newer users to DSniff instead.

Conclusion: For a new user I'd give PCredz 3 stars out of 5. For a developer it earns 4 stars out of 5 as it is a lot nicer than playing around with direct calls to SCAPY (another nice Python network tool).

CredSLayer

CredSlayer was inspired by PCredz with a focus on extensibility. It is able to use Wireshark dissectors and can support many more different protocols out of the box.

Installation:

While CredSLayer is written in Python, it relies upon tshark (the Wireshark command line interface) to do all the heavy lifting for parsing the packets. The following instructions are for getting it to work on a WSL Ubuntu install

Make sure you have Python >= 3.4 installed
Install tshark

sudo apt install tshark

pip3 install credslayer

Operation and Results:

To run CredSlayer against a packet capture file just run it with the packet capture file specified

credslayer decrypted_pcap.pcap

When I ran it, it worked perfectly the first time I tried, and extracted the same data that DSniff did (Telnet, HTTP Auth, and FTP Auth), while missing the same "content" secrets that DSniff missed. The biggest difference though was that Credslayer printed its results in a well defined, easily parsed format.

Now you might be wondering, "How can I make this fit even better into my workflows", or "Can I create a pipeline to make use of the found credentials". Heck, I can even make a callback to the beginning of this post: "Can I reduce my Time-To-Excel to as close to zero as possible". The good news is that not only can you import CredSLayer into your other Python programs, but it has documentation on how to do that as well!

What about adding in your own regular expressions to look for data in the session contents? CredSLayer has you covered there as well!

Overall Review:

I'm a huge fan of CredSLayer. It's not magic. For example, manual analysis is still the way to go during these competitions since none of the automated tools found the password reset e-mails, the HTTP "Password Vault" or the download of the FTP password list. But given the constraints that "secrets" are highly subjective and context sensitive, CredSLayer offers an impressive mix of built-in capabilities and extensibility. I like the code, I like the architecture, and I probably owe the developer a beer. This is code that brings me joy.

Conclusion: Five out of five stars. No notes. Will recommend. Will use in my day job.