Often residential networks operate some DPI devices either on LAN or WAN. Is there any way to fingerprint the DPI device(s) from the responses, headers or some other behavior on the network?
Which are some easy tests to do that do not require extra software but rather curl or other *NIX commands?
Also any available tools that can be used to fingerprint DPIs will be as well very useful.
There is a research paper from 2013 on the topic of fingerprinting DPIs: A Method for Identifying and Confirming the Use of URL Filtering Products for Censorship.
Some web filter companies have category “test pages” that are guaranteed to be blocked, if the filter is configured to block that particular category. I don’t know of a full list, but here are a few:
For example, you can try downloading
If the first one is blocked but the others are not, then you know it is a Sophos device. (Note the Netsweeper one isn’t working for me currently.)
There are certain network fingerprints you can look for, but I don’t know of a centralized list of these. For example, nsphostname= in a block page URL means Netsweeper. In 2016 I wrote some regexes for patterns that were then common in OONI web measurements. You can find them in the file ooni-tor-blocks/classify.py from https://archive.org/details/ndss16doyousee or git clone https://www.bamsoftware.com/git/ooni-tor-blocks.git. Some examples:
if re.match("^http://.*/webadmin/deny/", get_header(response, "Location", "")):
return True, "302-NETSWEEPER"
if re.search("<meta name=\"author\" content=\"Blue Coat systems\">\n<meta name=\"description\" content=\"Denied Access Policy\">\n", body):
return True, "403-BLUECOAT"
if re.search("<li>McAfee Global Threat Intelligence has determined</li>", body):
return True, "403-MCAFEE"
Thank you for all the resources, I wasn’t aware that WhatWeb is able to identify DPIs.
Any clues on how to trigger some responses from DPIs similar to the OONI tests HTTP invalid request and header field manipulation?
BTW how Sophos tests works with HTTPS? I see that all their tests links are on HTTPS URLs.
You can do some simple tests with nc or Ncat. For example here is test_random_invalid_method:
$ printf 'ABCD / HTTP/1.0\r\n\r\n' | ncat -v example.com 80
The header field manipulation tests are harder because they rely on having a cooperating server that can check the request and see if it has been modified from what the client sends. If you could find a server that tells you what headers it received, you could use that. http://wtfismyip.com/headers almost works, except that it canonicalizes case in the header names. It could work for detecting some kinds of manipulation.
$ printf 'GET /headers HTTP/1.0\r\nACCept-LANGuage: en-US\r\n\r\n' | ncat wtfismyip.com 80
HTTP/1.0 200 OK
Content-Type: text/plain
Date: Tue, 03 Mar 2020 09:09:38 GMT
Content-Length: 23
Accept-Language: en-US
That’s a good question. I don’t know. Maybe those tests are intended for testing a host-based firewall, rather than a middlebox.