X-Forwarded-For and the ISA Firewall: Track your Originating Client through a Web-proxy Chain and on Your IIS
When the 2000 ISA firewall was released, it was seen by the firewall community as an upgrade version of Microsoft Proxy Server 2.0 and subsequently its credibility in the firewall and security space has been continuously questioned by old world ‘hardware’ firewall admins. The ‘dinosaur class’ still sees the ISA firewall as a web proxy and caching device with a bit of firewall capabilities slapped on for affect. A large amount of this misinformation was propagated by competitive firewall vendors and the usual brigade of militant anti-Microsoft people (ABMer’s) who weren’t particularly hindered by trivial things (like facts).
For those in on ‘the know’, this is obviously an unfair and blatantly incorrect assessment of the firewall. The ISA firewall was designed from scratch to be a comprehensive network and application layer inspection firewall. The web proxy and caching functionality was the bolted on the core firewall technology, not the other way around.
This incorrect impression, that the ISA firewall is some kind of Web proxy only device, has achieved one thing though, and that is that the firewall has become the world’s most deployed web proxy and caching solution. There is little doubt that the ISA firewall performs exceptionally well as a Web proxy and caching server. However, like all solutions, there are some features found in competitor technologies that the ISA firewall does not have. A feature missing from the firewall, as well as other Microsoft products such as IIS, is support for the X-Forwarded-For: HTTP and HTTPS header field.
The X-Forwarded-For (XFF) HTTP header is a de facto industry standard for identifying the originating IP address of a client connecting to a Web server through an HTTP proxy. Without this field, any connection through the proxy would reveal only the originating IP address of the proxy, effectively turning the proxy server into an anonymizing service. This makes it very difficult, if not impossible, to reliably determine the requests originating client IP address once it passes through a web proxy chain. To answer the ‘So what?’ question, as Microsoft ISA Server does not natively support the X-Forwarded-For field, it becomes very difficult to log and detect abusive accesses that have passed through your ISA Server proxy chain.
There are reasons why this feature is not natively supported by Microsoft products such as the ISA firewall and IIS.
- First, X-Forwarded-For usage is a de facto industry standard and NOT an RFC official standard.
- Secondly, historically there have been many security flaws associated with X-Forwarded-For HTTP header support. Many implementations fell victim to spoof attacks where systems were given spoofed X-Forwarded-For information and they inadvertently processed a rule or action based on this information.
- X-Forwarded-For IP information is clear text inside a HTTP header; it is NOT signed and is NOT authenticated. This can pose a huge security risk if allow and deny security decisions are made based on the data stored in the X-Forwarded-For header especially if the data originates from the Internet.
- Another historic security issue with the technology is that internal IP address information could be revealed to the Internet, which could unwittingly divulge information about the internal infrastructure.
The guys at Winfrasoft have produced 2 products which allow your Microsoft infrastructure to recognize and utilize the X-Forwarded-For field while at the same time negating the security concerns associated with this technology. One product is designed for ISA firewalls, while the other is designed for IIS. It is not often that I assess products designed for technologies other than the ISA firewall. However, there is a significant synergy between the ISA firewall and IIS deployments, as many IIS Web servers publish web sites behind the firewall.
Winfrasoft X-Forwarded-For for ISA firewalls enables ISA firewall admins to track and log HTTP and HTTPS web traffic from its original source through multiple forward and reverse proxies and analyse the information through ISA Server Logging. This product provides the ISA firewall with the same functionality of other web proxy devices and load balancers such as Squid, Apache, F5 Big-IP, Blue Coat, Cisco Cache Engine and Netcache.
Winfrasoft X-Forwarded-For for IIS enables IIS administrators to log HTTP and HTTPS web traffic from its original source through multiple proxies and analyse the log data with standard web site reporting tools. This allows IIS web servers to be securely published behind layer 7 firewalls and load balancers such as the ISA firewall (with X-Forwarded-For for ISA firewalls installed) and the usual suspects listed above.
Winfrasoft X-Forwarded-For for ISA Server
Installation is a simple process that requires limited user input so I will not go into any details on this operation. Once installed, X-Forwarded-For for ISA Server appears in the Web Filters section of the Add-ins node in the left pane of the ISA firewall console. Through the ISA firewall console, the Web filter can be enabled or disabled and moved up and down the priority list.
Once X-Forwarded-For for ISA Server is installed on all the ISA firewalls in your web proxy chain, then the magic happens. As a client request passes through your web proxy chain, the X-Forwarded-For HTTP header is added (should the XFF field not exist) or modified (if XFF does exist) and the ISA firewall includes the real client’s IP address to the X-Forwarded-For HTTP header.
To test the X-Forwarded-For for ISA in a Forward Proxy, I have configured the following infrastructure in my lab environment.
This is the log from the Upstream Proxy (192.168.0.254) server showing a HTTP request to the world’s best web site :-). The originating client IP address can be seen in the Source field. Under Filter Information, the X-Forwarded-For HTTP header field has been created and the IP addresses of the web proxy servers that processed the request are recorded.
There is a potential security concern with this as the HTTP packet going out to the web could be listing your internal IP addresses, however, the default behaviour of X-Forwarded-For for ISA Server is to remove the X-Forwarded-For field data as the packet leaves the last web proxy in a web proxy chain, as Upstream Proxy does is in this scenario. This can be verified by using a network analyzer, such as NetMon 3.2 (Microsoft Network Monitor 3.2)
If you have a transparent upstream packet sniffer, then the behaviour of X-Forwarded-For for ISA Server can be configured to leave the X-Forwarded-For information in the HTTP packet as it leaves the last web proxy server in a web chain. Obviously, I would only recommend this if the upstream sniffer has the ability to remove the X-Forwarded-For header before the request hits the Internet.
To test the X-Forwarded-For for ISA in a Reverse Proxy, I have configured the following infrastructure in my lab environment.
This is the log from the Downstream2 Proxy (192.168.0.200) server showing a HTTP request from an Internet client to a web page published on an IIS sitting behind my reverse web proxy chain. The originating client IP address can be seen in the Source field. Under Filter Information, the X-Forwarded-For HTTP header field has been created and the IP addresses of the web proxy servers that processed the request are recorded.
Unlike in a forward proxy scenario, as the HTTP request is staying internally, there are no security issues with keeping internal addresses in the HTTP packet header so the last web proxy server does not strip the X-Forwarded-For field.
This is also a good thing because if you have an Apache (perish the thought!) Web server configured behind your web proxy chain, then the logs can be inspected on that server and the route for the request determined. This neatly leads me into the second product in the Winfrasoft X-Forwarded-For suite, X-Forwarded-For for IIS.
Winfrasoft X-Forwarded-For for IIS
As with the X-Forwarded-For for ISA Server, the installation process is very simple and again the user input required is very limited. X-Forwarded-For for IIS is an ISAPI filter that, once installed, injects itself into the ISAPI Filters property page for all web sites on your IIS server.
The following depicts my lab environment used to test X-Forwarded-For for IIS. In the examples below, from my Client PC (192.168.10.76), I am requesting a web page hosted on my IIS Web Server (192.168.0.1). My IIS server is IIS version 6.0 running on a Windows 2003 server, but for all you early adopters, X-Forwarded-For for IIS also supports IIS 7.0 on Windows 2008.
As mentioned previously, Internet Information Server does not natively support X-Forwarded-For, therefore investigating the logs on a standard install of IIS will show that all requests came from Downstream2 (192.168.0.200).
This is a W3C formatted log entry from Web Server (192.168.0.1). The c-ip field stores the originating client IP address.
#Software: Microsoft Internet Information Services 6.0 #Version: 1.0 #Date: 2008-09-09 16:37:24 #Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query s-port cs- username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status 2008-09-07 16:37:24 W3SVC1 192.168.0.1 GET /Default.htm - 80 - 192.168.0.200 Mozilla/ 4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.NET+CLR+2.0.50727) 200 0 0
From this experiment, you can see that the originating client IP address has been lost, effectively making the logging on the web server useless.
X-Forwarded-For for IIS can be configured using trust lists to either show the full X-Forwarded-For header field or the first non-trusted IP address in the IIS logs.
IP Address Trust List
As mentioned above, X-Forwarded-For for IIS can be configured to display the first non-trusted IP address. To understand this concept, we need to establish the differences between trusted and non-trusted IP addresses. In my lab environment, I know the IP addresses of all my proxy servers, so these are the trusted addresses which I’ll add to the trust list. My trust list, in this example, will be:
TrustList= 192.168.0.254, 192.168.0.200, 192.168.0.100
Any IP address not found in my trust list will be treated as a non-trusted IP address.
Trust lists mitigate the security issues seen with proxy servers that support X-Forwarded-For headers. Proxy servers which support X-Forwarded-For simply append the IP address they receive a request from to the header and forward it on. As there is no signing of the data in this field, any IP address can be spoofed into the header. This could invalidate any reports generated based on the X-Forwarded-For headers.
When the IIS filter processes the X-Forwarded-For header it checks the IP address of the first proxy based on its layer 4 address. If it is trusted, the verification process continues with the IP addresses listed in the X-Forwarded-For header until the first un-trusted IP address is found. This is then used as the c-ip in the web server log as it was the first Internet routable address that entered the trusted proxy chain - which means that any spoofed IP addresses listed prior are dropped.
Scenario 2 - Using XFF with Trust List
TrustList= 192.168.0.254, 192.168.0.200, 192.168.0.100
X-Forwarded-For: 192.168.0.254, 192.168.0.200, 192.168.0.100
This is the W3C formatted log entry from Web Server (192.168.0.1) using the above trust list. Again, the c-ip field stores the originating client IP address.
#Software: Microsoft Internet Information Services 6.0 #Version: 1.0 #Date: 2008-09-09 16:44:12 #Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query s-port cs- username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status 2008-09-07 16:37:24 W3SVC1 192.168.0.1 GET /Default.htm - 80 - 192.168.10.76 Mozilla/ 4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.NET+CLR+2.0.50727) 200 0 0
From this experiment, you can now see that the first non-trusted client IP address is logged as the originating client IP address.
Scenario 3 - Using XFF without a configured Trust List
X-Forwarded-For: 192.168.0.254, 192.168.0.200, 192.168.0.100
In this example, I have removed my trust list and again requested the web page hosted on the web server. As there is no trust list, X-Forwarded-For for IIS works differently and now inserts the entire X-Forwarded-For header into the W3C log c-ip field as well as the IP address of the server that passed the HTTP request to the IIS Web Server.
#Software: Microsoft Internet Information Services 6.0 #Version: 1.0 #Date: 2008-09-09 16:46:54 #Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query s-port cs- username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status 2008-09-07 16:37:24 W3SVC1 192.168.0.1 GET /Default.htm - 80 - 192.168.10.76,+192.168.0.254,+192.168.0.100,+192.168.0.200 Mozilla/ 4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.NET+CLR+2.0.50727) 200 0 0
The X-Forwarded-For product suite provides your ISA firewall and IIS infrastructure with the same X-Forwarded-For implementation and logging functionality found in other competitive products. Winfrasoft has gone a step further to ensure that the security risks associated with this technology have been removed.
These two products, which can be installed independent of each other, will greatly enhance your abilities to track down HTTP and HTTPS requests and their benefits become very apparent when you are trying to analyze or detect malicious attacks. X-Forwarded-For for ISA Server and X-Forwarded-For for IIS also allow you to gather necessary data critical in your audit and compliance strategies.