There are many reasons why you’d want to front your Apache-based site behind a reverse proxy but one of the sacrifices you make in speed and security, you would by default lose a sensible way to extract the real IP (or at least have a very good chance to) of the client making the request. This is because your reverse proxy or load balancer in many situations becomes the client and to Apache this means you will only ever see the IP (the internal IP if its local to your Apache server farm) which become useless if you are doing analytics on the Apache logs or making decisions for the request based on the IP (for example Geo decisions, or ModSecurity).
To circumvent this, you’d want to use mod_rpaf (http://stderr.net/apache/rpaf/). This module is easily integrated into your Apache environment using the following method (Apache 2.x):
sudo apxs -i -c -n mod_rpaf-2.0.so mod_rpaf-2.0.c
And then configuring Apache to pick up this module:
# if DSO load module first: LoadModule rpaf_module modules/mod_rpaf-2.0.so RPAFenable On RPAFsethostname Off RPAFproxy_ips 10.10.10.1 RPAFheader X-Forwarded-For
This says look in the X-Forwarded-For header for the existence of the IPs listed in the RPAFproxy_ips line, and if found, pick up the external IP.
Unfortunately in practice this doesn’t work so well for a couple of reasons:
1. When auto-scaling an environment such as in Amazon, and you’re using something like ELB, you won’t know what internal IP the ELB will talk to Apache on – causing you to not be able to auto-configure mod_rpaf at install/run-time.
2. It makes an assumption on the external IP: it just takes the last IP seen on the X-Forwarded-For line.
Number 2 caused me issues because it is possible to have an internal IP address as the last entry on the X-Forwarded-For line – and if inspected properly, the external IP would be somewhere along the X-Forwarded-For line. To get around this and the ability to have more flexibility in the mod_rpaf config so that the code invoked to expose the real IP when an internal address was seen I created the following patch:
--- mod_rpaf-2.0.c 2011-06-23 13:51:53.000000000 +0100 +++ mod_rpaf-2.0.c.new 2011-06-24 16:08:18.000000000 +0100 @@ -71,6 +71,7 @@ #include "http_protocol.h" #include "http_vhost.h" #include "apr_strings.h" +#include "string.h" module AP_MODULE_DECLARE_DATA rpaf_module; @@ -136,10 +137,14 @@ } static int is_in_array(const char *remote_ip, apr_array_header_t *proxy_ips) { - int i; + int i,len; + char tmp[16]; char **list = (char**)proxy_ips->elts; for (i = 0; i nelts; i++) { - if (strcmp(remote_ip, list[i]) == 0) + len=strlen(list[i]); + strncpy(tmp,remote_ip,len); + tmp[len]=''; + if (strcmp(tmp, list[i]) == 0) return 1; } return 0; @@ -155,6 +160,7 @@ static int change_remote_ip(request_rec *r) { const char *fwdvalue; char *val; + int i; rpaf_server_cfg *cfg = (rpaf_server_cfg *)ap_get_module_config(r->server->module_config, &rpaf_module); @@ -183,7 +189,10 @@ rcr->old_ip = apr_pstrdup(r->connection->pool, r->connection->remote_ip); rcr->r = r; apr_pool_cleanup_register(r->pool, (void *)rcr, rpaf_cleanup, apr_pool_cleanup_null); - r->connection->remote_ip = apr_pstrdup(r->connection->pool, ((char **)arr->elts)[((arr->nelts)-1)]); + for(i=arr->nelts-1; i >= 0; i--) { + if (is_in_array( apr_pstrdup(r->connection->pool, ((char **)arr->elts)[i]), cfg->proxy_ips) == 0 ) + r->connection->remote_ip = apr_pstrdup(r->connection->pool, ((char **)arr->elts)[i]); + } r->connection->remote_addr->sa.sin.sin_addr.s_addr = apr_inet_addr(r->connection->remote_ip); if (cfg->sethostname) { const char *hostvalue;
Patch and compile/install as follows:
patch -p1 < patch_mod_rpaf sudo apxs -i -c -n mod_rpaf-2.0.so mod_rpaf-2.0.c
This allows you to use a modified config file which allows you to run Apache behind, say, ELB and it extracts the last seen external IP (i.e. not 10. 172. or 192.168.). Of course, edit to suit your particular environment:
# if DSO load module first: LoadModule rpaf_module modules/mod_rpaf-2.0.so RPAFenable On RPAFsethostname Off RPAFproxy_ips 10. 172. 192.168. RPAFheader X-Forwarded-For
When it sees a line like
X-Forwarded-For: 192.168.100.227, 209.88.21.195, 10.58.59.219, 192.168.123.123
It will pick out 209.88.21.195 as the real IP.
Thank you for this solution!
Just hit a snag with installation & was hoping you could advise.
On ubuntu 10.04, using mod_rpaf source from https://launchpad.net/ubuntu/lucid/+source/libapache2-mod-rpaf,
I’m hitting the following error:
root@amazon-ip:/tmp/mod_rpaf-0.5# patch mod_rpaf-2.0.c patch_mod_rpaf(Patch is indented 1 space.)
patching file mod_rpaf-2.0.c
patch: **** malformed patch at line 8: module AP_MODULE_DECLARE_DATA rpaf_module;
Tried to mod the c source directly, but now i’m hitting this on compile:
/usr/bin/apxs2 -i -c -n mod_rpaf-2.0.so mod_rpaf-2.0.c
/usr/share/apr-1.0/build/libtool –silent –mode=compile –tag=disable-static i486-linux-gnu-gcc -prefer-pic -DLINUX=2 -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -D_LARGEFILE64_SOURCE -D_REENTRANT -I/usr/include/apr-1.0 -I/usr/include/openssl -I/usr/include/xmltok -pthread -I/usr/include/apache2 -I/usr/include/apr-1.0 -I/usr/include/apr-1.0 -c -o mod_rpaf-2.0.lo mod_rpaf-2.0.c && touch mod_rpaf-2.0.slo
mod_rpaf-2.0.c:130:15: error: empty character constant
mod_rpaf-2.0.c:195: warning: initialization from incompatible pointer type
apxs:Error: Command failed with rc=65536
Any ideas? It’s looking to me like the Ubuntu source is different than what you compiled from, but http://stderr.net/apache/rpaf/ is 404’ing for me.
Got the patch to compile (‘ ‘, not ”… duh!), but it’s still not filtering wildcard IPs like “10.”… just passes ELB ip straight thru.
Hmm. I’ll be testing this in anger in a few days fronting a busy live website, so if there is something amiss I’ll find it then.
My setup is Apache 2.2.19, plus I’ve got ModSecurity installed – which is the reason I needed mod_rpaf working correctly behind something like ELB – but without knowing what that internal IP the ELB will be talking to the Apache instances on.
sudo tcpdump -i eth0 -s 1514 ‘tcp port 80’ -w capture.cap may help when you view the capture through Wireshark to see if the X-Forwarded-For header is being set.
I’ve also found curl to help too:
tail your apache access logs and run curl:
curl -H “X-Forwarded-For: 10.10.10.1 174.235.127.235 10.10.10.2” http://yourserver/
should write 174.235.127.235 to your access log for example.
I’ve been running this successfully in anger on a site that was getting 000,000s hits an hour and didn’t see any issues with mod_rpaf. I had 6 Apache 2.2 servers running ModSecurity and Mod_rpaf behind ELB and proxying requests through to the datacentre serving the content. In ModSecurity I was seeing the real external IP.
I don’t understand what do you mean by “Got the patch to compile (‘ ‘, not ”… duh!)”. I got the same issue… What should I change in the patch? Thanks!
I tried to replace ” to ‘ – same thing…
And it does appear stderr.net is offline – and has been for a good few days (reason why my patch is on here and not sent to the developer). I’ll upload the source tarball when I get chance.
I tried to ping the site owner but no response as of yet.
or you mean white-spaces?
Pingback: Mod_rpaf source download « System Administration and Architecture Blog
I’ve uploaded the original mod_rpaf code temporarily to http://www.linuxservices.co.uk/mod_rpaf-0.6.tar.gz
Could you give source file for that patch? I think that WP messed the source in the box
You can find the pre-built RPM here: http://www.linuxservices.co.uk/mod_rpaf-0.6-3.x86_64.rpm.
You can find the patch here: http://www.linuxservices.co.uk/mod_rpaf_xforward_for.patch
Thanks again for posting the patch Kevin.
Kevin, those links aren’t valid anymore, still have them handy?
Thanks
Ah, sorry – I moved the hosting and I’ve failed to move some artifacts. I’ll upload these on Monday as soon as I can.
Any luck on getting it transferred over?
Getting the same error as the first poster –
File to patch: mod_rpaf-2.0.c
patching file mod_rpaf-2.0.c
patch: **** malformed patch at line 8: module AP_MODULE_DECLARE_DATA rpaf_module;
Any Ideas?
Hi Guys – sorry about the hosting of the files issue – I’ve uploaded these now.
RPM: http://www.linuxservices.co.uk/mod_rpaf-0.6-3.x86_64.rpm
Patch: http://www.linuxservices.co.uk/mod_rpaf_xforward_for.patch
Original source tarball: http://www.linuxservices.co.uk/mod_rpaf-0.6.tar.gz
Kev
Thank you so much for this, big big contribution.
This was a huge help under ELB. Thank you very much for posting a complete solution to the problem.
Could you give source file for that patch? The links that you posted on Jan 10th are not longer available.
Thank you in advance!
This should be back up and running now. Last time I host some stuff on a beta cloud solution 😉
Hello there! This blog post could not be written much better!
Looking through this article reminds me of my previous roommate!
He always kept preaching about this. I am going to send this post
to him. Fairly certain he’s going to have a good read. Many thanks for sharing!
ANother way of getting real IP (rightmost ip) from XFF is by using Apache SetEnvIf and regex
LogFormat “%{CLIENTIP}e %l %u %t \”%r\” %{Host}i %>s %b %D \”%{DecURID}i\”” common
SetEnvIf REMOTE_ADDR “(.+)” CLIENTIP=$1
SetEnvIf X-Forwarded-For “.*(\D|^)((\d{1,3}\.){3}\d{1,3})” CLIENTIP=$2
CustomLog “logs/access_log” common
Does your blog have a contact page? I’m having trouble locating it
but, I’d like to shoot you an email. I’ve got some ideas for
your blog you might be interested in hearing. Either way, great blog
and I look forward to seeing it expand over time.
Can I simply say what a relief to uncover someone who actually knows what they’re discussing over the internet.
You actually understand how to bring a problem to light and make it important.
More and more people must read this and understand this
side of the story. I was surprised that you’re
not more popular given that you surely have the gift.
You need to take part in a contest for one of the greatest websites on the net.
I am going to recommend this website!
Thanks for some other magnificent post. The place else may anybody get that type of information in such a perfect way of writing?
I’ve a presentation subsequent week, and I am on the search for such info.