Skip to content

Tag Archives: Apache

Creating a simple IP blacklist with mod_rewrite and RewriteMap


I’ve had a lot of URL spam (bots testing URLs for weaknesses and exploits). They haven’t been successful but i have configured my system to send me an e-mail when exceptions occurs and there have a been a lot of e-mails.

I wanted a simple solution with the following features:

  • No additional packages to install
  • No server restarts when the blacklist changes

In the given setup Apache is merely a proxy to my backend and a simple .htaccess inside the root directory with a lot of ip entries wouldn’t work. Instead this would have gone into the vhost definition which means server restarts / reloads.

I came up with the idea using the RewriteMap directive of mod_rewrite.

You can ask a RewriteMap for a value with anything ModRewrite and apache variables give you.

This is what my map (blacklist.txt) looks like:    b   b   b

I use httxt2dbm to create DBM Hash File

httxt2dbm -i blacklist.txt -o blacklist.dbm

and then in my vhost definition:

RewriteMap ipmap dbm:/etc/apache2/sites-available/blacklist.dbm 
RewriteCond ${ipmap:%{REMOTE_ADDR}} ^b$            [NC]
RewriteRule .* - [F,L]

That means:

Use the given blacklist.dbm as map named ipmap. Than ask for the value for the given remote address. If the value is “b” than deny access to all pages.

Simple, but effective.

Apache httpd, Tomcat und sendfile


I used to use mod_xsendfile by Nils Maier, who’s Homepage doesn’t seem to exist anymore, to send files from Ruby proxied by Apache respectively powered by modrails. Those files shouldn’t be in any public www directory as authorization needs to be checked, but are accessed very often so that streaming them is not an option.

To use this technique you need mod_xsendfile, which is attached to this post.

I just have rewritten my application from Ruby on Rails to Java and it’s easy to add the necessary headers in a HttpServletResponse:

response.setHeader("X-Sendfile", file.getAbsolutePath());

You may add other headers like Content-Type and the like but you must not modify the body, hence the flushBuffer.

This works quite well… As long as had my Apache httpd running with mpm-prefork.

Switching to Apache mpm-worker caused some problems. I cannot say with a final conclusion that mod_xsendfile was causing troubles but i started to see the wrong files (images in this case) delivered or not delivered at all.

My alternate solution was streaming the files using Channels from java.nio but CPU usage went nuts.

The solution now employed is the using Apache Tomcats asynchronous writes that are available since Tomcat 6. Their documentation is rather short but so is implementing them:

HttpServletRequest request = ...
HttpServletResponse response = ...
if(request != null && Boolean.TRUE.equals(request.getAttribute(""))) {		
  long l = file.length();
  request.setAttribute("org.apache.tomcat.sendfile.filename", absolutePath);
  request.setAttribute("org.apache.tomcat.sendfile.start", 0l);
  request.setAttribute("org.apache.tomcat.sendfile.end", l);
  response.setHeader("Content-Length", Long.toString(l));
} else if(use_xsendfile) {
  // see above
} else {
  // stream files

What i got wrong at first was setting those attributes in the response. That didn’t work. The must be set in the request and you must take care setting all of those and with the correct type (String respectively long). And that’s it.

The request attribute will be true when the connector is configured to either use the APR connector (org.apache.coyote.http11.Http11AprProtocol) or the non blocking Java connector (org.apache.coyote.http11.Http11NioProtocol) (the later one being easier to deploy as it has no external dependencies).

CPU usage for sending those files is now nearly 0.

WordPress 2.2 Update


Wordpress 2.2 arrived a day ago.
Updating was no problem at all. Prominent new features are wigdet support without the need for an external plugin and a working atom 1.0 feed.

I could drop two plugins: the widget plugin and the atom10 plugin. Nice. All others work as before (some important to mention are UTW, WP-Cache and the nice extended live archive(*).

A really nice site effect of the update: the WP Team finally managed to put a “SET NAMES blah” into wp-db.php, so that setting “UTF-8″ i.e. in wp-admin really means UTF-8 in the database. I always hacked my wp-db(**) for my other blog, since there are a lot of old posts from another legacy system that _are_ indeed UTF-8 but which where maltransformed by wp without that hack before 2.2.

I do have a local development system on my mac but i tend to test things twice or more often. Other people maybe don’t have or want that luxury and need to udate their blogs directly on the server. Either way, it’s a good idea, to put your blog to maintenance while updating your wp core files. This is easily accomplished with .htacess and mod_rewrite. First create a file “maintenance.html” for your visitors. Than look up your external ip and add the following to your .htaccess:

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{REMOTE_ADDR} ! [NC]
RewriteRule ! maintenance.html$ maintenance.html [R=301,L]

and replace with your external ip. Remember commenting these entries when you’re done.

(*) If you have any questions about my plugins, don’t hesitate to ask, i just don’t want a plugin list in case of bugs in any plugin.

(**) The hack was: Add the following to __construct in your wp-db.php (./wp-includes/) :

	$this->query("SET NAMES 'utf8'");

Rails mit FastCGI und Apache unter Mac OS X


Die vorherigen Artikel Ruby on Rails mit Mac OS X und Rails: Rolling into production on Mac OS X beschäftigten sich mit der Installation von Ruby on Rails unter Mac OS X bzw. mit der Produktivumgebung von Rails.

Prämisse für mich war jedesmal, dass der eingebaute Mac OS X Apache genutzt wird und nicht ein zweiter Serverprozess.

Heute soll es um die FastCGI Integration gehen. CGI war mir bis jetzt zum Testen nicht zu langsam. Fängt man aber mit AJAX Spielerein an und nutzt Rails mit CGI, kann man das auch getrost ganz sein lassen. Vorweg: FastCGI zu CGI ist ein Unterschied wie Tag und Nacht.

Was braucht man? (Die Links verweisen direkt auf die benötigten Dateien)

Die Installation ist relativ einfach.


sudo make install


apxs -o -c *.c 
sudo apxs -i -a -n fastcgi

Ruby Bindings

ruby install.rb config
ruby install.rb setup
ruby install.rb install

Fast geschafft. Es folgt ein Apache Neustart mit

sudo apachectl graceful

Anschliessend gibt es einen kleinen Pitfall. Man muss zwei Tempverzeichnisse für FastCGI chmoden:

chmod 777 /tmp/fcgi_ipc
chmod 777 /tmp/fcgi_ipc/dynamic

sonst geht es in die Hose.

Anschliessend zur Apache Konfiguration. Diese habe ich wieder auf zwei Dateien aufgeteilt. In der httpd.conf wird das Modul mod_fastcgi geladen. Der Installer hat diesen Eintrag bereits automatisch erzeugt. Wenn nicht, folgende Zeilen ergänzen:

LoadModule fastcgi_module     libexec/httpd/

Danach den Handler einrichten:

<IfModule mod_fastcgi.c> 
FastCgiIpcDir /tmp/fcgi_ipc/ 
AddHandler fastcgi-script .fcgi 

und wir sind fast fertig. Die eigentliche Rails Anwendung wird wieder in der Userspezifischen httpd.conf eingestellt, zu finden unter /etc/httpd/users/name.conf:

<IfModule mod_fastcgi.c>
   FastCgiServer "/Users/msimons/dev/rails/test/public/dispatch.fcgi" -
initial-env RAILS_ENV=production -processes 15 -idle-timeout 60

Die Werte fürs Timeout und die Anzahl Prozesse sind aus dem Buch “Agile Web Development with Rails”.

Die Anwendung nutzt fastcgi, sobald man die entsprechende Rewrite Rule in .htaccess im public folder setzt:

RewriteRule ^(.*)$ dispatch.fcgi [QSA,L]

bzw. die ursprüngliche Regel

RewriteRule ^(.*)$ dispatch.cgi [QSA,L]


Danach ist ein letzter Apache Neustart fällig und man sieht seine Anwendung auf einmal in einem ganz anderen Tempo.

Rails: Rolling into production on Mac OS X


Damit die

SetEnv RAILS_ENV production

Direktive in der httpd.conf oder einer lokalen .htaccess Datei funktioniert, muss das Modul “env_module” im Apache aktiviert werden. Dies wird unter Mac OS X nicht standardmässig getan.

Folgende Zeilen in der /etc/httpd.conf einkommentieren:

LoadModule env_module         libexec/httpd/
AddModule mod_env.c