info.michael-simons.eu

MySql compatible AES encryption / decryption in Java

MySQL has an aes_encrypt/aes_decrypt pair of functions (Encryption and Compression Functions) that enable encryption and decryption of data using the official AES algorithm.

The functions are easy to use (select AES_ENCRYPT(‘text’,’password’)) and the result is easy to store (insert into secrets values HEX(AES_ENCRYPT(‘text’,’password’))) as hex values.

I used this technique for a while but i wanted to have a more database agnostic version of this encryption and tried to build the same methods with java.

Although it was relatively easy to find the exact cipher mode (which is AES/ECB/PKCS5Padding), i had a real hard time figuring out how the key is calculated from the given password (the key must be 16bytes long, per default MySql uses AES-128). It turns out that the MySQL algorithm just or’s the bytes of a given passphrase against the previous bytes if the password is longer than 16 chars and just leaves them 0 when the password is shorter than 16 chars. So you can generate a secret key spec in Java for an aes_encrypt/decrypt compatible cipher like so:

import java.io.UnsupportedEncodingException;
 
import javax.crypto.Cipher;
import javax.crypto.spec.SecretKeySpec;
 
import org.apache.commons.codec.binary.Hex;
 
public class Demo {
	public static SecretKeySpec generateMySQLAESKey(final String key, final String encoding) {
		try {
			final byte[] finalKey = new byte[16];
			int i = 0;
			for(byte b : key.getBytes(encoding))
				finalKey[i++%16] ^= b;			
			return new SecretKeySpec(finalKey, "AES");
		} catch(UnsupportedEncodingException e) {
			throw new RuntimeException(e);
		}
	}
 
	public static void main(String... args) throws Exception {
		// Decrypt
		final Cipher decryptCipher = Cipher.getInstance("AES");	        				
		decryptCipher.init(Cipher.DECRYPT_MODE, generateMySQLAESKey("your super secret passphrase", "UTF-8"));
		System.out.println(new String(decryptCipher.doFinal(Hex.decodeHex("56A34D7AB6225616799F6559AA388F07C2C9E53983111BDD5F49F36461AAD789".toCharArray()))));
 
		// Encrypt
		final Cipher encryptCipher = Cipher.getInstance("AES");	        				
		encryptCipher.init(Cipher.ENCRYPT_MODE, generateMySQLAESKey("your super secret passphrase", "UTF-8"));		
		System.out.println(String.format("Select aes_decrypt(unhex('%s'), 'your super secret passphrase');", new String(Hex.encodeHex(encryptCipher.doFinal("Hallo nach Aachen".getBytes("UTF-8")))))); 
	}
}

You need Commons Codec to run these.

This isn’t probably the most secure solution from a cryptographic point of view but it just replicates the built-in MySql function for other databases or just for interoperability. I hope to save someone else time with this as i spent about days about those view lines.

Filed in English posts, Java | Comments (14) »

Apache httpd, Tomcat und sendfile

I used to use mod_xsendfile by Nils Maier, who’s Homepage doesn’t seem to exist anymore, to send files from Ruby proxied by Apache respectively powered by modrails. Those files shouldn’t be in any public www directory as authorization needs to be checked, but are accessed very often so that streaming them is not an option.

To use this technique you need mod_xsendfile, which is attached to this post.

I just have rewritten my application from Ruby on Rails to Java and it’s easy to add the necessary headers in a HttpServletResponse:

response.setHeader("X-Sendfile", file.getAbsolutePath());
response.flushBuffer();

You may add other headers like Content-Type and the like but you must not modify the body, hence the flushBuffer.

This works quite well… As long as had my Apache httpd running with mpm-prefork.

Switching to Apache mpm-worker caused some problems. I cannot say with a final conclusion that mod_xsendfile was causing troubles but i started to see the wrong files (images in this case) delivered or not delivered at all.

My alternate solution was streaming the files using Channels from java.nio but CPU usage went nuts.

The solution now employed is the using Apache Tomcats asynchronous writes that are available since Tomcat 6. Their documentation is rather short but so is implementing them:

/*
HttpServletRequest request = ...
HttpServletResponse response = ...
*/
 
if(request != null && Boolean.TRUE.equals(request.getAttribute("org.apache.tomcat.sendfile.support"))) {		
  long l = file.length();
  request.setAttribute("org.apache.tomcat.sendfile.filename", absolutePath);
  request.setAttribute("org.apache.tomcat.sendfile.start", 0l);
  request.setAttribute("org.apache.tomcat.sendfile.end", l);
  response.setHeader("Content-Length", Long.toString(l));
  response.flushBuffer();
} else if(use_xsendfile) {
  // see above
} else {
  // stream files
}

What i got wrong at first was setting those attributes in the response. That didn’t work. The must be set in the request and you must take care setting all of those and with the correct type (String respectively long). And that’s it.

The request attribute org.apache.tomcat.sendfile.support will be true when the connector is configured to either use the APR connector (org.apache.coyote.http11.Http11AprProtocol) or the non blocking Java connector (org.apache.coyote.http11.Http11NioProtocol) (the later one being easier to deploy as it has no external dependencies).

CPU usage for sending those files is now nearly 0.

Filed in English posts, Java | Comments (0) »

Creating a better PathMatcher for Spring 3

Spring 3 has excellent support for mapping URLs to @Controller methods through the @RequestMapping annotation. This works quite well and i especially like the fact having the mapping right next to the method and not in some other config file like routes.rb.

My goal was to have urls like http://foobar.com/resource, http://foobar.com/resource.html, http://foobar.com/resource.zip etc. This is no problem at all thanks to the ContentNegotiatingViewResolver.

The solution has only one draw back: The format is not known to the controller. Yes, this shouldn’t be a controller concern in most cases but what if you have a format that you don’t want to be available to all users? Maybe an nice zip download of your resources? Handling authentication in a view? I don’t think so.

So my first attempt looked like this

@RequestMapping("/resource.{format}")
public String resource(
		final @PathVariable String format,
		final HttpServletRequest request,
		final Model model
)

That didn’t work because it wouldn’t work for the default text/html resource http://foobar.com/resource so i added

@RequestMapping("/resource")
public String resource(
		final Model model
) {
  this.resource('html', model);
}

That worked for http://foobar.com/resource but not for http://foobar.com/resource.zip… “format” was always html. Hmmm…

After much googling and reading StackOverflow.com i found the “useDefaultSuffixPattern” option on org.springframework.web.servlet.mvc.annotation.DefaultAnnotationHandlerMapping. If set to true (which is the default) the mapping “/resource/” will also map to “/resource/” and “/resource.*”. Although both useful i tried disabling it through my spring-cfg.xml like

<bean class="org.springframework.web.servlet.mvc.annotation.DefaultAnnotationHandlerMapping">
    	<property name="order" value="0" />
    	<property name="useDefaultSuffixPattern" value="false" />
</bean>

Enter the next problem: First, i didn’t work either. Second, all urls where mapped twice. Without the default suffix pattern and with. I spend 2 hours trying to locate the place where the spring config was loaded twice. In the end it was that one line that caused me trouble:

  <mvc:annotation-driven/>

That tag enables a lot of stuff in Spring 3, like the @Controller programming model and many other goodies. What it also does is establishing an AnnotationHandlerMapping that cannot be overwritten. So the next thing i did was browsing through the Spring sources to see what it does and redid with Spring beans in my config file (code follows later).

With that implemented, my urls still didn’t work, for all cases the URL without .{format} was called.

As i was already deep down in the Spring sources i had a look at the default path parser and matcher called AntPathMatcher. There is nothing wrong with the parsing code but the “getPatternComparator” method that “Given a full path, returns a Comparator suitable for sorting patterns in order of explicitness.” had some flaws, at least for my use case.

It sorts the patterns by explicitness and that explicitness is (among others) defined by how many placeholders for path variables are present. So my “/resource” is more explicit that “/resource.{format}”. With that in mind, i extend the path matcher like so:

This PathMatcher delegates most of his methods to the default AntPathMatcher but overwrites the getPatternComparator. If you have a look at the sources you’ll see that it is also partly copied. In the last else branch you’ll see that i sort both patterns by length, strip the default suffix (.*) and check wether the longer pattern starts with the other one. If it does i check wether the difference is just a .{format} (hardcoded). If that’s true, than the pattern with the format suffix is more explicit. Otherwise, i’ll use the default algorithm.

To get this to work, you cannot use the mvc:annotation-driven tag as the PathMatcher is a property of the AnnotationMappingHandler which in turn cannot be overwritten. So to get the same functionality like in Spring 3.0.5 with my PathMatcher use

As you can see i left the useDefaultSuffixPattern option enabled as it works very well with my PathMatcher and i didn’t want to care about mapping “/resource”, “/resource/” etc…

I really hope that the gists will save someone some time. I cannot imagine that i’m the only one having this kind of requirement. The solution is really simple but the way to it was not that easy.

Filed in English posts, Java, Webdevelopment | Comments (1) »

All roads lead to Rome…

…or: Adding Atom links to an RSS feed generated by ROME.

I’m using ROME to create RSS and Atom feeds for a project.

While ROME has excellent support for creating either RSS or Atom feeds, there is no build-in immediate support for Atom elements inside an RSS feed like

<atom:link href="http://example.com/example.rss" rel="self"/>

The first thing nowadays is to Google some keywords and finding someone who had the same problem. I found jettro and started with his solution but to me, the solution might work but is wrong.

Some points:

• The copyFrom method means exactly the opposite
• The generator doesn’t need to add the namespace
• I want to add more than a link some time

I’ve implemented an AtomContent class that holds a list of com.sun.syndication.feed.atom.Link but is easy extensible.

This content is managed by an AtomModule like so

public interface AtomModule extends Module {
	/** The public namespace URI */
	public final static String ATOM_10_URI = "http://www.w3.org/2005/Atom";
	/** as used in the namespaced prefixed with "atom" */
	public final static Namespace ATOM_NS = Namespace.getNamespace("atom", ATOM_10_URI);
 
	/** Gets the included content */
	public AtomContent getContent();
 
	/** Sets the included content */
	public void setContent(final AtomContent content);
}

This interface is accompanied by an implementation AtomModuleImpl that is used to provide instances of AtomContent and especially an AtomModuleGenerator that is used to generate XML elements via JDOM.

The generator implements ModuleGenerator and if added through a rome.properties files automatically adds the appropriate namespace to the feeds. The rome.properties looks like this:

rss_2.0.feed.ModuleGenerator.classes=ac.simons.syndication.modules.atom.AtomModuleGenerator
rss_2.0.item.ModuleGenerator.classes=ac.simons.syndication.modules.atom.AtomModuleGenerator

As you see, i’ll only added the generator to RSS 2.0. Also a parser is not available at the moment as i didn’t need one.

The following snippet demonstrates the usage of the AtomModule:

final AtomContent atomContent = new AtomContent();
atomContent.addLink(new SyndicationLink().withRel("self").withHref("http://example.com/example.rss").getLink());
atomContent.addLink(new SyndicationLink().withRel("alternate").withType("text/html").withHref("http://example.com/example.html").getLink());
feed.getModules().add(new AtomModuleImpl(atomContent));

I published the code as java-syndication along with some other helper classes. If i need more elements, i’ll add them, but in the meantime, the code is usable and maybe of some value for people building RSS feeds with Java and ROME.

Filed in English posts, Java | Comments (0) »

Server push, jquery atmosphere and the throbber of doom

I’m using the Atmosphere framework to implement server push (a.k.a. Comet) in a Spring 3 application.

Atmosphere includes a fine jQuery plugin to handle the client side code. The plugin works quite well and Jean-Francois, the author, has a good tutorial in his blog: Using Atmosphere’s JQuery Plug In to build applications supporting both Websocket and Comet.

The code works but causes webkit browsers (Chrome and Safari) to show the throbber (that little spinning thingy in tab or address bar). In iOS browsers that loading progress bar will not disappear.

I found some suggestions to get rid of the “throbber of doom” but none of them worked for me.

So here is my solution which is tested with jQuery 1.4.2 and jQuery.atmosphere 0.6.3:

<script>  	
	// See https://developer.mozilla.org/en/JavaScript/Reference/Global_Objects/Function/bind
	if(!Function.prototype.bind) {
		Function.prototype.bind = function( obj ) {
			var slice = [].slice,
			args = slice.call(arguments, 1), 
			self = this, 
			nop = function () {}, 
			bound = function () {
				return self.apply( this instanceof nop ? this : ( obj || {} ), args.concat( slice.call(arguments) ) );    
			};
 
			nop.prototype = self.prototype;
			bound.prototype = new nop();
			return bound;
		};
	}
 
	var subscribe = function() {		 	
		$.atmosphere.subscribe(
			'/subscription/url',
			function(response) {				
				$('body').append(response.responseBody + '<br>');						
			},
			$.atmosphere.request = {								
				transport: 'websocket',
				fallbackTransport : 'long-polling'
			}
		);
	};
 
	$(window).load(function() {				
		setTimeout(subscribe.bind(document),500);
	});
</script>

The trick is: Wait for the whole page to be loaded (including all resources) by using $(window).load instead of $(document).ready. Then set a timeout to your subscription function. The timeout will change the execution context of the function to the global object (in most cases window). This needs to be corrected to the document for the atmosphere jQuery plugin to work. Here comes bind() to the rescue. This is a Javascript 1.8.5 function and it is not available in all browsers (see bind) and therefore added to the prototype.

So far i’ve tested this in Firefox 3.6, Chrome 8, Safari 5 and Internet Explorer 8 and had no problems.

Filed in English posts | Comments (0) »