Java implementation of Excels statistical functions NORMINV

I was recently in need of a java implementation of the inverted normal distribution function, named “NORMINV” in Excel 2003 and later: NORMINV.

I only found a JavaScript implementation in kmpm’s repository norminv.js. It has quite a history, coming over C# from C++. I have the impression nobody wants to deal with the mathematics 😉

So here is my Java implementation of NORMINV. As an added bonus it implements org.apache.poi.ss.formula.functions.FreeRefFunction to be used as a User defined function in Apache POI:

import org.apache.poi.ss.formula.OperationEvaluationContext;
import org.apache.poi.ss.formula.eval.EvaluationException;
import org.apache.poi.ss.formula.eval.NumberEval;
import org.apache.poi.ss.formula.eval.OperandResolver;
import org.apache.poi.ss.formula.eval.ValueEval;
import org.apache.poi.ss.formula.functions.FreeRefFunction;
 
/**
 * Java Implementation of http://support.microsoft.com/kb/827358/en-us implementing 
 * Apache POIs {@link FreeRefFunction} to be used as a user defined function.
 * 
 * Mathematics found here: 
 * https://gist.github.com/kmpm/1211922/
 * https://gist.github.com/kmpm/1211922/raw/a11e0dfc9fab493bcdadc669f3213d11f1897ebf/norminv.js
 * 
 * Register for a given workbook with:
 * 
    <code>
		final UDFFinder udfs = new DefaultUDFFinder(new String[]{ "MS_NormInv" }, new FreeRefFunction[]{ new NormInv() }) ;        
		workbook.addToolPack(new AggregatingUDFFinder(new UDFFinder[] {udfs}));
    </code>
 * @author michael.simons
 */
public class NormInv implements FreeRefFunction {
	public ValueEval evaluate(ValueEval[] args, OperationEvaluationContext ec) {		
		try {
			final ValueEval p = OperandResolver.getSingleValue(args[0], ec.getRowIndex(), ec.getColumnIndex()) ;
			final ValueEval mu = OperandResolver.getSingleValue(args[1], ec.getRowIndex(), ec.getColumnIndex()) ;
			final ValueEval sigma = OperandResolver.getSingleValue(args[2], ec.getRowIndex(), ec.getColumnIndex()) ;			
			return new NumberEval(this.compute(OperandResolver.coerceValueToDouble(p), OperandResolver.coerceValueToDouble(mu), OperandResolver.coerceValueToDouble(sigma)));
		} catch (EvaluationException e) {
			return e.getErrorEval();
		}			 		
	}
 
	/**
	 * Original C++ implementation found at http://www.wilmott.com/messageview.cfm?catid=10&threadid=38771
	 * C# implementation found at http://weblogs.asp.net/esanchez/archive/2010/07/29/a-quick-and-dirty-implementation-of-excel-norminv-function-in-c.aspx
	 *    Compute the quantile function for the normal distribution.
	 *
	 *    For small to moderate probabilities, algorithm referenced
	 *    below is used to obtain an initial approximation which is
	 *    polished with a final Newton step.
	 *    For very large arguments, an algorithm of Wichura is used.
	 *
	 *  REFERENCE
	 *
	 *    Beasley, J. D. and S. G. Springer (1977).
	 *    Algorithm AS 111: The percentage points of the normal distribution,
	 *    Applied Statistics, 26, 118-121.
	 *
	 *     Wichura, M.J. (1988).
	 *     Algorithm AS 241: The Percentage Points of the Normal Distribution.
	 *     Applied Statistics, 37, 477-484.
	 * @param p
	 * @param mu
	 * @param sigma
	 * @return
	 */
	public double compute(double p, double mu, double sigma) {
		if(p < 0 || p > 1) 
			throw new RuntimeException("The probality p must be bigger than 0 and smaller than 1");		
		if(sigma < 0)
			throw new RuntimeException("The standard deviation sigma must be positive");
		if(p == 0)
			return Double.NEGATIVE_INFINITY;		
		if(p == 1)
			return Double.POSITIVE_INFINITY;		
		if(sigma == 0)
			return mu;		
		double  q, r, val;
 
		q = p - 0.5;
 
		/* 0.075 <= p <= 0.925 */
		if(Math.abs(q) <= .425) {
			r = .180625 - q * q;
			val =
		         q * (((((((r * 2509.0809287301226727 +
		                    33430.575583588128105) * r + 67265.770927008700853) * r +
		                  45921.953931549871457) * r + 13731.693765509461125) * r +
		                1971.5909503065514427) * r + 133.14166789178437745) * r +
		              3.387132872796366608)
		         / (((((((r * 5226.495278852854561 +
		                  28729.085735721942674) * r + 39307.89580009271061) * r +
		                21213.794301586595867) * r + 5394.1960214247511077) * r +
		              687.1870074920579083) * r + 42.313330701600911252) * r + 1);
		}
		/* closer than 0.075 from {0,1} boundary */
		else {
		     /* r = min(p, 1-p) < 0.075 */
			 if (q > 0) {
				 r = 1 - p;
			 } else {
				 r = p;
			 }
 
			 r = Math.sqrt(-Math.log(r));
			 /* r = sqrt(-log(r))  < ==>  min(p, 1-p) = exp( - r^2 ) */
 
			 if (r < = 5) { /* <==> min(p,1-p) >= exp(-25) ~= 1.3888e-11 */
				 r += -1.6;
				 val = (((((((r * 7.7454501427834140764e-4 +
		                     .0227238449892691845833) * r + .24178072517745061177) *
		                   r + 1.27045825245236838258) * r +
		                  3.64784832476320460504) * r + 5.7694972214606914055) *
		                r + 4.6303378461565452959) * r +
		               1.42343711074968357734)
		              / (((((((r *
		                       1.05075007164441684324e-9 + 5.475938084995344946e-4) *
		                      r + .0151986665636164571966) * r +
		                     .14810397642748007459) * r + .68976733498510000455) *
		                   r + 1.6763848301838038494) * r +
		                  2.05319162663775882187) * r + 1);
			 } else { /* very close to  0 or 1 */
				 r += -5;
				 val = (((((((r * 2.01033439929228813265e-7 +
		                     2.71155556874348757815e-5) * r +
		                    .0012426609473880784386) * r + .026532189526576123093) *
		                  r + .29656057182850489123) * r +
		                 1.7848265399172913358) * r + 5.4637849111641143699) *
		               r + 6.6579046435011037772)
		              / (((((((r *
		                       2.04426310338993978564e-15 + 1.4215117583164458887e-7) *
		                      r + 1.8463183175100546818e-5) * r +
		                     7.868691311456132591e-4) * r + .0148753612908506148525)
		                   * r + .13692988092273580531) * r +
		                  .59983220655588793769) * r + 1);
		      }
 
		      if (q < 0.0) {
		          val = -val;
		      }
		  }
 
		  return mu + sigma * val;		
	}
}

Read the complete article »

| Comments (7) »

21-Feb-13


mysqldump / mysql tips

Two tips to make your life with mysqldump easier and your backups better:

The following command creates a full backup of your mysql server including all databases and structures (including views and(!) stored procedures) along with the data:

mysqldump -uroot -proot --opt --routines --add-drop-database --default-character-set=utf8 --create-options --all-databases | \ 
bzip2 > backup.sql.bz2

You can also choose which databases to export with the –databases option:

mysqldump -uroot -proot --opt --routines --add-drop-database --default-character-set=utf8 --create-options --databases db1 db2 | \ 
bzip2 > backup.sql.bz2

But i prefer the complete version that can be used to restore the whole server including the mysql schema.

What if you want to restore only one database or transfer a database from this backup? Easy. mysql has a -D or –database parameter that is “default database to use” and -o respectively –one-database, the later one means “skip all sql commands that are not used while the default database is in use.

So to restore only db1 from the first backup just use

bzcat backup.sql.bz2 | mysql -uroot -proot -D db1 -o

and you’re done. Be aware that the db1 must exist when using this command, so you may want to create it as an empty database first.

To restore your whole server, just pipe the whole dump to mysql:

bzcat backup.sql.bz2 | mysql -uroot -proot

| Comments (2) »

14-Feb-13


Hidden Java gems: java.text.Normalizer

Java has a build-in java.text.Normalizer class to transform Unicode text into an equivalent composed or decomposed form. Dafuq?

The letter ‘Á’ can be represented in a composed form

U+00C1 LATIN CAPITAL LETTER A WITH ACUTE

and a decomposed form

U+0041    LATIN CAPITAL LETTER A
U+0301    COMBINING ACUTE ACCENT

Normalizer handles this for your:

import java.text.Normalizer;
import java.text.Normalizer.Form;
 
public class NormalizerExample {	
	public static void main(String[] args) {
		String s = Normalizer.normalize("Á", Form.NFD);
		System.out.println("Decomposed:");
		for(int i=0;i<s.length();++i)
			System.out.println(Integer.toHexString((int)s.charAt(i)));
		s = Normalizer.normalize(s, Form.NFC);
		System.out.println("Composed:");
		for(int i=0;i<s.length();++i)
			System.out.println(Integer.toHexString((int)s.charAt(i)));
	}
}

Output:

Decomposed:
41
301
Composed:
c1

Normalizer is available since JDK6.

What is this good for?

I use it to build nice slugs, seen here, like so:

String name = "Die Ärzte 2013!";
 
// Decompose unicode characters
String slug = Normalizer.normalize(name.toLowerCase(), Form.NFD)
// replace all combining diacritical marks and also everything that isn't a word or a whitespace character
	.replaceAll("\\p{InCombiningDiacriticalMarks}|[^\\w\\s]", "")
// replace all occurences of whitespaces or dashes with one single whitespace 
	.replaceAll("[\\s-]+", " ")
// trim the string
	.trim()
// and replace all blanks with a dash
	.replaceAll("\\s", "-");

| Comments (4) »

25-Jan-13


Schei� encoding: Java, MySQL and multi-byte UTF-8 support

UTF-8 has always been a multi-byte encoding but you probably had to handle only 2 byte (16bit) UTF-8 characters. With the raise of Emojis 4 byte characters rose as well so handling 4 byte UTF-8 characters is not only of interest for handling exotic languages but also for the needs of average users who want to post fancy smilies with their phones.

I won’t go into detail too much but only note some tips and caveats for supporting 4 byte UTF-8 characters in a Java / MySQL ecosystem. You’ll find the basic setup for your MySQL database, considerations about MySQL performance, connecting your Java program to the database and finally a little information about handling 4 byte UTF-8 strings in java:

Read the complete article »

| Comments (10) »

21-Jan-13


Add touch support to jQuery FancyBox 1.3.4

I’m using jQuery FancyBox on dailyfratze.de. I still use version 1.3.4 which is released under MIT and GPL license, if i remember correctly version 2 has not been always available under an open license but i’m maybe wrong. But nevertheless, version 2 also lacks touch support like 1.3.4 does.

This can be changed if you’re using Modernizr for detecting HTML5 and CSS3 features and also very small jQuery Touchwipe Plugin (there maybe more feature rich plugins but for the given goal, this is enough).

The change is pretty simple and the core of it is:

if(Modernizr.touch)
	content.touchwipe({
	     wipeLeft: function() { $.fancybox.next(); },
	     wipeRight: function() { $.fancybox.prev() },		     
	     min_move_x: 20,
	     min_move_y: 20,
	     preventDefaultEvents: true
	});

On wipe left show the next picture in a gallery, on wipe right the previous. There’s some more to disable the default click handlers, but i don’t want to bore you.

Here is a version of FancyBox already patched: jquery.fancybox-1.3.4.with_touch_support.js and here is a unified patch: jquery.fancybox-1.3.4.touch_support.patch.

Don’t forget to include modernizr with touch event support and jquery-touchwipe. To see how it works, visit dailyfratze.de and touch one of the the little magnifiers.

| Comments (4) »

12-Dec-12