info.michael-simons.eu

Maven snippets

This post will contain some Maven snippets that i use and forget regularly (similar to my Git Post).

Download the sources of all dependencies of a project if available:

mvn dependency:sources

Download the JavaDoc for all dependencies if available:

mvn dependency:resolve -Dclassifier=javadoc -Dtype=jar

Check for updates of all direct dependencies:

mvn versions:display-dependency-updates

Skip signing of artifacts (i.e. for a local install of a snapshot)

mvn -Dgpg.skip=true install

If you need to branch from a tag and fix some bugs, i recommend to checkout the tag and then use

mvn release:branch -DbranchName=my-branch -DupdateBranchVersions=true -DupdateWorkingCopyVersions=false

This creates a new branch “my-branch” and updates the version to x.x.x-SNAPSHOT in the branch and leaves the working copy as is.

Run single tests or single methods in tests: Nice, official documentation right here

If you want to run single (integration) tests but the build fails because you didn’t run unit tests, add

-DfailIfNoTests=false

Execute a goal of a single, configured plugin with a given execution id:

mvn asciidoctor:process-asciidoc@generate-docs

List goals by the order they will execute

mvn fr.jcgay.maven.plugins:buildplan-maven-plugin:list

Group goals by phase

mvn fr.jcgay.maven.plugins:buildplan-maven-plugin:list-phase

Group goals by plugin

mvn fr.jcgay.maven.plugins:buildplan-maven-plugin:list-plugin

Last update: 2018/01/09

Filed in English posts, Java | Comments (0) »

26-Feb-13

Java implementation of Excels statistical functions NORMINV

I was recently in need of a java implementation of the inverted normal distribution function, named “NORMINV” in Excel 2003 and later: NORMINV.

I only found a JavaScript implementation in kmpm’s repository norminv.js. It has quite a history, coming over C# from C++. I have the impression nobody wants to deal with the mathematics 😉

So here is my Java implementation of NORMINV. As an added bonus it implements org.apache.poi.ss.formula.functions.FreeRefFunction to be used as a User defined function in Apache POI:

import org.apache.poi.ss.formula.OperationEvaluationContext;
import org.apache.poi.ss.formula.eval.EvaluationException;
import org.apache.poi.ss.formula.eval.NumberEval;
import org.apache.poi.ss.formula.eval.OperandResolver;
import org.apache.poi.ss.formula.eval.ValueEval;
import org.apache.poi.ss.formula.functions.FreeRefFunction;
 
/**
 * Java Implementation of http://support.microsoft.com/kb/827358/en-us implementing 
 * Apache POIs {@link FreeRefFunction} to be used as a user defined function.
 * 
 * Mathematics found here: 
 * https://gist.github.com/kmpm/1211922/
 * https://gist.github.com/kmpm/1211922/raw/a11e0dfc9fab493bcdadc669f3213d11f1897ebf/norminv.js
 * 
 * Register for a given workbook with:
 * 
    <code>
		final UDFFinder udfs = new DefaultUDFFinder(new String[]{ "MS_NormInv" }, new FreeRefFunction[]{ new NormInv() }) ;        
		workbook.addToolPack(new AggregatingUDFFinder(new UDFFinder[] {udfs}));
    </code>
 * @author michael.simons
 */
public class NormInv implements FreeRefFunction {
	public ValueEval evaluate(ValueEval[] args, OperationEvaluationContext ec) {		
		try {
			final ValueEval p = OperandResolver.getSingleValue(args[0], ec.getRowIndex(), ec.getColumnIndex()) ;
			final ValueEval mu = OperandResolver.getSingleValue(args[1], ec.getRowIndex(), ec.getColumnIndex()) ;
			final ValueEval sigma = OperandResolver.getSingleValue(args[2], ec.getRowIndex(), ec.getColumnIndex()) ;			
			return new NumberEval(this.compute(OperandResolver.coerceValueToDouble(p), OperandResolver.coerceValueToDouble(mu), OperandResolver.coerceValueToDouble(sigma)));
		} catch (EvaluationException e) {
			return e.getErrorEval();
		}			 		
	}
 
	/**
	 * Original C++ implementation found at http://www.wilmott.com/messageview.cfm?catid=10&threadid=38771
	 * C# implementation found at http://weblogs.asp.net/esanchez/archive/2010/07/29/a-quick-and-dirty-implementation-of-excel-norminv-function-in-c.aspx
	 *    Compute the quantile function for the normal distribution.
	 *
	 *    For small to moderate probabilities, algorithm referenced
	 *    below is used to obtain an initial approximation which is
	 *    polished with a final Newton step.
	 *    For very large arguments, an algorithm of Wichura is used.
	 *
	 *  REFERENCE
	 *
	 *    Beasley, J. D. and S. G. Springer (1977).
	 *    Algorithm AS 111: The percentage points of the normal distribution,
	 *    Applied Statistics, 26, 118-121.
	 *
	 *     Wichura, M.J. (1988).
	 *     Algorithm AS 241: The Percentage Points of the Normal Distribution.
	 *     Applied Statistics, 37, 477-484.
	 * @param p
	 * @param mu
	 * @param sigma
	 * @return
	 */
	public double compute(double p, double mu, double sigma) {
		if(p < 0 || p > 1) 
			throw new RuntimeException("The probality p must be bigger than 0 and smaller than 1");		
		if(sigma < 0)
			throw new RuntimeException("The standard deviation sigma must be positive");
		if(p == 0)
			return Double.NEGATIVE_INFINITY;		
		if(p == 1)
			return Double.POSITIVE_INFINITY;		
		if(sigma == 0)
			return mu;		
		double  q, r, val;
 
		q = p - 0.5;
 
		/* 0.075 <= p <= 0.925 */
		if(Math.abs(q) <= .425) {
			r = .180625 - q * q;
			val =
		         q * (((((((r * 2509.0809287301226727 +
		                    33430.575583588128105) * r + 67265.770927008700853) * r +
		                  45921.953931549871457) * r + 13731.693765509461125) * r +
		                1971.5909503065514427) * r + 133.14166789178437745) * r +
		              3.387132872796366608)
		         / (((((((r * 5226.495278852854561 +
		                  28729.085735721942674) * r + 39307.89580009271061) * r +
		                21213.794301586595867) * r + 5394.1960214247511077) * r +
		              687.1870074920579083) * r + 42.313330701600911252) * r + 1);
		}
		/* closer than 0.075 from {0,1} boundary */
		else {
		     /* r = min(p, 1-p) < 0.075 */
			 if (q > 0) {
				 r = 1 - p;
			 } else {
				 r = p;
			 }
 
			 r = Math.sqrt(-Math.log(r));
			 /* r = sqrt(-log(r))  < ==>  min(p, 1-p) = exp( - r^2 ) */
 
			 if (r < = 5) { /* <==> min(p,1-p) >= exp(-25) ~= 1.3888e-11 */
				 r += -1.6;
				 val = (((((((r * 7.7454501427834140764e-4 +
		                     .0227238449892691845833) * r + .24178072517745061177) *
		                   r + 1.27045825245236838258) * r +
		                  3.64784832476320460504) * r + 5.7694972214606914055) *
		                r + 4.6303378461565452959) * r +
		               1.42343711074968357734)
		              / (((((((r *
		                       1.05075007164441684324e-9 + 5.475938084995344946e-4) *
		                      r + .0151986665636164571966) * r +
		                     .14810397642748007459) * r + .68976733498510000455) *
		                   r + 1.6763848301838038494) * r +
		                  2.05319162663775882187) * r + 1);
			 } else { /* very close to  0 or 1 */
				 r += -5;
				 val = (((((((r * 2.01033439929228813265e-7 +
		                     2.71155556874348757815e-5) * r +
		                    .0012426609473880784386) * r + .026532189526576123093) *
		                  r + .29656057182850489123) * r +
		                 1.7848265399172913358) * r + 5.4637849111641143699) *
		               r + 6.6579046435011037772)
		              / (((((((r *
		                       2.04426310338993978564e-15 + 1.4215117583164458887e-7) *
		                      r + 1.8463183175100546818e-5) * r +
		                     7.868691311456132591e-4) * r + .0148753612908506148525)
		                   * r + .13692988092273580531) * r +
		                  .59983220655588793769) * r + 1);
		      }
 
		      if (q < 0.0) {
		          val = -val;
		      }
		  }
 
		  return mu + sigma * val;		
	}
}

import org.apache.poi.ss.formula.OperationEvaluationContext; import org.apache.poi.ss.formula.eval.EvaluationException; import org.apache.poi.ss.formula.eval.NumberEval; import org.apache.poi.ss.formula.eval.OperandResolver; import org.apache.poi.ss.formula.eval.ValueEval; import org.apache.poi.ss.formula.functions.FreeRefFunction; /** * Java Implementation of http://support.microsoft.com/kb/827358/en-us implementing * Apache POIs {@link FreeRefFunction} to be used as a user defined function. * * Mathematics found here: * https://gist.github.com/kmpm/1211922/ * https://gist.github.com/kmpm/1211922/raw/a11e0dfc9fab493bcdadc669f3213d11f1897ebf/norminv.js * * Register for a given workbook with: * <code> final UDFFinder udfs = new DefaultUDFFinder(new String[]{ "MS_NormInv" }, new FreeRefFunction[]{ new NormInv() }) ; workbook.addToolPack(new AggregatingUDFFinder(new UDFFinder[] {udfs})); </code> * @author michael.simons */ public class NormInv implements FreeRefFunction { public ValueEval evaluate(ValueEval[] args, OperationEvaluationContext ec) { try { final ValueEval p = OperandResolver.getSingleValue(args[0], ec.getRowIndex(), ec.getColumnIndex()) ; final ValueEval mu = OperandResolver.getSingleValue(args[1], ec.getRowIndex(), ec.getColumnIndex()) ; final ValueEval sigma = OperandResolver.getSingleValue(args[2], ec.getRowIndex(), ec.getColumnIndex()) ; return new NumberEval(this.compute(OperandResolver.coerceValueToDouble(p), OperandResolver.coerceValueToDouble(mu), OperandResolver.coerceValueToDouble(sigma))); } catch (EvaluationException e) { return e.getErrorEval(); } } /** * Original C++ implementation found at http://www.wilmott.com/messageview.cfm?catid=10&threadid=38771 * C# implementation found at http://weblogs.asp.net/esanchez/archive/2010/07/29/a-quick-and-dirty-implementation-of-excel-norminv-function-in-c.aspx * Compute the quantile function for the normal distribution. * * For small to moderate probabilities, algorithm referenced * below is used to obtain an initial approximation which is * polished with a final Newton step. * For very large arguments, an algorithm of Wichura is used. * * REFERENCE * * Beasley, J. D. and S. G. Springer (1977). * Algorithm AS 111: The percentage points of the normal distribution, * Applied Statistics, 26, 118-121. * * Wichura, M.J. (1988). * Algorithm AS 241: The Percentage Points of the Normal Distribution. * Applied Statistics, 37, 477-484. * @param p * @param mu * @param sigma * @return */ public double compute(double p, double mu, double sigma) { if(p < 0 || p > 1) throw new RuntimeException("The probality p must be bigger than 0 and smaller than 1"); if(sigma < 0) throw new RuntimeException("The standard deviation sigma must be positive"); if(p == 0) return Double.NEGATIVE_INFINITY; if(p == 1) return Double.POSITIVE_INFINITY; if(sigma == 0) return mu; double q, r, val; q = p - 0.5; /* 0.075 <= p <= 0.925 */ if(Math.abs(q) <= .425) { r = .180625 - q * q; val = q * (((((((r * 2509.0809287301226727 + 33430.575583588128105) * r + 67265.770927008700853) * r + 45921.953931549871457) * r + 13731.693765509461125) * r + 1971.5909503065514427) * r + 133.14166789178437745) * r + 3.387132872796366608) / (((((((r * 5226.495278852854561 + 28729.085735721942674) * r + 39307.89580009271061) * r + 21213.794301586595867) * r + 5394.1960214247511077) * r + 687.1870074920579083) * r + 42.313330701600911252) * r + 1); } /* closer than 0.075 from {0,1} boundary */ else { /* r = min(p, 1-p) < 0.075 */ if (q > 0) { r = 1 - p; } else { r = p; } r = Math.sqrt(-Math.log(r)); /* r = sqrt(-log(r)) < ==> min(p, 1-p) = exp( - r^2 ) */ if (r < = 5) { /* <==> min(p,1-p) >= exp(-25) ~= 1.3888e-11 */ r += -1.6; val = (((((((r * 7.7454501427834140764e-4 + .0227238449892691845833) * r + .24178072517745061177) * r + 1.27045825245236838258) * r + 3.64784832476320460504) * r + 5.7694972214606914055) * r + 4.6303378461565452959) * r + 1.42343711074968357734) / (((((((r * 1.05075007164441684324e-9 + 5.475938084995344946e-4) * r + .0151986665636164571966) * r + .14810397642748007459) * r + .68976733498510000455) * r + 1.6763848301838038494) * r + 2.05319162663775882187) * r + 1); } else { /* very close to 0 or 1 */ r += -5; val = (((((((r * 2.01033439929228813265e-7 + 2.71155556874348757815e-5) * r + .0012426609473880784386) * r + .026532189526576123093) * r + .29656057182850489123) * r + 1.7848265399172913358) * r + 5.4637849111641143699) * r + 6.6579046435011037772) / (((((((r * 2.04426310338993978564e-15 + 1.4215117583164458887e-7) * r + 1.8463183175100546818e-5) * r + 7.868691311456132591e-4) * r + .0148753612908506148525) * r + .13692988092273580531) * r + .59983220655588793769) * r + 1); } if (q < 0.0) { val = -val; } } return mu + sigma * val; } }

Read the complete article »

Filed in English posts, Java, Oracle | Comments (7) »

21-Feb-13

mysqldump / mysql tips

Two tips to make your life with mysqldump easier and your backups better:

The following command creates a full backup of your mysql server including all databases and structures (including views and(!) stored procedures) along with the data:

mysqldump -uroot -proot --opt --routines --add-drop-database --default-character-set=utf8 --create-options --all-databases | \ 
bzip2 > backup.sql.bz2

You can also choose which databases to export with the –databases option:

mysqldump -uroot -proot --opt --routines --add-drop-database --default-character-set=utf8 --create-options --databases db1 db2 | \ 
bzip2 > backup.sql.bz2

But i prefer the complete version that can be used to restore the whole server including the mysql schema.

What if you want to restore only one database or transfer a database from this backup? Easy. mysql has a -D or –database parameter that is “default database to use” and -o respectively –one-database, the later one means “skip all sql commands that are not used while the default database is in use.

So to restore only db1 from the first backup just use

bzcat backup.sql.bz2 | mysql -uroot -proot -D db1 -o

and you’re done. Be aware that the db1 must exist when using this command, so you may want to create it as an empty database first.

To restore your whole server, just pipe the whole dump to mysql:

bzcat backup.sql.bz2 | mysql -uroot -proot

Filed in English posts | Comments (2) »

14-Feb-13

Hidden Java gems: java.text.Normalizer

Java has a build-in java.text.Normalizer class to transform Unicode text into an equivalent composed or decomposed form. Dafuq?

The letter ‘Á’ can be represented in a composed form

U+00C1 LATIN CAPITAL LETTER A WITH ACUTE

and a decomposed form

U+0041    LATIN CAPITAL LETTER A
U+0301    COMBINING ACUTE ACCENT

Normalizer handles this for your:

import java.text.Normalizer;
import java.text.Normalizer.Form;
 
public class NormalizerExample {	
	public static void main(String[] args) {
		String s = Normalizer.normalize("Á", Form.NFD);
		System.out.println("Decomposed:");
		for(int i=0;i<s.length();++i)
			System.out.println(Integer.toHexString((int)s.charAt(i)));
		s = Normalizer.normalize(s, Form.NFC);
		System.out.println("Composed:");
		for(int i=0;i<s.length();++i)
			System.out.println(Integer.toHexString((int)s.charAt(i)));
	}
}

Output:

Decomposed:
41
301
Composed:
c1

Normalizer is available since JDK6.

What is this good for?

I use it to build nice slugs, seen here, like so:

String name = "Die Ärzte 2013!";
 
// Decompose unicode characters
String slug = Normalizer.normalize(name.toLowerCase(), Form.NFD)
// replace all combining diacritical marks and also everything that isn't a word or a whitespace character
	.replaceAll("\\p{InCombiningDiacriticalMarks}|[^\\w\\s]", "")
// replace all occurences of whitespaces or dashes with one single whitespace 
	.replaceAll("[\\s-]+", " ")
// trim the string
	.trim()
// and replace all blanks with a dash
	.replaceAll("\\s", "-");

Filed in English posts, Java | Comments (4) »

25-Jan-13

Schei� encoding: Java, MySQL and multi-byte UTF-8 support

UTF-8 has always been a multi-byte encoding but you probably had to handle only 2 byte (16bit) UTF-8 characters. With the raise of Emojis 4 byte characters rose as well so handling 4 byte UTF-8 characters is not only of interest for handling exotic languages but also for the needs of average users who want to post fancy smilies with their phones.

I won’t go into detail too much but only note some tips and caveats for supporting 4 byte UTF-8 characters in a Java / MySQL ecosystem. You’ll find the basic setup for your MySQL database, considerations about MySQL performance, connecting your Java program to the database and finally a little information about handling 4 byte UTF-8 strings in java:

Read the complete article »

Filed in English posts, Java | Comments (10) »

21-Jan-13