info.michael-simons.eu

From relational databases to databases with relations

In the summer of 2018, I joined Neo4j. This seems odd at first, my “love” for relational databases and SQL is known. I didn’t have this slide in my jOOQ presentation for two years now without reason. But: Looking at the jOOQ and SQL talks from the perspective of early 2000s, they also seemed odd at first. Back then, I never thought doing that much with databases. That changed a lot.

My experience is that all the data your deal with usually has a much longer lifetime than any of your applications sitting on top of that. Knowing one or more database management systems is essential. Being able to query them even more: What Neo4j and relational databases have in common: A great, declarative way to tell them which data to return and not how to return them. In the case of a relational database, this is obviously SQL, which had quite the renaissance for a few years now. Neo4j’s lingua franca is Cypher.

I get to play with Cypher a lot, but in the end this is not what I am working on at Neo4j. My work is focused on our Object Graph Mapper, Neo4j-OGM and the related Spring Data Module, Spring Data Neo4j. We have written about that a bit on medium. Given my experience with Spring, Spring Data and Spring Boot, the role suddenly makes much more sense.

People who entered the IT-conference circus may know the merry-go-round (or should I say “trap”?) of “talks stress me out a lot” – “hey, this is great, just enter another CfP”. I fell for it again and proposed a talk with the above title to several conferences. Now, I have to come up with something.

Back in 2016, when I first held my talk Database centric applications with Spring Boot and jOOQ, I started to write my story down. Back then, it helped me a lot, so here we go again. Join me on my way from relational databases to databases with relations.

Content

• Part 1: What are we talking about here? (This post)
• Part 2: How to get data into Neo4j?
• Part 3: Accessing data stored in Neo4j on the JVM
• Part 4: Modeling a domain with Spring Data Neo4j and OGM
• Part 5: Running interesting queries with ease

What are we talking about here?

The domain will be music. I have been tracking my musical habits for more than 10 years now at my side project Daily Fratze and I enjoy looking back to what I listened like this months but 5 years ago.

The guy on the left, Edgar F. Codd invented the relational model back in the 1970s. A relation in this model doesn’t describe a relation like the one between two people or a musician associated with a band who released a new track. A relation in the relational model is a table itself. Foreign keys between tables ensure referential integrity, but cannot define relations themselves. I put down some thoughts a while back in this German deck. This is what a relation looks like in a relational database:

One kind of sport to do with relational databases is the process of normalizing data. There are several normal forms. Their goal is to keep a database redundancy free, for several reasons. Back in the 1970, disk space being one of them. First normal form (1NF): All attributes should be atomic. 2NF is 1NF plus no functional dependencies on parts of any candidate keys. That is: There must not be a pair of attributes appearing twice in a relation’s tuples. 3NF forbids transitive dependencies (“Nothing but the key, so help me Codd”) and it gets complicated from there on. I have to say though, that normalization up to 3NF is still relevant today in a relational systems, at least if you’re a friend of (strong) consistent data.

However, as my colleague Rik van Bruggen points out in his book “Learning Neo4j”, relational databases are quite anti-relational.

Why is this? Relations in NF can be queried in many, many ways. Each query send via SQL returns a new relation, what’s the problem? It depends. In a strictly analytical use case, there’s often not a problem. Recreating object hierarchies however, joining things back together, is. A handful of joins is not hard to understand, even without a tool, but self referential joins or a sheer, huge amount, is. It also gets increasingly hard on the database management system.

This is where graphs can come into play. Graphs are another mathematical concept, this time from Graph theory. Sometimes
people call a chart graph by coincident, but this is wrong. A graph is a set of objects with pairs of objects being related. In mathematical terms, those objects are vertices and the relations between them, edges. We call them nodes and relationships in the Neo4j database.

Neo4j is a Property Graph. A property graph adds labels and properties to both nodes and relationships:

The above picture is from our excellent post What is a Graph Database?.

One takeaway from that post is: Neo4j is referred to as a native graph database because it efficiently implements the property graph model down to the storage level. Or in technical terms: Neo4j employs so called index-free adjacency, which is the most efficient means of processing data in a graph because connected nodes physically point to each other in the database. This obliterates the needs for complex joins, either directly or via intersection tables. One just can tell the database to retrieve all nodes connected to another node.

This is not only super nice for simple aggregations of things, but especially for many graph algorithms.

So what has this todo with my talk? My SQL talk was all about doing analytics. That is, retrieving data like in this image from a relational database with build-in analytic functions. Computing running totals, differences from previous windows and so on (read more here).

First of all I’m gonna analyze how to create a graph structure from the very same dataset I used in the SQL talk. There are different tools out there with different approaches. I’ve chosen a technique that resonated for various reasons with me. As I want to enrich the existing dataset, I’ll model a domain around it, with Java, putting Neo4j-OGM to use. I’ll show how Spring Data Neo4j helps me not having to deal with a lot of cruft. In the end, I’ll show that I can build my own music recommendation engine based on 10 years of tracking my musical habits by applying some of the queries and algorithms possible with Neo4j.

Next step: Loading data.

Pictures in this post: Codd and Table – Wikipedia, Property Graph – Neo4j, Featured image – Sarah Cervantes.

Filed in English posts, Java | Comments (3) »

Validate nested Transaction settings with Spring and Spring Boot

The Spring Framework has had an outstanding, declarative Transaction management for years now.
The configurable options maybe overwhelming at first, but important to accommodate many different scenarios.

Three of them stick out: propagation, isolation and to some lesser extend, read-only mode (more on that a bit later)

• propagation describes what happens if a transaction is to be opened inside the scope of an already existing transaction
• isolation determines among other whether one transaction can see uncommitted writes from another
• read-only can be used as a hint when user code only executes reads

I wrote “to some lesser extend” regarding read-only as read-only transactions can be a useful optimization in some cases, such as when you use Hibernate. Some underlying implementations treat them as hints only and don’t actually prevent writes. For a full description of things, have a look at the reference documentation on transaction strategies.

Note: A great discussion on how setting read-only to true can affect performance in a positive way with Spring 5.1 and Hibernate 5.3 can be find in the Spring Ticket SPR-16956.

Some of the transactions settings are contradicting in case of nested transaction scenarios. The documentation says:

By default, a participating transaction joins the characteristics of the outer scope, silently ignoring the local isolation level, timeout value, or read-only flag (if any).

This service here is broken in my perception. It explicitly declare a read-only transaction and than calls a save on a Spring Data repository:

import org.springframework.transaction.annotation.Transactional;
import org.springframework.stereotype.Service;
 
@Service
public class BrokenService {
	private final ThingRepository thingRepository;
 
	public BrokenService(ThingRepository thingRepository) {
		this.thingRepository = thingRepository;
	}
 
	@Transactional(readOnly = true)
	public ThingEntity tryToWriteInReadOnlyTx() {
		return this.thingRepository.save(new ThingEntity("A thing"));
	}
}

This can be detected by using a PlatformTransactionManager that supports validation of existing transactions. The JpaTransactionManager does this as well as Neo4js Neo4jTransactionManager (both extending AbstractPlatformTransactionManager).

To enable validation for JPA’s transaction manager in a Spring Boot based scenario, just make use of the provided PlatformTransactionManagerCustomizer interface. Spring Boots autoconfiguration calls them with the corresponding transaction manager:

import org.springframework.boot.autoconfigure.transaction.PlatformTransactionManagerCustomizer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.transaction.support.AbstractPlatformTransactionManager;
 
@Configuration
class TransactionManagerConfiguration {
 
	@Bean
	public PlatformTransactionManagerCustomizer<AbstractPlatformTransactionManager> transactionManagementConfigurer() {
		return (AbstractPlatformTransactionManager transactionManager) -> transactionManager
			.setValidateExistingTransaction(true);
	}
}

In the unlikely scenario you’re not using Spring Boot, you can always let Spring inject an EntityManagerFactory or for example Neo4j’s SessionFactory and create and configure the corresponding transaction manager yourself. My Neo4j tip does cover that as well.

If you try to execute the above service now, it’ll fail with an IllegalTransactionStateException indicating “save is not marked as read-only but existing transaction is”.

The question if a validation is possible arose in a discussion with customers. Funny enough, even working with Spring now for nearly 10 years, I never thought of that and always assumed it would validate those automatically but never tested it. Good to have learned something new, again.

Featured image on this post: Transaction by Nick Youngson CC BY-SA 3.0 Alpha Stock Images.

Filed in English posts | Comments (0) »

Donating to Médecins Sans Frontières (Ärzte ohne Grenzen)

Some weeks ago, my friends Judith and Christian, who write great Steampunk, Fantasy and in the recent time science fiction books for whom I wrote this little Kotlin app had a good idea:

And I was like:

My book Spring Boot 2 has been a bestseller this year and I still got quite a decent revenue from arc42 by example that I co-authored with Gernot Starke and Stefan Zörner.

Keeping my promises, here are the numbers: My share of royalties of the arc42 by example book has been 112,15€ from June to August. Gernot, Stefan and I split revenues, so that is only my share. Gernot himself already donates through Leanpubs causes. I don’t have numbers from my publisher for Spring Boot Book. I therefore decided to round this number to 250€:

Me and my family have been incredibly lucky the last years and I’ more than happy that one can give. We live in Germany and despite some idiots on various (social) media, it’s really fortunate to live here. Health care is working, social system as well. We don’t have war but clean water, food and everything. We should not forget that this is by far not self-evident for many, many people on this planet-

Filed in English posts | Comments (0) »

On becoming a Java Champion

July 2018 marks a personal highlight of mine. Just a bit after Rabea brought the news to our very own EuregJUG, the Java Champions account send this tweet out:

My name along this Java illuminaries. When I started this blog here more than twelve years ago, that was something I never even dreamed of. In 2006 the Java Champions program already existed but me, just been from university and vocal training for about 4 years or so, had no clue at all.

While getting my feet wet, I took inspiration from many of the people in the program, from their code, blog posts and talks. Not knowing that I would be working with them later in my life, even being direct colleagues with one of the founders like Eberhard Wolff. Even better: I’m lucky enough to call some of them my friends.

Java Champion was a long shot. I’m hopefully not the worst software engineer and architect out there, but I’m very far from knowing all the things. Quite the contrary. Did I know what a Java bridge method is until Gunnar brought this up on twitter? Do I know much about theoretical computer science? Hell no. I’m very sure that many people who I find very inspiring could forget more than I ever knew and still know more than I do.

So I must have done something else right and I’m happy with that. I just want to write some points down that might help others on their way:

On growth

Somewhen back in autumn 2014 I got Prokura in my company. I’m still not aware of an English word for that, but it means something along the lines that I can make and execute business decisions. My Prokura was only restricted in the way that I could not have closed the company.

Looking back, it was my bosses saying “we trust you to run this thing and also, this is our way of saying here, you’re explicitly technical lead, too.” Sadly, it didn’t come with a manual. At this point I was already more than 12 years at ENERKO INFORMATIK. I had grown in this time, but mostly on many technical levels. Things I learned include SQL, PL/SQL, XML, XSLT, Java (obviously), Spring (more obvious), we did Groovy at some point, not speaking of all the Swing based stuff I wrote and AFAIK ENERKO still runs an Oracle Database Dictionary based ORM I invented.

But could I lead a team? That was hard for me for several reasons. The company didn’t have had much fluctuation (and still hasn’t), so one did basically “grew up” with one another and it’s a weird situation if one person suddenly changes, either internally or externally.

Back then, I somewhen added the title line of a Nick Cave song to my personal site: “You’ve got to just, Keep on pushing, Keep on pushing, Push the sky away.” That has been my motto for quite some time.

I needed to grow beyond technical, “hard skills”. I tried and surprisingly, the feedback I got after leaving ENERKO INFORMATIK in 2017 was better than the impression I had of myself. I managed to hire two new engineers and they are still with the company which makes me quite happy.

In the end, I still felt I didn’t manage to achieve to anything. It’s weird how self perception and perception from others diverge. I left the company and I am working now at Neo4j, where I work in a small team with Gerrit Meier on Neo4j OGM and Spring Data and I couldn’t be happier with it.

Things come with a price

Most of the things “Spring” I learned on working with and on Daily Fratze, a personal photo site I’m running since 2005. I really love that stuff and I learned so much by developing it. But: It ruined my sleep in the days between 2010 and 2013 (2013 was the year my 2nd kid was born), it made parts of my family time with the my 1st kid very hard. I wanted to “finish” stuff and basically didn’t do anything else in my spare time.

Partially the same happened during late 2014 and early 2015, coincident with the events described unter “Growth” when I developed biking.michael-simons.eu and a lot more of Spring Boot related stuff inside my company. I was at home but I wasn’t there. I came back from the office, ate, and went into my office.

At some point, I needed to step back and also visited a doctor to help me to cope with sleep issues and dark moods.

We have a good family life most of the time, though. My wife always kept my back and I tried to be awake early every morning to be there for my kids. It’s by far not self evident that someone tolerates a partner that uses every minute awake when the kids are in their beds for coding, reading technical stuff and so on.

I gave a lot of talks since the end of 2015 with some success. The talks I enjoyed the most have been on Spring Boot and database related stuff. All the things I can explain in the middle of the night. However: I was and I still am super nervous at least a week before a talk. That feeling doesn’t seem to go away. Doing those talks is much more work for me than writing things (for example a book as explained in this post).

On mentoring

I was astonished that during the last 3 years or so many people came to me and thanked me for inspiration. That’s one of the reasons I’m trying to write this down here. Something like being a Java Champion or also success in general is most often not something that happens in a void, without help and support.

Everything I did and keep on doing: I would not be able to do this without having support in my life, a growing sense of what is good for my health (both mind and body) and without having had good mentors in my life.

There was my boss Rainer Barluschke at ENERKO INFORMATIK who taught me that there’s so much more than technical problems to solve. That it’s worth going a detour if the end result fits. Who even introduced me to some topics that seems to be more of the esoteric kind back than, like spiritual growth.

My ex-colleague Silke Böhler, who challenged me in JavaLand 2015 with some good food for thoughts and later on with a line “work is fun, but has to be taken seriously anyway…”. Apart from that: It has been the little things that last and helped along the way.

I already mentioned the support of my family, but also a kid can be a mentor. It’s hard to describe, but having someone near me that most of the time is in a good mood compared to myself, helps on focussing and accentuating the good things.

Summing things up…

Don’t give up trying to reach your goals because other peoples success seems to be so easily achieved. In the end, people of a group engage in the same game, but start with different preconditions. Look for opportunities where you are

• Allowed to learn
• Be able to fulfill a meaningful task, with all given due diligence and seriousness
• Be part of a team, IT is not a single players sport

And also

• Find a good mentor
• Become a mentor: Pass on what you learned
• Keep interest in other things outside your job… Not everything is related to IT

Right now, I feel at peace with myself for the first time in about 5 years. Going over the midlife crises? Who knows… I’m thankful that I actually could push my sky away by magnitudes and the last year will be a year I will always remember.

For the near future I’m super thrilled to work on cool stuff with Neo4j and Spring Data, with having the latest release of Neo4j OGM 3.1.1 just out of the door and see what happens next.

I made some innuendo to some people (Ralf 😉) that I do have ideas for a next book and as a spoiler: If put this together, I’ll try to bring this post here, something along that one and some other ideas into a form that might be worth reading for more people.

Filed in English posts | Comments (0) »

Spring Boots Configuration Metadata with Kotlin

Last week I decided to raffle a copy of my book (see Twitter) and I wrote a small Spring Boot Command Line Runner to raffle the retweeting winner as one does (see raffle-by-retweet, feel free to reuse this).

I wrote the application in Kotlin. Notice the use of @ConfigurationProperties in my application:

The lateinit attributes are not as nice as I want them to be, but I heard support for data-classes is coming. Anyway. A super useful thing with those configuration property classes are the metadata that can be generated for your IDE of choice, see Configuration Metadata. In a Java application it’s enough to add org.springframework.boot:spring-boot-configuration-processor as compile time dependency respectively as annotationProcessor dependency in a Gradle build.

For Kotlin, you have to use the kotlin-kapt-plugin. It takes care of annotation processing in Kotlin and Spring Boots annotation processor has to be declared in its scope like this:

apply plugin: 'kotlin-kapt'
 
dependencies {
    // Other dependencies omitted
 
    kapt("org.springframework.boot:spring-boot-configuration-processor")
}

To make IDEA recognize the generated sources (and also use the same folders for classes as the Gradle build), you can this as well:

apply plugin: 'idea'
 
idea {
    module {
        def kaptMain = file("${project.buildDir}/generated/source/kapt/main")
        sourceDirs += kaptMain
        generatedSourceDirs += kaptMain
 
        outputDir file("${project.buildDir}/classes/main")
        testOutputDir file("${project.buildDir}/classes/test")
    }
}

Find the full build script here.

Filed in English posts | Comments (0) »