Synchronizing Neo4j causal cluster bookmarks

Neo4j cluster is available in the Neo4j enterprise version and of course in Neo4j Aura.

A cluster provides causal consistency via the concept of bookmarks:

On executing a transaction, the client can ask for a bookmark which it then presents as a parameter to subsequent transactions. Using that bookmark the cluster can ensure that only servers which have processed the client’s bookmarked transaction will run its next transaction. This provides a causal chain which ensures correct read-after-write semantics from the client’s point of view.

When you use a Session from any of the official drivers directly (for example from the Neo4j Java Driver, or at least version 6 of Spring Data Neo4j, you hardly every see bookmarks yourself and there’s mostly no need to.

While Spring Data Neo4j 5.x still required some manual setup, SDN 6 does not need this anymore.

So in theory, all applications running against Neo4j should be fine in terms of “reading their own writes”.

But what about multiple instances of an application running against a cluster, for example to scale the application, too?

This does actually require some work. In the following example I refer to the Java driver, but the API of the other drivers and of course, the behavior of the server is identical.

When creating a new session the driver allows to pass in a collection or iterable of Bookmarks and not just a single object. Those bookmarks are then taken all to the cluster. As soon as the first transaction in this session starts, the routing will make sure that the requests go to a cluster member that has reached at last the latest transaction defined by the collection of bookmarks. There is no need to keep the bookmarks in order.. That allows us just collect new bookmarks and pass them on. We don’t have to do anything on the client side about sorting.

That work is done already internally by Spring Datas implementation of Bookmark managers.

But what can we do with that information to make sure multiple instances of the same application read the latest writes?

As soon as SDN becomes aware of a new bookmark, we must grab it and push it into exchange. A Redis pubsub topic, or a JMS queue configured as pubsub or even some Kafka configuration will do.

I have created to projects to demonstrate such a setup with both SDN 6 and the prior version, SDN5+OGM:

Common setup

Both projects require a locally running Redis instance. Please consult the Redis documentation for your OS or use a Docker container.

Conceptional, every messaging system that supports pubsub should do.

Spring Data Neo4j 6

The example projected is here: bookmark-sync-sdn6.

SDN 6 publishes bookmarks received from the driver as ApplicationEvents, so we can listen on them via ApplicationListener. For the other way round, the Neo4jTransactionManager can be seeded with a Supplier>.

The whole setup is as follows, please read the JavaDoc comments:

Spring Data Neo4j 5 + Neo4j-OGM

The example projected is here: bookmark-sync-sdn5.

In SDN5+OGM we can use the BookmarkManager interface provided by SDN5. We run a completely custom (and also better implementation) than the default one relying on Caffeine-Cache (which is not necessary at all).

The principle is the same, though: When new bookmarks are received via the transaction system, they will be published, new bookmarks received on the exchange will be the new seed. Please note, in SDN5 bookmark support must be enabled with @EnableBookmarkManagement

Note: Don’t combine SDN6 and SDN5 config! Those configurations above are for two separate example projects.

Photo by Chiara F on Unsplash

| Comments (1) »

12-Jul-21


What if Metallica went into Java programming?

Yesterday, Maciej shared this and I answered

after that, things escalated a bit… 🙂

While I thought, if Metallica would have been Spring developers, they would have written about Disposable Beans, not Heroes, other people have been more creative:

From Gerald came songs about memory leaks or the lack thereof: Creeping Death (Memory Leak) or The Memory Remains (no Memory Leak).
Java is a managed manage, so of course there’s the garbage collection with Harvester of Garbage. We have (at least 3 boolean values), which is Sad But True (Boxed Booleans).
Anyway, there’s seems to be a running thread… well, Until it Sleeps. If there are some weak references, they thought I Disappear.

Most important bit however is Coffee in the Jar.

Mario chimed in as well and pointed out to the One singleton, the less friendly garbage collector named Seek and Destroy. I would have said that this one pulls The Shortest Straw for you, but that one also applies to a Comparable. In the end, Nothing else matters anyway, while jitting the hot path.

Christoph being the awesome Spring developer he is, brought up of course The Master Of Beans as well as The Bean Within. Everything has a beginning and the end, and so have beans: Init & Destroy.

My favorite one however came from dear Eirini: The bean that should not be, which sums up some struggles of setting up some projects where nicely 🙂

The twitter thread made my day, some light heartened fun for a change.

Have some fun while listening to the songs above:

Featured image on the post from Wikimedia commons under a Creative Commons license.

| Comments (0) »

27-May-21


Releasing Maven based projects to Maven central

I published my first library on Maven central about 2013. A server side embedding tool for webpages based on OEmbed. I remember how happy and proud I was having published something in binary form “for all eternity”.

Maven central is the canonical, default artifact repository for the built tool of the same name, Apache Maven. Until configured otherwise, Maven tries to resolve dependencies from there.

The company behind sponsoring and running Maven central is Sonatype. Back in 2013 (and also 2018) account approval to release things to central was manual and involved resolving of ownership of the reverse DNS name. All of that is centered around preventing the hijacking of coordinates (read along here) to prevent people from tricking other people into using malicious software.

The whole process to be a producer starts here: The central repository: Producers. It’s explained in great detail.

Releasing to central also involves going through something called staging repositories and the process associated with it. it can be done through an UI (OSS Sonatype) or via plugins for Maven.

This week, another company, JFrog, announced that they are shutting down Bintray / JCenter. I was aware that Bintray and JCenter are around and are often used within Gradle projects. Apart from that, I only used one of JFrogs products, Artifactory, in a company as a local Maven central mirror and local deploy and release target.

People seem to have used JCenter and Bintray because the release process seems less strict and they found the Maven central way of doing things too hard. Other voices have been raised that Maven central is often slow. I cannot confirm the later, though.

I am writing down the following remarks to demonstrate that it is not that hard to publish your libraries on Maven central after getting across the initial setup.

First of all, read the above link “Producer” to get your coordinates registered for you.

I did not go through the UI of Sonatype for quite some time. The libraries I put myself onto central are all released via the Maven release plugin.

To get this up and running, your pom.xml has to fit some requirements. Especially, the meta data has to be complete. This should be the first step.

Further requirements are: Javadoc and sources must be present. The artifacts must not come in a snapshot form and they must not depend on anything not being on Maven central. Also, the artifacts must signed via GPG (See all of this here).

For my personal needs I have configured to a local release profile in my ~/.m2/settings.xml containing my username on oss.sonatype.org and encrypted password. This goes into the list of <servers> like this:

<server>
  <id>ossrh</id>
  <username>g.reizt</username>
  <password>XXXXX</password>
</server>

Also in settings.xml, the GPG credentials.

<profile>
    <id>gpg</id>
    <properties>
            <gpg.keyname>KEYNAME</gpg.keyname>
<!--        <gpg.passphrase>XXXX</gpg.passphrase> -->
<!-- Or better via an agent -->
    </properties>
</profile>

This turns out to be one of the hardest part to get right. I always have to look this up for CI or a new machine.

So, for the projects pom: Make sure you follow the requirements for the meta data and configure the necessary plugins for JavaDoc, sources and signature.

The libraries I put on central have basically all this information:

<build>
	<plugins>
		<plugin>
			<groupId>org.sonatype.plugins</groupId>
			<artifactId>nexus-staging-maven-plugin</artifactId>
			<version>${nexus-staging-maven-plugin.version}</version>
			<extensions>true</extensions>
			<configuration>
				<serverId>ossrh</serverId>
				<nexusUrl>https://oss.sonatype.org/</nexusUrl>
				<autoReleaseAfterClose>true</autoReleaseAfterClose>
			</configuration>
		</plugin>
		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-source-plugin</artifactId>
			<version>${maven-source-plugin.version}</version>
			<executions>
				<execution>
					<id>attach-sources</id>
					<goals>
						<goal>jar-no-fork</goal>
					</goals>
				</execution>
			</executions>
		</plugin>
		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-javadoc-plugin</artifactId>
			<executions>
				<execution>
					<id>attach-javadocs</id>
					<goals>
						<goal>jar</goal>
					</goals>
				</execution>
			</executions>
			<configuration>
				<detectOfflineLinks>false</detectOfflineLinks>
				<detectJavaApiLink>false</detectJavaApiLink>
				<source>${java.version}</source>
				<tags>
					<tag>
						<name>soundtrack</name>
						<placement>X</placement>
					</tag>
				</tags>
			</configuration>
		</plugin>
		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-release-plugin</artifactId>
			<version>${maven-release-plugin.version}</version>
			<configuration>
				<autoVersionSubmodules>true</autoVersionSubmodules>
				<useReleaseProfile>false</useReleaseProfile>
				<releaseProfiles>release</releaseProfiles>
				<tagNameFormat>@{project.version}</tagNameFormat>
				<goals>deploy</goals>
			</configuration>
		</plugin>
	</plugins>
</build>
 
<profiles>
	<profile>
		<id>release</id>
		<build>
			<plugins>
				<plugin>
					<groupId>org.apache.maven.plugins</groupId>
					<artifactId>maven-gpg-plugin</artifactId>
					<version>${maven-gpg-plugin.version}</version>
					<executions>
						<execution>
							<id>sign-artifacts</id>
							<phase>verify</phase>
							<goals>
								<goal>sign</goal>
							</goals>
						</execution>
					</executions>
				</plugin>
			</plugins>
		</build>
	</profile>
</profiles>

org.sonatype.plugins:nexus-staging-maven-plugin does all the heavy lifting behind the scenes as it hooks up to the release phases. I keep the gpg plugin in a separate profile so that users of my libraries are not pestered with it when they just want to build some stuff locally.

After all this is done, you can release your things in a two step process via `mvn release:prepare` followed by a `man release:perform`. The tooling will guide you through setting the current version and update your things to a new snapshot version.

I won’t go into the discussion whether the repeated tests runs are meaning full or not or whether not using the release plugin at all makes sense or not. I currently maintain projects than are run with CI friendly versions and released via other tooling to central and a couple of things released as described above.

| Comments (1) »

05-Feb-21


Do some puzzles sometimes

Wait, I here you say “This guy writing there, didn’t he write about not having time and energy for site projects?”:

Well, yes, sadly that’s the case and I did cancel a couple of things for good (BTW, I am looking for someone kind to take over the leadership of Aachens Java User Group EuregJUG, maybe there will be a time again for meetings).

On the other hand, I try not to go completely nuts and the last couple of weeks have not made this easy. It’s grey, cold (which I don’t even mind), but wet, wet, wet, muddy, muddy and then some: Running and especially cycling is a bit hard. Normally that would keep me on the track.

Instead, I started puzzle on Advent of Code again. I have a dedicated repository for my solutions, find it at michael-simons/aoc. Why more coding?

What I really like about the puzzles is the fact that they absolutely have nothing todo with frameworks, annotations, microservices, DDD platforms, build systems or moving JSON between endpoints. Just plain, logical puzzles to tinker with. A bit like doing a crossword thing each day.

In my repository, I tackled every puzzle with what’s available in a language, no libraries. Setup in a way that someone who want’s to run needs only to install one thing, the language. The repository contains Java (of course), Kotlin, Rust, Ruby, Go, SQL, Cypher, PHP, Scala, Typescript and some other things. I guess I managed to be more idiomatic in some languages than others, though. In most of the puzzles you’ll realize that you need various ideas, mathematical concepts, algorithms again and again. It’s helpful to compare how easy or hard is to implement those in various languages.

Up until this week I did run some circles around languages like Clojure or Lisp in general or things like Haskell. I find them intimidating at first sight and I don’t have a university background with knowledge about their theoretical concepts.

I started to read up on Clojure a bit and while I do not yet understand everything, I was able to create the following script

(def input (clojure.string/trim-newline (slurp "input.txt")))
 
(def freq (frequencies input))
(def starOne
    (- (get freq \() (get freq \))))
(println starOne)
 
(def starTwo
    (count
        (take-while (fn [p] (not= p -1))
            (reductions (fn [sum num] (+ sum num)) 0 (map {\( 1 \) -1} input)))))
(println starTwo)

It reads an input file into memory and uses the frequencies function to compute the frequencies of different characters in it. It assigns the difference of occurrences of `(` and `)` to the a variable named starOne and prints it.

The second part counts the number of iterations needed to map all opening brackets to 1 and closing brackets to -1 and summing them up (in several reductions) until one of them hits -1.

Many important things are already in there: Reading files, working with maps and lists, applying functions to every item in a list, calling functions and defining anonymous functions. I can work with that.

Fast forward a couple of days, having a look at I Was Told There Would Be No Math. Well, the math is actual super simple in that. Read a file with lines like 2x3x4. They give you the dimension of a parcel (length, width, height).

Compute according to some rules paper and ribbon needed to wrap those parcels or presents. Paper area is given by “find the surface area of the box, which is 2*l*w + 2*w*h + 2*h*l. The elves also need a little extra paper for each present: the area of the smallest side.”

The good object oriented person and the happy Java 15 user with preview feature I am I started to create a record to model that thing:

record Present(int l, int w, int h) {
    int surfaceArea() {
        return 2 * (l * w + w * h + h * l);
    }
 
    int slack() {
        return Math.min(l * w, Math.min(w * h, h * l));
    }
 
    int volume() {
        return l * w * h;
    }
 
}

I mean, basic math, right? Solution to the first question is just summing surface area and slack up like var starOne = presents.stream().mapToInt(p -> p.surfaceArea() + p.slack()).sum();. Easy, right?

Always thinking in objects primes you to things. Here to length, width, height. I should have realized what I am doing when I computed the smallest area (computed 3 areas and chose the smallest one): It doesn’t matter which value is assigned to length, width and height: I can just sort them, take the two smallest and multiple them. I realized that when I computed the smallest perimeter of the parcel:

int smallestPerimeter() {
    return Stream.of(l, w, h).sorted().limit(2).mapToInt(v -> 2 * v).sum();
}

Enter the Clojure solution. It felt very clumsy to define a type just for that.

Here’s what I came up with instead

(use '[clojure.string :only (trim-newline split-lines split)])
 
(def input 
    "I use again slurp to read the file and two library functions
     to trim the newlines and split the whole thing into a list. 
     An anonymous function is used on each line to split line by 
     the letter `x` and map the values to an int. Those ints are than
     sorted and the variable `input` will be a lazy list of int arrays."
    (map (fn [v] (sort (map bigint (split v #"x"))))
    (split-lines (trim-newline (slurp "input.txt")))))
 
(defn paper
    "As I know that the array is sorted, I can deconstruct it into the 
     3 values contained. The riddle for the paper is that the smallest area 
     is in there 3 and not 2 times. As the smallest area is defined by the first
     two elements, we multiple them 3 instead of 2 times like the rest."
  [dimensions]
  (let [[l w h] dimensions]
    (+ (* 3 l w) (* 2 l h) (* 2 w h))))
 
(defn ribbon  
    "Same idea as above: The smallest perimeter is defined by the 2 smallest values.
     The volume is of course the product of all 3."
  [dimensions]
  (let [[l w h] dimensions]
    (+ (* 2 l) (* 2 w) (* l w h))))
 
(println (reduce + (map paper input)))
(println (reduce + (map ribbon input)))

Find my prose inside the program.

I do like the approach not using to many data structures.

In the end: It is good to have a look outside the box sometimes and reset your brain with fresh ideas.

Thanks to Tim, Stefan and Jan for the multiple times in which you brought Clojure into my bubble.

Title picture by Bannon Morrissy on Unsplash

| Comments (1) »

03-Feb-21


Minecraft terminology for Java developers

My eldest kid – 11 at the time of writing – has been into Minecraft for some time now. I tried to motivate him on various ages to do some kinda programming with me. We tried Scratch, Lego Mindstorms (one of the few sets that is gathering dust) and a couple of other things. We didn’t have much fun with any of those. The nicest thing done together have been actually a couple of the Advent of Code challenges in which I tried to reduce to setup of things to a bare minimum (aka Texteditor).

I don’t blame the kid for failure at all… I am not a good teacher but what is worse, I have not much interest myself in tinkering with a game, never had (basically the same with regards to cycle, I prefer doing the thing actually), so I was quite happy that the kid himself wants to do stuff.

A bit of terminology

I was a bit surprised how many different kinds of Minecraft client and servers are out there:

First of all, there’s the Java based, “vanilla” server you can download here: original, “vanilla” Minecraft Java server. This edition of the server does not support custom plugins. It does however support Minecraft Forge. Forge is a modification loader for the vanilla server. You would program against the Forge API and that API encapsulates away the interaction with the server code.

Only the Minecraft Java edition client can connect against the Java server. This is a bit sad, as many friends of my kid would only have access to a gaming console. The Minecraft edition available on Switch, X-Box or Playstation does not support the Java server. They only connect to something called “Bedrock” edition.

Custom mods with MCreator

Back to Forge: Forge as the modloader must be installed into your Java Minecraft server via an installer that fits the server version. You get the installers here. After that you can pick and choose from a plethora of already existing mods at curseforge.

As far as I understood this, mods can also be installed client side only, but I don’t see the point of that when playing with multiple people on a server.

Anyway, we want to create our own mods. Mods can change many aspects of the games. They can include new commands, new recipes, blocks, bioms, creates, enchantments and more.

Adam Ness pointed me to MCreator. From the site: “MCreator is open source software used to make Minecraft Java Edition mods, Minecraft Bedrock Edition Add-Ons, and data packs using an intuitive easy-to-learn interface or with an integrated code editor.” This was exactly what I was looking for. A full fledged IDE but dedicated to Minecraft and the environment.

After download MCreator will setup a workspace for you. It uses Gradle behind the scenes to setup everything, including a development client and server. The nice thing here: You or your kid won’t need a Minecraft client license at that point. The documentation of the thing is stellar, it has an exhaustive Wiki and a good Knowledgebase.

MCreator contains tooling to package the mod and distribute it.

Custom plugins for the Spigot Minecraft server

I could stop here, as MCreator and the ability of mods to change a vanilla server is everything my kid was looking for, but for completeness and for fellow Java developers, here’s more.

There once was a fork of the Minecraft server called “Bukkit” but a little fluster cluck happened and now there’s Spigot. Spigot is something like a fork of the Minecraft server, but depends on the original binary (read the first link about to understand way). It does everything that the vanilla server does, but more. Especially it supports full fledged plugins.

To get started with developing plugins of any kind – btw, I know of one that uses Neo4j and Neo4j-OGM, yes, I helped the plugin author as part of my day job already – you first need to build the Spigot server. They don’t offer prebuild binaries due to the license mess (I found binaries at getbukkit.org but I am unsure about that being legit).

Building Spigot is pretty much trivial when you have a recent JDK installed on your machine. Grab the Spigot BuildTools.jar, save it somewhere and run in that directory java -jar BuildTools.jar to get a new server. I am unsure if you need to have Git installed for your machine (I have on all machines), but the build tools use it to clone a bunch of repositories.

After a while, you’ll find spigot-1.x.y.jar inside the same directory (at the time of writing, 1.16.4). This is your fresh Minecraft server supporting custom plugins.

How to write plugins? You can start of with a basic Maven project. The Spigot Plugin API lives under the following coordinates in the provided scope: org.spigotmc:spigot-api:1.16.4-R0.1-SNAPSHOT. They are not on central, so you would need to add the Spigot-Repo, too:

<repositories>
    <repository>
        <id>spigot-repo</id>
        <url>https://hub.spigotmc.org/nexus/content/repositories/snapshots/</url>
    </repository>
</repositories>
 
<dependencies>
    <dependency>
        <groupId>org.spigotmc:spigot-api:1.16.4-R0.1-SNAPSHOT</version>
        <scope>provided</scope>
    </dependency>
</dependencies>

The Spigot wiki gives more details.

The server version and naming a is a mess, not to speak about the licensing issues. But apart from that, I found the API pleasant to use: Spigot Java API docs. It gives full control about basically everything you can do in the game and I was able to create a simple plugin very fast.

To build the whole thing, you would need of course Maven or Gradle and some idea to set this up. I did install the Plugman plugin. This allows to load / unload / reload other plugins without restarting the server every time which cuts down on the feedback loop.

Summary

To sum this up: The various forks, versions, unclear naming of things regarded Minecraft is intimidating. People without much knowledge will eventually stumble from one YouTube “tutorial” to the next and download all sort of things in various qualities… After managing that initial step, things are not that bad and one can do neat things.

MCreator gives modders of all age great tooling without too much diversions in terms of installing things to express their minds about custom interactions in Minecraft. In a classroom situation or a scenario where a non-developer modern wants to try out things, I would recommend that.

Java developers will probably enjoy the Spigot API more. There’s even an IntelliJ plugin that creates full projects for you with the required setup.

| Comments (2) »

03-Jan-21