GraalVM on macOS

I have been doing some research with GraalVM and I noticed that RC14 brings an Info.plist inside it’s contents folder. That’s a nice thing to have, as it automatically registers it if you unzip it into /Library/Java/JavaVirtualMachines as a JDK 8 implementation, but a nasty surprise when using /usr/libexec/java_home -v1.8 to configure your JAVA_HOME, which now point to the GraalVM. Probably it’s not much of a problem, but I stumbled upon it as one of my favourite tools, jQAssistant suddenly stopped working.

The fix is easy and I described it already for JDK 9 ea years ago:

Just delete or rename the info.plist file:

export GRAALVM_HOME=/Library/Java/JavaVirtualMachines/graalvm-ce-1.0.0-rc14/Contents/Home
sudo mv $GRAALVM_HOME/../Info.plist $GRAALVM_HOME/../Info.plist.bak

This way you can have a standard JDK 8 having around that is selected via a standard java_home call and explicitly using GraalVM.

| Comments (0) »

27-Mar-19


Using Thymeleaf inline JavaScript with D3.js

I gave this talk several times now and every time I got asked how I created this visualization, clustering the genres I’m listening to by decade and country:



I just did a super spontaneous video for that. Key take away: Just use server side rendered HTML through Thymeleaf, obviously running on Spring Boot, render the content model as inlined Javascript and be good to go. Want an API later on?

Structure your Spring MVC controller and your service or repository layer in a way that makes it actually reusable.

Anyway, here’s the short clip and below the relevant code snippets:

Spring MVC controller, GenresController.java, either returning a String, denoting the template to be rendered, or the Model as just the response body:

@GetMapping(value = { "", "/" }, produces = MediaType.TEXT_HTML_VALUE)
public ModelAndView genres() {
 
	var model = genresModel();
	return new ModelAndView("genres", model);
}
 
@GetMapping(value = { "", "/" }, produces = MediaType.APPLICATION_JSON_UTF8_VALUE)
@ResponseBody
public Map<String, Object> genresModel() {
 
	var genres = this.genreRepository.findAll(Sort.by("name").ascending());
	var allSubgrenes = genreRepository.findAllSubgrenes();
	var top10Subgenres = allSubgrenes.stream().sorted(comparingLong(Subgenre::getFrequency).reversed()).limit(10).collect(toList());
	return Map.of(
			"genres", genres,
			"subgenres", allSubgrenes,
			"top10Subgenres", top10Subgenres
	);
}

The Thymeleaf template genres.html. The standard controller above renders it when / is requested, accepting text/html.

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:layout="http://www.ultraq.net.nz/thymeleaf/layout"
      xmlns:th="http://www.thymeleaf.org"
      layout:decorate="~{layout}">
<head>
    <title th:text="#{genres}">Placeholder</title>
    <script th:inline="javascript">
        var subgenres = /*[[${subgenres}]]*/ {};
    </script>
</head>
<body>
<!-- Some more content -->
</body>
</html>

Calling this thing application.js is way too much. It’s a simple function, reading in the global subgenres variable and displays it as a nice D3.js chart.

Remember: This is also no silver bullet and I absolutely don’t have anything agains todays Javascript development. I just feel more at home near a database.

| Comments (0) »

31-Jan-19


Documentation as code, code as documentation

This morning, I read this kind tweet by Raumzeitfalle:

This refers to arc42 by example. There will be a second edition of this book soon, with new contributions from Gernot and new, Ralf.

My example in this book, biking2 is still up to date. I used this now several times at customers and every time, I also recommended my approach of generating the documentation as part of the build process. You see the live output here.

What makes those special are the following facts:

  • The docs are written in Asciidoctor, see: src/docs.
  • I’m using the Asciidoctor Maven plugin. That means they are an essential part of the build and aren’t left to die in a Confluence or whatever.
  • Even more, I use maven-resources-plugin to copy them into the output dir from which they are put into the final artifact.
  • While the documentation is strictly structured by the ideas of arc42, here come’s the fun parts:

    • I’m using jQAssistant to continuously verify my personal quality goals with that project: jQAssistant is a QA tool, which allows the definition and validation of project specific rules on a structural level. It is built upon the graph database Neo4j.
      jQAssistant integrates also in your build process. It first analyzes your sources and dependencies and writes every information it can discover through various plugins into an embedded Neo4j instance. That is the scan step. In the following analyze step, it verifies the data agains build-in or custom rules. You write custom rules as Cypher queries. Now the interesting fact: Those concept and rules can be written in Asciidoctor as well. This one, concepts_structure.adoc, declares my concept of a config and support package. Those concepts are executed in Neo4j and add labels to certain nodes. The label is then used in this rule (structure.adoc):

      MATCH (a:Main:Artifact)
      MATCH (a) -[:CONTAINS]-> (p1:Package) -[:DEPENDS_ON]-> (p2:Package) <-[:CONTAINS]- (a)
      WHERE not p1:Config
        and not (p1) -[:CONTAINS]-> (p2)
        and not p2:Support
        and not p1.fqn = 'ac.simons.biking2.summary'
      RETURN p1, p2

      “Give me all the packages of the main artifact but the config and the summary package that depend on other packages that are not contained in the package itself.” If that query returns an entry, the rule is violated. I use that rule to enforce horizontal, domain specific slices.

      Now, the very same, executable rule becomes part of my documentation (see building block view) by just including them. Isn’t that great?

    • I also include specific information from my class files in the docs. Asciidoctor allows including dedicated part of all text files. For example, I use package-info.java files like this directly in the docs: “bikes”. I did this to a much bigger extend in this repo, find the linked articles there. I love Asciidoctor.
    • Last but not least, I use Spring REST docs in my unit tests. Spring REST docs combines hand-written documentation written with Asciidoctor and auto-generated snippets produced with Spring MVC Test. A simple unit test like this gets instrumented by a call to document like show in this block. This describes expectations about parameters and returned values. If they aren’t there in the request or response, the test fails. So one cannot change the api without adapting documentation. You might have guess: The generated Asciidoctor snippet as included again in the final docs, you find it here.

I started working on the documentation in early 2016, after a great workshop with Peter Hruschka and Gernot Starke. This is now 2 years ago and the tooling just got better and better.

Whether you are writing a monolithic application like my example, micro services or desktop applications. There’s no reason not to document your application.

Here are some more interesting projects, that have similar goals:

Also, as an added bonus, there’s DocGist, that creates rendered Asciidoctor files for you to share. Thanks Michael for the tip.

| Comments (5) »

05-Dec-18


Passion.

In “my” industry people speaking a lot of doing things “with passion or not at all.” I’m wonder, are we aware of what passion meant?

The word itself means both in Greek and Latin “to suffer”, which fits very well with the some of the crazy work ethics out there.

I for myself try to stay with the “good feelings” that arose with from the Stoic and might add to them: Engagement instead of passion. I’ll always do my best, but I will not suffer. Or at least, I try not to.

Just a late night thought.

| Comments (0) »

30-Nov-18


Modeling a domain with Spring Data Neo4j and OGM

This is the forth post in this series and I want to keep it short and simple.

A domain can be modeled in many ways and so can databases. As long as I deal with them, I always preferred the approach: Database (model) first. Usually, data is much longer around than applications and I don’t want my first application instance or version define the model for all eternity.

Using an Object-Graph-Mapper or Object-Relational-Mapper can be slightly danger. One tends to write down some class hierarchy and just let the tool do its magic. In the end, there are schemes that sometimes are very hard to read for humans. The danger might be a bit smaller with an OGM as hierarchies and connections map quite nicely onto a graph, but still, I don’t want that to be the default.

My domain can be summarized with a few sentences:

  • There are Artists, that might be highlighted as Bands or Solo Artists
  • Bands have Member
  • Bands are founded in and solo artists are born in countries
  • Sometimes Artists are associated with other Artists
  • Albums are released by Artists in a year which is part of a decade
  • Albums contain multiple tracks that have been played several times in a month of a given year

I spare you the logical ER-diagram for a relational database here and jump straight to the nodes. I highlighted all the important “classes” and their relations.

Modelling data with Neo4j feels a lot like modeling on a whiteboard. And actually, it really is: The whiteboard model ends up being the physical model in the end with Neo4j.

Neo4j is a property graph database. It stores Nodes with one or more Labels, their Relationships with a type among each other and properties for both nodes and relationships.

A label starts with a colon and is usually written with a initial upper case letter, i.e. :Artist and :Album, the type of a relationship is written with a colon and than all uppercase, :RELEASED_BY and properties in camel case, without a colon, i.e. name and firstName. The above list translates in my application to a model like this:



I really find it fascinating how that model reads: Pretty much the same as my verbal description.

How to model this with Neo4-OGM and Spring Data Neo4j? You might want to recap the previous post to get an idea of the moving parts.

Value objects that happen to be persisted

In my domain the most simple objects are probably the year and the decade of the year. Those objects are value objects, they don’t have an identity. A year with a given value is as good as another instance with the same value.

@NodeEntity("Decade")
public class DecadeEntity {
 
    @Id @GeneratedValue
    private Long id;
 
    @Index(unique = true)
    private long value;
 
    public DecadeEntity(long value) {
        this.value = value;
    }
}
 
@NodeEntity(label = "Year")
public class YearEntity {
 
    @Id @GeneratedValue
    private Long id;
 
    @Index(unique = true)
    private long value;
 
    @Relationship("PART_OF")
    private DecadeEntity decade;
 
    YearEntity(final DecadeEntity decade, final long value) {
        this.decade = decade;
        this.value = value;
    }
}

I did model both of them to use them further on in aggregates, for example in the relationship RELEASED_IN, but I don’t see providing dedicated repositories for them. They only have a meaning in connection with other nodes. Things to notice here are: I followed a naming convention: All classes that are mapped to something inside the graph ends in “Entity”. Thus I have to use label attribute of @NodeEntity (or the default attribute) to specify a “nice” label, i.e. @NodeEntity(label = "Year"). I use @Index for completeness. One can configure Neo4j-OGM to automatically create indexes, but TBH, I prefer to create them by hand. There’s also one outgoing relationship from year to decade. A year is part of a decade: @Relationship("PART_OF").

You also notice that I didn’t model any of the other outgoing relationships from the year: Like all the albums released in this year, all the months with play counts in that year or the foundation years of band.

While all of Neo4j, Neo4j-OGM and Spring Data Neo4j could map relationships many levels deep, I don’t think it’s wise from an application performance point of view. I’d rather explicitly select the stuff I need.

A common base class for aggregates

I’m known for my dislike of having common base entities:

For this project however, I included one for several reasons: To not pester every other entity with the technical id, to audit interesting entities and plain simple, to have an example that actually uses our (Spring Data Neo4j) support of Spring Data’s auditing in inheritance scenarios. This is what the class looks like:

public abstract class AbstractAuditableBaseEntity {
    @Id @GeneratedValue // 1
    private Long id;
 
    @CreatedDate // 2
    @Convert(NoOpLocalDateTimeConversion.class) // 3
    private LocalDateTime createdAt;
 
    @LastModifiedDate // 2
    @Convert(NoOpLocalDateTimeConversion.class)
    private LocalDateTime updatedAt;
}
  1. Maps Neo4j’s technical (internal) id to an attribute
  2. Audits the creation respectively modification date of an entity
  3. Due to some issues with the transport, we have to force Neo4j-OGM here to not convert anything. Neo4j 3.4 supports Java 8 time types natively

Simple aggregates or entities

I see something like the Genre as a simple entity:

@NodeEntity("Genre")
public class GenreEntity extends AbstractAuditableBaseEntity {
    @Index(unique = true)
    private String name;
 
    public GenreEntity(String name) {
        this.name = name;
    }
}

It is useful to maintain on its own, has an identity but no mapped relationships. I usually declare a repository for such an entity:

import org.springframework.data.neo4j.repository.Neo4jRepository;
 
public interface GenreRepository extends Neo4jRepository<GenreEntity, Long> {
}

While it’s often preferable to extend such a repository interface not from the concrete store, I do think it’s better having the concrete store at hand in the case of Neo4j. While the concrete implementation brings a lot of CRUD method one doesn’t need all the time, it also brings in overloaded versions of them that take the depth into account as well.

To mitigate the large surface of repository methods, it’s often a good idea to reduce the repositories visibility to a minimum.

I don’t see a problem using such an entity and repository directly from a controller, for example like this:

@Controller
@RequestMapping("/genres")
public class GenreController {
    private final GenreRepository genreRepository;
 
    public GenreController(GenreRepository genreRepository) {
        this.genreRepository = genreRepository;
    }
 
    @GetMapping(value = { "", "/" }, produces = MediaType.TEXT_HTML_VALUE)
    public ModelAndView genres() {
 
        var genres = this.genreRepository.findAll(Sort.by("name").ascending());
        return new ModelAndView("genres", Map.of("genres", genres));
    }
 
    @GetMapping(value = "/{genreId}", produces = MediaType.TEXT_HTML_VALUE)
    public ModelAndView genre(@PathVariable final Long genreId) {
 
        var genre = this.genreRepository.findById(genreId)
            .orElseThrow(() -> new NodeNotFoundException(GenreEntity.class, genreId));
 
        var model = Map.of(
            "genreCmd", new GenreCmd(genre),
            "genre", genre
        );
        return new ModelAndView("genre", model);
    }
 
    @PostMapping(value = { "", "/" }, produces = MediaType.TEXT_HTML_VALUE)
    public String genre(@Valid final GenreCmd genreForm, final BindingResult genreBindingResult) {
        if (genreBindingResult.hasErrors()) {
            return "genre";
        }
 
        var genre = Optional.ofNullable(genreForm.getId())
            .flatMap(genreRepository::findById)
            .map(existingGenre -> {
                existingGenre.setName(genreForm.getName());
                return existingGenre;
            }).orElseGet(() -> new GenreEntity(genreForm.getName()));
        genre = this.genreRepository.save(genre);
 
        return String.format("redirect:/genres/%d", genre.getId());
    }
 
    static class GenreCmd {
 
        private Long id;
 
        @NotBlank
        private String name;
 
        GenreCmd(GenreEntity genreEntity) {
            this.id = genreEntity.getId();
            this.name = genreEntity.getName();
        }
 
        public GenreCmd() {
        }
 
        public Long getId() {
            return this.id;
        }
 
        public String getName() {
            return this.name;
        }
 
        public void setId(Long id) {
            this.id = id;
        }
 
        public void setName(String name) {
            this.name = name;
        }
    }
}

Find the sources here: GenreEntity, GenreRepository and the GenreController.

One use case is definitely “give me all the albums having a specific main genre.” To implement such a case, I rather access that sub collection from the owning site of the relationship. Here, the albums:

@NodeEntity("Album")
public class AlbumEntity extends AbstractAuditableBaseEntity {
 
    @Relationship("RELEASED_BY")
    private ArtistEntity artist;
 
    private String name;
 
    @Relationship("RELEASED_IN")
    private YearEntity releasedIn;
 
    @Relationship("HAS")
    private GenreEntity genre;
 
    private boolean live = false;
 
    public AlbumEntity(ArtistEntity artist, String name, YearEntity releasedIn) {
        this.artist = artist;
        this.name = name;
        this.releasedIn = releasedIn;
    }
}

The repository for that entity declares a method like this:

interface AlbumRepository extends Neo4jRepository<AlbumEntity, Long> {
    List<AlbumEntity> findAllByGenreNameOrderByName(String name, @Depth int depth);
}

It makes use of Spring Data Neo4j’s traversal feature: It traverses the relationship from album to genre and asks for all albums having a relationship of type “HAS” to a genre with the given name.

That’s also a method that I’d wrap in a service, as the method name is super technical and I don’t want that:

@Service
public class AlbumService {
    private final AlbumRepository albumRepository;
 
    public List<AlbumEntity> findAllAlbumsWithGenre(GenreEntity genre) {
        return albumRepository.findAllByGenreNameOrderByName(genre.getName(), 1);
    }
}

Complex aggregates

The Artist is a complex thing. It exists in three different forms: An unspecified artist, solo artist and as bands.

While Neo4j-OGM allows you to add a list of labels to your domain and thus allowing one entity to be mapped to several labels, I don’t like that approach.

Bands and solo artists have quite different attributes, as you can see in the sources linked above and I don’t want them to mix up.

By declaring the Artist class with @NodeEntity("Artist") and the band, which extends from it, with @NodeEntity("Band") and solo artist accordingly, bands and solo artists are stored with this two labels. Polymorphic queries works to some extend with a repository for the base entity, but as Neo4j-OGM applies schema based loading, stuff can be missing from the result.

While polymorphic queries are not a daily use case, this one is:

A band has one or more members:

@RelationshipEntity("HAS_MEMBER")
public static class Member {
    @Id @GeneratedValue
    private Long memberId;
 
    @StartNode
    private BandEntity band;
 
    @EndNode
    private SoloArtistEntity artist;
 
    @Convert(YearConverter.class)
    private Year joinedIn;
 
    @Convert(YearConverter.class)
    private Year leftIn;
 
    Member(final BandEntity band, final SoloArtistEntity artist, final Year joinedIn, final Year leftIn) {
        this.band = band;
        this.artist = artist;
        this.joinedIn = joinedIn;
        this.leftIn = leftIn;
 
    }
}

And in BandEntity:

@NodeEntity("Band")
public class BandEntity extends ArtistEntity {
 
    @Relationship("FOUNDED_IN")
    private CountryEntity foundedIn;
 
    @Relationship("ACTIVE_SINCE")
    private YearEntity activeSince;
 
    @Relationship("HAS_MEMBER")
    private List<Member> member = new ArrayList<>();
 
    public BandEntity(String name, @Nullable String wikidataEntityId, @Nullable CountryEntity foundedIn) {
        super(name, wikidataEntityId);
        this.foundedIn = foundedIn;
    }
 
    BandEntity addMember(final SoloArtistEntity soloArtist, final Year joinedIn, final Year leftIn) {
        Optional<Member> existingMember = this.member.stream()
            .filter(m -> m.getArtist().equals(soloArtist) && m.getJoinedIn().equals(joinedIn)).findFirst();
        existingMember.ifPresentOrElse(m -> m.setLeftIn(leftIn), () -> {
            this.member.add(new Member(this, soloArtist, joinedIn, leftIn));
        });
 
        return this;
    }
 
    public List<Member> getMember() {
 
        return Collections.unmodifiableList(this.member);
    }
}

As you see: No setters for the member and a getter that returns an unmodifiable list. Thus adding (and removing) members goes only through the band. The modified band is then returned. As Neo4j-OGM and Spring Data Neo4j don’t do dirty tracking and don’t save things automatically at the end of a transaction, we have to take care here. Again, I recommend a service layer:

@Service
public class ArtistService {
        @Transactional
        public BandEntity addMember(final BandEntity band, final SoloArtistEntity newMember,  final Year joinedIn, @Nullable final Year leftIn) {
                return this.bandRepository.save(band.addMember(newMember, joinedIn, leftIn));
        }
}

This is again from ArtistService.

To close this up, one final example: When entities are being deleted through Neo4j-OGM, it deletes only relationships, not the target nodes of the relationships. You have to decide wether you want “dangling” nodes in your database or not. Sometimes this is ok, sometimes not. As of today, Neo4j itself has no foreign key constraint on the relationship. And how so? It’s complete ok that a node exists for its own.

In my domain here however, albums without an artist and tracks without albums serve no purpose. To delete them when I delete an artist, I do this again through a service. The session in the following snippet is the autowired OGM session. It’s completely ok to access it. Spring Data Neo4j takes care that it participates in ongoing transactions:

@Service
public class ArtistService {
    private final Session session;
 
    @Transactional
    public void deleteArtist(Long id) {
        this.findArtistById(id).ifPresent(a -> {
                session.delete(a);
                session.query("MATCH (a:Album) WHERE size((a)-[:RELEASED_BY]->(:Artist))=0 DETACH DELETE a", Map.of());
                session.query("MATCH (t:Track) WHERE size((:Album)-[:CONTAINS]->(t))=0 DETACH DELETE t", Map.of());
                session.query("MATCH (w:WikipediaArticle) WHERE size((:Artist)-[:HAS_LINK_TO]->(w))=0 DETACH DELETE w", Map.of());
        });
    }
}

Yes, there is Cypher hidden away in a class. Sometimes there are compromises to be taken, and this one is a comprise that’s ok for me. There’s also JCypher, maybe that would be something to try out in the future.

With all the things here in this post, it’s easy to write a nice application, that deals not only with CRUD, but already presents all the interesting associations:





The complete application is available on GitHub as “bootiful music”. It has some rough edges, also ops wise, but the repository along with the posts of this series should help to get you started.

I’d like to thank Michael a lot for the idea of this query, which results in nice micro genres or categories:



| Comments (1) »

02-Nov-18