info.michael-simons.eu

Why do we overwork and burnout?

This is the third post in my series of Running free… A developers story of development.. Here I will speak about my surroundings in German society and in a German company and why I overworked nevertheless and what I learned from my behaviour and mistakes therein.

This piece has been written before the #COVID-19 crisis. Speaking about a safety net feels so much weirder now and the need for one became so much more obvious.

Why do we overwork?

I will describe the surroundings and conditions you’ll find these days in Germany regarding work. Why? Because it is important to understand that self inflicted overwork is not necessary all the times. Technically, it is not necessary.

The question “Why do we overwork?” has been raised by a lot of people and also addressed by a lot.

I found one interesting paper by Lonnie Golden and Morris Altman: “Why Do People Overwork? Over-Supply of Hours of Labor, Labor Market Forces and Adaptive Preferences”

That paper of course addresses both externally and intrinsic imposed work hours and overemployment. The authors are experts on that field and you should probably trust their input more than mine.

Let’s stick with the authors definition of overwork as work beyond a persons own capacity that is self-sustainable in terms of physical or mental well-being and overemployment as work beyond their initially preferred or agreed extend of commitment toward working hours.

Overwork refers specifically to the cumulative consequences of operating at overcapacity, additional hours spent at work eventually creates fatigue or stress so that the worker’s physical or mental health, well-being health or quality of life is not sustainable in the longer run.

–Lonnie Golden and Morris Altman, Why Do People Overwork

Keep in Mind: Wage vs Salary

A couple of things said in the current slides will not apply for everyone. Freelancers and contractors usually have agreed wages per hour and are potentially in a better position to control their number of working hours. Or at least: Each working hour of a person with a wage based income will result in a higher income. For people with a fixed salary this is of course seldom true.

I’m unfamiliar with the situation outside Germany, but in my country we find various scenarios. At times over hours are included in the salary. Sometimes only a fixed number is included, the other ones are payable at a given amount. Or the over hours can be redeemed as off hours and days.

But the important thing to keep in mind here is: The fact that one has a directly related, positive feedback to over hours (aka more money) respectively is in control about the number of hours, reduces the perceived feeling of being overworked.

This doesn’t however change the adverse effects of being overwork on

• Personal well being
• Family and social life
• Risks of accidents

for example.

Directly imposed work hours and overemployment

Here are a couple of things that directly come to mind when thinking about over hours. Over hours can be explicitly or implicitly requested by the employer. In our industry that often happens at the end of a “final” sprint, before a release (well, shouldn’t those big bang release be gone with microservices and all that anyway?) or as the infamous crunch time.

Implicit over hours are more subtle. Their reasons vary from a project being under stuffed, people being constantly distracted from work and trying to keep up etc. One reason might be even old and sluggish hardware, shops giving old laptops with spinning disk drives out to people just to cut costs.

I tend to be ok with the occasional explicitly demanded hour. On the one hand I get paid well but on the other hand, I always had an interested in the success of the companies I worked for. If I wouldn’t have that, I couldn’t work to my full potential.

The last point on this slide is an issue of course: A lot of people involuntary do over hours and over employment and still don’t make ends meet. Probably that’s not such a big issue in a well paying industry like hours in a western society, but it’s a general problem.

At least in Germany, a couple of things mediate the later issue and I’m gonna briefly speak about them so that you get an idea about the safety net that helps you when making job wise decisions and planning in Germany. Why do I think this is important? Because it gives you freedom and room for decisions.

Safety net in Germany

Many people think of safety nets in cases of falling down… I like that image here a lot as well: Falling down starts by getting of the rails and a good safety net prevents this.

So, what do we have in Germany that helps keeping you on the track, giving you time finding new jobs and actually taking care of yourself and your family?

So first of all, we have Arbeitslosengeld which means unemployment benefit. This is paid for at least 6 months when you had been employed for 12 months. It is paid for 12 months when you had been employed for more than 24 months before being unemployed. There is a penalty of 2 months when you quit compared to when you had been given notice.

The gross amount of Arbeitslosengeld is 67% of your previous average income over a given period (until a maximum value). Most people won’t be able to keep their living standard on the amount, but it gives you a long enough period looking for a new job without worrying all to much, at least in our industry and filter bubble.

After the period of unemployment benefit people can claim Arbeitslosengeld II or Hartz IV. The things here a lot more complicated and the actual amount one receive is much lower. The law was an attempt to bring together social welfare and unemployment help. Arbeitslosengeld II comes with strings attached, people are obliged to improve their job situation, take offered jobs or they will sanctions.

A thing that keeps a lot of worries at bay is of course public healthcare. I don’t have to worry to call an ambulance, I don’t need to pay bills upfront. Most of the time this works pretty decent.

Last but not least: Most people here rely on state retirement pension. Private investing huge amount of income is still quite uncommon in Germany. There have been a lot of attempts to change that, but I’m sceptic here and I am unsure how that will work out. Most people here don’t plan on retire like before 40 or something, but work until eligible to receive a pension.

Family

One important aspect is the fact that Germany tries to protect families and encourages both parents to do both: family work and income related work. That could be better to large extend – at least compared to the Scandinavian area – but we’re not that bad.

What do we have in place? From the day of the pregnancy being announced at work until 4 months after birth mothers cannot be dismissed by law. Then there is maternity protection, which basically prohibits forcing a mother to work 6 weeks before birth and 8 weeks after birth.

For the longer term aspect of work versus family, the aspect of general parental leave protection is more important. Parents are protected by law against dismissal and similar when they go on parental leave, up-to 36 months per kid and parent. And that’s not all: For a maximum period of 14 month, there’s Elterngeld, which is at max 65% of the parents net income.

Why am I stating this here? While there different circumstances that makes things harder, but there shouldn’t be a reason why one parent – usually the dad – runs back to work asap. It’s definitely not a project that’s more important than a kid and often not the money.

So again, why?

Assuming we have don’t are continually required to do over time by our bosses, don’t need to work around the clock to compensate other issues of the company and can make ends meet without living in constant fear of losing a job:

Why do we over work?

Possible reasons for self inflicted over working

I came up with a number of possible reasons, and most of them are my own.

There are some pretty solid reasons… Working for a promotion, doing some experiments, maybe spiking something. A less valid reason is because everyone is doing it due to management seeing the busy people as more productive or a Hero culture where always someone really knowledgeable comes in and saves the day (on the price for them to burn out).

Some people forget what we had a couple of pages ago: We are not our code. And we are not our work. We don’t need to take everything personal and grind us to pieces on fixing something. I hear my colleague laughing over there in Brunswick. I’m in need for constant reminder not taking every bug personally.

And of course, there’s alway the thin line between being passionate for something and suffering for something. The origin of the word passion is exactly that: Pain.

I’m deliberately not saying that these reasons are from my past, I’m just the type of person prone to let passion go into pain.

A stoic, caring approach would help here.

Why do we I overwork?

Now, let’s jump into my personal reasons and the issues I’m constantly facing and make an “I” out of this we.

I feel responsible, whether it’s just perceived or for real. That feeling is stronger the less I feel there’s someone else jumping in or skilled enough to solve it.

That leads me to thinking there is no-one skilled or willingly enough to take care of an issue.

When faced with problems I’m trying to solve, I’m quite often too stubborn to ask for help, which of course is being myself implementing my own hero culture.

Apart from that, I just like the stuff I’m working on and I’m somewhat a perfectionist. Together, that’s an easy way to just loose myself in the zone and completely forgetting about time and my surroundings.

For me it was important to realize the items here that needed to be addressed for me to feel better.

Things I can change

So of course the feeling of being responsible is not a bad feeling. It’s quite important, it’s a sign that you care.

But remember, being responsible and doing everything on your own are not the same, so learn to delegate.

The assumption there is no-one to delegate to or no-one else skilled enough for a problem or not willing to solve it, is just wrong. Of course, if you’re working completely in isolation, that is the case, but that seems quite rare. So, make sure there other people, trust them and trust into their skills and assume a common goal.

If you are not working directly together and are in sort of customer / supplier relationship, try seeing an issue from their perspective. They want an issue solved as much as you do, but maybe they just don’t have the knowledge to do so.

The third bullet point should be perfectly clear: Just don’t be stubborn, asking for help is NOT a failure.

Flashback

After studies I started my career in a small company with less than 20 people, including 2 CEOs and a cleaner. I entered at a time when Oracle Forms Client server was the given tool in that company.

While the company was an Oracle shop – from Database to Forms to Designer and back – times where changing, Forms Client Server was about to be deprecated and I was really lucky that the CEO was very open to new things and always supported us with research and development without an immediate monetizable effect.

A bit fast forward 2 or 3 years, I found myself in a couple of Java courses and learning Java SE (Swing) as well as having a peek into web applications with J2EE (this is not a mistake, Jakarta EE was called that way back then). We didn’t go the Jakarta EE route, but landed in Spring and Oracle APEX world, but that’s story in its own right

Unrelated to the tech stack: Many of those small shops ran their own infrastructure back then, but without a supportive IT department and just doing things on the fly. Remember, that was pre cloud era and one used to host all things yourself. Guess who felt responsible in the end?

The result: I somewhat aggregated all the Java (and later) Spring knowledge in the company and later on, most of the administrative IT things.

That is nice for your ego, hopefully good for your paycheck and is fun to a certain point and amount of work, but eventually, you’ll be a single point of failure, for the company but also for what you can actually manage to achieve.

Lessons learned

A system having a single point of failure is not resilient. The single point of failure is under too much pressure.

Don’t be that single point of failure!

It is not enough to accumulate all the knowledge to grow. Growth needs room. There is no room when you’re the single point of failure.

The same is true for everyone around you: When you walk around with an ego bigger than your head, nobody else can grow, make their own mistakes and learn as you did. Good people will either leave or end up being frustrated.

When you work together, assume a common goal. Of course there’s always personal interest involved, but that is inevitable. Would working together without trust in common goals be reasonable at all?

The Things I won’t change

I still like the stuff I work on, even more these days. Also, losing oneself in an immersive task is just great.

To care about something and do it with 100% is both a strength and a flaw when overdone. It allows you to be good at something but also can burn away all your strength.

It’s all about the dosage and avoiding the pain of too much passion:

Passion needs fuel to burn

There is a fantastic blog titled Passion and burnout from Codecentrics Nandor Gyerman. Nandor starts with the suffering aspect as well and takes it a step further: Passion is an intense feeling of fire, an act of self-immolation.

The fire needs fuel to burn. That fire can be sustainable or not, depending on what you burn on the altar of passion: Time, energy, common sense, money, habits, sleep or finally sanity?

Enjoy stoic virtue more than uncontrollable passion

My approach here is – and by any measurement, i still fail on this quite often – to enjoy things I do with a stoic virtue more than a burning passion that eventually will devour me.

Care about things, but don’t forget to care about yourself. Otherwise passion will eventually burn you out.

Remember the key thing about life on earth is change.

“What about your previous shop?”

By the end of 2014 I was a general manager (“Prokurist” to be precise), and it turned out: It just wasn’t my role. There are lot of important takeaways:

• Making someone a manager just by order doesn’t work, it needs preparation
• Being a manager on the other hand requires effort and work and doesn’t work – like software development – out of the box
• If you not intrinsic motivated, things don’t work out

I found myself in a constant overload: Still trying to be a good developer and at the same time struggling to be a manager. I learned the hard way that authority given by order is worth nothing, it doesn’t work that way.

Even when a role changes significantly by outer mechanism, your position in the company between colleagues won’t do with the same momentum.

So in 2017 we parted ways and it was good to do so.

I could try out a couple of things afterwards and I wasn’t too afraid to change anymore

My old boss and I still speak regularly and I’m happy. They are doing as good as I am. I can see first hand, how things change for the better, when there is enough room for everyone.

One thing I took with me from those years: Having a mentor at a company, who trusts in you and supports you is invaluable. It opens doors to opportunities, learning and so much more.

If you happen to look for something new, this would be one aspect that would be important for me.

Continue with the forth and final part: More lessons learned.

Filed in English posts | Comments (0) »

Miles are my meditation

This is the second post in my series of Running free… A developers story of development.. In this part I will focus on what actually helped me defocus my head from spiralling around work related issues and problems.

Cognitive therapy – Miles are my meditation

In 2017, I was writing my second book, running the EuregJUG was a great success and on the outside, I was as successful as it get’s in my company, but I was feeling worse every day.

I tried a couple of things that I thought would be expected of a successful mid thirty guy (aka a dude with a full blown midlife crisis) and define myself better:

• Went to fancy barbershops and got myself expensive haircuts
• And an expensive watch
• Drank more than ever
• Tried to be what I thought is manly

Result: I looked stupid, spent to much money, got fat, sit in front of a computer even more. I needed a big shift of focus in my life, something like a cognitive therapy

I was always an avid cyclist and still managed to ride a bike nearly everyday. That was however also work related as I did commute by bike. Not much of a focus shift.

JCrete 2017

Sometimes all it needs is a good conversation and people being role models. I had a couple of those at JCrete 2017, especially with Felix and Heinz. I’m mentioning this here because of two reasons:

• The conversations people lead are important and have often an effect.
• If possible, go visit an unconference. JCrete is one of the most famous, but a couple of
  more have appeared in the last years, such as
  JAlba, JWild or JSpirit.

These unconferences offers a market place like proposal and selection of topics. Many of those pretty hard core technology wise but also with topics such as presented here.

Running – Is your bike broken

I always said that I start running on the day when all my bikes are broken.

I actually tried a couple of times to run more than a kilometer and I usually hit something like 3k and ended always at the point where everything hurt and while I’m usually quite stubborn, I couldn’t convince myself do go further.

Standard solution until then was trying to motivate me with buying more gear, but that was never longterm sustainable.

One doesn’t need much

The nice thing about running is: You can get pretty far with a decent pair of running shoes. It’s a good idea to go into a shop and get some guidance for selecting a pair. I needed one that gave me a bit of balance but not that much cushioning.

I have a couple of running shirts but depending on the lengths of your run, I don’t think they matter that much. I tend to sweat a lot, so I’m more for the lightweight sports gear that transports humidity away from the body. A fitness tracker is nice, but not required. I would even say a heart rate monitor is not necessary. Usually you will notice when you are over pacing, but alas, I’m not a doctor of medicine.

Anyway, this time, I stuck with my old pair of running shoes and set a couple of other goal:

SMART goals

Goals are important.

I really think that we all have a ton of intrinsic motivation in us. To access it we need goals and an environment that allows us to pursue those goals. Goals in the context of this talk here aren’t of course not only sport goals, but also professional ones. Those can be: Learning a new language (of course, both programming and spoken languages), solving the task of your job in an optimal way, making a relevant step in your career.

There are a lot of silly acronyms in project management and methodology but here’s one I really like: SMART goals. Those are goals that are:

• Specific
• Measurable
• Achievable
• Realistic
• Timely

Let’s see: I wanted to be able to run 10k by the end of 2017 and get my weight down to 80kg again. That makes already two specific and measurable goals. Are they achievable? Of course. A healthy person at my age should be able to run 10k in a reasonable amount of time and 80kg is pretty much my optimal body mass index. Both goals are realistic (in contrast to let’s say being able to compete against world class athletes). And finally, it was in the summer of 2017, 6 months would be perfectly long and short enough to achieve those goals.

Did I reach them?

There’s a nice run at the end of each year in my place. This is where I wanted to see if I reached my goal.

That was 2017:

10k in about 50 Minutes. Not bad at all. I spare you the view of my scale, but I reached the weight goal, too.

Fast forward to 2019:

Where to find motivation?

I already gave that answer: Miles are my meditation. Doing long distance things has a calming and relaxing effect. Runners or cyclist high is a thing.

It’s like turning off the repetitive thoughts in your head that circle around issues, anxiety and problems.

It forces you to focus on your breathing, your body, yourself. On the next step hill or mile in front of you. Not some abstract thing in the future.

Competition?

I like doing races because they make me stretch. It’s not like that I’m trying to reach a certain place, but running with a lot of folks and a timer ticking actually increases your pace by a whole magnitude.

Also: It’s great running in places that are reserved otherwise for cars etc.

Fun fact: I never considered participating in a cycling road race. I find group rides with more than 5 or 6 people mentally straining enough already. In a peloton you’re usually super close to each other and you really have to be aware. Kinda defeats the purpose of switching ones head of for a while.

What about Medals? 🏅

Remember what I said at the beginning about outside appreciation: Medals, physical and virtual, are also that, so enjoy them, but they don’t matter anyway (at least when you’re not a professional racer, I guess).

In June 2018 I joined Strava…

and things escalated a bit. I never thought that a platform like Strava would change my life that much.

If it’s not on Strava, it didn’t happen

I smiled about that joke at first, but then gamification kicked in. See 2018, 2019 and now 2020:

In 2019, I went totally bonkers… Here again a goal: Doing a Strava “Gran Fondo” aka a 100k cycling tour each month and taking pictures. That was a fun thing to do… I even created a small book from it and donated the revenue.

Nice memories

I have been thinking about these now for some time.

I have been discussing Strava and gamification with my wife and also the kids. The kids love this physical medals and yes, even when I said earlier today, don’t rely too much on outside appreciation, kids being proud of their parents is a hell of a good appreciation.

And actually, having some real tokens fits the experience of doing a halve or a full marathon a lot more.

Being outside

The biggest motivation for me however is being outside. Doesn’t matter if on bikes or running. While I took all three pictures here near my home during pretty good weather, I ran and cycle the last two years with

• rain
• more rain
• storm
• snow
• and everything in between

I found it much easier to go out running in bad conditions, the effort required to clean messed up gear is just smaller. Cycling usually means more inertia as you need to plastique wrap yourself and most of the time, wash the ride.

I used to listen to music the first couple of weeks while running but eventually stopped it. I don’t listen to podcasts. I try to don’t think about anything.

I cannot stress this enough: The brain also needs room to wonder and ruminate.

Zwift and generally indoor sports never clicked with me. They allow a lot of people to make the most of their time and I get this. Would I use something indoor sports, I would missing the bit of letting my brain go. Of course I would watch talks or TV shows.

How did the running influence me?

Feeling stronger and healthier

As plain as it is: I feel stronger and healthier.

More relaxed

Also, as simple as that: Working out, feeling yourself, your body and the surroundings and a hot shower afterwards does wonders, much more than the a lonesome leisure beer does.

For me, it worked wonders avoiding depersonalization.

More resilient

It made me more resilient: I needed to overcome an initial pain point.

The longer distances taught me that pace control is important. I can rush all the things I want in the beginning, if I cannot reach my goal, it’s in vain.

Let me tell you this story: I ran my first marathon in April 2019, started with an achievable goal time and try to run a constant pace to reach that. The second one, in October the same year I was like “Go all in”. That worked well enough for the first half of the thing and ended with me more or less crawling behind the finish line. In the end, I was only a minute faster than on the first one but at the same time, wore myself down.

Anyway, it’s the same with work: Make sure you’re in a place where you can find your pace. A good place will you give the time for that.

Positive feedback

The realization that I’m actually good at something I always disliked is eye opening.

If it works with sports, it probably works with other things, too. Things that might seem out of your comfort zone, too.

Like: Being more open in work, accepting help, accepting challenges that involves more than superficial reflections and so on.

Focus shift

Let’s focus on that one.

I had enough time to think while running and realized a couple of things:

• I was more than overworked
• I felt depersonalized and while being angry at a lot of things, actual success and good projects at work left me cold.
• Lot’s of stuff was very robotic and tiresome at the same time
• I took the anger with me home
• While being stuck in it, I wasn’t able to articulate that.

So let’s raise the question: How did I let it come to this, even in surroundings where it would not be technically necessary. Why do we overwork?

Continue with part 3: Why do we overwork and burnout?

Filed in English posts | Comments (0) »

Running free… A developers story of development.

This is gonna be the first of four parts accompanying my talk with the same name: Running free… A developers story of development. which I held in March 2020 at the inaugural launch of dev.next. I’m grateful that Venkat Subramaniam gave me the opportunity to do this.

Sadly, dev.next got cancelled respectively postponed due to the 2019–20 coronavirus pandemic.

I had created a readable version of the talk in hope anyone finds this motivating or can draw something for themselves out of it. I’m happy to have it now and I’m gonna share it in five parts over the coming weeks. If anyone finds it useful, I love to hear your feedback.

Overview

• Running free… A developers story of development (this)
• Miles are my meditation (Part 2)
• Why do we overwork and burnout? (Part 3)
• More lessons learned (Part 4)

Running free… A developers story of development.

This is “Running free”, a talk about a developers story of development. In this post I will tell you bits and pieces of my story that I hope you might find useful.

This talks needs a couple of warnings. The first – and shorter warning – it’s a personal thing, the standard about me page doesn’t apply. The whole talk is probably an about page.

I am gonna share a lot of personal stories and more important, personal recipes and strategies to cope and tackle various things. It should be obvious, that these approaches may work or may not work for you. They work in most cases for me, but your milage will vary.

Am I the right person for this?

I’d say, it depends… On a good day, of course I am. I’m confident in what I am doing, I am aware that I’m good at what I am doing. On a bad day, probably not.

There are actually tons of days where I

• Feel bad about me
• Feel bad about the stuff I do
• Overwork
• Stress out (myself, my family and colleagues (Hello, Gerrit… ))

But… let’s focus on the good days and keep it with Dajana here:

While I actually have training and I am certified to train other people for a job in computer science, i am not a personal coach and I have never been trained in coaching other people. I can only share my experiences the same way a lot of people shared their experience with me.

The talk will contain a couple of quotes from people I like and whose input helped me to rethink a couple of things. People and the network you’ll build over the course of a career are much more important than technology. Always remember that: Value relationships, in both directions.

So, let’s get started: who is this developer?

About me (Standard Edition)

Let’s start with the usual “About me” slide. My name is Michael Simons. What do we have here:

I work for Neo4j, the graph database company. We have a lot of cool things out there. First of all, the graph database with the same name and then the thing that I’m working on with my friend Gerrit Meier: The Neo4j-OGM and Spring Data integration.

Of course I have to brag around that I’m one of the Java Champions. It’s one of these things that made me really proud, actually. How did it come to this?

The slide explains it as well: I’m the current lead for a Java user group named EuregJUG, which I have been running now in the 5th year with great success. In that role I’m also actively involved with the JavaLand conference. I’m also the author of a couple of books: The first german Spring Boot 2 book and “Arc42 by example”,

Before I was at Neo4j, I worked most of my professional life at ENERKO INFORMATIK in Aachen, Germany. In between I briefly worked as a senior consultant for the well known German consultancy INNOQ.

In good moments I am able to honestly say that I’m proud about all of these things. In some others, my imposter syndrome kicks in and doesn’t let me realize that this is actually something.

At this point you’d usually find the company slide or slides, depending on how much marketing and legal material one is obliged to show but I like to quote Sébastien from RedHat here:

We are not just developers.

Sébastien runs the Riviera Dev in France and they have a track named this way about exactly that topic.
We spoke briefly about this talk here and the track name immediately clicked with me for a couple of reasons.

Hopefully it is clear that thinking about yourself as a Java or .NET, as Go or Rust developer or whatever
language you prefer these days, narrows the number of options and choices you might have.
So from a professional point of view it is of course a good choice to think as “I’m a developer”.

But thinking of oneself as just a developer and probably focussing on gaining appreciation from that single source makes you vulnerable and insecure.

First of all, we are humans, with different needs and hopefully, different interests and resulting from that, different inputs.

About me (Additional Edition)

So, I am a father of two, a husband to a beautiful wife, I cycle, run, read and do a lot of more things. I am probably a few more other things.

The things I do change over time, sometimes but less often, the things I am do too. It’s important to distinguish between something you need to be and something you do. Those are not the same (and please don’t get me started about “I have a wife” vs “I am a husband” or the same with “I have two kids” vs “I am a parent”)!

Sometimes it is appropriate to include an additional slide like this one, sometimes it isn’t. I think this time it was.

Keep a slide like this in mind next time you feel bad about code, a project or work in general.

I am We are not…

Actually, I think we could avoid a whole category of problems in our job when we could remember a couple of easy things here:

• We are not the code we wrote nor the project we maintain.
• We are not the architecture of a system we are responsible for.

If we don’t keep these things in mind we’re in danger of drawing all our self worth from external opportunities. This makes reasoning and speaking about technical problems so much harder, even when the other people in the room are good colleagues and your project has a good discussion culture.

Why? Because getting self worth and appreciation from only a single thing is a single point of failure and as such, a pain in the ass. Even if it does not fail, it bothers and bugs us.

As alway, there are smarter people in the room. When I tweeted about this thoughts – quite exactly a year ago – Roman Kennke, Java VM Hacker and Principal Software Engineer at RedHat, responded like this:

Don’t rely on any outside appreciation

I appreciate it very much that Roman replied with can. I mean, this is endboss-level-hard. Anyone of us likes a casual “thank you”, a “well done”. Even more so some verbalization that something we did was helpful.

By all probability not only the younger people in the audience but anyone who is in one form or the other on social media looks for hearts, stars, thumb ups, followers and what not.

Stuff like this becomes addictive and you will have a hard time getting off from it. I’m not saying it’s completely bad, not at all, but mind the dosage.

Find things that you enjoy, preferable a broad range of things, make yourself less fragile and more resilient.

For me, those couple of statements already required some kind of focus shift. Getting my head and my thoughts away from recurring patterns and thoughts. A cognitive therapy, if you like, so let’s start with that.

Continue with part 2: Miles are my meditation.

Filed in English posts, RL | Comments (1) »

Polyglot programming on the GraalVM

(*) As of now R, Ruby, Python and JavaScript 😉

GraalVM consists of a Java Compiler, a VM and a JDK based on HotSpot/OpenJDK, implemented in Java. These days it’s all about additional execution modes compared to the standard JVM, mainly ahead-of-time compilation and the creation of native images. Native images are an important piece to create Java based applications that are easier to scale in very elastic ways and large deployments.

However, the GraalVM has a couple of more distinctive features:

• Truffle Language Implementation framework and the GraalVM SDK, to implement additional programming language runtimes
• LLVM Runtime and JavaScript Runtime

The content of this post is based on the Truffle framework and with it, the polyglot aspect of GraalVM.

There have been a couple of great talks about Graal and especially Polyglot. Have a look at Polyglot Applications with GraalVM from @mesirii and JS, Rust, Python and the Holy Graal by @larsr_h.

The GraalVM is polyglot in many different ways: You can embedded a supported guest language in a host language through a so called polyglot context but you can also call access all supported languages from a supported, dynamic language by running the GraalVM version of the interpreter with the --polyglot.

Inside the GraalVM manual you’ll find the polyglot reference as well as the embedding reference. The embedding reference is mostly about running supported languages from Java programs. This won’t be our concern in this post. This is about the polyglot reference. The examples in the manual are designed for first selecting a start language and then a target language. The examples follow the pattern that a host languages executes some simple code in the guest language.

Me friend Michael however nerd snipped me with a different idea: How to access Neo4j from R? Or use the multi database features from Neo4j 4.0 from a language for which we (I work at Neo4j at the drivers team) haven’t yet come up with a driver?

GraalVM polyglot interoperability for the win. You find information about GraalVM’s polyglot interoperability per language inside the manual, for example here for Ruby.

In short: The interpreters for the supported languages all comes with an API to interact with all other supported languages.

That allows us the Neo4j Java Driver in it’s current 4.0.0 mainline to from R, Python, Ruby and for completeness from JavaScript (the 4.0 JavaScript driver for Neo4j is already there) as well.

Michael and I have setup a repository named “neo4j-graalvm-polyglot-examples” that demonstrates this approach.

Setting up Neo4j

These example are about access to Neo4j from various languages. I find the easiest way to have an instance up in running in no time is Docker.

docker run --publish=7474:7474 --publish=7687:7687 -e 'NEO4J_AUTH=neo4j/secret' neo4j:4.0.1

gives you a running instance. After that, you can open http://localhost:7474/browser/?cmd=play&arg=movies and install our Movie graph to have a dataset to work with.

Other options include our Desktop edition which you find among other downloads here: https://neo4j.com/download/

Setting up GraalVM

Michael Hunger uses SDKMan! for downloading and installing GraalVM, I went to the GraalVM Downloads page at GitHub and got the JDK 11 edition of GraalVM 20.0.0.

After downloading and installing, you should have a valid GRAALVM_HOME and a JAVA_HOME pointing to the former:

✗ echo $GRAALVM_HOME 
/Library/Java/JavaVirtualMachines/graalvm-ce-java11-20.0.0/Contents/Home
✗ echo $JAVA_HOME
/Library/Java/JavaVirtualMachines/graalvm-ce-java11-20.0.0/Contents/Home
✗ java -version
openjdk version "11.0.6" 2020-01-14
OpenJDK Runtime Environment GraalVM CE 20.0.0 (build 11.0.6+9-jvmci-20.0-b02)
OpenJDK 64-Bit Server VM GraalVM CE 20.0.0 (build 11.0.6+9-jvmci-20.0-b02, mixed mode, sharing)

GraalVM comes with gu, the GraalVM Component Updater. gu is used to install additional packages. We use it to install R, Ruby and Python packages as well as the native image tool for GraalVM:

$GRAALVM_HOME/bin/gu install R
$GRAALVM_HOME/bin/gu install Ruby     
$GRAALVM_HOME/bin/gu install Python
$GRAALVM_HOME/bin/gu install native-image

The list of installed components should now look likes this:

$GRAALVM_HOME/bin/gu list                
 
ComponentId              Version             Component name      Origin 
--------------------------------------------------------------------------------
graalvm                  20.0.0              GraalVM Core        
R                        20.0.0              FastR               github.com
llvm-toolchain           20.0.0              LLVM.org toolchain  github.com
native-image             20.0.0              Native Image        github.com
python                   20.0.0              Graal.Python        github.com
ruby                     20.0.0              TruffleRuby         github.com

JS respectively Node come by default with GraalVM.

Running the examples

The examples live in neo4j-graalvm-polyglot-examples. Clone this repository via standard Git means. The Neo4j driver lives under the Maven coordinates org.neo4j.driver:neo4j-java-driver. The driver has a single dependency to the reactive stream API. To make the download easier, the repository comes with a Gradle build that works both under Windows and Linux. Get and export the required dependencies via

./gradlew downloadDependencies
export CLASSPATH=lib/neo4j-java-driver-4.0.0.jar:lib/reactive-streams-1.0.2.jar

All examples can now be run like this:

# R
$GRAALVM_HOME/bin/Rscript --jvm --vm.cp=$CLASSPATH neo4j-graalvm-fastr-example.R
 
# Javascript
$GRAALVM_HOME/bin/node --jvm --vm.cp=$CLASSPATH neo4j-graalvm-javascript-example.js
 
# Python
$GRAALVM_HOME/bin/graalpython --jvm --vm.cp=$CLASSPATH neo4j-graalvm-python-example.py
 
# Ruby
$GRAALVM_HOME/bin/truffleruby --jvm --vm.cp=$CLASSPATH neo4j-graalvm-ruby-example.rb

They all execute the following query

MATCH (:Person {name:$name})-[:ACTED_IN]->(m)<-[:ACTED_IN]-(coActor) 
RETURN DISTINCT coActor

The query has one parameter called name and gives you all the actors that acted in the same movie like the request actor.

I take neo4j-graalvm-fastr-example.R as example, but the idea is the same for all the examples above.

First, you have to import the required classes via GraalVM’s Java API:

graphDatabase <- java.type('org.neo4j.driver.GraphDatabase')
authTokens <- java.type('org.neo4j.driver.AuthTokens')
config <- java.type('org.neo4j.driver.Config')

With our static factory method, a database connection is opened:

driver <- graphDatabase$driver('bolt://localhost:7687', authTokens$basic('neo4j', 'secret'), config$defaultConfig())

It looks similar in all the other languages. The Neo4j drivers are session oriented. That means, the driver instance is a long living object you keep around and the session is used for your tasks. The driver takes care of connection pooling.

Executing the above query looks like this in R:

query <- '
    MATCH (:Person {name:$name})-[:ACTED_IN]->(m)<-[:ACTED_IN]-(coActor) 
    RETURN DISTINCT coActor
'
 
session <- driver$session()
# The R list (which behaves like an associative array) is automatically converted to a Java Map 
coActorsRecords <- session$run(query, list(name="Tom Hanks"))$list()

This gives you a list of records that can be processed further.

Recap

The GraalVM is a fascinating piece of software. We are not combining trivial libraries in those examples, but loading a driver that manages a connection pool based on an embedded Netty.

While it is of course preferable to have everything “natively” to your language, a polyglot environment like this gives you the opportunity to use foreign functions without that much effort. It wouldn’t be that hard to wrap our Java driver with idiomatic code for someone that has actual R knowledge to make it feel like first class R citizen.

Another use case is to have access to all features of Neo4j 4.0, for example the multi database feature. This is not yet available in the Python driver or the community driven Ruby driver. Here’s an example on how to use them: Python and Ruby.

The official, native Neo4j drivers that are already fully Neo4j 4.0 are here:

• Neo4j Java Driver
• Neo4j JavaScript Driver
• Neo4j .NET Driver

The Python driver will be released the coming weeks with full 4.0 support.

Filed in English posts | Comments (2) »

Spring Data Neo4j, Neo4j-OGM and OSIV

TL;DR: Don’t use Open Session in View with a Neo4j-Cluster

If you use Spring Data Neo4j and Neo4j-OGM in your Spring Boot application connectect against a Neo4j cluster (via bolt+routing://, neo4j:// or multiple bolt-uris), configure spring.data.neo4j.open-in-view explicitly to false like this: spring.data.neo4j.open-in-view=false

What is Open Session in View?

Vlad Mihalcea has written extensive amounts about why the Open Session in View Pattern is an Anti pattern these days.

Vlad is well known for his work in JPA and especially hibernate world. What the hell has this to do with Spring Data Neo4j and Neo4j’s Object Graph Mapping (Neo4j-OGM)? It turns out, a lot. We have to bring a couple of things together.

Neo4j routing

This is maybe unexpected, but the first thing we have to understand is Neo4j’s routing mechanism. Neo4j databases are able to run as clusters. The instances of those clusters use the Raft consensus algorithm. Part of that algorithm are leaders, followers and read replicas.

My colleague David allen has a nice post out there on how to query a Neo4j cluster: Querying Neo4j Clusters.

The important stanza for this post is:

The leader is responsible for coordinating the cluster and accepting all writes. Followers help scale the read workload ability of the cluster and provide for high-availability of data. Should one of the cluster machines fail, and you still have a majority, you can still process reads and writes. If your cluster loses the majority it can only serve (stale) reads and has no leader anymore.

Optionally, you can have any number caches in the form of read replicas. They are read-only copies of your database for scaling out read-query load. They are not officially part of the cluster, but rather are “tag along” copies that get replicated transactions from the main cluster.

When you connect to a cluster one of the Neo4j drivers does the job of routing for you everytime you open a connection via the bolt+routing:// respectivly neo4j://. It is said that “the driver is connected to a cluster”. The driver has a routing table, knowing leader, follower and read replicas.

The driver does not parse Cypher on the client side. The only information it has available to pick a host from the routing table is whether you want a read session or a read-write session respectivly transaction. Therefor it is very important to make that choice consciously! Otherwise all request will go to the leader, as that instance will be able to answer all of them.

Neo4j-OGM

Neo4j-OGM is able to support Neo4j cluster and routing via it’s Bolt transport. There are a couple of convience methods on the Neo4j-OGM session that let’s you specify whether it’s a read or a read-write statement you want to run. Neo4j-OGM actually does look into your Cypher and issues a warning if you ask for a read-only transaction but have something like MERGE, CREATE or UPDATE in your Cypher.

So up here it’s all good.

Entering Spring’s @Transactional

Spring’s @Transactional is a great piece of code. It instructs the framework to execute your code in a transactional around aspect: A transaction is opened before your code and closed afterwards, commiting if your code ran without error, rolling back otherwise. A transaction manager takes care that this also works with a explicit TransactionTemplate.

The annotation has an attribute named readOnly. In contrast to a superficial look at it, it doesn’t prevent writes to happen against a database. And how could it? @Transactional is a general purpose mechanism, working with JPA, Neo4j and other databases. It would need mechanisms to find out about your query for all of those. readOnly is merely an indicator to be passed on to the underlying store so that this store configure characteristics as needed. Vlad has written down what it does for Hibernate.

For Neo4j-OGM it just configures the default mode of the Driver’s session: read only or read-write. As we learned above, this is super important to prevent the leader from being hammered by all the queries.

Spring Data Neo4j and Neo4j-OGM supports this in all of the following scenarios, given the following repository and service:

import java.util.Collection;
import java.util.Collections;
import java.util.List;
 
import org.neo4j.ogm.session.Session;
import org.springframework.data.neo4j.annotation.Query;
import org.springframework.data.neo4j.repository.Neo4jRepository;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
 
public interface MovieRepository extends Neo4jRepository<Movie, Long> {
 
	@Transactional(readOnly = true)
	@Query("MATCH (m:Movie) RETURN m.tagline as value")
	List<SomeResult> findCustomResultWithTxOnRepository();
 
	// Not read only
	@Query("MATCH (m:Movie) RETURN m.tagline as value")
	List<SomeResult> findCustomResult();
}
 
 
@Service
public class SomeService {
 
	private final MovieRepository movieRepository;
 
	private final Session session;
 
	public SomeService(MovieRepository movieRepository, Session session) {
		this.movieRepository = movieRepository;
		this.session = session;
	}
 
	@Transactional(readOnly = true)
	public Collection<SomeResult> findCustomResultWithTransactionOnService() {
 
		return movieRepository.findCustomResult();
	}
 
	@Transactional(readOnly = true)
	public Collection<Movie> findMoviesViaLoadAllOnSession() {
 
		return session.loadAll(Movie.class);
	}
 
	@Transactional(readOnly = true)
	public Collection<Movie>findMoviesViaCustomQueryOnSession() {
 
		return (Collection) session.query(Movie.class, "MATCH (m:Movie) RETURN m", Collections.emptyMap());
	}
}

All of those understand that none of the queries needs to go to a leader.

Entering OSIV

For Neo4j-OGM, there’s also an OpenSessionInViewInterceptor that get’s configured via Spring Boot by default in a web application. It makes sure there’s an ongoing Neo4j-OGM session along with it’s caches and mapping in place for the whole request.

Why is this bad in a cluster scenario? Because tied to the Neo4j-OGM session there’s the driver session. To allow the user to do anything they want in the request, those sessions need to be read-write sessions. Once marked as read-write, you cannot make turn read only ever again.

That means if you call any of the above scribbeld methods from a REST endpoint, there will already be an ongoing session and the readOnly indicator will be ignored!

So if you connect a Spring Data Neo4j + Neo4j-OGM application against a Neo4j cluster: Go to your Spring Boot configuration (either properties or YAML or Environment) and explicity configure:

spring.data.neo4j.open-in-view=false

I have raised an issue with the Spring Boot team to discuss changing the default: Open session in view is problematic with Neo4j..

Filed in Java | Comments (2) »