Code Retreats and the Game of Life

Code Retreats are a good way to improve our skills as Software Craftsmen. Soon there is the #gdcr15, reason enough to make the Game of Life a topic of a blog post.

Here is an example in Java:

Game of Life in Java

Java. Java? Java + extensions!

Hm, Java? That is not Java. Well, it looks like somewhat like Java, but that tables, that colored constants, that initialization of the generation array, that operations? It’s not hard to notice: there are some extensions made to Java. That takes for sure some years or at least months for a single person to create such extensions! No, not really. Uh yeah, if you take the OpenJDK and put your extensions there - but we don’t. We take a Language Workbench. To be more precise: we take the MPS Language Workbench. Now it is easy to modularize and put together programming languages.

In the following sections we will have a closer look on each of the new language concepts. You can get the source on my GitHub account.

New Type: Coordinate

The Gol class starts like a quite normal Java class with a main method. But already the second method, run(), contains something special. As you see, Coordinate is like a build-in Java type. It could also be a class named Coordinate, but there is a tiny difference: look at the initialization of the arraylist. These are no calls to constructors, these are the notation (aka “Concrete Syntax”) of a Literal. Just like 5 for an integer or “foo” for a string.

What do we need to create a new type? Surprisingly not that much. Of course the type itself:

Coordinate Type

Note the little blue arrow. The CoordinateType extends something called “Type”. This is the exact Type all other Java types like integer, long etc. extend too. Because of that our new CoordinateType fits nicely into the existing Java type system.

Next there is the Literal:

Coordinate Literal

Note again the blue arrow. The CoordinateLiteral extends an Expression - yes, again the very same Expression which is the base for all Java expressions.

Now we only have to tell the type system, that our CoordinateLiteral is of type CoordinateType:

CoordinateLiteral is of type CoordinateType

With some MPS type system syntax, this basically declares the the type as described above.

That’s it. After five minutes or so we are able to use a new Java type. It does not yet do something useful, therefore we have to declare the semantics via a generator. But that is the topic of a different section down below.

Syntax sugar: alive and dead

In the next method, nextGeneration(…), a two colorful constants catch the eye. ‘alive’ and ‘dead’ are used like constants or enums, but have a custom color. We could also assign other properties like underlined text, italic or bold font, a different font size and so on. Can normal Java IDEs do that too? No - and perhaps that would be a nice issue to be filed in Eclipse, NetBeans or Intellij IDEAs backlogs: assigning representation properties to constants and enums.

And it’s really no big deal to get domain specific styled constants:

AliveExpression

Again we extend an Expression. But that does not explain the different style of the text. This leads to a topic I did not mention so far: every new language concept needs or may declare how it is presented. Needs or may, eh? Yes, if we inherit from a language concepts which provides a decent representation we can go with it. If we don’t inherit or want to override the inherited style, we need to define a so called ‘editor’. Let’s have a look at the editor for the alive expression:

Alive Expression Editor

Hm, not that impressive. We see that it’s the ‘editor for concept AliveConcept’. And it’s the default one. Yes, we may declare more than one representation for a concept, e.g. for color-blind people. But let’s focus for now on only one editor. We also see that the #alias# should be shown. Alias? Yes, please go back to the Alive Expression picture. There it is: ‘alias: alive’. But how comes the color in? We can declare styles in a different tool window:

Alive Expression Editor Style

And the same goes for Dead Expression with the style ‘text-foreground-color : red’.

Decision Table

Now we come to one of the most powerful and most interesting things of custom representations via own ‘editors’. We are not limited to text like in normal IDEs. We could define also - tables! And not only tables, but even Java Swing components are allowed. So we could replace the alive and dead concepts with checkboxes. Or add some JavaDoc with explaining pictures. But for now let’s focus on the tables. In the nextGeneration(…) method we see a decision table. A content cell from the middle is taken if the column header and the row header both evaluate to true. If no constellation evaluates to true, the default value is taken. In fact, this decision table represents the core algorithm of the Game of Life:

Decision Table

Depending on if the current generation contains a cell and how many neighbors it has, it will be added to the next generation or not. And because the decision table also extends the Expression and the Type is declared boolean, the table can be used as condition in a regular if statement.

Mapping Table

In the neighbors() method we see again a table. This time no decision table but a mapping table. A single cell and a table were combined:

Mapping Table

The bold plus sign indicates that it is no normal summation operation but an adapted version for the table: the left side of the plus - here a variable named ‘cell’ of type Coordinate - is added to every entry in the table. This results into nine new Coordinates. The middle one is the original cell itself, so it is no neighbor and will be subtracted again.

Extending Operations: plus and minus

Subtracted? A single value via minus operation from a collection in Java? This is only possible because we can also put additional semantics on operations as language extensions. Once again the type system feature of MPS comes to help and let’s us overload operations:

Overloaded Operations

The first of the two new rules declares for plus and minus operators that the summation and subtraction of two things of type Coordinate is allowed and results into a Coordinate type.

The second new rule declares that a subtraction of a Coordinate from a Coordinate array is allowed and results into a Coordinate array.

So far it’s very nice that we can program with the extensions. But until now that program does not run.

Generate Java code

To let a program run, it has to be compiled or interpreted. Java programs were compiled into byte code. Until now we did not define how the bytecode for our new language concepts has to be generated. But do we really want to generate bytecode directly like Java does for the built-in keywords?

Let’s take a step back first and see what we did: we stacked a new language on top of an existing one. We created a new layer. Our new language extends the Java language. We should not bypass layers on the language stack and generate code for the language layer direct below the new language. Not that we are not constrained to extend only one language. A new one can extend as many languages as it wants. That’s why I chose the term ‘language layer’.

That said, we don’t generate bytecode, instead we generate Java code. For ‘alive’ and ‘dead’ it’s dead simple: ‘alive’ is replaced by a boolean ‘true’ and ‘dead’ by a boolean ‘false’.

Not really surprising is what we generate for the Coordinate type. In Java we would represent it by a class named Coordinate and so we generate it. To make that work, we have the Runtime Solution named ‘gol.runtime’. It contains only one class - the Coordinate. Exactly that Coordinate class is used as the generation target for the Coordinate type.

So if the Coordinate type is translated to the Coordinate class, the Coordinate literals is translated to a constructor call of the Coordinate class. Quite natural.

The nice thing about having the Coordinate class in the runtime model is that we can use it’s ‘sum’, ‘minus’ and ‘removeFromArray’ methods as generation targets for our overloaded operations ‘plus’ and ‘minus’. It is exact the same thing as the ‘add’ and ‘subtract’ methods in BigInteger and BigDecimal.

Now we are nearly done. But the hardest part comes as last. Translating the decision and mapping tables to valid Java code is not that simple. Therefor I would suggest to watch that video for a good and extensive explanation.

Conclusions

What did we learn? It’s quite simple to extend Java. Why should we do this? Nearly every Java programmer has some favorite missing language features, don’t you?

But there is a second and for me the more important reason: domain concepts can be integrated into a Java dialect which is specially created for the project in that domain. For the Game of Life, we added ‘alive’, ‘dead’, the Coordinate. This may be seen only as some syntactic sugar. But look again at the program: the color- and meaningful ‘alive’ and ‘dead’ words catch the eye. There is no syntactic clutter around creating a new Coordinate - no verbose Java ‘new Coordinate(1, 1), just a (1, 1). Image you can write a math formula just as - a math formula and not as a long Java methods with many words an no single math symbol. Notation matters, and notation is nothing else than the concrete syntax of a language. With language extensions we can define our own concrete syntax. A program can be much more optimized for reading. And source code is ten times more read than written. Or even more often.

Motivation

Let’s assume you visited a Code Retreat, read a book, visited a workshop or a conference - or whatever. Now you are burning to introduce Unit Tests into your project - and hit the wall hard. Yes, you are in a brownfield project. Who is not? Yes, there is no fairy offering you three wishes. Who saw one? So there is no easy way of transforming a we-are-not-writing-unit-tests-project into a TDD project. Where to begin? You could just starting with TDD from now on for every new line of code. That is one way and not the worst one. (This would be to not doing Unit Tests.) But there is an alternative: find methods or classes in the project which need Unit Tests most and start with them. It’s not really hard to do that because JaCoCo provides all you need: the test coverage with covered and missed branches. There is an interesting correlation between the number of branches of a method and the test-me-begging. Every branch adds a voice in the chorus singing test test me song. Now JaCoCo generates reports not only as HTML but also as CSV and XML. You could just use a scripting language of your choice, invest some Mana and find what you are looking for. Indeed I was quite successful with Groovy parsing the XML report, running in an ant build on the CI server as a watchdog for test coverage.

But technology gets better. Now there is no need for fiddling around with XML and scripts anymore - a simple database query does all the work. And it offers also the great power of combining the test coverage with other metrics and static code analysis. In this article I’ll explain the technical side of getting started with Unit Tests in a brownfield project. The basis is once again jQAssistant. I assume there is already a maven build in place, but not a single test yet. Then JaCoCo and the Kontext E JaCoCo plug-in for jQAssistant were added to the project and some basic rules implemented.

Find an example project

First let’s find some example project for this exercise. You think there is no such project with a considerable size anymore? Not true mate, unfortunately there are enough of them. Otherwise this article would make no sense, no? So let’s choose PlantUML. It comes with some 2500 classes and not a single Unit Test. For playing around with it I checked out the source and put it into a GitHub repo.

Now we are ready for the real hands-on experience. We will add a rudimentary test infrastructure so that there is at least one Unit Test to go on with. Than we add the static analysis and the test coverage checker. When both is in place, we bring it together in the same database and create coverage rules using database queries. Last but not least we will have a look on various ways for further improvements.

Ok, let’s begin!

Add rudimentary test infrastructure

If there is no Unit Test yes, we need to set up the very basic configuration. In the pom, we need a dependency on JUnit:

1
2
3
4
5
<dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.12</version>
</dependency>

and the surefire plugin:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.18</version>
        <configuration>
                <includes>
                        <include>**/*Test.java</include>
                </includes>
        </configuration>
        <dependencies>
                <dependency>
                        <groupId>org.apache.maven.surefire</groupId>
                        <artifactId>surefire-junit47</artifactId>
                        <version>2.18</version>
                </dependency>
        </dependencies>
</plugin>

We create a new folder for our tests, configure the test source folder in the pom if it does not follow the convention and create a dummy test which should fail:

1
2
3
4
5
6
public class DummyTest {
    @Test
    public void thatShouldFail() throws Exception {
        Assert.assertTrue(1 > 2);
    }
}

If we run ‘mvn verify’ we should see our test fail. If not, something is still wrong with our configuration which needs to be fixed. Now we correct the test and celebrate our success. The first green bar!

Ok, now we are eager to write more tests for real production classes - but how to find the classes which deserve them most? We add a test coverage checker.

Add test coverage checker

The configuration of the JaCoCo test coverage checker is quite simple: we add the JaCoCo maven plugin:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
<plugin>
        <groupId>org.jacoco</groupId>
        <artifactId>jacoco-maven-plugin</artifactId>
        <version>0.7.5.201505241946</version>
        <executions>
                <execution>
                        <id>default-prepare-agent</id>
                        <goals>
                                <goal>prepare-agent</goal>
                        </goals>
                </execution>
                <execution>
                        <id>default-report</id>
                        <phase>prepare-package</phase>
                        <goals>
                                <goal>report</goal>
                        </goals>
                </execution>
                <execution>
                        <id>default-check</id>
                        <goals>
                                <goal>check</goal>
                        </goals>
                        <configuration>
                                <rules>
                                </rules>
                        </configuration>
                </execution>
        </executions>
</plugin>

If we run now ‘mvn verify’, there should be a folder target/site/jacoco’ created containing reports as html, csv and xml. We can browse the html report and look for favourite units to test. Hm, what to look for? One goal of Unit Tests is to prevent regression. Regression when we change something. Does regression occur on one liners like getters and setters? Not so likely. Or is it raising it’s ugly head in these monster methods with lots and lots of branches? If we just add a little additional branch here and change that one there slightly? Oh yes, most likely some corner case is broken now - or was always broken. So let’s say for each method where the number of branches exceeds a limit we need a certain test coverage. The first candidates are easy to spot and we can begin to write tests for legacy code.

For starting not bad, but in day to day work we do not want to check this manually. Or do we run our Continuous Integration server for nothing? So let’s automate this and introduce automatically checked rules.

Add static analysis

Maybe you are already running jQAssistant for keeping the architecture in sync with the documentation or some other static analysis. If not you could at least risk a look on it - it’s open and very flexible. That makes it very powerful. We want to make use of this power.

Getting started is quite simple: follow the ‘Maven’ section of the Get Started Guide. Just like in the section for the first test, let the rule fail (e.g. by setting the WHERE clause to t.name =~ “.*TesTT”) to see if all is working correctly.

Ok, but how do we bring the test coverage results and the jQA database together?

Import test coverage into jQAssistant database using the Kontext E JaCoCo plug-in

This is also not hard. Just a little change here and there… Let’s do this step by step. First, we need to modify the pom. In the jqassistant-maven-plugin, we add a dependency for the ‘jqassistant.plugin.jacoco’:

1
2
3
4
5
6
7
<dependencies>
        <dependency>
                <groupId>de.kontext-e.jqassistant.plugin</groupId>
                <artifactId>jqassistant.plugin.jacoco</artifactId>
                <version>1.0.0</version>
        </dependency>
</dependencies>

and set a property with the name of the JaCoCo XML file:

1
2
3
<scanProperties>
        <jqassistant.plugin.jacoco.filename>jacoco.xml</jqassistant.plugin.jacoco.filename>
</scanProperties>

and include target/site/jacoco into the set of scanned directories:

1
2
3
4
5
<scanIncludes>
        <scanInclude>
                <path>target/site/jacoco</path>
        </scanInclude>
</scanIncludes>

so that the jqassistant-maven-plugin now looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
<plugin>
        <groupId>com.buschmais.jqassistant.scm</groupId>
        <artifactId>jqassistant-maven-plugin</artifactId>
        <executions>
                <execution>
                        <goals>
                                <goal>scan</goal>
                                <goal>analyze</goal>
                        </goals>
                        <configuration>
                                <failOnViolations>true</failOnViolations>
                                <scanProperties>
                                        <jqassistant.plugin.jacoco.filename>jacoco.xml</jqassistant.plugin.jacoco.filename>
                                </scanProperties>
                                <scanIncludes>
                                        <scanInclude>
                                                <path>target/site/jacoco</path>
                                        </scanInclude>
                                </scanIncludes>
                        </configuration>
                </execution>
        </executions>
        <dependencies>
                <dependency>
                        <groupId>de.kontext-e.jqassistant.plugin</groupId>
                        <artifactId>jqassistant.plugin.jacoco</artifactId>
                        <version>1.0.0</version>
                </dependency>
        </dependencies>
</plugin>

In our rules file (my-rules.xml if you followed the Geting Started Guide) we have to add some constraints for the test coverage:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
<constraint id="test:TestCoverageForLowComplexity">
    <description>...</description>
    <cypher><![CDATA[
    match (cl:JacocoClass)--(m:JacocoMethod)--(c:JacocoCounter {type: 'COMPLEXITY'})
    where c.missed+c.covered >= 2 and c.missed+c.covered <= 3 and not(m.signature ='boolean equals(java.lang.Object)') and not(m.signature ='int hashCode()')
    with m as method, cl.fqn as fqn, m.signature as signature, c.missed+c.covered as complexity
    match (m)--(branches:JacocoCounter {type: 'BRANCH'})
    where m=method and branches.covered*100/(branches.covered+branches.missed) < 0
    return fqn, signature, complexity, branches.covered*100/(branches.covered+branches.missed) as coverage
    ]]></cypher>
</constraint>

<constraint id="test:TestCoverageForMediumComplexity">
    <description>...</description>
    <cypher><![CDATA[
    match (cl:JacocoClass)--(m:JacocoMethod)--(c:JacocoCounter {type: 'COMPLEXITY'})
    where c.missed+c.covered >= 4 and c.missed+c.covered <= 5 and not(m.signature ='boolean equals(java.lang.Object)') and not(m.signature ='int hashCode()')
    with m as method, cl.fqn as fqn, m.signature as signature, c.missed+c.covered as complexity
    match (m)--(branches:JacocoCounter {type: 'BRANCH'})
    where m=method and branches.covered*100/(branches.covered+branches.missed) < 80
    return fqn, signature, complexity, branches.covered*100/(branches.covered+branches.missed) as coverage
    ]]></cypher>
</constraint>

<constraint id="test:TestCoverageForHighComplexity">
    <description>...</description>
    <cypher><![CDATA[
    MATCH (cl:JacocoClass)--(m:JacocoMethod)--(c:JacocoCounter {type: 'COMPLEXITY'})
    WHERE c.missed+c.covered > 5 AND NOT(m.signature ='boolean equals(java.lang.Object)') AND NOT(m.signature ='int hashCode()')
    WITH m AS method, cl.fqn AS fqn, m.signature AS signature, c.missed+c.covered AS complexity
    MATCH (m)--(branches:JacocoCounter {type: 'BRANCH'})
    WHERE m=method AND branches.covered*100/(branches.covered+branches.missed) < 90
    RETURN fqn, signature, complexity, branches.covered*100/(branches.covered+branches.missed) AS coverage
    ]]></cypher>
</constraint>

and add the constraints to the default group:

1
2
3
4
5
6
<group id="default">
    <includeConstraint refId="my-rules:TestClassName" />
    <includeConstraint refId="test:TestCoverageForLowComplexity" />
    <includeConstraint refId="test:TestCoverageForMediumComplexity" />
    <includeConstraint refId="test:TestCoverageForHighComplexity" />
</group>

That’s no magic. It’s Cypher, the query language of the Neo4j database which is used by jQAssistant.

And now, for now you don’t need to understand every detail of that. The important thing is: I suggest to group the methods by ‘complexity’. The more branches a method has, the more complex it is. Three complexity levels should be enough: low, medium, and high. Methods with low complexity are so trivial (getters, setters) that tests would bring no gain. No coverage is needed. Then there are the medium and high ones. What medium and high means in your project - its up to you to find adequate numbers. In this example let’s start with four and five branches for medium and higher than 5 for high complexity.

Now we run ‘mvn verify’ again. And are overwhelmed. By the time that is consumed for the check. By the amount of missing tests. Well, not really surprising for a fairly large codebase, written without any tests or other explicitly checked design constraints.

As a first aid to get back the control over the build time, let’s deactivate the checks for low and medium complexity by commenting out the ‘includeConstraint’ tags in the group section. Methods with low complexity have no enforced test coverage anyway. And we change in the rule for high complexity the

1
WHERE c.missed+c.covered > 5

into

1
WHERE c.missed+c.covered > 50

so that only the monsters show up. Let’s run ‘mvn verify’… better. Now you can go on and play with the numbers. You can also add a ‘SKIP x’ to the rule and replace x by a certain number. That ignores the first x results so that the check gets passed. Why would we do this? In that case we could also deactivate the rule completely, no? Not really. With this query we express more or less the following: we have a test coverage as a target, but the current code base is as bad as it is and we accept it. But we do not want to make it even worse, so every additional method with a too high complexity or too less test coverage gets reported. You can do a similar thing also with some

1
AND NOT cl.fqn =~'com.example.module.*'

after the first WHERE and before the WITH to exclude packages or classes. By the way: the ‘equals’ and ‘hashCode’ methods are excluded by default because in most cases they are complex but generated by the IDE. If you did a code review and come to the conclusion that the reviewed methods is complex but very readable and not really error prone, you should exclude them in the same way.

Improve the coverage rules

If we look at our test coverage rules, we see lots of duplication. Only some numbers change. We employ the Separation of Concerns principle and separate what changes - the test coverage threshold for a range of branches per method - from what is invariant. We declare the complexity ranges and their coverage thresholds in separated concepts and use a common query for all ranges. This looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
    <concept id="jacoco:TestCoverageMediumRange">
        <description>Define ranges for test coverage.</description>
        <cypher><![CDATA[
        CREATE
            (n:TestCoverageRange {min : 40, max : 49, coverage : 80 })
        RETURN
            n
    ]]></cypher>
    </concept>

    <concept id="jacoco:TestCoverageHighRange">
        <description>Define ranges for test coverage.</description>
        <cypher><![CDATA[
        CREATE
            (n:TestCoverageRange { min : 50, max : 999999, coverage : 90 })
        RETURN
            n
    ]]></cypher>
    </concept>


<constraint id="test:TestCoverageForConfiguredComplexity">
    <requiresConcept refId="jacoco:TestCoverageMediumRange"/>
    <requiresConcept refId="jacoco:TestCoverageHighRange"/>
    <description>...</description>
    <cypher><![CDATA[
    MATCH (tcr:TestCoverageRange)
    WITH tcr.min AS mincomplexity, tcr.max as maxcomplexity, tcr.coverage AS coveragethreshold
    MATCH (cl:JacocoClass)--(m:JacocoMethod)--(c:JacocoCounter {type: 'COMPLEXITY'})
    WHERE c.missed+c.covered >= mincomplexity AND c.missed+c.covered <= maxcomplexity
    AND NOT(m.signature ='boolean equals(java.lang.Object)') AND NOT(m.signature ='int hashCode()')
    AND NOT(cl.fqn =~ 'net.sourceforge.plantuml.sudoku.dlx_solver.*')
    WITH m AS method, cl.fqn AS fqn, m.signature AS signature, c.missed+c.covered AS complexity, coveragethreshold as coveragethreshold
    MATCH (m)--(branches:JacocoCounter {type: 'BRANCH'})
    WHERE m=method AND branches.covered*100/(branches.covered+branches.missed) < coveragethreshold
    RETURN complexity, coveragethreshold, branches.covered*100/(branches.covered+branches.missed) AS coverage, fqn, signature
    ORDER BY complexity, coverage
    ]]></cypher>
</constraint>

In the constraint, I excluded the package ‘net.sourceforge.plantuml.sudoku.dlx_solver’ from the check. I doubt that this is relevant for drawing UML diagrams.

Now all is set up for writing Unit Tests and employing the CI server to watch out for missing tests. Ok ok, you may want to add more supporting libraries for testing like Hamcrest Matchers or a mocking framework. And for every iteration, play again with the numbers and/or excluded packages and classes to find a new set of testworthy methods.

You find a copy of the PlantUML project with all the stuff discussed here on my GitHub repo into the branch ‘unittest’.

In software, there is always architecture. You can’t have no architecture. But with it’s documentation, there are two options: you can have one or not. If you have one, there are two options: the documentation matches with the real architecture or not. If your documentation matches with the real architecture, there are two options: it stays in sync or not. If it should or has to stay in sync, there are two options: watch it manually or… automatically. Automatically? Has Word(TM) a “Run all tests” button nowadays? No, it hasn’t. And we don’t use Word(TM).

But we use Jenkins as a Continuous Integration server. And we use jQAssistant with some additional plug-ins. With all of that in place only a small additional effort is needed to check the consistency between architecture documentation and the coded one.

As friends of small but powerful tools we use PlantUML. There you can create class diagrams, especially with packages. It’s not needed to put classes into the packages, you can just use the packages to describe the high level architecture. E.g. like this:

1
2
3
4
5
6
7
8
9
10
@startuml

skinparam packageStyle rect

package de.kontext_e.jqassistant.plugin.plantuml.scanner {}
package de.kontext_e.jqassistant.plugin.plantuml.store.descriptor {}

de.kontext_e.jqassistant.plugin.plantuml.scanner --> de.kontext_e.jqassistant.plugin.plantuml.store.descriptor

@enduml

With only a few lines of code, these diagram descriptions can be parsed in a jQAssistant plugin like this. It will be part of the next release 1.1.0 of the Kontext E plugin suite for jQAssistant which will be compatible with jQAssistant 1.1.0 version.

Now we have the wanted… er documented and real architecture together in the same database. The last piece of the puzzle is the jQAssistant rule which matches both of them:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
<constraint id="dependency:WrongDirection" severity="critical">
    <requiresConcept refId="dependency:Package"/>
    <description>Finds package dependencies which are in the wrong direction according to the documentation.</description>
    <cypher><![CDATA[
        MATCH
            (p1:PlantUmlPackage)-[:MAY_DEPEND_ON]->(p2:PlantUmlPackage),
            (p3:Package)-[:DEPENDS_ON]->(p4:Package)
        WHERE
            p1.fqn = p4.fqn
            AND p2.fqn = p3.fqn
        RETURN
            p3.fqn + "-->" + p4.fqn AS WrongDirection;
    ]]></cypher>
</constraint>

Was not that hard, no? Ok, a few lines more and we can also scan Asciidoc where PlantUML diagrams are embedded. But that’s… that is… that IS “Executable Architecture Documentation”.

Not a friend of Ascii? So for jQA and Neo4j Cypher we can’t help you, the rules are Ascii art.

But for the documentation part - well, of course you could use Word(TM) and a nice UML tool for the model pictures. And then you export your pictures not only in png but also the models as XMI. With the jQA standard XML plug-in and some Cypher magic or a dedicated XMI import plugin you also get the model into the database.

But these coarse grained building blocks are better managed by maven/gradle subprojects and/or splitted Spring contexts you ask? Why this waste of energy, creativity, memory and computing power you ask? Yes of course you are right - for greenfield projects. Consider a brownfield big ball of mud and you want to bring in structure and carve out modules. Then you are happy when you can define rules and have such a powerful and flexible mechanism like jQA/Cypher queries to define exceptions from the rules for the current state of code.

And these package dependencies may also be defined on a much more finer grained level for intra-module structure. You have maven modules for your slices? And want to enforce the right access for in-slice layers? Or have additional rules for calls from a layer of one slice to another layer of a different slice? (Plant-)UML and jQAssistant are your friends.

Now you have two options: business as usual or make your lazy architecture documentation finally something useful for you.

Our German speaking readers already know from the last post that we contribute to the jQAssistant project. We do so because we use jQAssistant in normal everyday work. There are a bunch of architecture and design rules to keep our projects clean and in shape.

From time to time there is the need to have a more powerful search mechanism than even the awesome “Search Structurally” feature of IntelliJ IDEA provides. Wouldn’t it be nice to have the power of database queries to find classes? The good news is: you have! Today we published our IntelliJ plugin to GitHub. You find it at GitHub It’s licensed under the GPLv3. The bad news is: it’s still an early version, so you don’t find it in the JetBrains Plugin Repository. You have to clone the project an build the plugin yourself. Please follow the instructions in the README file.

Feel free to file issues and send pull requests.

Was ist jQAssistant?

jQAssistant ist ein Werkzeug zur Sicherung der Codequalität. Es scannt die Artifakte des Projekts - Klassen, Property-Dateien, XML-Descriptoren usw. - und speichert die Daten in der Graphendatenbank Neo4J. Auf diesen Daten werden Analysen durchgeführt. Regeln können definiert und deren Einhaltung automatisch geprüft werden. Zum Beispiel können Namenskonventionen für Klassen, die Abhängigkeitsbeziehungen zwischen Paketen oder die Verwendung unerwünschter Klassen (z.B. java.util.Date) geprüft werden.

Die Datenbank kann aber auch als Server gestartet werden. Mit einer Webschnittstelle kann man den Code erforschen oder neue Regeln ausprobieren.

Die Mächtigkeit dieses Ansatzes geht weit über bisherige Werkzeuge dieser Art hinaus. Eine Graphendatenbank ist hervorragend für diesen Zweck geeignet. Über die Neo4j-Abfragesprache “Cypher” ist es leicht möglich, Suchen zu definieren, die in einer IDE nicht möglich sind. Mittels Cypher werden auch die definierten Regeln geprüft. Alle Daten stehen dem Nutzer direkt zur Verfügung.

Entwickelt wird jQAssistant von der Firma Buschmais, ansässig in Dresden. Der initiale Commit auf GitHub erfolgte im März 2013. Auf Maven Central ist der Meilenstein 3 von Version 1.0 publiziert, auf GitHub kann man schon den Snapshot von Meilenstein 4 beziehen. Das Final Release wird nicht mehr lange auf sich warten lassen.

Aktuell gilt die Apache Licence 2.0. Es ist jedoch beabsichtigt, aus Lizenzkompatibilitätsgründen auf die GPL umzusteigen.

Weitere Metriken

Mittels der importierten Standarddaten kann man schon sehr gut strukturelle Prüfungen und Abfragen definieren. Im Buildprozess fallen aber noch eine Reihe weiterer Daten an. Es liegt nahe, auch diese in die jQAssistant-Datenbank zu importierten. Da jQAssistant explizit für Erweiterungen mittels Plugins entworfen wurde, ist das Importieren nicht schwer. Bei Kontext E haben wir deshalb jQAssistant-Plugins für die wichtigsten Testartefakte geschrieben.

Üblicherweise wird die Testabdeckung nach Zeilen und Zweigen für Unit Tests gemessen. Häufig werden damit auch Vorgaben verbunden, deren Verletzung den Buildprozess abbrechen. Wir verwenden jacoco, welches die Ergebnisse als XML exportieren kann. Diese XML-Datei lesen wir mit einem Scanner-Plugin ein. Einige Cypher-Statements stellen die Verbindungen zu den vorhandenen Klassen und Methoden her.

Sehr verbreitet ist die statische Codeanalyse mit FindBugs. Viele Fehlermuster werden damit gefunden. Auch FindBugs schreibt die Ergebnisse als XML, welches wir mit einem zweiten Scanner-Plugin einlesen. Die jQAssistant-Daten werden wieder mittels Cypher mit den FindBugs verbunden.

Als weiteres wertvolles Tool zur statischen Codeanalyse setzen wir Checkstyle ein. Es überrascht nicht, dass dieses eine XML-Datei schreibt, die wir einlesen und die Daten mit den vorhandenen verknüpfen.

Bisher haben wir die von den Werkzeugen ermittelten Metriken jeweils einzeln weiterverarbeitet. Mal mit XSLT, mal mit Groovy-Scripten. Damit wurden aus den Rohdaten Indikatoren gewonnen, die eine sinnvolle Codequalitätssicherung ermöglichen. Der CI-Server bricht den Build ab, wenn die Indikatoren auf eine zu schlechte Codequalität hindeuten.

Jetzt werden die Daten nicht mehr einzeln ausgewertet, sondern sind miteinander verbunden und an die Codestruktur gebunden. Das ermöglicht eine neue Dimension der Auswertung und die Verfeinerung der Indikatoren. Außerdem können neue Indikatoren eingeführt werden, die bisher mangels passender Datenlage nicht möglich waren.

Als Beispiel sei eine Codestelle genannt, die so geschrieben ist, wie eine vorgeschriebene Bibliothek es verlangt. Checkstyle und FindBugs warnen zu recht, aber die Codestelle ist sehr detailliert getestet. Damit wird sie akzeptabel.

Obwohl weder jQAssistant noch unsere Plugins final als 1.0-Version veröffentlich wurde, sind sie aber schon so stabil und funktionsreich, dass sie im täglichen produktiven Einsatz wertvolle Dienste leisten. Wenn der CI-Server sagt, dass an einer Stelle nachgebessert werden muss, dann ist das auch fast immer der Fall.

Stand der Implementierung

Die Plugins sind wie jQAssistant selbst in der Verwendung sehr stabil, aber die Programmierschnittstellen ändern sich noch. Das bezieht sich einerseits auf die Java-API, andererseits aber auch auf das Datenmodell in der Datenbank. Das bedeutet, dass nicht immer die letzten GitHub-Commits von jQAssistant und den Plugins miteinander kompatibel sind. Mit dem Erreichen der finalen Version 1.0 sind diese Probleme dann aber ausgeräumt und man kann beides als normale Dependency im Projekt eintragen.

Lizenz

Für die Kontext E Plugins für jQAssistant wurde noch keine Lizenz deklariert, d.h. vorerst unterliegt die Software dem gesetzlichen Urheberrechtsschutz. Vorgesehen ist aber, dass sie dieselbe Lizenz wie jQAssistant bekommen. Wie oben erwähnt ist die GPL zu erwarten, so dass auch die Plugins unter die GPL gestellt werden.

Beziehen und benutzen

Die Kontext E jQAssistant Plugins werden auf GitHub gehostet. Bis eine offizielle Version über Maven Central beziehbar ist, muss man sich den Quellcode von GitHub besorgen und per gradlew selbst bauen. Voraussetzung ist, dass der aktuelle Snapshot von jQAssistant im lokalen Maven Repository installiert ist.

Im Zielprojekt muss man dann jQAssistant und die Plugins als Dependencies eintragen. jQAssistant selbst kommt mit einer engen Maven-Integration. Kontext E hat einen Kommandozeilen-Runner gespendet. Somit kann man seine Projekte auch per IDE, mit einem Batch-Skript, per ant, gradle oder von anderen Tools und eigenen Projekten aus starten.