Some while ago I wrote how we make sure that the documented and the actual architecture match using the idea of Executable Architecture Documentation. This time I want to re-visit this again, but now a little bit more as a tutorial.

An ultra-short recapitulation of the idea

arc42 is a widely used template for architecture documentation. Chapter 5 describes the Building Block View UML Package diagrams can be used to picture the building blocks. There is the current version of the arc42 template on GitHub. Out of the template in Asciidoc several other formats can be generated, but we go with Asciidoc. PlantUML is a Domain Specific Language to describe UML diagrams in textual notation and tooling to make standalone or embedded graphics out of this. jQAssistant scans Java projects into a Neo4j database and can be easily extended by plugins. Kontext E provides a Plugin for reading PlantUML package diagrams.

So the main building blocks described in the architecture documentation can be put into the same database where the actual architecture artifacts are located. Only one simple Cypher query is needed to compare the documented and actual package dependencies.

Add the PlantUML plugin to the jQAssistant configuration

First we need to add the Kontext E PlantUML plugin to jQAssistant in the build file. In the following sections I’ll give examples for Maven and Gradle

Maven

As described in the jQAssistant documentation you add another dependency to the jQA Maven plugin:

1
2
3
4
5
6
7
    <dependencies>
        <dependency>
            <groupId>de.kontext-e.jqassistant.plugin</groupId>
            <artifactId>jqassistant.plugin.plantuml</artifactId>
            <version>1.1.4</version>
        </dependency>
    </dependencies>

Gradle

Given you have a separate configuration for jQAssistant plugins:

1
2
3
    configurations {
        jqaRt
    }

you add the PlantUML plugin this way:

1
	jqaRt("de.kontext-e.jqassistant.plugin:jqassistant.plugin.plantuml:1.1.4")

Put the PlantUML diagram into arc42 Building Block View chapter

It is quite easy to put a UML diagram into an Asciidoc file. For the jQA PlantUML plugin itself it could look like this:

    ["plantuml","MainBuildingBlocks.png","png"]
    -----
    package de.kontext_e.jqassistant.plugin.plantuml.scanner {}
    package de.kontext_e.jqassistant.plugin.plantuml.store {
        package de.kontext_e.jqassistant.plugin.plantuml.store.descriptor{}
    }
    
    de.kontext_e.jqassistant.plugin.plantuml.scanner ---> de.kontext_e.jqassistant.plugin.plantuml.store
    
    -----

It gets rendered into a UML Package Diagram:

.

As you noticed, the example is from the jQAssistant PlantUML plugin itself. The rendered Asciidoc example Building Block View chapter looks like this:

Add the architecture documentation directory to the scan path

To make jQA also scan the architecture documentation, it must be configured to look into the very same directory.

Maven

Given the architecture documentation resides in ‘doc/architecture’, simply add this folder as a scan target like this:

1
2
3
4
5
6
7
    <configuration>
        <scanIncludes>
            <scanInclude>
                <path>doc/architecture</path>
            </scanInclude>
        </scanIncludes>
    </configuration>

Gradle

This is a snippet for scanning a multi-module Gradle project, again with an additional directory to scan for the architecture documentation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
    task(jqascan, type: JavaExec) {
      main = 'com.buschmais.jqassistant.scm.cli.Main'
      classpath = configurations.jqaRt
      args 'scan'
      args '-p'
      args 'jqassistant/jqassistant.properties'
      args '-f'
    
      rootProject.subprojects {
        args 'java:classpath::'+it.name+'/build/classes/main'
        args 'java:classpath::'+it.name+'/build/classes/test'
        args it.name+'/build/reports'
        args it.name+'/src/main'
        args it.name+'/src/test'
      }
    
      args 'doc/architecture'
    }

After scanning the project we get a graph like this:

That’s a screenshot of the Neo4j graph browser which I pimped a little bit to make it more expressive. If you try it out exactly at this point, you will notice that in the original scan there is no dependency from scanner to store in the real architecture. This brings us directly to the following section.

jQAssistant Concepts and Constraints

Now as all information is put into one database, expected and actual state can be matched using jQA Concepts and Constraints. In jQAssistant, Concepts and Constraints can be given in XML and Asciidoc. I’ll give the examples for both of them.

XML

In the style of this XML example we add the following snippets.

First, we need a package level as described here:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
    <concept id="package:PackageLevel">
        <requiresConcept refId="dependency:Package"/>
        <description>
            Set the level property of a package, 
            e.g. 1 for de, 2 for de.kontext_e and so on
        </description>
        <cypher><![CDATA[
			MATCH
				(p:Java:Package)
			WITH
				SPLIT(p.fqn,".") AS splitted, p
			SET
				p.level=SIZE(splitted)
			RETURN
				splitted, SIZE(splitted);
		]]></cypher>
    </concept>

This package levels are used to add some transitive package dependencies:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
    <concept id="dependency:TransitivePackageDependencies">
        <requiresConcept refId="package:PackageLevel"/>
        <requiresConcept refId="dependency:Package"/>
        <description>
            Add a DEPENDS_ON relationship to parents of a package P 
            from other packages up to the same level of the source package.
        </description>
        <cypher><![CDATA[
            MATCH
                (p:Java:Package)-[:DEPENDS_ON]->(p2:Java:Package),
                (parent:Java:Package)-[:CONTAINS*]->(p2:Java:Package)
            WHERE
                p.level <= parent.level
            CREATE UNIQUE
                (p)-[:DEPENDS_ON]->(parent)
            RETURN
                p.fqn, parent.fqn;
        ]]></cypher>
    </concept>

which come handy to find package dependencies in the wrong direction with this little Constraint:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
    <constraint id="dependency:WrongDirection" severity="critical">
        <requiresConcept refId="dependency:Package"/>
        <requiresConcept refId="dependency:TransitivePackageDependencies"/>
        <description>
            Finds package dependencies which are in the wrong direction 
            according to the documentation.
        </description>
        <cypher><![CDATA[
            MATCH
                (p1:PlantUml:Package)-[:MAY_DEPEND_ON]->(p2:PlantUml:Package),
                (p3:Java:Package)-[:DEPENDS_ON]->(p4:Java:Package)
            WHERE
                p1.fqn = p4.fqn
                AND p2.fqn = p3.fqn
            RETURN
                p3.fqn + "-->" + p4.fqn AS WrongDirection;
        ]]></cypher>
    </constraint>

Don’t forget to add all of that to the default group:

1
2
3
4
5
6
7
    <group id="default">
        <includeConcept refId="package:PackageLevel"/>
        <includeConcept refId="dependency:TransitivePackageDependencies"/>
        <includeConstraint refId="dependency:WrongDirection" 
                           severity="critical"/>
    </group>

That’s it.

Asciidoc

And now in the style of the Asciidoc example:

    [[default]]
    [role=group,includesConstraints="dependency:WrongDirection(critical)"]
    - <<package:PackageLevel>>
    - <<dependency:TransitivePackageDependencies>>
    - <<dependency:WrongDirection>>

    [[package:PackageLevel]]
    .Set the level property of a package, e.g. 1 for de, 2 for de.kontext_e and so on.
    [source,cypher,role=concept,requiresConcepts="dependency:Package"]
    ----
    MATCH
        (p:Java:Package)
    WITH
        SPLIT(p.fqn,".") AS splitted, p
    SET
        p.level=SIZE(splitted)
    RETURN
        splitted, SIZE(splitted);
    ----

    [[dependency:TransitivePackageDependencies]]
    .Add a DEPENDS_ON relationship to parents of a package P from other packages up to the same level of the source package.
    [source,cypher,role=concept,requiresConcepts="package:PackageLevel"]
    ----
    MATCH
        (p:Java:Package)-[:DEPENDS_ON]->(p2:Java:Package),
        (parent:Java:Package)-[:CONTAINS*]->(p2:Java:Package)
    WHERE
        p.level <= parent.level
    CREATE UNIQUE
        (p)-[:DEPENDS_ON]->(parent)
    RETURN
        p.fqn, parent.fqn;
    ----

    [[dependency:WrongDirection]]
    .Finds package dependencies which are in the wrong direction according to the documentation.
    [source,cypher,role=constraint,requiresConcepts="dependency:TransitivePackageDependencies",severity=critical]
    ----
    MATCH
        (p1:PlantUml:Package)-[:MAY_DEPEND_ON]->(p2:PlantUml:Package),
        (p3:Java:Package)-[:DEPENDS_ON]->(p4:Java:Package)
    WHERE
        p1.fqn = p4.fqn
        AND p2.fqn = p3.fqn
    RETURN
        p3.fqn + "-->" + p4.fqn AS WrongDirection;
    ----

Running the check

No special things have to be done to run the actual check. It is done in the normal jQAssistant run. So it works on the local machine as well as in the CI process quite easily.

Making HTML (or PDF) out of Asciidoc

One widely used toolchain is Asciidoctor with Asciidoctor Diagram to generate a nice set of HTML documents. There is also Asciidoctor PDF for a direct generation of PDF.

A complete example

These were a lot of snippets. Therefore I created a very basic example that contains all to make it run but nothing else. You can download the sources on GitHub, explore and play around.

If you do a

1
mvn verify

you’ll find four constraint violations.

Stay tune for follow up posts. They explain why the example is called “Uneven Modules” and what can be done for taming the architecture with jQAssistant.

Some closing words

There are of course many other arc42 chapters. Some of them are not suited to be checked automatically, but others may contain also checkable contents like the Design Decisions chapter. For example rules like ‘no java.util.Date anymore’ or ‘use log4j instead of java.util.logging’ are quite easy to enforce.

The Concepts and Constraints above do not cover two things:

• packages that were found but not documented
• packages that were documented but not found

I’m not sure if that is needed and/or useful. Everyone should decide this for the concrete project situation.

So now the CI build creates HTML/PDF architecture documentation and uses the same document to check the defined against the actual architecture. How crazy is this?

In this post you can read that it may surprisingly quick and simple to make use of jQAssistant for verifying module dependencies in C++.

Over the last year I was busy developing some C++ application. As usual it starts nice and clean, gets into production - and evolves. Although I employed analyses tools like Valgrind and CPPCheck I missed something like jQAssistant to check architectural rules. As I updated the architecture documentation again I noticed that the thing I really wanted was to check module dependencies. Sure, there are tools like CppDepend, Sonargraph etc., but I also wanted to take advantage of the Executable Architecture Documentation described in a previous post.

My first attempt was to use the Clang Tools for reading the source code and dump the AST to a file. It turns out that a C++ AST is considerably more complex than one may think in the first place. While I began to write a jQA plugin for that, it may take a while - but I wanted something working now.

My second thought was to dump the CLion PSI tree. But that would not work for CI builds.

But finally I had a nice idea reading the LLSA book: Carola Lilienthal describes that the Sotograph uses regular expressions to determine modules, patterns, layers etc. Because my project structure is very simple - in the src folder are subfolders for each module, containing only files and no submodules - it turns out that analyzing the #include declarations should be sufficient. As the Agile Principle says: “the simplest thing that could possibly work”. The effort for an experiment is very low, so I gave it a try to find out if it could possibly work.

I created a “plaintext” plugin for jQAssistant which does exactly that: import the plain text line by line into the jQA database. jQAssistant comes already with the notion of a “File” and a “Directory”. So I can create a relationship between two files which are connected by an #include with this Cypher statement:

1
2
3
4
5
6
7
8
    MATCH
        (x:File:Plaintext), (d:Directory)-->(f:File:Plaintext)-->(l:Line:Plaintext)
    WHERE
        l.text=~'#include.*' and l.text=~'.*../.*' and d.fileName=~'/.*' and l.text=~('#include.*'+x.fileName+'.*')
    MERGE
        (f)-[:DEPENDS_ON]->(x)
    RETURN
        d.fileName, f.fileName, l.text, x.fileName

Next step is to connect the directories where the connected files are located:

1
2
3
4
5
6
7
8
    MATCH
        (d1:Directory)-->(a), (d2:Directory)-->(b)
    WHERE
        (a)-[:DEPENDS_ON]->(b) and d1.fileName=~'/.*' and d2.fileName=~'/.*'
    MERGE
        (d1)-[:DEPENDS_ON]->(d2)
    RETURN
        a.fileName, b.fileName, d1.fileName, d2.fileName

But wait: I told you that I organized my source code so that one directory contains one module. So let’s mark the directories as modules:

1
2
3
4
5
6
7
8
    MATCH
        (d:Directory)
    WHERE
        d.fileName=~'/.*'
    SET
        d:Module
    RETURN
        d.fileName

Now it is simple to find dependencies

1
2
3
4
5
6
    MATCH
        (d1:Cpp:Module)-[:DEPENDS_ON]->(d2:Cpp:Module)
    RETURN
        d1.fileName, d2.fileName
    ORDER BY
        d1.fileName

or direct cycles

1
2
3
4
5
6
7
8
    MATCH
        (d1:Cpp:Module)-[:DEPENDS_ON]->(d2:Cpp:Module)
    WHERE
        (d2)-[:DEPENDS_ON]->(d1)
    RETURN
        d1.fileName, d2.fileName
    ORDER BY
        d1.fileName

This works indeed astonishingly well for my purpose. The effort was really very low because jQA brings a nice plugin concept and Neo4j Cypher supports the regular expressions. I gained interesting insights into how the architecture developed and what unintended dependencies I created while adding more features. Now it’s time to pay back some Technical Debt…

In her exciting talk Applying Java 8 Idioms to Existing Code, Trisha Gee states that demoing the refactoring to returning java.util.Optional could be a mess because so much caller statements have to be changed. You may find yourself easily in a similar situation every now and then if you have some spare time in the project and think by yourself: “Would’nt it be nice to do a quick refactoring now?” If you decide to return an Optional and get 538 compiler errors because of a wrong return type you can forget about ‘quick’. So how do you find a nice spot fitting in your refactoring time box?

It is just one simple Neo4j Cypher query if you are already using jQAssistant in your project. Let’s suppose you want to know how many callers are of method located in the de.kontext_e.techblog.service package. Here is the query:

1
2
3
4
5
6
7
8
    MATCH 
        (caller:Method:Java)-[:INVOKES]->(callee:Method:Java)<-[:DECLARES]-(t:Type) 
    WHERE 
        t.fqn=~'de.kontext_e.techblog.service.*' 
    RETURN 
        t.fqn, callee.name, count(caller) AS callers
    ORDER BY 
        callers

That’s it. Now you can assess the impact and choose wisely.

In her German book “Langlebige Software-Architekturen” Carola Lilienthal tells the story of an architect who wants to know which public methods are currently not called from outside the package (p. 117).

Sure, with a powerful tool like Sotograph she is using this is no problem - if you have the money. But can you achieve this also with Open Source tools? Having read some of my previous posts (especially the ones about jQAssistant) you already know the answer: yes, of course! I’ll show you how easy that is.

For this post, I created a little demo project with a Service and a Client calling this Service:

The Service has two methods

1
2
3
4
5
6
    public class Service {
        public void calledFromDifferentPackage(){
            onlyCalledInPackage();
        }    
        public void onlyCalledInPackage(){}
    }

and the Client calls one of them from a different package

1
2
3
4
5
    public class Client {    
        public void call() {
            new Service().calledFromDifferentPackage();
        }
    }

I scanned the project into a jQAssistant database and started the server for exploration. Now I can query for the public methods not accessed from a different package in three easy steps.

First step: put a label ‘Public’ on the public methods

1
2
3
4
5
6
            MATCH
                (c:Type:Class)-[:DECLARES]->(m:Method)
            WHERE
                m.visibility='public'
            SET
                m:Public

Second step: put a label ‘UsedFromDifferentPackage’ on methods which are called from a different package

1
2
3
4
5
6
7
8
9
            MATCH
                (t1:Type)-[:DECLARES]->(m:Method),
                (t2:Type)-[:DECLARES]->(p:Method:Public),
                (package1:Package)-[:CONTAINS]->(t1),
                (package2:Package)-[:CONTAINS]->(t2),
                (m)-[:INVOKES]->(p)
            WHERE
                package1.fqn <> package2.fqn
            SET p:UsedFromDifferentPackage

Third step: query for the methods which have no label ‘UsedFromDifferentPackage’

1
2
3
4
5
6
            MATCH
                (c:Type)-[:DECLARES]->(u:Method:Public)
            WHERE NOT
                u:UsedFromDifferentPackage
            RETURN
                c.fqn, u.name

Of course I could have done this in one more complex step. But I decided to separate the concerns in this way because most likely I would add some WHERE clauses in the third step to exclude public APIs, unscanned entry points, uninteresting packages, or examine only some submodules. As a nice side effect, the first two steps can be easily transformed into jQAssistant concepts and the third one into a jQAssistant constraint.

I published the source code on GitHub.

Code Retreats and the Game of Life

Code Retreats are a good way to improve our skills as Software Craftsmen. Soon there is the #gdcr15, reason enough to make the Game of Life a topic of a blog post.

Here is an example in Java:

Java. Java? Java + extensions!

Hm, Java? That is not Java. Well, it looks like somewhat like Java, but that tables, that colored constants, that initialization of the generation array, that operations? It’s not hard to notice: there are some extensions made to Java. That takes for sure some years or at least months for a single person to create such extensions! No, not really. Uh yeah, if you take the OpenJDK and put your extensions there - but we don’t. We take a Language Workbench. To be more precise: we take the MPS Language Workbench. Now it is easy to modularize and put together programming languages.

In the following sections we will have a closer look on each of the new language concepts. You can get the source on my GitHub account.

New Type: Coordinate

The Gol class starts like a quite normal Java class with a main method. But already the second method, run(), contains something special. As you see, Coordinate is like a build-in Java type. It could also be a class named Coordinate, but there is a tiny difference: look at the initialization of the arraylist. These are no calls to constructors, these are the notation (aka “Concrete Syntax”) of a Literal. Just like 5 for an integer or “foo” for a string.

What do we need to create a new type? Surprisingly not that much. Of course the type itself:

Note the little blue arrow. The CoordinateType extends something called “Type”. This is the exact Type all other Java types like integer, long etc. extend too. Because of that our new CoordinateType fits nicely into the existing Java type system.

Next there is the Literal:

Note again the blue arrow. The CoordinateLiteral extends an Expression - yes, again the very same Expression which is the base for all Java expressions.

Now we only have to tell the type system, that our CoordinateLiteral is of type CoordinateType:

With some MPS type system syntax, this basically declares the the type as described above.

That’s it. After five minutes or so we are able to use a new Java type. It does not yet do something useful, therefore we have to declare the semantics via a generator. But that is the topic of a different section down below.

Syntax sugar: alive and dead

In the next method, nextGeneration(…), a two colorful constants catch the eye. ‘alive’ and ‘dead’ are used like constants or enums, but have a custom color. We could also assign other properties like underlined text, italic or bold font, a different font size and so on. Can normal Java IDEs do that too? No - and perhaps that would be a nice issue to be filed in Eclipse, NetBeans or Intellij IDEAs backlogs: assigning representation properties to constants and enums.

And it’s really no big deal to get domain specific styled constants:

Again we extend an Expression. But that does not explain the different style of the text. This leads to a topic I did not mention so far: every new language concept needs or may declare how it is presented. Needs or may, eh? Yes, if we inherit from a language concepts which provides a decent representation we can go with it. If we don’t inherit or want to override the inherited style, we need to define a so called ‘editor’. Let’s have a look at the editor for the alive expression:

Hm, not that impressive. We see that it’s the ‘editor for concept AliveConcept’. And it’s the default one. Yes, we may declare more than one representation for a concept, e.g. for color-blind people. But let’s focus for now on only one editor. We also see that the #alias# should be shown. Alias? Yes, please go back to the Alive Expression picture. There it is: ‘alias: alive’. But how comes the color in? We can declare styles in a different tool window:

And the same goes for Dead Expression with the style ‘text-foreground-color : red’.

Decision Table

Now we come to one of the most powerful and most interesting things of custom representations via own ‘editors’. We are not limited to text like in normal IDEs. We could define also - tables! And not only tables, but even Java Swing components are allowed. So we could replace the alive and dead concepts with checkboxes. Or add some JavaDoc with explaining pictures. But for now let’s focus on the tables. In the nextGeneration(…) method we see a decision table. A content cell from the middle is taken if the column header and the row header both evaluate to true. If no constellation evaluates to true, the default value is taken. In fact, this decision table represents the core algorithm of the Game of Life:

Depending on if the current generation contains a cell and how many neighbors it has, it will be added to the next generation or not. And because the decision table also extends the Expression and the Type is declared boolean, the table can be used as condition in a regular if statement.

Mapping Table

In the neighbors() method we see again a table. This time no decision table but a mapping table. A single cell and a table were combined:

The bold plus sign indicates that it is no normal summation operation but an adapted version for the table: the left side of the plus - here a variable named ‘cell’ of type Coordinate - is added to every entry in the table. This results into nine new Coordinates. The middle one is the original cell itself, so it is no neighbor and will be subtracted again.

Extending Operations: plus and minus

Subtracted? A single value via minus operation from a collection in Java? This is only possible because we can also put additional semantics on operations as language extensions. Once again the type system feature of MPS comes to help and let’s us overload operations:

The first of the two new rules declares for plus and minus operators that the summation and subtraction of two things of type Coordinate is allowed and results into a Coordinate type.

The second new rule declares that a subtraction of a Coordinate from a Coordinate array is allowed and results into a Coordinate array.

So far it’s very nice that we can program with the extensions. But until now that program does not run.

Generate Java code

To let a program run, it has to be compiled or interpreted. Java programs were compiled into byte code. Until now we did not define how the bytecode for our new language concepts has to be generated. But do we really want to generate bytecode directly like Java does for the built-in keywords?

Let’s take a step back first and see what we did: we stacked a new language on top of an existing one. We created a new layer. Our new language extends the Java language. We should not bypass layers on the language stack and generate code for the language layer direct below the new language. Not that we are not constrained to extend only one language. A new one can extend as many languages as it wants. That’s why I chose the term ‘language layer’.

That said, we don’t generate bytecode, instead we generate Java code. For ‘alive’ and ‘dead’ it’s dead simple: ‘alive’ is replaced by a boolean ‘true’ and ‘dead’ by a boolean ‘false’.

Not really surprising is what we generate for the Coordinate type. In Java we would represent it by a class named Coordinate and so we generate it. To make that work, we have the Runtime Solution named ‘gol.runtime’. It contains only one class - the Coordinate. Exactly that Coordinate class is used as the generation target for the Coordinate type.

So if the Coordinate type is translated to the Coordinate class, the Coordinate literals is translated to a constructor call of the Coordinate class. Quite natural.

The nice thing about having the Coordinate class in the runtime model is that we can use it’s ‘sum’, ‘minus’ and ‘removeFromArray’ methods as generation targets for our overloaded operations ‘plus’ and ‘minus’. It is exact the same thing as the ‘add’ and ‘subtract’ methods in BigInteger and BigDecimal.

Now we are nearly done. But the hardest part comes as last. Translating the decision and mapping tables to valid Java code is not that simple. Therefor I would suggest to watch that video for a good and extensive explanation.

Conclusions

What did we learn? It’s quite simple to extend Java. Why should we do this? Nearly every Java programmer has some favorite missing language features, don’t you?

But there is a second and for me the more important reason: domain concepts can be integrated into a Java dialect which is specially created for the project in that domain. For the Game of Life, we added ‘alive’, ‘dead’, the Coordinate. This may be seen only as some syntactic sugar. But look again at the program: the color- and meaningful ‘alive’ and ‘dead’ words catch the eye. There is no syntactic clutter around creating a new Coordinate - no verbose Java ‘new Coordinate(1, 1), just a (1, 1). Image you can write a math formula just as - a math formula and not as a long Java methods with many words an no single math symbol. Notation matters, and notation is nothing else than the concrete syntax of a language. With language extensions we can define our own concrete syntax. A program can be much more optimized for reading. And source code is ten times more read than written. Or even more often.