Grails in large projects – Part 2

This is a follow-up of Part 1, so if you haven’t read that, please do so first 😉

In Part 2 I will track down all issues that came up around GORM, Hibernate and transactions:

GORM and its Active Record pattern

To be honest, when I first saw the possibility to write

def p = Person.findByFirstNameAndLastName(“John”, “Doe”) 

everywhere in the code, even with content assist from the incredible Intellij IDEA, I was really impressed. I liked the concept that all the boilerplate code that usually needs to be written in the persistence layer is not required any more (even if it’s not too much when using Spring Data), but this is actually also the big problem.

With great power comes great responsibility.

This is not only true to Spiderman, but also to every Grails developer. If it is possible to write Person.get(4) anywhere, then it will be written anywhere. In services, in controllers, in taglibs, even in gsps. Inside a transaction, outside a transaction, everywhere. Same for queries that I mentioned above. It’s simply because it is much more convenient to do this simple static method call than fetching the data from any service that I would first need to inject. And the worst thing in it: you simply cannot restrict it. You can’t restrict anyone to do

new Person(lastName:’Poison’, firstName: ‘Ivy’).save(flush:true)

from anywhere in the code. It’s even not so easy to find out all usages of such code, which is especially painful when doing refactorings.

Now, in smaller projects with 1-3 developers this is not so big an issue, as (hopefully) every developer knows the whole codebase, but when you have a distributed team with more than 10 developers, it is essential to restrict things like that.

Many possibilities to do the same

GORM is powerful and lets you do many things. The following code snippets all do the same:

Person.findByFirstName(“John”)
Person.where { firstName == “John” }.find()
Person.createCriteria { eq (‘firstName’, ‘John’) }.find()
new DetachedCriteria(Person).build { eq ‘firstName’, ‘John’ }.find()
Person.find(“from Person p where p.firstName = ‘John’”)

Probably I forgot some but these seem to be the most important ones. Now it is clear to me that you also have those possibilities when running Spring Data and Hibernate, but in Grails everything looks quite similar, and you can call every code from everywhere, and every developer does it differently and mixes up the strategies. Still, everyone should know the differences between those techniques, which leads me to the

Testability of GORM

Dynamic Finders, Where Queries and Criteria can be tested with unit tests by mocking the domain classes, whereas HQL criteria can’t. You need integration tests for them. But when running integration tests, Grails starts up the whole application context, you have no chance to only initialize those beans that you need in your test and then run the queries against an H2 database, like we would do in regular spring applications. Especially when the application gets big this can lead to a startup time of 30 seconds to a minute for one single integration test. Execution of subsequent integration tests is faster though, as they share the same application context, but in that case you need to cleanup everything again (e.g. local caches) after each test.

Back to unit-tests, there seem to be some significant performance improvements from Grails 2.2 to 2.3, but unfortunately we can’t upgrade due to some major issues that we reported (I will come to that point in one of the next blog entries). Besides that you need to be aware that when testing queries (finders, where-clauses, criteria) in unit tests, you do that against an entirely different environment than in production. Grails holds some kind of advanced HashMaps as data storage and executes everything in memory. In most of the cases this behaves exactly the same as in production, but there are some points where it doesn’t.

One example:

class Person {
User user
}

In production you can call:

def userId = Person.get(1).userId

which is convenient as the User entity is not fetched. But in unit tests, this does not work, you need to call:

Person.get(1).user.id

Besides that, you need to mock all domain classes that are somehow involved, so that the GORM methods are applied on the classes. At least in Grails 2.2 this leads to significant memory consumption when running some 100 tests in a row.

But even if you are running only one single unit test, for every execution Grails starts up and even one unit test never needs less than 2-3 seconds. If I remember back the good old Java times running JUnit or TestNG where I could execute some 100 tests in that time, I get tears in my eyes.

And when your code also needs to do HQL queries, then you need integration tests anyways. Same, when you hook on the different GORM event methods (beforeUpdate, afterUpdate), I will talk about those methods more in detail in a few seconds.

But what really frustrates me most is that if you are doing errors in where-queries or criteria (e.g. injecting Non-Domain-Objects as query parameters, calling groovy methods inside the where-queries and so on), then Grails fails silently. It doesn’t warn you that you can’t call this method inside a where-clause (as this finally needs to be translated to SQL), it just silently fails without any warnings and returns incorrect results.

So, if I started a project from scratch here or needed to design Grails 3.0, then I would take GORM out of the game and replace it with Spring Data. Having Repositories as Java Interfaces that can easily be mocked with Mockito in Unit-Test, having POJOs with JPA annotations as entities. I then have one clear data access layer, I have one interface per entity where I see all the data access code including all queries (except if I use Hibernate criteria, but this is not required in most of the case) which leads to a clear layered architecture. I didn’t yet check if this would be possible right now (replacing GORM with Spring Data), but for sure it wouldn’t be Grails’ best practice.

Services in domain classes

Also something that you can see in some popular plugins: services in domain objects. The Spring Security Plugin uses this technique to inject the SpringSecurityService to the User to encode the password before saving or updating the entity.

I can remember a discussion at my former employer where one colleague wanted to implement a doSerialize()-Method on every entity and/or DTO with the reason that every class should know itself how to serialize (binary, as XML, to any other stream,…). In my opinion, this is bad hit against the principle “separation of concerns”. Why should a POJO be aware of XML serialization? And why should a User know how to encode its password? Why should it even know that this is required? There is a service layer which is responsible for that. Having a service in the entity makes it dependent on that service, thus harder to test, and so on. Same for the beforeUpdate() method, which brings me to the next topic:

Implementing GORM events on the entities

The methods beforeInsert(), beforeUpdate(),… on the domain classes seem to be a convenient way to react properly on events that are triggered by GORM right aways before or after an entity is saved or loaded.

The documentation already warns you about the possible sideeffects: “Do not attempt to flush the session within an event (such as with obj.save(flush:true)). Since events are fired during flushing this will cause a StackOverflowError.”

But if you get some lines down in the docu you will find the following code:

class Person {
def securityService
String firstName
String lastName
String lastUpdatedBy
static constraints = {
lastUpdatedBy nullable: true
}
def beforeUpdate() {
lastUpdatedBy = securityService.currentAuthenticatedUsername()
}
}

So we have a service injected in the domain object, called in the beforeUpdate() method. Guess what’s happening when you try to save such an object outside of a transaction in case the SecurityService is transactional? Inside of beforeUpdate you open a new transaction and you may end up in the troubles mentioned above. With those few lines you can do so much pain if you are not careful because it’s absolutely not transparent what is going on behind.

Also never try to do GORM queries from those event methods, as they will also flush the session in advance.

So our recommendation:

  • Do not use the GORM event methods
  • Do not use services in domain objects

Possibilities to access the Hibernate session

The next thing is a tiny parameter that can be passed to the save()-Method as mentioned above, to flush the hibernate session. You can run in big troubles when doing this, as perfectly explained in this blog article. The experience that we made is that if developers notice that some entities are not stored correctly, they try some flushes until it works.

Unfortunately the default controller that is generated by Grails’ scaffolding mechanism (grails generate-controller) does exactly that; it does manipulation of the entities directly in the controller, not using any transaction, but manual flushes. Yes, you can enhance all those scaffolding templates and generate services, but why does Grails deliver that code as best practice? Probably because it is short and cool-looking on the first sight.

Also, the possibility to easily open a new hibernate session (Person.withSession(…)) is something that gives the developer access to low level infrastructure which he usually should not care of.

Default transaction settings

Grails’ convention is to have services transactional and controllers non-transactional. Even though this can be overridden by using Spring’s @Transactional, I do not understand why Grails decided it to do exactly like this, because this leads to multiple transactions when a controller method accesses more than one service method (which is the usual case for bigger pages). Personally, I think that generally one web request should lead to exactly one transaction on the database, but if you use the default transaction settings you will most likely have much more than one (and remember, a transaction means a roundtrip to the database).

If you are calling transactional services from a taglib, then it is even worse. Because even if you annotate your controller with @Transactional then at the time of rendering the taglib, this transaction is already closed and every rendering of a taglib will open a new transaction. If you are iterating over elements in a GSP, rendering them with a taglib that calls a transactional service, then you have n roundtrips to the database, even if you don’t fetch data from there.

Now one could say: ok, simply do not call services from TagLibs. Well, then I say: This is what we enforced. But you will see that in many best practices from Grails and in featured plugins, last but not least it is possible, and what is possible, will be done.

So our recommendations:

  • make all services non-transactional by default;
  • use @Transactional on those service methods (only on the methods, not on the whole class) where it is required.
  • use @Transactional on controller methods where it is required.
  • Do not call services from taglibs, at least no transactional ones.

I’d also recommend everyone to configure the logging system in Config.groovy for development mode like this:

debug 'org.hibernate.SQL'
debug 'org.hibernate.transaction'
debug 'org.springframework.transaction.support'

Now open any page in your grails application and have a look at your logging console. Did you expect that?

What we also did is to write a small widget that prints the hibernate statistics of each web request on top of each page (number of transactions, flushes, entities fetched, cache hit….) in development mode. Probably I’m gonna extract this code into a grails plugin and make it available on our Github account. It also helped us a lot to reduce the unrequired database hits.

Another example: If you install the spring security plugin and run the s2-script, then you will get a User domain class implementing the UserDetails interface from Spring Security. The GormUserDetailsRepository will load this entity and put it into the SecurityContext; if configured, then your web server will try to serialize this object into the session cache. This is problematic in many ways, you again have lots of unrequired database accesses when fetching the current user or evaluating the roles for that user, and even if everything is in the 2nd level cache, Hibernate is busy doing unrequired flushes. Finally, we ended up in implementing our own UserDetailsService with our own simple implementation of UserDetails as a POJO, without any database access and a local Guava Cache.

That’s it so far from the data access layer, probably some more things come up in the next few posts concerning these topics as well.

In the third blog post I will talk about our experiences with Grails as a framework itself, Groovy, modularization and the build system.

Hope you liked it, waiting for your comments. See ya!

Vorheriger Beitrag
Grails in large projects – Part 1
Nächster Beitrag
Frisch aus dem Druck!

Related Posts

No results found

7 Kommentare. Hinterlasse eine Antwort

Thanks for sharing your experience which is very valuable for me because I’m currently learning Grails.

Looking forward to the upcoming parts.

Antworten

If you go deeper down the rabbit hole you will notice that even though the criteria API is supposed to work in unit tests it doesn’t do so as soon as you do joins or use properties of entities in a single criterion – but Grails won’t fail, it just silently returns empty or wrong result sets in unit tests.

If you combine @Transactional with @Secured annotations you cannot (easily) control the precedence and by default permission checks will be done before the transaction is started, which means that they are done in a different transaction. Besides violating atomicity this means that the security context is basically loaded twice from the database (not cache!) – once for the permission check and once in the controller/service itself where you most likely want to know who made the request.

… and then there is that thing that GORM dynamically adds id, version, and errors fields to entities – even multiple times once for each class in your entity hierarchy. That in fact does mean that a class has multiple fields with the same name (!) and it is more or less random which field is accessed when accessing entity.id in your code. If you want to reliably get the id of your entity you have to use getId() explicitly.

I could go on forever …

Thanks, Klaus, for summarizing all that!

Antworten

[…] Eine neue Sprache zu lernen habe ich nicht mehr geschafft, obwohl es mit Erlang, Dart und Scala doch einige mir noch unbekannten Sprachen gibt, die mich interessieren. Immerhin bin ich mit Grails weiter gekommen und je mehr ich mich damit beschäftige, desto größer wird meine Skepsis. Ich habe mich mit Frameworks noch nie so richtig wohl gefühlt, die versprechen einem die “ganze” Arbeit abzunehmen. Irgendwann muss man dann doch Zeit aufwenden um zu verstehen, was da genau im Hintergrund passiert … das ist bei Hibernate auch so. Ich bin aber immer mehr davon überzeugt, dass diese Zeit besser investiert ist, wenn man die Konzepte dahinter lernt (SQL im Fall von Hibernate) und das Gelernte selbst anwendet. Meine Berufskollgen von Catalysts sind einer ähnlichen Ansicht zu dem Thema bezogen auf Software-Entwicklung mit Grails. […]

Antworten

Hello,

Great introduction articles, still waiting for your 3rd post, i have read about Grails applications modularization and still looking forward your point of view, good luck in your future projects ..

Antworten

Im studying grails but I dont like it because its very complex to deploy on real world java ee aplication server like weblogic. The guilty isnt groovy but the rails shit… I really dont like this framework, I would prefer using groovy with ext.js and webservices, I think would be much more productive.
Im a big fan of groovy.
cya.

Antworten

Hi!

“What we also did is to write a small widget that prints the hibernate statistics of each web request on top of each page (number of transactions, flushes, entities fetched, cache hit….) in development mode.”

Have you released that widget yet? What classes did you monitor to retrieve such stats? It would be really, really helpful 🙂

Thanks!

Antworten

Hello,
Would love to see that plugin come to life.
Thanks

Antworten

Schreibe einen Kommentar

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit * markiert.

Bitte füllen Sie dieses Feld aus
Bitte füllen Sie dieses Feld aus
Bitte gib eine gültige E-Mail-Adresse ein.
Sie müssen den Bedingungen zustimmen, um fortzufahren

Menü