Friday Repost: Making Grails work behind an proxy server

The Friday Repost series are copies from my earlier writings. Since I don’t want to loose them, and they might prove useful to others, I’m reposting them on this blog.

Making Grails work behind an NTLM firewall (or using Grails without Internet)

Currently, I’m in the process of taking over an existing Grails project and migrating it to a different location. This location has been setup by me, so I have full control over it, except for one tiny detail: the Proxy Server.

The application used to work great, and installing plugins was a breeze: just do a good old ‘grails install-plugin fitnesse’, and the plugin would be installed. However, since this Proxy Server is not just any Proxy Server, but a Microsoft NTLM Proxy Server, we tried quite hard to make Java work with it, but at the moment the score is 1-0 for the Proxy Server. This gives us the following problems:

1) Our dependencies cannot be downloaded
2) Our plugins cannot be be downloaded
3) Some plugins cannot be installed anymore

Downloading dependencies
In our BuildConfig.groovy, we specified some dependencies like joda time and xstream. These dependencies are normally downloaded, but due to our Proxy issue, that doesn’t work.

Our solution to this is to use Nexus. Nexus is able to pass the firewall, and can download our dependencies. It serves as a transparent Maven proxy, which can also be used by Ivy, and solves our problem nicely. Besides, it’s a good idea to install a local Maven proxy anyway, mostly because of reliability, but also because of speed and making sure everyone uses the same dependencies.

Downloading plugins
Unfortunately, the Grails plugins are not in a Maven repository. However, a solution to this is to download the plugin zip (eg grails-fitnesse-0.2.zip), rename it (to fitnesse-0.2.zip), put it in the lib directory (even though I’m really not a fan of putting binaries in version control systems), and register it in the application.properties and BuildConfig.groovy.

The BuildConfig.groovy should look like this:

As you can see, everything is configured here: the dependencies are here, the location to nexus is configured, and there’s a plugin closure. This “plugins” closure defines the plugins, and allows you to place them in the ‘lib’ directory. Be sure to rename the plugin zip to remove the ‘grails’ part, else it doesn’t work. Maybe these zips are also available from a Maven repository, which would solve the issue of putting binaries in our source control system, but I haven’t been able to find them yet.

Plugin installation
Some plugins, like the jQuery plugin, but also like my own plugin, the syntax-highlighter, download resources from the Internet when installing them. I haven’t been able to work around that yet, but my current solution is to just not use those plugins. I downloaded the jQuery javascript libraries and attached them to the project myself, and since we only use jQuery which does that, this solution is also an adequate fix.

Conclusion
So, as you can see, with some effort, it’s possible to make Grails work behind a Proxy Server, or without internet at all. This is a good solution for build servers, so you can have reproducible builds without the need for an Internet connection.

Friday Repost: Preventing spam with Groovy and Grails

The Friday Repost series are copies from my earlier writings. Since I don’t want to loose them, and they might prove useful to others, I’m reposting them on this blog

When I created the jworks.nl website in Grails, one of my requirements was to have a blog. Since the website is created in Grails, and the Grails application doesn’t integrate easily with a standard blogging solution like WordPress, I decided to create my own blogging solution. But, since this proved to be much more painful than initially anticipated, I migrated the website to a new design and decided to use WordPress instead. However, the plugin creation was fun to do, so I’m sharing my experiences here.

The current blog consists of little more than just a BlogPost class, and a Comment class. This worked pretty well, and I got the blog up in no time. However, my blog was found by the spam bots, and it got spammed a little. In the first couple of days, I got 3 spam messages a day, which could be easily managed by hand (however still quite annoying). Then, the spam load increased to 10, 100 and then a 1000 per day. Then I got really annoyed, and decided it was time for a solution!

My solution was not to choose a Captcha. Captcha’s tend to bother the user, making it harder to post feedback. That’s not something I’d look forward to. So I created my own solution, based on how WordPress does it.

WordPress has their own spam blocking service, which is called Akismet. The nice thing about Akismet is that their service isn’t exclusive for WordPress, but they provide an API so others can also use it! Which is exactly what I did: I created a Grails plugin for Akismet, called (surprise surpise!): Grails Akismet! I’m sure you didn’t see that coming!

It’s very simple to use. Just do a ‘grails install akismet’, add your site plus the Aksimet key to your Grails configuration, and you’re good to go! The plugin provides a Grails Service, which can be injected into most Grails artifacts (like services, domains, and controllers). By using the ‘check for spam’ method in the AkismetService, a call to the online Akismet service is done, which determines if the comment is spam or not. Very easy, and very non-intrusive!

So, if you want to get started with the plugin, please check out the documentation and let me know what you think!

Friday Repost: Indexing documents with Tika and Grails

The Friday Repost series are copies from my earlier writings. Since I don’t want to loose them, and they might prove useful to others, I’m reposting them on this blog

In one of the previous blogposts, I described a way to easily upload images by using Grails commands. This blogpost further builds on that, by providing a generic way to index uploaded documents. These documents can later be retrieved by using the search option provided by the Grails Search plugin.

To index documents, we’re going to need two components: one of these components is the Grails Search plugin, as described above. The other component is Tika. Tika is a document indexer library which can index many kinds of documents like Text, Word, Excel, Powerpoint, but also more exotic document formats, like MP3 and FLV. Combined with Lucene, this provides a powerful combo.

So, how do we get this to work? Well, first we need the application from the previous blogpost. I refactored it a little to represent the new generic format better. I renamed all references from image to document (so now we have a CreateDocumentCommand instead of a CreateImageCommand, as well as all other references), and I moved the upload.gsp to the views directory, since redirection and forwarding works better that way. After the refactoring was done, I first installed the searchable plugin by typing:

grails install searchable

I also needed Tika, so I added it to the BuildConfig.groovy in the grails-app/conf directory, like this:

Grails and extra XML libraries really don’t mix well, so to prevent some nasty class loading issues, better exclude them here. What we need to do now is to tell searchable to index our documents when we upload them. This is quite trivial really, we can just annotate the Document class (which in the previous application was called ‘Image’) with a Tika converter. Then we have to create the converter, register it in Spring, and we should be done. So, first things first: let’s annotate the Document class.

In the Document class, add a searchable closure with a converter for the byte array, like this:

We now need to create the TikaConverter. We’ll create a TikaConverter.groovy in the src/groovy/com/example/converter directory. The class should look like this:

This class is responsible for extracting text from any kind of document uploaded to the system. To make this converter available, we have to register it in the Spring context, and then we’re ready to go. To register the Converter in the Spring context, add the following lines to grails-app/conf/spring/resources.groovy:

Now everything should be set in place. Browse to: http://localhost:8080/document-indexing/upload/show to upload a document. Any kind of document will do, as long as it contains some text. When the uploading went alright, the document will be available in the Lucene index. You can test this by going to http://localhost:8080/document-indexing/searchable and type in some text from the document you have just uploaded. The result should be a result page similar to Google.

GStringImpl cannot be cast to java.lang.String

Last week, we got a nasty error in our application:

java.lang.ClassCastException: org.codehaus.groovy.runtime.GStringImpl cannot be cast to java.lang.String

The stacktrace was not helpful at all, and showed that the error was caused by a NullPointerException in Melody, which was a false message. After some searching around, we found the following code to be the culprit:

What happens here, is that a GString is used as a map key for rendering the model in the view. It’s never a good idea to use GStrings as a Map key, since the hashCode of GStrings are not guaranteed to be stable. A better approach is to use a String instead, which can be done using the following code:

The above is our current fix, and we changed our CodeNarc GString as Map key to a prio 1 rule. Whenever a prio 1 error occurs, our build will fail, preventing future errors like this.

Testing Grails applications with Geb and JsTestDriver on a headless Bamboo CI server

In our project we are using two testing tools – Geb and JsTestDriver. Geb is a tool for functional testing of web applications and is basically a layer on the top of Selenium’s WebDriver so that the test you write are more groovy and more fun to write. JsTestDriver on the other hand is a tool for unit testing javascript in your project. Together with Sinon.JS which is a javascript mocking framework they give you an easy solution to get your javascript code unit tested.

Both of the aforementioned frameworks need real browsers to run. This is good on one side, because you are actually testing your application in real environment in which it will be used. On the other hand it is nasty if your continuous integration server is headless. You might ask how to run a browser if there is no window management / graphical environment installed. Fortunatelly there is a solution – Xvfb which provides a virtual screen. So now, all you need is to install it, make sure that it is started during booting and configure the testing tools to use a virtual screen instead of a real one.

Xvfb is quite popular and is probably packaged for most of the mainstream Linux distros – on Ubuntu you can easily install it using apt-get. Unfortunately it doesn’t come with service scripts nor is started during booting by default. After googling for a while I was able to find the folowing startup script for Xvfb which I put into ‘/etc/init.d/xvfbd’:

As you can see it is extremely simple as it biggest objective is to get the Xvfb running during booting. The approach to stopping is maybe not the cleanest one but my bash skills aren’t the best. Nevertheless note that in XVFB_OPTIONS variable we’re saying that we want Xfvb to create a screen number 99 for us. It’s important as we will use that value later on. Having a service script we now just have to link it from ‘/etc/rc*.d’ directories so it’s picked up during startup and shutdown of the server.

If you use Geb for testing a Grails application then running the tests is a piece of cake – all you have to do is call: ‘grails test-app functional:’. This is not the case for JsTestDriver tests which are executed using a standalone jar which is not integrated with Grails what so ever. As running the JsTestDriver involves setting a lot of switches and one of the parameters is actually the path to your browser I decided to create a bash script for it and then wrap it into a grails script so that it’s easier to run from Bamboo. That’s the source to the bash script which lands in ‘test/js’ subdirectory of your Grails application together with the JsTestDriver.jar and jsTestDriver.conf files:

And here comes the really simple ‘RunJsTests.groovy’ script:

Finally the Bamboo configuration. Apart from running the tests the most important part is to set the ‘DISPLAY’ environment variable in your Bamboo Grails task to ‘:99’. This will tell the browsers that are started to use the display number 99 which is handled by Xvfb. T

Our Grails Stats: but what are yours?

We’re currently working on a Grails project for over a year now, with, on average, around 8 people including testers. I thought it would be nice to share our project statistics, but I’m wondering how our statistics compare to yours.

Some information about our project:

  • Java Helpers: this is 100% generated (CXF/JAXB code).
  • Scripts: this consists of some scripts to set the build number, generate the above CXF code based on some WSDLs, and run our Javascript unit tests
  • Domain/Service: Most of our business logic is in the domain or service layer. There is still to much code in our controllers, and we are working on that to make our controller layer smaller.

I hacked the script to also display our GSP and Javascript files. The Javascript is not entirely accurate, since it also includes JQuery and some JQuery libraries. Our functional (Geb) tests are missing though. These are quite new in our application, but are taking a bigger and bigger portion of our tests.

So, I’m curious to see what your project statistics looks like. If you run:

grails stats

You’ll get the same format as above. Please post it as a comment, even if it’s only a small project, along with some project details you might find interesting to share, and let’s see if we can create some nice graphs out of this!!

Excluding fields and properties from marshalling using annotations in Grails

Recently the flexible and configurable marshalling system of Grails was mentioned on Twitter by Graeme Rocher as needing some more documentation. That was probably the outcome of the interesting blog post by J-Wicz on the matter. Rob Fletcher has also blogged about it quite some time ago.

The issue got my attention because in our project I needed to exclude some fields from marshalling on a DTO. The reason for that might be various, for me it was a fact that the DTO contained some intermediate data which was then used to determine a value that is being send to the frontend in a JSON format. One of the fields had to also be formatted so I created a getter for it but the original value was still marshalled which was not desired – you’re not using it then don’t send it to frontend, less data means better performance. For others it might be the fact that a marshalled object contains a big data structure which is not needed in the frontend.

I wanted the solution to be reusable so that it could be applied to multiple classes without the need to create a new marshaller for every one of them. I decided to mark the fields with a @DoNotMarshall annotation which would be then detected in my marshaller and the fields and properties annotated with it won’t be marshalled. I used the information from the abovementioned blog posts as the starting point.

So the first thing is the two annotations – one from marking the fields and accessors to be skipped from marshalling and the second to mark the class on which we want to configure the marshalling. I will explain why the second is needed later on.

The code for DoNotMarshall annotation:

And for the ConfigureMarshalling annotation:

Next we’ll need a test for our new marshaller:

Few things to note about the test class:

  • in setUp() we are registering our marshaller using JSON#registerObjectMarshaller(ObjectMarshaller<JSON>) method – you will also need to do the same in the bootstrap if you want to use the marshaller in a Grails application
  • AnnotatedClass and NonAnnotatedClass are our test classes for which we define which fields should be skipped when marshalling
  • there are two tests – one for testing if the fields and properties marked with @DoNotMarshall are not marshalled and one for testing if the @DoNotMarshall is ignored if the class itself wasn’t marked with @ConfigureMarshalling

And finally the marshaller implementation:

I decided to extend from ClosureOjectMarshaller (yes, it’s an ‘Oject’ marshaller, apparently a typo) which allows to specify a closure which is called with the marshalled object as the parameter and shall return another object (in my case a map) that will be marshalled. Thanks to that I don’t have to worry about any JSON building. I’m just returning a map from the closure where keys are property names and values are the respective property values and the result is later on JSONified for me.

The code inside the closure is a modified version of the code from GroovyBeanMarshaller with the sole difference in not marshalling the fields and properties on which the DoNotMarshall annotation is detected.

Finally note that I’ve overridden the supports method – thanks to that the marshaller will be used for any class that bears the ConfigureMarshalling and is a GroovyObject but for non annotated classes the original (in most cases the GroovyBeanMarshaller) marshaller will be used. As the AnnotationDrivenObjectMarshaller creates a map which is then marshalled it has the performance penalty of that extra step so if all the fields are to be marshalled it’s better to let the framework use the original marshaller.

To use this marshaller you need to drop it together with the annotations in the src/groovy directory of your Grails application and register it in the bootstrap as previously mentioned in the description of the test class. As the bootstrap is not available in test classes it might be a good idea to register marshallers in a static method of some utility class. You can then call it both from the bootstap as well as from the unit tests – this way you can easily replicate the marshalling configuration of a running application in the tests.

Happy annotating and marshalling!

Friday Repost: Remove Groovy MetaClass methods

The Friday Repost series are copies from my earlier writings. Since I don’t want to loose them, and they might prove useful to others, I’m reposting them on this blog

Whenever you are using metaclasses in Groovy or Grails unit tests, you might use Groovy’s metaClass power to mock the behavior of methods. However, Grails doesn’t fork the JVM for each test, so a nice side effect of this is that the mocked method will be available for the complete lifetime of the VM. Since Grails first executes unit tests (the test which usually contain the mocked code), and then followed by the integration tests, these tests can fail because you are maybe still using a mocked version of the method instead of the real one! This can often happen when mocking Grails methods, like dynamic finders, or named queries. The trick is to clean up these methods.

This is done by registering the metaClass when extending GrailsUnitTestCase. You’ll have to call the registerMetaClass(clazz) method, which does the following internally:

You can use this code in your tests in the following way, for example in the UserTests.groovy:

This way, the User class is registered with Grails to be cleaned up afterwards. This is done automatically in the teardown of the test, so you’re sure that no custom defined methods are available after running the test.

So, beware that whenever you use Groovy’s meta magic, be also sure to clean it up afterwards, since it might affect other parts of the code. You can check out my other blog post for a good solution to not forget the cleanup part!

Friday Repost: Easy file uploading in Grails

The Friday Repost series are copies from my earlier writings. Since I don’t want to loose them, and they might prove useful to others, I’m reposting them on this blog.

If you want to upload files with Grails, there are multiple ways of doing so. However, there’s at least one way which makes it really easy do so, while keeping the code as clean as possible. This blog describes the steps to take to get the file upload to work, while maintaining a very clean codebase.

To get started, first create a new Grails application by running the following command:

grails create-app imageupload

After you have created your application, create a new file called ‘UploadImageCommand.groovy’ in the src/groovy/com/example/command directory

The UploadImageCommand.groovy should look like this:

Using commands for all changes in the system is usually a good thing. This prevents any malicious code from entering the system, and also describes the
intent of the system by using sensible naming for the commands, in this case ‘Upload Image’.

Now we need two more things: the persistent object (the Domain Object) and the Controller. Actually, I would recommend using a Service for all state changes
of the system, because of the transactional support in services and for providing a uniform way of state changes. However, that’s beyond the scope of this blog, so we’ll
leave it for later and focus on the image uploading for now!

The domain class

To create the domain class, typing in the following:

grails create-domain-class com.example.domain.Image

This creates an Image.groovy file in the grails-app/domain folder. Open the file, and type in the following contents:

Also, create a controller:

grails create-controller com.example.controller.Image

This creates an ImageController.groovy file in the grails-app/controllers folder. Open the file, and type in the following contents:

Now, all you need is a simple form which will upload your image (or actually any kind of document). The easiest way to do this is create a new file called
upload.gsp and place it in the web-app directory . Make the file look like this:

Now, type grails run-app

And browse to http://localhost:8080/imageupload/upload.gsp. When you upload a file there, you will see the counter increase in the log file. Clean, fast and easy!

Testing Grails controller actions that use bindData method and render validation errors as JSON

Let’s assume that we have a controller action in a grails application that updates a user domain instance. We want name and surname to be updateable but login and password shouldn’t be updateble. To achieve that we will use the controller’s bindData() method with whitelisting. The action will be called using ajax so we also want to render JSON with validation errors in response if there were any. So our controller might look like that:

And the example User class might look like that:

Our next step will be unit testing the action. We want to make sure that only the whitelisted fields are copied from parameters (testUpdate) and also that the validation errors are rendered (testUpdateValidationError):

It turns out that those tests won’t pass. But not because of the fact that we have bugs in the code, it’s just that grails doesn’t do enough magic for our controller action to be unit testable out of the box.

First problem is that the bindData() method is not mocked in ControllerUnitTestCase. Fortunately thanks to this stackoverflow response we know how to easily mock it. To make it reusable let’s create a mixin out of it:

Now all that we have to do to make it work is to apply it on our test class and call the mockBindData() method in the setUp() method.

Second problem is that by default the JSON converter doesn’t know how to convert bean validation errors. But that’s easily fixable as well – all we have to do is register validation error marshaller in the testUpdateValidationError test:

After the aforementioned changes our passing test class should look like that: