An integrated SBT + IDEA Scala Development Setup

Posted by Mathias in [scala]

12 Oct 2010

One of the more discussed topics at the Scala Lift Off in London last week was the Scala development setup favored by the various people. There are quite a few options, preferences depend to a large extend on the individual likings. However, there appeared to be a clear preference for SBT as the build tool of choice and a lot of people use Jetbrains IntelliJ IDEA along with its rapidly maturing Scala Plugin for writing Scala code.

So, since documentation on somewhat more complex dev setups involving SBT and IDEA is still somewhat sparse, I decided to publish my recent experiences with SBT, IDEA and a multi-module Scala project. Contrary to what I said in an earlier post SBT actually seems to offer a few advantages over building with IDEAs Scala Plugin or other build tools like Apache Buildr, the main ones being compilation speed and strong support from an active Scala community.

In this post I am going to lead you through setting up a multi-module Scala project from scratch, have it build with SBT, integrate it as much as currently possible with IntelliJ IDEA and put it under version control with git. My experiences with this setup stem from OS/X but there should be no reason why it shouldn’t work in a very similar fashion on other platforms.

This is a somewhat lengthy post, with all the steps spelled out in detail in order to properly support people new to IntelliJ IDEA. If you want to skip through the nitty-gritty-ness and simply look at the final end-product you can simply clone this github repo as a template for your own endeavors.

Ok, let’s get started. The sample project contains three sub projects (aka project “Modules” in IDEA speak):

  • Backend
  • Frontend
  • Utils

They don’t actually contain any meaningful code, they merely serve as stubs for your own modules.

Setup Directory Structure

Open a terminal, create a new empty directory for your project and cd into it.
Then create the following sub directory structure:

  Backend
    src
      main
        resources
        scala
      test
        resources
        scala
  Frontend
    src
      main
        resources
        scala
      test
        resources
        scala
  Utils
    lib
    src
      main
        resources
        scala
      test
        resources
        scala
  lib
    scala
  project
    build
    plugins

In order to create this structure in one go run this command in your project root directory:

mkdir -p Backend/src/main/resources Backend/src/main/scala Backend/src/test/resources Backend/src/test/scala Frontend/src/main/resources Frontend/src/main/scala Frontend/src/test/resources Frontend/src/test/scala Utils/lib Utils/src/main/resources Utils/src/main/scala Utils/src/test/resources Utils/src/test/scala lib/scala project/build project/plugins

SBT can set up a project directory structure by itself, however this does not seem to work for projects containing sub projects. It also does not create the lib directories we are going to use later.

Setup SBT Project

I’m assuming you already have SBT installed. If not go ahead and do so. (At the time of writing the current version is 0.7.4.) In order to manually setup an SBT project we need to create three files. So go ahead, fire up your favorite text editor (not yet IDEA) and create project/build/Project.scala with this content:

import sbt._
class Project(info: ProjectInfo) extends ParentProject(info) {
  lazy val backend   = project("Backend", "Backend", new BackendProject(_))
  lazy val frontend  = project("Frontend", "Frontend", new FrontendProject(_))
  lazy val utils     = project("Utils", "Utils", new UtilsProject(_))

  class BackendProject(info: ProjectInfo) extends DefaultProject(info) {
    val dependsOnUtils = utils
  }

  class FrontendProject(info: ProjectInfo) extends DefaultProject(info) {
    val dependsOnUtils = utils
  }

  class UtilsProject(info: ProjectInfo) extends DefaultProject(info) {
  }
}

This is the main project “buildfile” which defines the main projects and the three sub projects. As you probably have guessed the module projects “Backend” and “Frontend” depend on the “Utils” module. For the time being no other dependencies are defined.

Create a stub project/plugins/Plugins.scala with this content (not really required for the minimal setup, but a good placeholder for your future plugin definitions):

import sbt._
class Plugins(info: ProjectInfo) extends PluginDefinition(info) {
  // plugin definitions/dependencies go here
}

Finally create project/build.properties with this content:

project.organization=ExampleCompany
project.name=ExampleProject
sbt.version=0.7.4
project.version=0.1.0
build.scala.versions=2.8.0
project.initialize=false

With all this in place we are done with the SBT setup.
Now run sbt update in the project root directory. You will see SBT download a few artifacts, issue four warnings about “No dependency configuration found” and end with the line
[success] Build completed successfully.
If you want you can fire up the SBT interactive mode by running sbt and then issue a compile and/or test command at the prompt. SBT will run the tasks (which do not do anything yet) and report success.

Setup Scala Libraries

One of my development principles is to always get the source code for all libraries my projects depend on and make them available to the IDE. Not only does this greatly ease debugging (since you can simply step into all libraries), it also reduces your dependence on proper documentation for your dependencies (since you can simply look up what the method you are calling actually does) and generally improves your code reading abilities. You can get great insight into the inner workings of other peoples code and many times learn a thing or two from it.
The Scala library itself is no exception, so let’s get its sources:

  • Go to http://www.scala-lang.org/downloads and download the scala-2.8.0.final.tgz or scala-2.8.0.final.zip.
  • Unpack and move the scala-compiler.jar and scala-library.jar from the lib sub directory of the archive into the lib/scala directory underneath your project root.
  • Repeat the latter with the scala-library-src.jar from the src sub directory of the archive.

Your lib/scala directory should now contain the three files scala-compiler.jar, scala-library.jar and scala-library-src.jar.

Setup IDEA Project

With the SBT side and Scala artifacts all ready we now need to create an IDEA project definition that integrates as much as possible with the SBT side. We want to be able to do the following three things:

  • Edit all source code from within IDEA in order to be able to leverage all the features of the Scala plugin.
  • Compile either with SBT or IDEAs Scala plugin. Normally SBT will be the better choice (since it’s faster and offers continuous/triggered compilation) but sometimes it’s nice to be able to “see” compiler errors directly in IDEA in order to be able to directly jump to the error location.
  • Run tests with either SBT or IDEA. Again SBT is great for quickly and regularly running all or a large part of your tests, however, being able to run a test from within IDEA gives us “clickable” stack traces and proper diffing (e.g. when run with the TestNG runner).

One option of achieving this setup is to use SBT-IDEA plugin, which does its best to create a proper IDEA setup from an existing SBT project definition. At the time of this writing its developers are very active and respond to issues quickly. Definitely worth checking out.
However, in this post I am following the manual route, which is a good way to really understand all the inner working of a multi-module IDEA project. Something that will also help you in the case of unexpected problems with a setup created by the plugin.
So, for an IDEA setup that fulfills all the above requirements follow these steps (I am not showing screenshots but rather rely on “menu/dialog element” paths for brevity):

  1. Install the latest IDEA release (9.0.3 at the time of this writing, IDEA X will not be significantly different, you can also use an IDEA X EAP build if you want).

  2. Install the latest Scala Plugin (via “File/Settings/Plugins/Available”), restart IDEA.

  3. Create the project

    • Open the “File/New Project” wizard
    • Choose “Create project from scratch” and click “Next”
    • Do not enter a name (IDEA will do this automatically)
    • Click the ”…” button at the end of second line, navigate to your project root directory and click “Ok”
    • Leave the “Project file format” at “.idea (directory based)”
    • Uncheck “Create module” (we will create the modules in a separate step)
    • Click “Finish” (IDEA will open the “Project Structure” dialog after a short moment)
  4. Create the first module

    • In the “Project Structure” dialog click the ”+” icon at the top of the dialog window to open the “Add Module” wizard
    • Choose “Create module from scratch” and click “Next”
    • Do not enter a name (IDEA will do this automatically)
    • Click the ”…” button at the end of the “Content root” line, navigate to the “Backend” directory underneath your project root directory and click “Ok”
    • Make sure the module type is set to “Java Module” and click “Next”
    • Choose “Do not create source directory” and click “Next”
    • Select the 1.6 JDK (1.5 would probably work as well) or configure a JDK if you haven’t done so already, click “Next”
    • Do NOT check the Scala facade on the next screen (we will add it later), just click “Finish”
  5. Configure the source, test and excluded directories for the module

    • On the “Sources” tab of the module expand all directories in the tree on the right
    • Mark the resources and scala directories underneath src/main (ctrl-click) and click on “Sources”
    • Mark the resources and scala directories underneath src/test (ctrl-click) and click on “Test Sources”
    • Mark the target directory and click on “Excluded”
    • Switch to the “Paths” tab
    • Choose “Use module compile output path”
    • Use the ”…” button on the “Output path” line and select the “Backend/target/scala_2.8.0/classes” directory, click “Ok”
    • Use the ”…” button on the “Test output path” line and select the “Backend/target/scala_2.8.0/test-classes” directory, click “Ok”
    • Click “Apply” in the lower right corner of the “Project Structure” dialog
  6. Repeat steps 4 and 5 for the “Frontend” and “Utils” modules.

  7. Also exclude the lib directory underneath the “Utils” module by marking it in the modules “Sources” tab dir tree on the right and clicking “Excluded”

  8. Add a module dependency from module “Backend” to module “Utils”

    • Select the “Backend” module, switch to the “Dependencies” tab
    • Click the “Add…” button on the right, choose “Module dependency”
    • Select the “Utils” and click “Ok”
  9. Repeat step 8 for the “Frontend” module

  10. Create the “Scala Library” and “Scala Compiler” libraries

    • Select “Libraries” under “Project Settings” in the list control on the left of the “Project Structure” window
    • Click the ”+” icon at the top of the dialog window to open the “Create Library” dialog
    • Enter “Scala Library” and click “Ok”
    • Select the “Utils” module in the “Choose Modules” dialog and click “Ok”
    • Click “Attach Classes…” on the right and navigate to the lib/scala sub directory underneath your project root
    • Select the scala-library.jar and click “Ok”
    • Click “Attach Sources…” on the right and navigate to the lib/scala sub directory underneath your project root
    • Select the scala-library-src.jar and click “Ok”
    • Click the ”+” icon at the top of the dialog window to open the “Create Library” dialog
    • Enter “Scala Compiler” and click “Ok”
    • In the “Choose Modules” dialog click “Cancel” (we only need the compiler library for the Scala facet, not for any modules)
    • Click “Attach Classes…” on the right and navigate to the lib/scala sub directory underneath your project root
    • Select scala-compiler.jar and scala-library.jar, click “Ok” (the compiler also needs access to the Scala library)
    • Click “Apply” in the lower right corner of the “Project Structure” dialog (IDEA will start updating project indices)
  11. Create Scala facets for all modules

    • Select “Modules” in the “Project Settings” on the left
    • Right-click “Backend”, choose “New/Scala”
    • Select “Scala Compiler” in the drop-down for “Compiler library”
    • Check “Deprecation warnings” and click the bottom right “Apply” button
    • Repeat the last three steps for the “FrontEnd” and “Utils” modules
  12. Make the dependency of the “Utils” module on the “Scala Library” transitive

    • Select the “Utils” module, switch to the “Dependencies” tab
    • Check the “Export” checkbox to the left of the “Scala Library” entry in the dependency table, click “Apply”
  13. Close the “Project Structure” window by clicking on “Ok”

Your IDEA project is now (almost) properly set up. We can now add additional library dependencies and, later on, some sample code.

Setup Dependency Libraries

In order to test our setup we will add some example code. Since we also want to test the dependency handling we are going to use something that relies on Configgy, a very popular config library for Scala.
Configgy is a good (or rather bad) example also in another way: It does not publish source or doc jars by itself. A bummer for everyone wanting to easily tie in the source code (see above). My common solution to this problem (and there are many libraries that exhibit this behavior) is to quickly jar up the sources myself. For Configgy I have done it too, so download this ZIP and unpack its contents to Utils/lib/configgy/.
When a library does not make its sources readily available via a Maven or Ivy repo you are basically on your own. Since in my dev environment dependency management it not a huge problem I commonly do not rely on public artifact repositories but rather download or build the respective jars myself. This also has the advantage of resulting in very lightweight and fast SBT project definitions, since there is no repository/version configuration to be done. I simply put all the library jars my project depends on into the lib directory underneath a module sub directory and SBT will automatically find and use them. Additionally they are then easily made available to IDEA.

For this example we have put the Configgy jars into the lib directory underneath the “Utils” module. SBT recognizes all jars in this path as a dependency of “Utils” and makes them available to all other module depending on “Utils” transitively. This means, you can easily “feed” jars to any module in your project by having one common module that every other module depends on and using only its lib sub directory as dependency “repository”. Of course, if this is too coarse-grained for your taste you can also use lib directories underneath the other modules.

Making the libraries (in this case Configgy) available to IDEA requires a bit more work. Contrary to the “Scala Library” above we create the dependency on Configgy as a “Module Library” in IDEA, in order to mimic the workings on the SBT side:

  • Open the “Project Structure” dialog (via “File/Project Structure”) and select “Modules” on the left
  • Select the “Utils” module and switch to the “Dependencies” tab
  • Click on the “Add” button on the right and choose “Module Library”
  • Enter “Configgy” as library name and use the buttons on the right to add the main jar (“Attach Classes”), source jar (“Attach Sources”) and doc jar (“Attach Documentation”) in the Utils/lib directory underneath your project root.
  • Click “Ok” to close the “Configure Module Library” dialog
  • Check the “Export” checkbox next to “Configgy” in order to make the dependency transitive
  • Click “Ok” to close the “Project Structure” dialog

In order to demonstrate test integration between SBT and IDEA we need two other dependencies: TestNG and ScalaTest. You can download them from here and here, put them into Utils/lib/testng (Utils/lib/scalatest respectively) and set them up as Module Libraries for module “Utils” inside of IDEA in analogy to the instruction for Configgy above.

Add Sample Code

In order to test this setup we add three small files to the “Utils” module.

  • Expand the “Utils” module in the “Project” tree view on the right and right-click onto the src/main/scala directory
  • Select “New/Scala Class” and enter “org.example.NameConfigured”
  • Replace the file contents with this:
package org.example
import net.lag.configgy.Config

class NameConfigured(config: Config) {
  val name: String = config("name")
}
  • Right-click onto the src/test/scala directory and select “New/Scala Class”
  • Enter “org.example.NameConfiguredTest”
  • Replace the file contents with this:
package org.example
import net.lag.configgy.Config
import org.scalatest.testng.TestNGSuite
import org.scalatest.matchers.ShouldMatchers
import org.testng.annotations.Test

class NameConfiguredTest extends TestNGSuite with ShouldMatchers {
  @Test
  def testConfiggyConfiguration() {
    val config = Config.fromResource("test.conf", getClass.getClassLoader)
    new NameConfigured(config).name should equal ("Mathias")
  }
}
  • Finally, right-click onto the src/main/resources directory and select “New/File”
  • Enter “test.conf”
  • Replace the file contents with this:

name = “Mathias”

This is all the code we need in order to test the testing and resource handling.

Test the Setup

Back in the terminal go into SBT interactive mode by running sbt and kick off the test with test. This will compile everything and run the (single) test in module “Utils”, which should succeed without problems.

Now switch over to IDEA where we will try to have the tests run and succeed as well without having to resort to the Scala compiler (since SBT already compiled everything):

  • Select “Run/Edit Configurations” from the menu
  • Click the “Edit Defaults” button in the bottom left area of the dialog
  • Go through all entries on the left and, for each, deselect the “Make” checkbox at the bottom of the panel on the right
  • Close the two open dialogs with “Ok”

Now, we should be ready to run the NameConfiguredTest in the “Utils” module. Right-click the NameConfiguredTest class underneath the “Utils” module in the project view on the left and choose “Run/NameConfiguredTest” (TestNG). The test will start but throw an net.lag.configgy.ParseException: Can't find resource: test.conf.
The reason is that SBT copies the test resources from the Utils/src/test/resources directory into Utils/target/scala_2.8.0/test-resources but IDEA does not know about this path, it only runs the test with the Utils/target/scala_2.8.0/test-classes directory in the class path.
Luckily, this can be fixed with a little “undocumented” feature:

  • In IDEA double-click the Utils.iml file underneath the “Utils” module
  • Change line 13 from
    <output-test url="file://$MODULE_DIR$/target/scala_2.8.0/classes" />
    to
    <output-test url="file://$MODULE_DIR$/target/scala_2.8.0/classes:$MODULE_DIR$/target/scala_2.8.0/resources" />
  • Change line 14 from
    <output-test url="file://$MODULE_DIR$/target/scala_2.8.0/test-classes" />
    to
    <output-test url="file://$MODULE_DIR$/target/scala_2.8.0/test-classes:$MODULE_DIR$/target/scala_2.8.0/test-resources" />
  • Select “File/Save All” and reload the project

Basically IDEA allows the test output “directory” to contain more than one entry separated by the path separator (which is ’:’ on UNIX-based systems and ’;’ under Windows). By tweaking that entry we can force-feed the the correct classpath also used by SBT to the IDEA test runner. Rerun the NameConfiguredTest, which should now go through smoothly.
In order to also fix the classpath problem for the other modules you need to apply the same change to their .iml files.

As a final test run the “Build/Rebuild Project” menu item. IDEA should properly rebuild the whole project without any errors. This means we have achieved all our goals: We can use all the features of IDEAs Scala plugin, can build with either SBT or IDEA and run our test with either SBT or IDEA. If you run your Scala test with TestNG under IDEA you even have access to the built-in diff support for assertEquals(String, String), something I have come to really like (finding “bugs” in larger multi-line strings, e.g. containing some tree printout, is a breeze when IDEAs excellent diff tool is available).

Setup GIT

The final missing piece is setting up version control. I use git and frankly, so should you. I really cannot think of any good reason why you might want to pick another VCS for a new project setup.
Let’s add the following .gitignore file to the project root directory:

/.idea/workspace.xml
/project/boot/
target/
lib_managed/
src_managed/
test-output/

The new directory-based IDEA project definition lends itself better for being put under version control, since the project definition is split across many smaller files, resulting in a lower probability of merge conflicts if several team members change settings simultaneously. However, the .idea/workspace.xml only contains very volatile settings (like your editor state) and should not be under VCS. So, to initialize your shiny new git repo just run

  • git init
  • git add .
  • and git commit -m "Initial commit"

With everything set up there should now be nothing stopping you from coding away in a very productive Scala development environment. (One final hint: For maximum efficiency I suggest you read up on SBTs triggered execution.)

Any feedback, improvement ideas and the like are, of course, as always, very welcome.

Cheers,
Mathias

[Update 2010-10-13]
Updated the “Setup IDEA Project” section for better coverage of the SBT-IDEA plugin.

View Comments