Fluxus Frequency

How I Hacked The Mainframe

Setting Up a Client-Side JavaScript Project With Gulp and Browserify

This post originally appeared on Engine Yard.

Introduction

For JavaScript developers, it can be hard to keep up to date with the latest frameworks and libraries. It seems like every day there’s a new something.js to check out. Luckily, there is one part of the toolchain that doesn’t change as often, and that’s the build process. That said, it’s worth checking out your options every now and then.

My build process toolset has traditionally been comprised of RequireJS for dependency loading, and Grunt. They’ve worked great, but recently I was pairing with someone who prefers to use Gulp and Browserify instead. After using them on a couple of projects, I’m coming to like them quite a bit. They’re great for use with Backbone, Angular, Ember, React, and my own hand-rolled JavaScript projects.

In this post, we’ll explore how to set up a clientside JavaScript project for success using Gulp and Browserify.

Defining the Project Structure

For the purposes of this post, we’ll pretend we’re building an app called Car Finder, that helps you remember where you parked your car. If you want to follow along, check out the code on GitHub.

When building a full application that includes both an API server and a clientside JavaScript app, there’s a certain project structure that I’ve found often works well for me. I like to put my clientside app in a folder one level down from the root of my project, called client. This folder usually has sibling folders named server, test, public, and build. Here’s how this would look for Car Finder:

1
2
3
4
5
6
7
8
9
car-finder
|- build
|- client
   |- less
|- public
   |- javascripts
   |- stylesheets
|- server
|- test

The idea is to do our app developent inside of client, then use a build task to compile the JS and copy it to the build folder, where it will be minified, uglified, and copied to public to be served by the backend.

Pulling In Dependencies

To get up and running, we’ll need to pull in some dependencies.

Run npm init and follow the prompts.

Add browserify, gulp, and our build and testing dependencies:

1
2
npm install --save-dev gulp gulp-browserify browserify-shim gulp-jshint gulp-mocha-phantomjs \
gulp-rename gulp-uglify gulp-less gulp-autoprefixer gulp-minify-css mocha chai

If you’re using git, you may want to ignore your node_modules folder with echo "node_modules" >> .gitignore.

Shimming Your Frameworks

You’ll probably want to use browserify-shim to shim jQuery and your JavaScript framework so that you can write var $ = require('jquery') into your code. We’ll use jQuery here, but the process is the same for any other library (Angular, Ember, Backbone, React, etc.). To set it up, modify your package.json like so:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
{
  "name": "car-finder",
  "author": "Ben Lewis",
  "devDependencies": {
    "gulp-rename": "^1.2.0",
    "gulp": "^3.8.10",
    "gulp-mocha-phantomjs": "^0.5.1",
    "gulp-jshint": "^1.9.0",
    "gulp-browserify": "^0.5.0",
    "browserify": "^6.3.4",
    "browserify-shim": "^3.8.0",
    "mocha": "^2.0.1",
    "gulp-minify-css": "^0.3.11",
    "gulp-uglify": "^1.0.1",
    "gulp-autoprefixer": "^2.0.0",
    "gulp-less": "^1.3.6",
    "chai": "^1.10.0"
  },
  "browserify-shim": {
    "jquery": "$"
  },
  "browserify": {
    "transform": [
      "browserify-shim"
    ]
  }
}

If you’re getting JSHint errors in your editor for this file, you can turn them off with echo "package.json" >> .jshintignore.

Setting Up Gulp

Now that we have the gulp package installed, we’ll configure gulp tasks to lint our code, test it, trigger the compilation process, and copy our minified JS into the public folder. We’ll also set up a watch task that we can use to trigger a lint and recompile of our project whenever a source file is changed.

We’ll start by requiring the gulp packages we want in a gulpfile.js that lives in the root of the project.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
// Gulp Dependencies
var gulp = require('gulp');
var rename = require('gulp-rename');

// Build Dependencies
var browserify = require('gulp-browserify');
var uglify = require('gulp-uglify');

// Style Dependencies
var less = require('gulp-less');
var prefix = require('gulp-autoprefixer');
var minifyCSS = require('gulp-minify-css');

// Development Dependencies
var jshint = require('gulp-jshint');

// Test Dependencies
var mochaPhantomjs = require('gulp-mocha-phantomjs');

Now we can start defining some tasks.

JSHint

To set up linting for our clientside code as well as our test code, we’ll add the following to the gulpfile:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
gulp.task('lint-client', function() {
  return gulp.src('./client/**/*.js')
    .pipe(jshint())
    .pipe(jshint.reporter('default'));
});

gulp.task('lint-test', function() {
  return gulp.src('./test/**/*.js')
    .pipe(jshint())
    .pipe(jshint.reporter('default'));
});
``

We'll also need to define a `.jshintrc` in the root of our project, so that JSHint will know which rules to apply. If you have a JSHint plugin turned on in your editor, it will show you any linting errors as well. I use [jshint.vim](https://github.com/wookiehangover/jshint.vim). Here's an example of a typical `.jshintrc` for one of my projects. You'll notice that it has some predefined globals that we'll be using in our testing environment.

json { “camelcase”: true, “curly”: true, “eqeqeq”: true, “expr” : true, “forin”: true, “immed”: true, “indent”: 2, “latedef”: “nofunc”, “newcap”: false, “noarg”: true, “node”: true, “nonbsp”: true, “quotmark”: “single”, “undef”: true, “unused”: “vars”, “trailing”: true, “globals”: { “after” : false, “afterEach” : false, “before” : false, “beforeEach” : false, “context” : false, “describe” : false, “it” : false, “window” : false } }

1
2
3
4
5
### Mocha

I'm a Test-Driven Development junkie, so one of the first things I always do when setting up a project is to make sure I have a working testing framework. For clientside unit testing, I like to use [gulp-mocha-phantomjs](https://www.npmjs.org/package/gulp-mocha-phantomjs), which we already pulled in above.

Before we can run any tests, we'll need to create a `test/client/index.html` file for Mocha to load up in the headless PhantomJS browser environment. It will pull Mocha in from our `node_modules` folder, require `build/client-test.js` (more on this in a minute), then run the scripts:

html <!doctype html> Mocha Test Runner

1
2
3
## Setting Up Browserify

Now we need to set up Browserify to compile our code. First, we'll define a couple of `gulp` tasks: one to build the app, and one to build the tests. We'll copy the result of the compile to `public` so we can serve it unminified in development, and we'll also put a copy into `build`, where we'll grab it for minification. The  compiled test file will also go into `build`. Finally, we'll set up a `watch` task to trigger rebuilds of the app and test when one of the source files changes.

javascript gulp.task(‘browserify-client’, [‘lint-client’], function() { return gulp.src(‘client/index.js’) .pipe(browserify({ insertGlobals: true })) .pipe(rename(‘car-finder.js’)) .pipe(gulp.dest(‘build’)); .pipe(gulp.dest(‘public/javascripts’)); });

gulp.task(‘browserify-test’, [‘lint-test’], function() { return gulp.src(‘test/client/index.js’) .pipe(browserify({ insertGlobals: true })) .pipe(rename(‘client-test.js’)) .pipe(gulp.dest(‘build’)); });

gulp.task(‘watch’, function() { gulp.watch(‘client//*.js’, [‘browserify-client’]); gulp.watch(‘test/client//*.js’, [‘browserify-test’]); });

1
2
3
4
5
6
7
There's one more thing we'll need to do before we can run our `gulp` tasks, which is to make sure we actually have `index.js` files in each of the folders we've it to look at, so it doesn't raise an error. Add one to the `client` and `test/client` folders.

Now, when we run `gulp browserify-client` from the command line, we see new `build/car-finder.js` and `public/javascripts/car-finder.js` files. In the same way, `gulp browserify-test` creates a `build/client-test.js` file.

## More Testing

Now that we have Browserify set up, we can finish getting our test environment up and running. Let's define a `test` Gulp task and add it to our `watch`. We'll add `browserify-test` as a dependency for the `test` task, so our `watch` will just require `test`. We should also update our watch to run the tests whenever we change any of the app _or_ test files.

javascript gulp.task(‘test’, [‘lint-test’, ‘browserify-test’], function() { return gulp.src(‘test/client/index.html’) .pipe(mochaPhantomjs()); });

gulp.task(‘watch’, function() { gulp.watch(‘client//*.js’, [‘browserify-client’, ‘test’]); gulp.watch(‘test/client//*.js’, [‘test’]); });

1
To verify that this is working, let's write a simple test in `test/client/index.js`:

javascript var expect = require(‘chai’).expect;

describe(‘test setup’, function() { it(‘should work’, function() { expect(true).to.be.true; }); });

1
2
3
4
5
6
7
8
9
Now, when we run `gulp test`, we should see Gulp run the `lint-test`, `browserify-test`, and `test` tasks and exit with one passing example. We can also test the `watch` task by running `gulp watch`, then making changes to `test/client/index.js` or `client/index.js`, which should trigger the tests.

## Building Assets

Next, let's turn our attention to the rest of our build process. I like to use `less` for styling. We'll need a `styles` task to compile it down to CSS. In the process, we'll use [gulp-autoprefixer](https://www.npmjs.org/package/gulp-autoprefixer) so that we don't have to write vendor prefixes in our CSS rules. As we did with the app, we'll create a development copy and a build copy, and place them in `public/stylesheets` and `build`, respectively. We'll also add the `less` directory to our `watch`, so changes to our styles will get picked up.

We should also uglify our JavaScript files to improve page load time. We'll write tasks for minification and uglification, then copy the minified production versions of the files to `public/stylesheets` and `public/javascripts`. Finally, we'll wrap it all up into a `build` task.

Here are the changes to the `gulpfile`:

javascript gulp.task(‘styles’, function() { return gulp.src(‘client/less/index.less’) .pipe(less()) .pipe(prefix({ cascade: true })) .pipe(rename(‘car-finder.css’)) .pipe(gulp.dest(‘build’)) .pipe(gulp.dest(‘public/stylesheets’)); });

gulp.task(‘minify’, [‘styles’], function() { return gulp.src(‘build/car-finder.css’) .pipe(minifyCSS()) .pipe(rename(‘car-finder.min.css’)) .pipe(gulp.dest(‘public/stylesheets’)); });

gulp.task(‘uglify’, [‘browserify-client’], function() { return gulp.src(‘build/car-finder.js’) .pipe(uglify()) .pipe(rename(‘car-finder.min.js’)) .pipe(gulp.dest(‘public/javascripts’)); });

gulp.task(‘build’, [‘uglify’, ‘minify’]);

1
2
3
4
5
6
7
8
9
If we now run `gulp build`, we see the following files appear:
- `build/car-finder.css`
- `public/javascripts/car-finder.min.js`
- `public/stylesheets/car-finder.css`
- `public/stylesheets/car-finder.min.css`

## Did It Work?

We'll want to check that what we've built is actually going to work.  Let's add a little bit of styling and JS code to make sure it's all getting compiled and served the way we hope it is. We'll start with an `index.html` file in the `public` folder. It will load up the development versions of our CSS and JS files.

html <!doctype html> Car Finder

1
We'll add some styling in `client/less/index.less`:

less body { background-color: DarkOliveGreen; }

1
Now we'll write our million dollar app in `client/index.js`:

javascript alert(‘I found your car!’);

1
2
3
4
5
Let's put it all together. Run `grunt build`, then `open public/index.html`. Our default browser opens a beautiful olive green screen with an alert box. Profit!

## One Task To Rule Them All

At this point, I usually like to tie it all together with a `default` Gulp task, so all I have to do is run `gulp` to check that everything's going together the way I expect, and start watching for changes. Since `test` already does the linting and browserifying, all we really need here is `test`, `build`, and `watch`.

javascript gulp.task(‘default’, [‘test’, ‘build’, ‘watch’]); “`

Wrapping Up

We’ve now set up our project to use Browserify and Gulp. The former took the headache out of requiring modules and dependencies, and the latter made defining tasks for linting, testing, less compilation, minification, and uglification a breeze.

I hope you’ve found this exploration of Gulp and Browserify has been enlightening. I personally love these tools. For the moment, they’re my defaults when creating a personal project. I hope this post helps make your day-to-day development more fun by simplifying things. Thanks for reading!

Instances, Classes, and Modules, Oh My!

This post originally appeared on Engine Yard and was later published on the Quick Left Blog.

Introduction

One of the biggest challenges of object oriented programming in Ruby is defining the interface of your objects. In other languages, such as Java, there is an explicit way to define an interface that you must conform to. But in Ruby, it’s up to you.

Compounding with this difficulty is the problem of deciding which object should own a method that you want to write. Trying to choose between modules, class methods, instance methods, structs, and lambdas can be overwhelming.

In this post, we’ll look at several ways to solve the Exercism Leap Year problem, exploring different levels of method visiblitiy and scope level along the way.

Leap Year

If you sign up for Exercism.io and start solving Ruby problems, one of the first problems you will look at is called Leap Year. The tests guide you to write a solution that has an interface that works like this:

1
2
Year.leap?(1984)
#=> true

A leap year is defined as a year that is divisible by four, unless it’s also divisible by 100. Apparently, centuries aren’t leap years. That is, unless they are centuries that are divisible by 400. So there are three rules for leap years:

1) If it’s divisible by 400, it’s an exceptional century and is a leap year. 2) If it’s divisible by 100, it’s a mundane century and in not a leap year. 3) Otherwise, if it’s divisible by 4, it’s a leap year, otherwise it’s not.

The First Approach: Using Class Methods

Here’s one simple solution to this problem. You can rearrange the booleans in a couple of different ways and it will still work. This is the version I came up with:

1
2
3
4
5
class Year
  def self.leap?(year)
    year % 4 == 0 && !(year % 100 == 0) || year % 400 == 0
  end
end

This is nice and compact, but not entirely easy to understand. I’m a pretty big fan of self-documenting code, so I used the extract method refactoring pattern to name these three rules.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
class Year
  def self.leap?(year)
    mundane_leap?(year) || exceptional_century?(year)
  end

  def self.mundane_leap?(year)
    year % 4 == 0 && !century?
  end

  def self.century?(year)
    year % 100 == 0
  end

  def self.exceptional_century?(year)
    year % 400 == 0
  end
end

Class Methods and Privacy

This is a lot more understandable, in my mind. But there’s a problem with this. mundane_leap?, century?, and exceptional_century? are all publicly exposed methods. They really only exist in support of leap?, and I’m not sure how reusable they are, with the possible exception of century?. If I write this test, it will pass:

1
2
3
  def test_exceptional_century
    assert Year.exceptional_century?(2400)
  end

I would like to make exceptional_century? private, so that it can’t be accessed outside of the Year class. I can try something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
class Year
  ...

  private

  def self.mundane_leap?(year)
    year % 4 == 0 && !century?
  end

  def self.century?(year)
    year % 100 == 0
  end

  def self.exceptional_century?(year)
    year % 400 == 0
  end
end

But, unfortunately, this won’t work. The test still passes, becuase the private keyword doesn’t work on class methods. Instead, I would have to use private_class_method after my method definition.

1
  private_class_method :mundane_leap?, :century?, :exceptional_century?

Now if I run that last test, it will raise an error.

All About the Eigenclass

In my mind, what we’ve now done is somewhat better, but there’s still a smell here. I’ll get to exactly what that is in just a moment, but for now I’ll say that it’s due to the fact that we’ve defined all of these methods on the singleton class, or eigenclass, of Year. If you don’t know about eigenclasses, you can read about them here.

We can define methods on an eigenclass by prepending each method definition with self., or we can use class << self or class << Year and nesting our method definitions inside of that block. Doing it this way makes it possible to use the private keyword, because we’re now working at the instance level of scope. If we were to introduce class << self, then, we could do away with our private_class_method call.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
class Year
  class << self
    def leap?(year)
      mundane_leap?(year) || exceptional_century?(year)
    end

    private

    def mundane_leap?(year)
      year % 4 == 0 && !century?(year)
    end

    def century?(year)
      year % 100 == 0
    end

    def exceptional_century?(year)
      year % 400 == 0
    end
  end
end

But we haven’t really changed much here. In my mind, there’s still a big smell, which is this. According to Wikipedia, a class is (emphasis mine) “ an extensible program-code-template for creating objects ”. Our Year class never creates a single instance. It’s just an eigenclass that happens to be able to create (mostly) useless Year instances.

So where should we be putting class-level methods that are not associated with any instance object?

Second Approach: Module Functions

In Ruby, the Class object inherits from Module. A module is basically a collection of methods, constants, and classes. Their primary feature is that they can be mixed into other modules and classes to extend their functionality. They’re also frequently used for namespacing (for example, the ActiveRecord constant is a module).

We can put our Year implementation into a module without changing much: just swap the word class for module. Instead of using class << self as we did for the class version, we can use extend self in our module, and get the same effect, allowing us to use the private keyword. extend takes the methods defined in a module and makes them into class-level methods on the target module (or class). This is in contrast with include, which mixes them in as instance-level methods. Thus, if we extend the module into itself, it gets all of it’s own methods at the class level.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
module Year
  extend self

  def leap?(year)
    mundane_leap?(year) || exceptional_century?(year)
  end

  private

  def mundane_leap?(year)
    year % 4 == 0 && !century?(year)
  end

  def century?(year)
    year % 100 == 0
  end

  def exceptional_century?(year)
    year % 400 == 0
  end
end

If we wanted to define some other methods in a Year class that had nothing to do with leap years, we could change the name of our module to Leap and mix it into a class called Year.

If we want to make the three leap year rule methods private, we now have another choice of how to do it. We can use module_function. module_function will make these methods available to the module, but when they get mixed into the class, they will be private. Module functions allow you to be selective with what can be called by the module itself, while still defining methods that can be mixed into other modules and classes.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
module Leap
  def leap?(year)
    mundane_leap?(year) || exceptional_century?(year)
  end

  def mundane_leap?(year)
    year % 4 == 0 && !century?(year)
  end

  def century?(year)
    year % 100 == 0
  end

  def exceptional_century?(year)
    year % 400 == 0
  end

  module_function :mundane_leap?, :century?, :exceptional_century?
end

class Year; extend Leap; end

Now, if we run the tests, they will all still pass, with the exception of the one we wrote that tries to call exceptional_century?

Third Approach: Instance Methods

I want to stay with the idea that we might want a Year class that can do other things unrelated to determining whether we are talking about leap year or not. Really, there are a bunch of different years, but they all have certain things in common (which era they fall in, whether they are leap, etc.). In my mind, this would be a good use case for instances instead of class level methods.

What it look like if we brought the responsibility for knowing whether a year is leap back into a Year class, but passing that behavior onto instances of year, instead of putting it on the eigenclass?

It’s a little weird to be able to do something like Year.new(2015).year, so I’ll name the state we’re going to store reckoning instead.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
class Year
  def self.leap?(year)
    self.new(year).leap?
  end

  attr_reader :reckoning

  def initialize(reckoning)
    @reckoning = reckoning
  end

  def leap?
    mundane_leap? || exceptional_century?
  end

  private

  def mundane_leap?
    reckoning % 4 == 0 && !century?
  end

  def century?
    reckoning % 100 == 0
  end

  def exceptional_century?
    reckoning % 400 == 0
  end
end

The instance-based approach has certain advantages. We don’t have to use anything exotic like private_class_method, class << self, or module_function, to make our methods private. This might make things easier to understand for future maintainers of our code.

We also gained access to the reckoning, that could be used to calculate other properites in the future. For example, if we wanted to write to_julian or to_hebrew methods, we’d already have the number we’d need to use to calculate that available to us.

Finally, we can now use a protected section as well. Methods in this section will only be visible to other instances of Year. We might use it to do something like this:

1
2
3
4
5
6
7
8
9
10
11
class Year
  ...

  protected

  def ==(other)
    self.recoking == other.reckoning
  end

  ...
end

Conclusion

We’ve looked at several different ways to write a program that can tell us whether a given year is a leap year. Each has its advantages and disadvantages.

We started with a class method that was a single-liner. There’s a lot of value in the compactness of year % 4 == 0 && !(year % 100 == 0) || year % 400 == 0. But it’s also fairly hard to understand. If we decide to split some of that logic into separate methods, now we’re faced with the problem of how to keep these new methods out of the interface of Year.

We used an eigenclass-based approach, simply hiding the implementation methods with private_class_method. We solved it using a module, treating the leap? method both as a mix-in and as a module_function. Finally, we pushed the functionality down to the instance level, which could help us in dealing with multiple Year objects down the road.

Which of these approaches is best?

It depends on lots of things: how you feel about privacy, how you expect the program to change in the future, how comfortable you are with scope and the more arcane methods in Ruby. At the very least, it’s nice to know what’s available to you when thinking about what methods you want to expose, and what level of scope you should use it on. I hope this post will factor into your thoughts when making these kind of decisions in the future.

P.S. What do you think? Did this make sense? Should I have used Structs or Lambdas? Throw us a comment below!

Measuring Client-Side JavaScript Test Coverage With Istanbul

This post originally appeared on Engine Yard.

It was also published on the Quick Left Blog.

Introduction

Testing is a vital part of any development process. Whether a project’s authors are hoping for scalability, fewer bugs, or just code that stands the test of time, a solid test suite is essential.

It’s all well and good writing tests, but how do you know that you’ve tested the code that matters? Did you cover the most important components? Did you test the sad path? The edge cases? What about the zero states?

Using Istanbul, you can generate a nice coverage report to answer these questions. In this article, we’ll look at how to get it set up in a typical clientside JS project.

A Note About Metrics

Before we get into the process of setting up Istanbul, I’d like to talk about code coverage as a metric. Metrics in programming can be very two-sided. On the one hand, they can be very useful in measuring velocity and predicting completion of new features. But, they can also become self-fulfilling prophecies leading to bad code.

For example, if a project manager or tech lead measures her programmers’ effectiveness by counting the lines of code they write, she will not find that those who wrote the most lines are the ones that wrote the best code. In fact, there’s a danger that some of the programmers will adopt a verbose style, using far more lines than necessary to get the same thing done, in order to bump their ranking.

Placing a strong emphasis on test coverage as a metric can lead to a similar problem. Imagine that a company adopts a policy that any pull request must increase or maintain the percentage of lines tested. What will be the result?

I imagine that in many cases, developers will write good tests to accompany their features and bugfixes. But what happens if there’s a rush and they just want to get the code in?

Test coverage tools only count the lines that were hit when the test suite was run, so the developer can just run the line in question during the test, while making a trivial assertion. The result? The line is marked covered, but nothing meaningful is being tested.

Rather than using test coverage as a measure of developer thoroughness, it makes a lot more sense to use coverage as a way of seeing which code isn’t covered (hint: it’s often else branches). That information can then be used to prioritize testing goals.

In summary: don’t use code coverage to measure what’s tested, use it to find out what isn’t.

Time to Test

Let’s imagine that we have a clientside JavaScript application written in Angular, Ember, or Backbone and templated with Handlebars. It’s compiled with Browserify and built with NPM scripts.

This application has been around for a couple of years, and due to business pressures, its authors have only managed to write a handful of tests. At this point, the app is well-established, and there is a testing setup in place, but there’s also a lot of code that’s untested.

Recently, the company behind the application closed a funding round, and they’re feeling flush. We’ve been brought in to write some tests.

Because we’re hotshots and we want to show off, we decide to begin by taking a snapshot of the current code coverage, so that we can brag about how many percentage points we added to the coverage when we’re done.

Setting up Istanbul

This is where Istanbul comes in. Istanbul is a code coverage tool written in JavaScript by Krishnan Anantheswaran of Yahoo!.

It can be a little tricky to set up, so let’s take a look at one way to do it.

There are four steps in our approach:

  1. Instrument the source code with Istanbul
  2. Run mocha-phantomjs, passing in a hooks argument
  3. Use a phantomjs hook file to write out the results when testing is complete
  4. Run the Istanbul cli to generate the full report as an HTML file

Let’s get started.

1. Instrument the Source

We’ll need to find a way to run our code through Istanbul after it’s been compiled, so the first step is to set up an NPM task that will pipe compiled code into a tool like browserify-istanbul.

Just in case you’re not using browserify, a variety of other Istanbul NPM packages exist for instrumenting code, including browserify-gulp, grunt-istanbul-reporter, and borschik-tech-istanbul.

For the moment, let’s imagine that we are using Browserify. We already have NPM tasks in place to compile our code for development/production and for testing. Here’s what they look like. Note that the -o option to browserify specifies the output file for the build and the -d option turns on debugging.

In package.json:

1
2
3
4
5
6
7
8
9
{
  ...
  "scripts": {
    ...
    "build": "browserify ./js/main.js -o html/static/build/js/myapp.js",
    "build-test": "browserify -r handlebars:hbsfy/runtime ./js/test/index.js -o html/static/build/js/test.js -d --verbose"
  }
  ...
}

We can use the build-test task as a template for a new build-test-coverage task.

Before we do that, we’ll want to make sure we pull in browserify-istanbul with npm install --save-dev browserify-istanbul.

Next, we’ll write the task in package.json. We’ll ignore the Handlebars templates and Node modules when we load everything into Istanbul. We’ll also use the -t option to Browserify to use a transform module.

1
2
3
4
5
6
7
8
{
  ...
  "scripts": {
    ...
    "build-test-coverage": "mkdir -p html/static/build/js/ && browserify -r handlebars:hbsfy/runtime -t [ browserify-istanbul --ignore **/*.hbs **/bower_components/** ] ./js/test/index.js -o html/static/build/js/test.js -d"
  }
  ...
}

With the browserify-istanbul package and the build-test-coverage script in place, we’ve got our code instrumented, and we’re ready to move on to step two.

2. Run mocha-phantomjs, Passing in a Hooks Argument

Now that the code is all built and ready to go, we need to pass it into a test framework. We’ll use write a cli script that spawns mocha-phantomjs in a child process. We’ll pass in a hooks argument that specifies a phantom_hooks.js file we’ve yet to write.

(Note: if you’re using gulp or grunt, you may want to check out gulp-mocha-phantomjs or grunt-mocha-phantomjs for this step.)

In js/test/cli.js:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
#!/usr/bin/env node

var spawn = require('child_process').spawn;

var child = spawn('mocha-phantomjs', [
  'http://localhost:9000/static/js/test/index.html',
  '--timeout', '25000',
  '--hooks', './js/test/phantom_hooks.js'
]);

child.on('close', function (code) {
  console.log('Mocha process exited with code ' + code);
  if (code > 0) {
    process.exit(1);
  }
});

With our cli script in place, we’ve now got a way to put our Istanbulified code into PhantomJS, so we’ll move to step three.

3. Use a PhantomJS Hook File to Write the Results

In the script we wrote in the last section, we passed a hooks file to mocha-phantomjs, but we hadn’t created it yet. Let’s do that now.

After all the tests have run, our hook will grab the __coverage__ property of the window, which contains the result of our coverage run, and write it to a coverage/coverage.json file.

We’ll load this data into the Istanbul CLI to generate a more readable report in the next step.

In js/test/phantom_hooks.js:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
module.exports = {
  afterEnd: function(runner) {
    var fs = require('fs');
    var coverage = runner.page.evaluate(function() {
      return window.__coverage__;
    });

    if (coverage) {
      console.log('Writing coverage to coverage/coverage.json');
      fs.write('coverage/coverage.json', JSON.stringify(coverage), 'w');
    } else {
      console.log('No coverage data generated');
    }
  }
};

With our coverage data saved, we’re ready to move on to the last step.

4. Run the Istanbul CLI to Generate the Full Report

The final step in our process is to take the coverage data generated in step three and plug it into the Istanbul CLI to generate a coverage report HTML file.

We’ll write an NPM script that executes the instanbul report command, passing it the folder where we saved our results (coverage), and specifying lcov as our output format. This option will save both lcov and html files.

1
2
3
4
5
6
7
8
{
  ...
  "scripts": {
    ...
    "coverage-report": "istanbul report --root coverage lcov"
  }
  ...
}

Now we have all the scripts we need to generate and view our coverage report.

Generating the Report

We’ll need to run each of the commands that we’ve defined before we’ll be able to view our results (you may want to wrap these in a single NPM task for convenience).

  • npm run build-test-coverage to compile our test code and load it into Istanbul
  • ./js/test/cli.js to run the tests with mocha-phantomjs and write the coverage.json file
  • npm run coverage-report to format the coverage results as an HTML file

Once we’ve completed these steps, we should see a new coverage/lcov-report folder containing an index.html file and a few assets to make it look pretty.

Viewing the Report

If we open the generated file in our browser, we’ll see an easy-to-read breakdown of our coverage.

There are four category columns at the top of the page, each telling us about a different aspect of our coverage.

  • Statements tells up how many of the statements in our code were touched during the test run.
  • Branches tells us how many of our logical if/else branches were touched
  • Functions tells us how many of our functions were touched
  • Lines tells us the total lines of code that were touched

There’s also a list of the folders in our project, each with a categorical coverage breakdown. You can also click on each of the folders to further break down the coverage file by file. If you then click on a file, you’ll see its contents highlighted in green and red to indicate covered and uncovered lines.

The ability to zoom in and out on our project and see the categorical breakdown at each level makes Istanbul particularly nice to work with. It also makes it easy to dig in and explore places in your code base that might benefit from some additional testing.

Wrapping Up

If you haven’t added code coverage to your JS projects yet, I highly recommend it. Getting everything up and running is a minimal time investment, and it can really pay off.

Although I would discourage you from measuring the success of your test suite based on percentage points only, getting an in-depth look at what you currently are and aren’t touching can really pave the way to uncovering bugs you didn’t even know you had.

I hope that this post has helped you get Istanbul up and running with a minimum of heartache. Until next time, keep testing!

Five Ruby Methods You Should Be Using

This post was the top featured article in Ruby Weekly #229.

It was originally published on Engine Yard and also appeared on the Quick Left Blog.

Introduction

There’s something magical about the way that Ruby just flows from your fingertips. why once said “Ruby will teach you to express your ideas through a computer.” Maybe that’s why Ruby has become such a popular choice for modern web development.

Just as in English, there are lots of ways to say the same thing in Ruby. I spend a lot of time reading and nitpicking people’s code on Exercism, and I often see exercises solved in a way that could be greatly simplified if the author had only known about a certain Ruby method.

Here’s a look at some lesser-used methods solve a specific problem very well.

Object#tap

Did you ever find yourself calling a method on some object, and the return value not being what you wanted it to? You were hoping to get back the object, but instead you got back some other value. Maybe you wanted to add an arbitrary value to a set of parameters stored in a hash. You update it with Hash.[], but you get back 'bar instead of the params hash, so you have to return it explicitly.

1
2
3
4
def update_params(params)
  params[:foo] = 'bar'
  params
end

The params line at the end of that method seems extraneous.

We can clean it up with Object#tap.

It’s easy to use. Just call it on the object, then pass tap a block with the code that you wanted to run. The object will be yielded to the block, then be returned. Here’s how we could use it to improve update_params:

1
2
3
def update_params(params)
  params.tap {|p| p[:foo] = 'bar' }
end

There are dozens of great places to use Object#tap. Just keep your eyes open for methods called on an object that don’t return the object, when you wish that they would.

Array#bsearch

I don’t know about you, but I do a lot of looking through arrays for data. Ruby enumerables make it easy to find what I need: select, reject, and find are valuable tools that I use daily. But when the dataset is big, I start to worry about the length of time it will take to go through all of those records.

If you’re using ActiveRecord and dealing with a SQL database, there’s a lot of magic that happens behind the scenes to make sure that your searches are conducted with the least algorithmic complexity. But sometimes you have to pull all of the data out of the database before you can work with it. For example, if the records are encrypted in the database, you can’t query them very well with SQL.

At times like these, I think hard about how to sift through the data with an algorithm that has the least complex Big O classification that I can. If you don’t know about Big O notation, check out Justin Abrahms’s Big-O Notation Explained By A Self-Taught Programmer or the Big-O Complexity Cheat Sheet.

The basic gist is that algorithms can take more or less time, depending on their complexity, which is ranked in this order: O(1), O(log n), O(n), O(n log(n)), O(n2), O(2n), O(n!). So we prefer searches to be in one of the classifications at the beginning of this list.

When it comes to searching through arrays in Ruby, the first method that comes to mind is Enumerable#find, also known as detect. However, this method will search through the entire list until the match is found. While that’s great if the record is at the beginning, it’s a problem if the record is at the end of a really long list. It takes O(n) complexity to run a find search.

There is a faster way. Using Array#bsearch, you can find a match with only O(log n) complexity. To find out more about how a Binary Search works, check out my post Building A Binary Search.

Here’s a look at the difference in search times between the two approaches when searching a range of 50,000,000 numbers:

1
2
3
4
5
6
7
8
9
10
11
12
require 'benchmark'

data = (0..50_000_000)

Benchmark.bm do |x|
  x.report(:find) { data.find {|number| number > 40_000_000 } }
  x.report(:bsearch) { data.bsearch {|number| number > 40_000_000 } }
end

         user       system     total       real
find     3.020000   0.010000   3.030000   (3.028417)
bsearch  0.000000   0.000000   0.000000   (0.000006)

As you can see, bsearch is much faster. However, there is a pretty big catch involved with using bsearch: the array must be sorted. This somewhat limits its usefulness, but it’s still worth keeping in mind for occasions where it might come in handy, such finding a record in by a created_at timestamp that has already been loaded from the database.

Enumerable#flat_map

When dealing with relational data, sometimes we need to collect a bunch of unrelated attributes and return them in an array that is not nested. Let’s imagine you had a blog application, and you wanted to find the authors of comments left on posts written in the last month by a given set of users.

You might do something like this:

1
2
3
4
5
6
7
8
9
10
module CommentFinder
  def self.find_for_users(user_ids)
    users = User.where(id: user_ids)
    user.posts.map do |post|
      post.comments.map |comment|
        comment.author.username
      end
    end
  end
end

You would then end up with a result such as:

1
[[['Ben', 'Sam', 'David'], ['Keith']], [[], [nil]], [['Chris'], []]]

But you just wanted the authors! I guess we can call flatten.

1
2
3
4
5
6
7
8
9
10
module CommentFinder
  def self.find_for_users(user_ids)
    users = User.where(id: user_ids)
    user.posts.map { |post|
      post.comments.map { |comment|
        comment.author.username
      }.flatten
    }.flatten
  end
end

Another option would have been to use flat_map.

This just does the flattening as you go:

1
2
3
4
5
6
7
8
9
10
module CommentFinder
  def self.find_for_users(user_ids)
    users = User.where(id: user_ids)
    user.posts.flat_map { |post|
      post.comments.flat_map { |comment|
        comment.author.username
      }
    }
  end
end

It’s not too much different, but better than having to call flatten a bunch of times.

Array.new with a Block

One time, when I was in bootcamp, our teacher Jeff Casimir (founder of Turing School) asked us to build the game Battleship in an hour. It was a great exercise in object-oriented programming. We needed Rules, Players, Games, and Boards.

Creating a represention of a Board is a fun exercise. After several iterations, I found the easiest way to set up an 8x8 grid was to do this:

1
2
3
4
5
class Board
  def board
    @board ||= Array.new(8) { Array.new(8) { '0' } }
  end
end

What’s going on here? When you call Array.new with an argument, it creates an array of that length:

1
2
Array.new(8)
#=> [nil, nil, nil, nil, nil, nil, nil, nil]

When you pass it a block, it populates each of its members with the result of evaluating that block:

1
2
Array.new(8) { 'O' }
#=> ['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O']

So if you pass an array with eight elements a block that produces an array with eight elements that are all 'O', you end up with an 8x8 array populated with 'O' strings.

Using the Array#new with a block pattern, you can create all kinds of bizarre arrays with default data and any amount of nesting.

<=>

The spaceship, or sort, operator is one of my favorite Ruby constructs. It appears in most of the built-in Ruby classes, and is useful when working with enumerables.

To illustrate how it works, let’s look at how it behaves for Fixnums. If you call 5<=>5, it returns 0. If you call 4<=>5, it returns -1. If you call 5<=>4, it returns 1. Basically, if the two numbers are the same, it returns 0, otherwise it returns -1 for least to greatest sorting, and 1 for reverse sorting.

You can use the spaceship in your own classes by including the comparable module and redefining <=> with logic branching to make it return -1, 0, and 1 for the cases you want.

Why would you ever want to do that?

Here’s a cool use of it I came across on Exercism one day. There’s an exercise called Clock, where you have to adjust the hours an minutes on a clock using custom + and - methods. It gets complicated when you try to add more than 60 minutes, because that will make your minute value invalid. So you have to adjust by incrementing another hour and subtracting 60 from the minutes.

One user, dalexj, had a brilliant way to solve this, using the spaceship operator:

1
2
3
4
5
6
7
8
  def fix_minutes
    until (0...60).member? minutes
      @hours -= 60 <=> minutes
      @minutes += 60 * (60 <=> minutes)
    end
    @hours %= 24
    self
  end

It works like this: until the minutes passed in are between 0 and 60, he subtracts either 1 or -1 from the hours, depending on whether the minute amount is greater than 60. He then adjusts the minutes, adding either -60 or 60 depending on the sort order.

The spaceship is great for defining custom sort orders for your objects, and can also come in handy for arithmetic operations if you remember that it returns one of three Fixnum values.

Wrapping Up

Getting better at writing code is a process of learning. Since Ruby is a language, a lot of the time I spend trying to improve is spent of reading “literature” (i.e. code on Exercism andGitHub), and reading (what is essentially) the dictionary for my language: (Rubydocs.

It’s so much easier to write expressive code when you know more methods. I hope that this collection of curiosities helped expand your Ruby vocabulary.

Seven Reasons I Love Minitest

This post was a featured article in Ruby Weekly.

It originally appeared on Engine Yard, and was also published on the Quick Left Blog.

Introduction

The other day at our company standup, I mentioned that I was eager to read an article on [Concurrency in Minitest] (http://chriskottom.com/blog/2014/10/exploring-minitest-concurrency/) that was featured in Ruby Weekly. One of my coworkers asked: “people still use Minitest?” My reply: “you mean you’re not using Minitest yet?”

I love Minitest. It’s small, lightweight, and ships with Ruby. It’s used by respected programmers like Aaron Patterson, Katrina Owen, Sandi Metz, and of course, DHH. Here’s a look at why Minitest remains a powerful and popular choice for testing Ruby code.

Witness the Firepower of this Fully Armed and Operational Testing Tool

Although I come from a family of programmers, I entered the profession by going to a bootcamp. I was a student in the second gSchool class. At that time, it was being taught by Jumpstart Lab. My instructors were Jeff Casimir, Franklin Webber, and [Katrina Owen] (https://twitter.com/kytrinyx).

My classmates and I were brought up with TDD from day one. We practiced it in everything we did. The tool we used was Minitest. When it was first introduced, I scoffed a little because of the name.

“Why are we using a ‘mini’ testing framework? I want to use what the pros use,” I complained to Katrina.

“Minitest is a fully-featured testing framework, and is used in plenty of production apps,” was her reply.

I’ve been using it ever since. Here are seven reasons I think Minitest is the bees’ knees.

1. It’s Simple

Minitest ships with Ruby because it’s easy to understand, and can be written by anyone who knows the language. I love this simplicity because it makes it easy to focus on designing code. Just imagine what you want it to do, and write your assertion, and make it pass.

I recently reached out to Katrina to ask her thoughts on what’s good about Minitest. Its simplicity was at the top of her list:

“It’s simpler. There’s no ‘magic’ (just plain Ruby) […] When there is “magic” then it’s very easy to assume that you can’t understand it […] You can read through the [Minitest] source code, and understand what is going on.”

I also asked Sandi Metz what she thought of Minitest. She said:

“Minitest is wonderfully simple and encourages this same simplicity in tests and code.”

Minitest is low in sugar. All you need to know is assert for booleans, and assert_equal for everything else. If you want to test a negative case, use refute or refute_equal instead. For everything else, just write Ruby.

2. It’s Extensible

The relatively “low-level” status of Minitest makes it easy to customize tests. Katrina observes:

“Because Minitest is so simple, it’s not too hard to extend. It feels like a tool that I can shape to my needs, rather than a tool that I need to use the way it was intended.”

If you want to repeat an assertion in multiple contexts, you can write a custom assertion for it, and call it in as many tests as you need.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
def assert_average_speed(swallow)
  # Do some additional work here
  assert_equal '11 MPS', swallow.speed
end

def test_african
  swallow = Swallow.new(type: 'African')
  assert_average_speed(swallow)
end

def test_european
  swallow = Swallow.new(type: 'European')
  assert_average_speed(swallow)
end

If you are testing something that requires nearly the same test to be run repeatedly, for example when testing a controller that has a user authentication before_action, you can create a shared example by including a module.

If you need even more control, you can create an object with custom behavior that inherits from Minitest::Test, and have your tests inherit from it. Doing so allows you to completely customize your test suite with new methods as you see fit.

Finally, Minitest comes with hooks to help you easily write extensions to the gem. You can define a plugin by adding a minitest/XXX_plugin.rb file to your project folder, and Minitest will automatically find and require it. You can use extensions to define command-line options, custom reporters, and anything else you want Minitest to be able to do.

3. It’s Flat If You Use Assert

The default syntax for Minitest is assert. Although the BDD expect syntax is available in mintest/spec, I recommend giving assert a try. Maybe you love expectations because they read like English, sort of. Although assert doesn’t try to imitate natural language, it’s actually quite intuitive to use. It also provides the benefit of making your test files flat.

With nested context, describe, and it blocks, it can be difficult to remember what it refers to, and which before blocks are accessible in your scope. I find myself scanning indentation in BDD tests to figure out what scope I’m working in.

When you use assert syntax, your test file is flat. You start with a test class, then you write a bunch of test methods inside of it. It’s clear that the only variables available are those defined in the setup method or in the test itself. This flatness also means it gets painful quickly if your test is tied to too many dependencies. As Katrina puts it, “the lack of nested contexts means that I’m faced with the appropriate amount of pain when I make bad choices. It quickly becomes ugly if I have too many dependencies. I like that.”

Using assert also makes it easy to document the desired behavior of your code: just name the test method to describe what you are testing. If you’re really worried you’ll forget what the test was for, you can output a message to the console if the test fails:

1
2
3
4
5
6
7
test 'it calculates the air-speed velocity of an unladen swallow' do
  swallow = Swallow.new(wing_span: 30, laden: false, type: 'European')
  expected = '11 MPS'
  actual = swallow.average_speed
  assert_equal expected, actual, 'The average speed of an unladen
    swallow was incorrect'
end

Minitest’s flatness is also beneficial when it comes to practicing a Test-Driven workflow. You can skip all of the tests in a file easily, without scanning through nested blocks. Then you can make them pass, one at a time.

4. It Lends Itself to A Good Test-Driven Workflow

Minitest is awesome for getting into a red/green/refactor loop. Write a test, watch it fail, make it pass, refactor. Repeat. A Minitest file is just a list of tests that are waiting for you to make them pass.

Plus, since you’re just writing Ruby, you can use a style like this to get the next test set up with a minimum of effort:

1
2
3
4
5
test 'something' do
  expected = # some code
  actual = # some simple result
  assert_equal expected, actual
end

If you want to repeat an assertion in different contexts, you can write a method for it, and call it in as many tests as you want to. Need a shared example? Include a module.

5. Minitest::Benchmark is Awesome

If you are dealing with large amounts of data, and performance is a concern, Minitest::Benchmark is your new best friend.

It lets you test your algorithms in a repeatable manner, to make sure that their algorithmic efficiency don’t accidentally get changed. You can collect benchmarks in “tab-separated format, making it easy to paste into a spreadsheet for graphing or further analysis”.

Here are a few of the assertions in Minitest::Benchmark that might be of interest:

  • assert_performance_constant
  • assert_performance_exponential
  • assert_performance_logarithmic
  • assert_performance_linear

6. It’s Randomized By Default

Running the tests in a different order each time can help catch bugs caused by unintended dependencies between examples. Here’s how Aaron Patterson described the benefit of randomized tests in an episode of Ruby Rogues:

“I’m sure you’ve been in a situation where your […] test fails in isolation, but when you run [the entire test suite] it works. [T]hat’s typically because one test set up some particular environment that another test depended on. […] Since Minitest runs the test in a random order, you can’t make one test depend another, so you’ll see an error case.”

7. It’s Faster

Minitest doesn’t create any matchers or example objects that have to be garbage collected. Many people have benchmarked Minitest against other frameworks, and it usually comes out at ahead. Often it’s a marginal difference, but sometimes it comes out significantly ahead.

Minitest also supports concurrent test runs. Although I thought this would lead to great speed gains, it turns out that it only makes a difference in JRuby and Rubinius. The Matz Ruby Implementation (MRI) doesn’t get speed gains when using concurrency. Still, it’s nice to know that the option is there, in case you are using JRuby or Rubinius, or the MRI changes in the future.

Have You Tried It?

I’ve talked to a lot of people that are surprised when I say I prefer Minitest. They often ask, “why should I switch to Minitest?” Perhaps a better question to ask is this one, posed by Ken Collins: “What is in [other testing frameworks] that you need that Minitest does not offer?”

Programming tools are a matter of individual taste. When I’ve asked programmers about Minitest, they’ve all expressed that they were not against RSpec or other frameworks. Sandi Metz said: “I’m agnostic about testing frameworks […] I also use RSpec and I find it equally useful in it’s own way.” Katrina Owen said: “I don’t dislike RSpec, but I do prefer Minitest.” In the end, I think most would agree that testing frameworks are a personal choice.

That said, if you haven’t used Minitest lately (or ever), why not check it out? If you’re curious, but feel like you still need some convincing, just try it! It’s a small investment. You’ll be up and running in a few minutes. Why not go try the first Ruby Exercism?

I hope this tour of Minitest’s features has been informative, and piqued your interest in this fabulous test framework. Is there anything I missed? Disagree? Let’s talk! Please share your point of view in the comments section, or tweet at me at @fluxusfrequency.

Other Resources

A Big Look At Minitest
An informative slide deck that explores the ins and outs of Minitest.

Bow Before Minitest
A more opinionated slide deck comparing Minitest with other testing tools.

Caching Asynchronous Queries in Backbone

This post originally appeared on Engine Yard.

It was also published on the Quick Left Blog.

Introduction

I was working on a Backbone project with Bob Bonifield recently, when we came across a problem. We were building an administration panel to be used internally by a client, and there were a couple of views that needed to display the same information about users in a slightly different way. To prevent unnecessary AJAX calls, we decided to cache the result of Backbone.Model#fetch at two levels of our application.

The result: less network time and a snappier user experience.

Here’s how we did it.

Caching the Controller Call

We decided to use Brent Ertz’s backbone-route-control package. It makes separation of concerns in the Backbone router easier by splitting the methods associated with each route into controllers.

I’ll show how we set up the UsersController to handle the first level of caching. In this example, we’ll use backbone-route-control. If you weren’t using it, you could accomplish the same thing in the Backbone router.

First, we set up the app view and initialized a new Router as a property of it, passing in the controllers we wanted to use.

1
2
3
4
5
6
7
8
9
10
11
// Main

var UsersController = require('controllers/users');

var app = new AppView();

app.router = new Router({
  controllers: {
    users: new UsersController(app)
  }
});

Next, we defined the router and set it up to use the UsersController to handle the appropriate routes.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
// Router

var BackboneRouteControl = require('backbone-route-control')

var Router = BackboneRouteControl.extend({
  routes: {
    'users/:id': 'users#show',
    'users/:id/dashboard': 'users#dashboard'
  },

  initialize: function(options) {
    this.app = options.app;
  }
});

return Router;

Caching the User at the Controller Level

After we got the router set up, we defined the UsersController and the appropriate route methods. We needed to wait until the user was loaded before we could generate the DOM, because we needed to display some data about the user.

We opted to cache the ID of the last user that was accessed by either the show or dashboard method, so that we wouldn’t repeat the fetch call when we didn’t need to. We set the result of the call to Backbone.Model#fetch (a promise) to a variable called userLoadedDeferred, and passed it down the the views themselves.

In doing so, we took advantage of the fact that, behind the scenes, fetch uses jQuery.ajax and returns a deferred object. When saving the result of a call to jQuery.ajax to variable, the value of the deferred’s .complete or .fail callback will always return the same payload after it has been fetched from the server.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
// UsersController

var UsersController = function(app) {
  var lastUserID,
      userLoadedDeferred,
      user,
      lastUser;

  return {
    show: function(id) {
      this._checkLastUser();

      var usersView = new UserShowView({
        app: app,
        user: lastUser,
        userLoadedDeferred: userLoadedDeferred
      });

      app.render(usersView);
    },

    dashboard: function(id) {
      this._checkLastUser();

      var usersView = new UserDashboardView({
        app: app,
        user: lastUser,
        userLoadedDeferred: userLoadedDeferred
      });

      app.render(usersView);
    },

    _checkLastUser: function(id) {
      if (lastUserId != id) {
        lastUserId = id;
        lastUser = new User({ id: id });
        userLoadedDeferred = lastUser.fetch();
      }
    }
  }
});

Caching the User at the Model Level

Although our UsersController was now caching the result of a fetch for a given user, we soon found that also needed to refetch the user to display their information in a sidebar view as well.

Since the UsersController and the SidebarView were making two separate calls to the User model fetch method, we decided to do some more caching in the Backbone Model. We opted to save the results of the fetch call for 30 seconds, only making a new server request if the timer had expired.

This allowed us to simply call fetch from within the view, without needing to know whether the User model was making an AJAX call or just returning the cached user data.

Here’s what the code looked like in the model:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
// User Model

var Backbone = require('backbone');
// Set the timeout length at 30 seconds.
var FETCH_CACHE_TIMEOUT = 30000;

var User = Backbone.Model.extend({
  fetch: function() {

    // set a flag to bust the cache if we have ever set it
    // before, and it's been more than 30 seconds
    var bustCache = !(this.lastFetched && new Date() - this.lastFetched < FETCH_CACHE_TIMEOUT);

    // if we've never cached the call to `fetch`, or if we're busting
    // the cache, make a note of the current time, hit the server, and
    // set the cache to this.lastFetchDeferred.
    if (bustCache) {
      this.lastFetched = new Date();
      this.lastFetchDeferred = Backbone.Model.prototype.fetch.apply(this, arguments);
    }

    // return the promise object that was cached
    return this.lastFetchDeferred;
  }
});

return User;

Busting the Cache

Later on in our development, we came across a situation where we needed to force a new fetch of User, right after updating some of their attributes. Because we were caching the result for 30 seconds, the newly updated attributes were not getting pulled from the server on our next fetch call. To overcome this, we neede to bust our cache manually. To make this happen, we changed our overridden fetch method to take an option that allowed us to force a refetch.

1
2
3
4
5
6
7
8
9
10
11
12
13
// User Model

  ...

  fetch: function(options) {
    // check if we passed the forceRefetch flag in the options
    var forceRefetch = options && options.forceRefetch;

    ...

    // updated the check to if the flag was passed
    if (!this.lastFetchDeferred || bustCache || forceRefetch) {
    ...

Conclusion

Caching the User model in this app reduced our network time by quite a bit. Initially, we were making two server calls per route, because we had to fetch the user to display data in both the main view and the sidebar. After saving the result of the fetch in the controller, we were now only calling to the server once per User ID.

With the addition of model-level caching, we were also able to remove the duplicated call between the main views and the sidebar view, by saving the results of the fetch call for 30 seconds.

Overall, we reduced four calls per route to one call per 30 seconds. Making these adjustments helped make our application behave a lot more smoothly, and reduced server load in the process.

P.S. Have you implemented anything like this before? What are some of the tricks you use to make Backbone more efficient? Tweet at me @fluxusfrequency.

Building a Ruby List Comprehension

This post was a featured article in Ruby Weekly.

It was originally published on Engine Yard and appeared on the Quick Left Blog.

Introduction

As developers, we’re in the business of continually bettering ourselves. Part of that process is pushing ourselves to learn and use better code patterns, try new libraries, and pick up new languages. For me, the latest self-learning project has been picking up Python.

As I’ve worked with it, I’ve discovered the joy of list comprehensions, and I’ve been wondering what it would take to implement a similar syntax in Ruby. I decided to give it a try. This exercise yielded several insights into the inner workings of Ruby, which we’ll explore in this post.

Snake Handling

I’m primarily a Rubyist. I’ve always enjoyed the natural way that Ruby flows off the fingers, and heard that Python was similar. It sounded easy enough, especially since this article promised I could learn Python in ten minutes.

It took a little while to get used to some of the differences in Python, like capitalizing booleans and passing self into all of my methods–but they were mostly superficial differences. As I solved more and more Python problems on exercism.io, the syntax began to feel natural. I felt like everything I wrote in Python had an analog in Ruby. Except list comprehensions.

You Cannot Grasp The True Form Of Lists

In Python, Coffeescript, and many functional languages, there is a really cool operation you can do called a list comprehension. It’s something like a map, but it’s written in mathematical set builder notation, which looks like this:

Instead of doing this:

1
(1..100).map { |i| 2 * i if i.even? }

You do this:

1
[2 * x for x in xrange(1, 100)]

You can even nest comprehensions:

1
return [[print(str(x) for x in y] for 2 * y in xrange(1, 100)]

These are so cool, in fact, that I decided to see if I could implement them in Ruby.

How It Works

My first thought in writing a list comprehension in Ruby was to make it look like Python. I wanted to be able to write square brackets with an expression inside of them. To make that happen, I would need to override the main []() method. It turns out that that method is an alias for the Array#[] singleton method. It’s not so easy to monkey patch, because you can’t really call super on it.

I decided to abandon this approach and create a new method called c(), that I would put in a module and include in main. Ideally, this method would take a single argument with this syntax: x for x in my_array if x.odd?. Since you can’t really pass an expression like this into a method and expect Ruby to parse it properly, I opted to pass it in as a string and parse it. I was into this idea, but not really interested in rewriting the Ruby source code.

Caught In A Loop

My first goal was to get the basic x for x in my_array part working.

I wrote a test:

1
2
3
4
5
6
7
8
9
10
class ComprehensionTest < Minitest::Test
  def setup
    extend ListComprehension
  end

  def test_simple_array
    result = c('n for n in [1, 2, 3, 4, 5]')
    assert_equal [1, 2, 3, 4, 5], result
  end
end

This was fairly straightforward to get passing.

1
2
3
4
5
6
7
8
module ListComprehension
  def c(comprehension)
    parts = comprehension.split(' ')
    if parts[0] == parts[2]
      eval(comprehension.scan(/\[.*\]/).last)
    end
  end
end

I continued working along, refactoring to have the module instantiate a Comprehension class and evaluate parts of the string:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
module ListComprehension
  def c(expression)
    Comprehension.new(expression).comprehend
  end
end

class Comprehension
  def initialize(expression)
    @expression = expression
  end

  def comprehend
    # Do some parsing and call `eval` on some stuff
  end
end

But pretty soon, I ran into a problem.

No Scope

When I defined a variable in the test scope, then tried to evaluate it in the comprehension argument string, my Comprehension couldn’t access it.

For example, I wrote this test:

1
2
3
4
5
def test_array_variable
  example = [1, 2, 3, 4, 5]
  result = c ('x for x in example)
  assert_equal [1, 2, 3, 4, 5], result
end

When I tried to call eval on example, I got:

1
NameError: undefined local variable or method `example' for #<Comprehension:0x007f9cec989fc8>

So how could I acceess the scope where example was defined? I did some googling, and discovered that you can access the calling scope a lot more easily from block than a string that you pass into a method.

With that in mind, I changed the interface of Comprehension to take a block instead of a string. To call c(), you would now write c{ 'x for x in example' } instead of c('x for x in example').

Inside of the Comprehension class, I did:

1
2
3
4
5
6
7
8
9
class Comprehension
  def initialize(&block)
    @comprehension = block.call
    @scope = block.send(:binding)
  end

  ...

end

Now I could call eval on the calling scope by doing:

1
2
3
4
5
def comprehend
  # some parsing
  collection = scope.send(:eval, parts.last)
  # carry out some actions on the collection
end

I had no idea you could access the calling scope like this in Ruby. It opened my eyes to the whole world of accessing callers and callees in a way that I normally don’t think about in Ruby.

You Obviously Worked Hard On These Plans, Kernel Klink

I wasn’t all that happy with having to define my c() method in a module and include it in the main scope. I really just wanted it to be available automagically if you required comprehension.rb.

After poking around a bit, I found an article on metaprogramming that showed me how you can monkey patch Kernel itself.

After changing module ListComprehension to module Kernel, I was able to remove the setup method entirely from my test suite. I didn’t realize it was this easy to get methods defined in the main scope. Even though it is probably very wrong in many situations, it’s cool to get an understanding of how Ruby itself is put together.

Lessons Learned

I set out to write a list comprehension in Ruby, and in a way, I failed. I was hoping to be able to write an expression inside of square brackets and have Ruby parse it. I ended up settling for a string inside of a block instead.

What’s more, my comprehension implementation is lacking several features. It doesn’t support method calls in the conditional, so you can’t write c{'x for x in (0..10) if Array(x)'}. You can’t pass arguments to the conditional either, so you can’t do c{'x for x in (1..10) if x.respond_to?(:even?)'}. You can’t access an index while you’re looping. And perhaps most disappointing of all, you can’t nest comprehensions.

But despite these shortcomings, I felt like this exercise was a great success, because I learned three things:

  1. When you call [] to create an array, you’re really calling Array#[], which delegates to Array#new.

  2. You can access the calling scope of a block with block.send(:binding).

  3. You can monkey patch module Kernel to get methods available in main.

To the the knowledge gained from this adventure was totally worth it. Although I did not create a library I would expect people to use, I learned a lot about how Ruby works, and had a great time solving the problem. To check out the full results, please visit my GitHub profile.

P.S. How would you have solved it? Maybe you’ve written a natively parsed list comprehension yourself? If so, tweet at me @fluxusfrequency.

Better SOA Development With Foreman and NGINX

This post was originally published on Engine Yard.

It also appeared on the Quick Left Blog.

MOAR!

Everyone knows more is better. More kittens, more money, more apps. Why settle for one Ruby project, when you can have three? We’ll take one Rails app for authorization and one to serve an API. Hey, let’s throw in a Sinatra proxy server serving up an AngularJS app to while we’re at it! Now we’re cookin’!

There are many ways organizations stand to gain by splitting their application into multiple projects running in symphony. If we’re being good programmers and following the Single Responsibility Principle (SRP), it makes sense to embrace it at all levels of organization, from our methods and classes up through our project structure. To organize this on the macro level, we can use a Service Oriented Architecture (SOA) approach. In this article, we’ll explore some patterns that make it easier to develop SOA apps with a minimum of headaches.

Service Oriented Architecture

In the mid-2000s, some programmers began to organize their applications in a new way. Led by enterprise behemoths like Microsoft and IBM, the programming community saw a rise in the use of Web Services: applications that provide data or functionality to others. When you stick a few of these services together, and coordinate them in some kind of client-facing interface, you’ve built an application using SOA. The benefits of this approach remain relevant in modern web development:

  • Data storage is encapsulated
  • You can reuse services between multiple applications (e.g. authentication)
  • You can monitor messages sent between services for business intelligence
  • Services are modular, and can be composed into new applications without repetition of common functionality

A Sample Project

To illustrate some processes that make it easier to develop an SOA project, we’ll imagine that we’re building a project called Panda, that is composed of three services:

PandaAPI: A RESTful API that serves data about Giant Pandas PandaAuth: Login page and user authentication service PandaClient: An AngularJS app sitting atop a Sinatra proxy server

Setting Up GitHub

To deal with an SOA project like this, it’s helpful to make sure you have everything well-structured on GitHub, so that all developers working on it can get up to speed quickly, and stay in sync with each other. I recommend creating an organization, and setting up all of the service project repositories under that organization. For Panda, I would start with a structure that looks like this:

1
2
3
4
5
panda-org (organization)
|- panda-auth (repo)
|- panda-api (repo)
|- panda-client (repo)
|- processes (repo)

The first three repos will hold the actual service projects, and the processes repo will hold scripts and processes that are shared between them.

Git Your S#*& Together

It can be pretty annoying to have your projects all out of sync. To make it easier to keep things up to date, here’s a bash script you can use to pull all of your projects down and update them in one go. Inside of the processes folder, touch a file called git_update.sh.

1
2
3
4
5
6
7
8
9
10
#git_update.sh

#!/bin/bash

BRANCH=$1
: ${BRANCH:="master"}

cd ../panda-auth   && git checkout $BRANCH && git pull origin $BRANCH
cd ../panda-api    && git checkout $BRANCH && git pull origin $BRANCH
cd ../panda-client && git checkout $BRANCH && git pull origin $BRANCH

When executing this script, you can specify the branch by running sh ./git_update <feature-name>.

We can do something similar for bundling and running migrations.

Create the bundle_and_migrate.sh file.

It should look like this:

1
2
3
4
5
6
7
8
9
10
#!/bin/bash

# Load RVM
[[ -s "$HOME/.rvm/scripts/rvm" ]] && source "$HOME/.rvm/scripts/rvm" # Load RVM into a shell session *as a function*
export PATH="/usr/local/bin:/usr/local/sbin:~/bin:$PATH"
[[ -s "$HOME/.rvm/scripts/rvm" ]] && . "$HOME/.rvm/scripts/rvm" # Load RVM function

cd ../panda-auth   && bundle && bundle exec rake db:migrate
cd ../panda-api    && bundle && bundle exec rake db:migrate
cd ../panda-client && bundle && bundle exec rake db:migrate

Now the projects are all updated and ready to go, and we want to begin developing some features. We could write another script to go start rails server in each of our directories, but there is a better way.

Call In The Foreman

The foreman gem is my favorite way to manage multiple applications: it runs them all in a single terminal session. It’s pretty simple to get set up, and saves you having to run a lot of shell sessions (and a lot of headaches).

First, we’ll need to gem install foreman, to make sure we have the global executable available. Then, we’ll set up a Procfile to tell it which processes we want it to run. We’ll create ours in the processes directory, since that’s where we’re keeping things that pertain to all of the projects in our SOA app.

1
2
3
4
5
#Procfile

auth:     sh -c 'cd ../panda-auth   && bundle exec rails s -p 3000'
api:       sh -c 'cd ../panda-api    && bundle exec rails s -p 3001'
client:   sh -c 'cd ../panda-client && bundle exec rails s -p 3002'

This will work great as long as all of your apps are running on the same gemset. If not, you will need to check out the subcontractor gem.

From the processes folder, run foreman start. Sweet. Now everything is all set up. Just open up your browser and navigate to http://localhost:3000. Oh, and pop open two more tabs for http://localhost:3001 and http://localhost:3002 in them.

Man, wouldn’t it be nice if we could just run everything under a single host name?

NGINX, NGINX #9

To get around the problem of having three different localhosts, we can use NGINX. If you’re not familiar with NGINX, it’s an Apache alternative that acts as “a web server, a reverse proxy server and an application load balancer” (from the official site). We can use it to serve up all three of our apps from the same host, and make things a whole lot easier on ourselves.

To install NGINX, I recommend using Homebrew. If you have Homebrew, installation is as simple as brew install nginx. If you don’t, you can try one of these alternatives.

Once NGINX is installed, we’ll want to locate our nginx.conf. If you installed using Homebrew, it will be located at /usr/local/etc/nginx/nginx.conf. Otherwise, you’ll want to use ack, mdfind, or another search tool to locate it.

Once you’ve located it, open it in your text editor and locate the server section. Find the block that starts with location / (line 43 for me) and replace it with the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
#nginx.conf

http {
  ...

  server {
    listen       8080;
    ...

    # Client
    location / {
      proxy_pass        http://127.0.0.1:3002;
      proxy_set_header  X-Real-IP  $remote_addr;
    }

    # Auth
    location /auth {
      proxy_pass        http://127.0.0.1:3000;
      proxy_set_header  X-Real-IP  $remote_addr;
    }

    # API
    location /api {
      proxy_pass        http://127.0.0.1:3001;
      proxy_set_header  X-Real-IP  $remote_addr;
    }

  ...
  }
  ...
}

Now start NGINX with the nginx command. With these proxy_pass settings in place, we should be able to visit see all of our apps from http://localhost:8080:

  • / takes us to the client app
  • /auth takes us to the auth app
  • api takes us to the API app

Dealing With Redirects

One last tricky part of developing SOA apps is figuring out how to deal with url redirects between our apps. Let’s say that you want the client app to redirect users to the auth app if they haven’t logged in yet.

You would probably want to start with something like this in the client app:

1
2
3
4
5
6
7
8
9
10
11
12
13
#app/controllers/application_controller.rb

class ApplicationController < ActionController::Base
  before_action :check_user

  private

  def check_user
    if !session[:user_id]
      redirect_to '/auth'
    end
  end
end

Looks good, and it should work locally.

But it could be a problem if some of your apps are served from subdomains in production. Fortunately, there’s an easy way to get around this.

Create a config/service_urls.yml file in each project. Inside of it, define the url for each app:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
#config/service_urls.yml

defaults: &defaults
  service-urls:
    panda-api: 'localhost:8080/api'
    panda-auth: 'localhost:8080/auth'
    panda-client: 'localhost:8080'

development:
  <<: *defaults

test:
  <<: *defaults

production:
  service-urls:
    panda-api: 'TBD'
    panda-auth: 'TBD'
    panda-client: 'TBD'

We’ll also need to register this configuration file in config/application.rb:

1
2
3
4
5
6
7
8
9
10
11
12
#config/application.rb

module PandaClient
  class Application < Rails::Application
    ...
    # In Rails 4.2, you can use:
    Rails.configuration.urls = config_for(:service_urls)[Rails.env]

    # For older Rails versions, use:
    Rails.configuration.urls = YAML.load_file(Rails.root.join('config', 'service_urls.yml'))[Rails.env]
  end
end

With this configuration in place, we can now update the url redirect to look like the following, and it will work in all environments.

That will look something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
#app/controllers/application_controller.rb

class ApplicationController < ActionController::Base
  ...
  def check_user
    if !session[:user_id]
      redirect_to auth_url
    end
  end

  def auth_url
    @auth_url ||= Rails.application.config.urls['service-urls']['panda-auth']
  end
end

With these changes in place, our applications will now redirect to the appropriate url in all environments.

All Your App Are Belong To Us

By now, you sould have a better idea of what it takes to develop an application using SOA principals. We’ve taken a look at using shell scripts to keep our files in sync, foreman to run several servers at once, and NGINX to pull everything together into a single host address that makes it easier to work with all our services in the browser.

Juggling several services can be pretty confusing, but if you start with the right set up, it makes things a lot easier. All your apps will be under control if you manage them from a central place, and using the strategies discussed in this article should help make the process less painful. Good luck!

P.S. What tricks do you use when you’re developing SOA apps? Did I leave anything out? If you think so, tweet at me @fluxusfrequency.

Getting Started With Active Job

This post was a featured article in Ruby Weekly.

It originally appeared on Engine Yard and was also published on the Quick Left Blog.

Introduction

With the announcement of Rails 4.2 came some exciting news: Rails now has built-in support for executing jobs in the background using Active Job. The ability to schedule newsletters, follow-up emails, and database housekeeping tasks is vital to almost any production application. In the past, developers had to hand-roll this functionality, and configuration varied between different queueing services. With the release of Rails 4.2, setting up jobs to be executed by workers at a later time is standardized. In this article, we’ll take a look at how to set up Active Job and use it to send a follow-up email to a new user.

Updating Rails

You’ll need Rails 4.2.0beta1 or greater if you want to Active Job available by default (in older versions of Rails, you can require it as a gem). This tutorial is based on Rails 4.2.0beta2 (edge Rails). If you want to use edge Rails, use gem 'rails', github: 'rails/rails' in your Gemfile, and run bundle update.

Setting Up Resque

In order to send emails outside of our main application process, we’ll need to make use of a queueing system. There are many choices of technology for setting up background workers available, and Active Job abstracts the differences between them. Today we’ll use Resque, as it’s widely-used and stable.

To use Resque, you’ll need to make sure you have Redis installed. If you don’t, I recommend getting it with Homebrew. Otherwise, you can follow the instructions for download from the official site. Once it’s set up, make sure redis-server is running.

The next step is to install and configure the Resque gem. We’ll also need resque-scheduler to use ActiveJob. Add them the Gemfile with gem 'resque' and gem 'resque-scheduler' and bundle install. We’ll also need to create a Resque configuration file:

1
2
3
4
#config/initializers/resque.rb

Resque.redis = Redis.new(:url => 'http://localhost:6379')
Resque.after_fork = Proc.new { ActiveRecord::Base.establish_connection }

We’ll also require the Resque and Resque Scheduler rake tasks, so we can start our workers and scheduler with rake:

1
2
3
4
5
6
7
8
9
10
11
#lib/tasks/resque.rake

require 'resque/tasks'
require 'resque/scheduler/tasks'

namespace :resque do
  task :setup do
    require 'resque'
    require 'resque-scheduler'
  end
end

We can now start a worker with QUEUE=* rake environment resque:work. If everything’s working right, we should be able to see it in the Resque console. Run resque-web and visit http://0.0.0.0:5678/overview. If you see “0 of 1 Workers Working”, all’s well. We’ll also need to boot up the scheduler in a separate process with rake environment resque:scheduler.

Creating the Mailer

Now that we have a worker, we need it an email to send. Let’s imagine that we want to send a follow up email to a user that recently registered for our site. We’ll create a UserMailer, with a follow_up_email method that takes an email address.

1
2
3
4
5
6
7
8
9
10
11
12
#app/mailers/user_mailer.rb

class UserMailer < ActionMailer::Base
  default from: 'noreturn@example.com'

  def follow_up_email(email)
    mail(
      to: email,
      subject: 'We hope you are enjoying our app'
    )
  end
end

We’ll also need to write a follow-up email template.

1
2
3
4
#app/views/user_mailer/follow_up_email.text

Hey, we saw that you recently signed up for our app.
We hope you're enjoying it!

Creating the Job

Now that we have a working mailer, we can set up Active Job. All we really need to do is configure it to use the Resque adapter.

1
2
3
#config/initializers/active_job.rb

ActiveJob::Base.queue_adapter = :resque

Next, we’ll create a job that tells the background worker to send the email. The conventions for a job include: giving it a queue_as, and defining a perform method.

1
2
3
4
5
6
7
8
9
#app/jobs/follow_up_email_job.rb

class FollowUpEmailJob < ActiveJob::Base
  queue_as :email

  def perform(email)
    UserMailer.follow_up_email(email).deliver_now
  end
end

Now when a user signs up, we can have the UsersController enqueue the job for execution at a later time. Although you would probably delay the job a few days in a real application, we’ll just wait 10 seconds for easier testing.

1
2
3
4
5
6
7
8
9
10
11
12
13
#app/controllers/users_controller.rb

class UsersController < ApplicationController
  def new
    @user = User.new
  end

  def create
    @user = User.create(user_params)
    FollowUpEmailJob.new(@user.email).enqueue(wait: 10.seconds)
    # redirect somewhere
  end
end

To make this work, we’ll need some routes and a view template:

1
2
3
4
5
#config/routes.rb

Rails.application.routes.draw do
  resources :users, only: [:new, :create]
end
1
2
3
4
5
6
#app/views/users/new.html.erb

<%= form_for @user do |f| %>
  <%= f.email_field :email %>
  <%= f.submit %>
<% end %>

Setting Up Mailcatcher

Before we try our job, we’ll want to make sure we can intercept the emails we’re expecting the mailer to send. To achieve this, we’ll use the mailcatcher gem. Do a global install with gem install mailcatcher, and run mailcatcher. Once we configure Action Mailer to send the emails to localhost:1025 via smtp, we’ll be able to view intercepted emails at http://127.0.0.1:1080.

1
2
3
4
5
6
7
#config/environments/development.rb

Rails.application.configure do
  ...
  config.action_mailer.delivery_method = :smtp
  config.action_mailer.smtp_settings = { :address => "localhost", :port => 1025 }
end

Trying It Out

Now everything is set up. To try it out, we’ll sign up as a new user and watch the job get enqueued in the queue, then catch the mail in Mailcatcher. At this point, we have four processes running:

  • mailcatcher
  • QUEUE=* rake environment resque:work
  • rake environment resque:scheduler
  • rails server

In your browser, view the Resque dashboard at http:/http://0.0.0.0:5678. In another tab, visit http://127.0.0.1:1080 to see the Mailcatcher dashboard.

Now, for the moment of truth. Visit localhost:3000/users/new and sign up as a new user. Ten seconds later, a new job will appear in the emails queue of the Resque dashboard. Just afterward, the email will appear in Mailcatcher.

Using Acive Job with Action Mailer

The pattern we’ve written here woks for scheduling any job. But, since ActiveJob is now baked into ActionMailer, we can also schedule the job directly with the UserMailer in the UsersController.

1
2
3
4
5
6
7
8
9
#app/controllers/users_controller.rb

class UsersController < ApplicationController
  ...
  def create
    ...
    UserMailer.follow_up_email(email).deliver_later!(wait: 10.seconds)
  end
end

Conclusion

Active Job makes scheduling background jobs easier. It’s also a great way to set up your job infrastructure without knowing too much about what queueing system you’re using. If you needed to switch to Sidekiq or Delayed Job in the future, it would be as simple as setting ActiveJob to the appropriate adapter.

It can be a little bit tricky to get Active Job set up correctly. Hopefully, this tutorial made the process a little more transparent for you.

Until next time,

Happy hacking!

Wrapping Your API in a Custom Ruby Gem

This post originally appeared on Engine Yard.

It also appeard on the Quick Left Blog.

Introduction

In the modern web, API-based projects are becoming the norm. Why? For one thing, APIs are necessary to serve Single Page Applications, which are all the rage right now. From a business standpoint, APIs give companies a new way to charge others for access to their data. If you are part of a company that offers such a service, a great way to generate interest in your API is to offer a Ruby gem that makes fetching and consuming your data easy for Ruby developers.

Today, we’ll take a look at how to wrap an imaginary API in a new Ruby gem and share it with the world. If you want to follow along at home, you can clone the project from my GitHub account.

Our API

Let’s pretend we have an application called Ben’s Benzes that serves data about cars for sale. We’re exposing a RESTful API so that developers from other companies can serve our car data on their websites. For our first iteration, here are the routes we’ve set up:

Get info about a certain car currently for sale: GET http://www.bensbenzes.com/api/v1/cars/active/:id

Get all the cars that are currently for sale: GET http://www.bensbenzes.com/api/v1/cars/active

Setting Up the Gem

We’ll be using bundler, so begin by making sure you have that installed (you probably do). We’ll create the gem by running bundle gem <gem-name> from the command line. I’m going to use benzinator as the name of my gem. That name is now taken, so you’ll have to come up with your own. Open up the project directory and you should see:

1
2
3
4
5
6
7
8
9
├── Gemfile
├── LICENSE.txt
├── README.md
├── Rakefile
├── benzinator.gemspec
└── lib
    ├── benzinator
    │   └── version.rb
    └── benzinator.rb

Let’s open up benzinator.gemspec and do a little configuration. Update the following lines with your name, email and a summary and description of the gem.

1
2
3
4
5
6
...
spec.authors       = ["Ben Lewis"]
spec.email         = ["blewis@example.com"]
spec.summary       = %q{Gem to wrap BensBenzes.com API}
spec.description   = %q{Gem to wrap BensBenzes.com API}
...

While we’re in here, let’s add the dependencies we’ll be using, right after bundler and rake. We’ll add testing tools as development dependencies, as well as hard dependencies on Faraday and json.

1
2
3
4
5
6
7
8
...
spec.add_development_dependency "minitest"
spec.add_development_dependency "vcr"
spec.add_development_dependency "webmock"

spec.add_dependency "faraday"
spec.add_dependency "json"
...

Writing a Test

We’ll need to make a test folder, and put a test_helper.rb into it. We’ll use the helper to pull in our gem and testing dependencies. We’ll also configure VCR and Webmock, which we’re using to stub out our server responses so that our gem isn’t dependent on access to the API for testing. See the VCR documentation for more about how this works.

1
2
3
4
5
6
7
8
9
10
#test/test_helper.rb
require './lib/benzinator'
require 'minitest/autorun'
require 'webmock/minitest'
require 'vcr'

VCR.configure do |c|
  c.cassette_library_dir = "test/fixtures"
  c.hook_into :webmock
end

Don’t forget to create the test/fixtures folder so VCR has somewhere to put the fixtures. With that done, we’ll write the first test for our gem. Our goal is to create an object called Benzinator::Car that exposes #all and #find methods to wrap calls to our API. Let’s start by making sure that object exists:

1
2
3
4
5
6
7
8
#test/car/car_test.rb
require './test/test_helper'

class BenzinatorCarTest < Minitest::Test
  def test_exists
    assert Benzinator::Car
  end
end

Creating The Wrapper Model

If we now run ruby test/car/car_test.rb, we get this error: uninitialized constant Benzinator::Car. Looks like it’s time to write some code:

1
2
3
4
5
#lib/benzinator/car.rb
module Benzinator
  class Car
  end
end

We’ll also need to require it in lib/benzinator.rb:

1
2
3
require_relative "benzinator/version"
require_relative "benzinator/car"
...

Now if we run the test, it passes. Let’s write another one to make sure that our Benzinator::Car model can give back the data for a car. Let’s imagine that an API call to http://www.bensbenzes.com/api/v1/cars/active/68 returns this JSON object:

1
2
3
4
5
6
7
8
9
"{
  \"id\": 68,
  \"make\": \"Honda\",
  \"model\": \"Civic\",
  \"year\": \"1996\",
  \"color\": \"Blue\",
  \"vin\": \"XXXXXXXXXXXXXX\",
  \"dealer_id\": 34
}"

We’ll want to make sure that our Benzinator::Car object has getter convenience methods all of the fields shown for each car. Given these API results, we could write a test like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
#test/car/car_test.rb
  ...
  def test_it_gives_back_a_single_car
    VCR.use_cassette('one_car') do
      car = Benzinator::Car.find(68)
      assert_equal Benzinator::Car, car.class

      # Check that the fields are accessible by our model
      assert_equal 68, car.id
      assert_equal "Honda", car.make
      assert_equal "Civic", car.model
      assert_equal "1996", car.year
      assert_equal "Blue", car.color
      assert_equal "XXXXXXXXXXXXXX", car.vin
      assert_equal 34, car.dealer_id
    end
  end
end

Running this test, we get undefined method 'find' for Benzinator::Car:Class. Let’s go define it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
#lib/benzinator/car.rb
require 'faraday'
require 'json'

API_URL = "http://www.bensbenzes.com/api/v1/cars/active"

module Benzinator
  class Car
    def self.find(id)
      response = Faraday.get("#{API_URL}/#{id}")
      attributes = JSON.parse(response.body)
    end
  end
end

Now the test says we forgot to make it a Benzinator::Car model:

1
2
Expected: Benzinator::Car
Actual: Hash

We can fix that, and make the attributes into getters at the same time.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
#lib/benzinator/car.rb
...
module Benzinator
  class Car
    attr_reader :id, :make, :model, :year, :color, :vin, :dealer_id
    def initialize(attributes)
      @id = attributes["id"]
      @make = attributes["make"]
      @model = attributes["model"]
      @year = attributes["year"]
      @color = attributes["color"]
      @vin = attributes["vin"]
      @dealer_id = attributes["dealer_id"]
    end

    def self.find(id)
      ...
      new(attributes)
    end
    ...
  end
end

That should take care of it. Now to test the #all method.

Let’s imagine that a call to http://www.bensbenzes.com/api/v1/cars/active responds with an array of 64 cars, with this Honda being the first one. We can rely on the API call giving back 64 cars today, but we hope there will be 6000 listed tomorrow. This is why we’re using VCR to save the result of the call as a fixture.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
#test/car/car_test.rb
class BenzinatorCarTest < Minitest::Test
  ...
  def test_it_gives_back_all_the_cars
    VCR.use_cassette('all_cars') do
      result = Benzinator::Car.all

      # Make sure we got all the cars
      assert_equal 64, result.length

      # Make sure that the JSON was parsed
      assert result.kind_of?(Array)
      assert result.first.kind_of?(Benzinator::Car)
    end
  end
end

Running this, we get undefined method 'all' for Benzinator::Car:Class. Let’s define it:

1
2
3
4
5
6
7
8
9
10
11
12
#lib/benzinator/car.rb
...
module Benzinator
  class Car
    ...
    def self.all
      response = Faraday.get(API_URL)
      cars = JSON.parse(response.body)
      cars.map { |attributes| new(attributes) }
    end
  end
end

Sweet success! The tests pass!

Publishing and Using Our Gem

Now that our gem works, we can publish it to RubyGems. This is a pretty easy process. First, we’ll bump the version to 1.0:

1
2
3
4
#lib/benzinator/version.rb
module Benzinator
  VERSION = "0.1.0"
end

Then we can bundle it up by running gem build benzinator.gemspec. This will create a benzinator-0.1.0.gem file in our project directory. To publish it , all we have to do is run gem push benzinator-0.1.0.gem. You’ll be prompted for your RubyGems username and password, which you’ll need to create if you don’t have an account yet. After you enter your credentials, your gem is live!

Now anyone can use our gem in their Ruby projects. All they have to do is add it to their Gemfile with gem 'benzinator', and run bundle.

Conclusion

Winning! Now the whole wide world can access the Ben’s Benzes API from their Ruby project, with convenience methods to make things easier to work with.

For our next iteration, we could get a lot more in depth. We might want to add the ability to create, edit or destroy cars. If we decided to that, we might first build sort of authentication process into the gem. Once users have gotten a taste of our data and rely on it, our API might get so popular that we need to limit the number of API calls a user can make per day. We could then write subscription service to allow users greater access to your data at a cost.

One last note: you might want to use this technique to wrap someone else’s API, too! If you’re using a service that doesn’t offer a gem for its API, you can always write one and release it as open source!

Happy hacking!