Page references to xUnit Test Patterns (XTP), especially Chapter 11 ("Test Doubles"), "Four-Phase Test" (pp. 358-361) and Practical Object-Oriented Design in Ruby (POODR), especially Chapter 9 ("Designing Cost-Effective Tests").

Other resources:

The thing we are testing (the SUT)

The thing we are testing is called a "System under test" (SUT) (XTP) or "object under test" (POODR, p. 195, Figure 9.1). We'll use SUT. It is fair to think about other things being under test, such as a class, object, method, or application, but the idea here is that the thing under test is defined by the scope of the test itself. Other objects may be involved, such as a "depended-on component" (DOC).

(POODR, Figure 9.1)

Testing messages received from others

When we receive a message from another object, we want to verify a state change in our own object.

Example: Suppose we have an object that converts fahrenheit to celsius. When we pass in a measurement in fahrenheit, we want to verify that the computation of the celsius value is correct.

require 'minitest/autorun'

class TemperatureConverter
  def f_to_c(f)
    (f - 32.0) * 5.0 / 9.0
  end
end

class TemperatureConverterTest < MiniTest::Test
  def setup
    @tc = TemperatureConverter.new
  end

  def test_f_to_c
    assert_in_delta 0.0, @tc.f_to_c(32), 0.01
  end
end

This is very easy. The SUT does not depend on other classes; the state change can be verified directly.

Testing messages received from others -- When to use a stub

Now let's say that we have a converter object that is capable of a variety of conversions. The way this is going to work is that we are going to pass in the type of conversion we want, along with a value to be converted, and we want to get the right result.

It will look something like this:

Converter.new(FToCConverter).convert(32)

Now our Converter class depends on a collaborator, FToCConverter. XTP calls this an "indirect input" (p. 125), and the name it gives to this kind of collaborator is a DOC -- a "depended-on component." We could test Converter with a specific Collaborator (FToCConverter) or we can try to test the SUT in isolation. If we can test the SUT in isolation, then there will be fewer dependencies on other objects, which will make our test less brittle.

When we test the converter, we are not attempting to establish whether the conversion is correct; instead, we want to verify that it can delegate the 32 to the specific converter and return a value. For the verification of the specific converter, we will write separate tests for that. Additionally, we may be writing the main converter first. We may not even know the range of specific converters we are going to want, or have one in hand.

Stubbing

A stub is an implementation that returns a canned answer (POODR, p. 210).

NOTE: We stub on the DOC, not on the SUT. For some guidance on this, see https://robots.thoughtbot.com/don-t-stub-the-system-under-test.

If we create a stub manually, we will want an instance of FToCConverter's convert method to return a value that we can verify. It might look like this:

require 'minitest/autorun'

class FToCConverter
  def convert(value)
    500
  end
end

class Converter
  def initialize(specific_converter_class)
    @specific_converter = specific_converter_class.new
  end
  def convert(value)
    @specific_converter.convert(value)
  end
end

class ConverterTest < MiniTest::Test
  def setup
    @c = Converter.new(FToCConverter)
  end

  def test_convert
    assert_in_delta 500, @c.convert(32), 0.01
  end
end

NOTE: In the real world, this is not quite how it's done. Why? Because in this test, we don't care about FToCConverter. All we care about is making available to the test an object that exposes a convert method that returns a specific value, so that we can validate that the main convert method on Converter leverages it. In short, we want to write the least amount of code to see that the plumbing is working. In MiniTest, it might look like this:

require 'minitest/autorun'

class FToCConverter
  def convert(value)
    500
  end
end

class Converter
  def initialize(specific_converter)
    @specific_converter = specific_converter
  end
  def convert(value)
    @specific_converter.convert(value)
  end
end

class ConverterTest < MiniTest::Test
  def specific_converter
    @specific_converter ||= FToCConverter.new
  end

  def setup
    @c = Converter.new(specific_converter)
  end

  def test_convert
    specific_converter.stub :convert, 400 do
      assert_in_delta 400, @c.convert(32), 0.01
    end
  end
end

Notice that here our implementation of FToCConverter#convert returns 500. But our stub of this method returns 400: and we verify against that. What this means is that we can check the plumbing of the delegation to the specific converter even if it's wrong; our stubbed test doesn't depend on what the actual class does at all: our test is completely independent. Additionally, we have moved the key verification value closer to the assertion, which is easier to read.

For here on out, we will use a different syntax in MiniTest. Here's the same test but using spec-style syntax using keywords such as "describe" and "expect."

require 'minitest/autorun'

class FToCConverter
  def convert(value)
    500
  end
end

class Converter
  def initialize(specific_converter)
    @specific_converter = specific_converter
  end
  def convert(value)
    @specific_converter.convert(value)
  end
end

describe Converter do
  let(:specific_converter) { FToCConverter.new }
  subject { Converter.new(specific_converter) }

  it "can delegate to a specific converter" do
    specific_converter.stub :convert, 400 do
      expect subject.convert(32).must_be_within_epsilon 400, 0.01
    end
  end
end

let and subject provide for dynamically creating methods that return what's defined in the blocks. With let, the name of the method comes from the symbol you pass in. For subject, the method is called subject. They are both lazy. So if you never use specific_converter the block { FToCConverter.new } will never run. Also, because they are lazy, you can reverse the order. Maybe people like to have the subject of the test at the top of the spec. I'll do that below.

Stubbing in MiniTest requires the the stubbed method actually exist in the instance being stubbed. Other testing frameworks are more lenient in this respect, which can result in more concise tests -- but at the expense of having test doubles that are too fake and don't match up to your real collaborators. For example, you might stub a method convert but over time your collaborators change that method name to transform: Your tests would continue to pass because you've stubbed a method that doesn't exist on the real objects.

The Four-Phase Test

It is conventional to think of tests as having four phases:

  1. Setup
  2. Exercise
  3. Verify
  4. Teardown

Typically teardown (releasing resources) is done for you by the framework. Here's our spec of Converter with comments showing the phases:

describe Converter do
  let(:specific_converter) { FToCConverter.new }               # setup
  subject { Converter.new(specific_converter) }                # setup

  it "can delegate to a specific converter" do
    specific_converter.stub :convert, 400 do
      converted_value = subject.convert(32)                    # exercise
      expect converted_value.must_be_within_epsilon 400, 0.01  # verify
    end
  end
                                                               # teardown
end

Testing messages sent to others -- when to mock

When we send a message to another object that results in a side effect, we want to verify the side effect.

In other words, we want to prove that a promised behavior change in a collaborator has been triggered.

Example:

Our Converter provides a means for there to be logging of the conversion.

Let's make a few changes to our design of Converter. First off, let's allow that it's easier to provide configuration parameters via a hash. We'll also provide sensible defaults -- a specific converter that does nothing and nil for the logger. We won't trigger a logger if no logger is set.

class Converter
  class PassThroughConverter
    def convert(value)
      value
    end
  end

  attr_reader :converter, :logger

  def initialize(args = {})
    @converter = args[:converter] || PassThroughConverter.new
    @logger    = args[:logger]
  end
  def convert(value)
    converted_value = converter.convert(value)
    log(value, converted_value)
    converted_value
  end

  private

  def log(value, converted_value)
     logger.log(value, converted_value) if logger
  end
end

Now, what do we want to verify? We do not want to verify that the log method on Converter gets called with value and converted_value -- what we want to know is whether the collaborator is sent the right message. In this case, we want to know if a logger instance would be sent the message log with the right values. At this point, we don't even have a logger class. We just know that it is going to expose a method log and expect that the method call will pass the value and its conversion.

To make this happen, we want to create a Mock. Notice that we are still using the stubbed converter. But now we write logger.expect to set up our expectations for what will happen on the collaborating object; and then we verify it afterward.

It is critical to understand that we are not mocking an object or a class; we are just verifying that the delegate is sent the log method with the right parameters. What this means is that we are verifying a "role" -- Just one aspect of the outgoing messages from the SUT.

I've annotated this with the four phases.

describe Converter, "delegation to logger" do
  subject { Converter.new(logger: logger) }         # setup
  let(:logger) { Minitest::Mock.new }               # setup
  let(:converter) { subject.converter }             # setup

  it "logs the value and the converted value" do
    converter.stub :convert, 400 do
      logger.expect(:log, nil, [32, 400])           # verify
      subject.convert(32)                           # exercise
    end
    logger.verify                                   # verify
  end
end

Other topics to add

  • Metz's method for verifying that an object conforms to an interface
  • RSpec's "behaves like" pattern
  • Fixtures

Terms

  • SUT - System under test
  • DOC - Depended-on component
  • "indirect input" - This is data that gets into the SUT from a DOC. I.e., we're not calling a method with parameters; inputs are getting into the SUT via some DOC. (126)
  • "indirect output" - We want to verify at an "observation point" that calls to the DOC are happening correctly (127).
  • "Stubbing" for Indirect Input - When the SUT makes calls to the DOC, it may take data from the DOC. This data would be an "indirect input." We want to simulate these indirect inputs. Why? Because the DOC may be unpredictable or unavailable. A thing that stands in for the DOC so as to provide indirect inputs to the SUT is called a stub. The stub receives the calls and returns pre-configured responses. We want to "install a Test Stub in place of the DOC" (129). Want to provide indirect inputs? We say that install the Test Stub to act as a "control point" (135). We call it a "control point" because we are trying to force the SUT down some path (524).
  • "Test Spies" or "Mocking" for Indirect Output - By "indirect output," we mean the calls the SUT makes to DOCs. Example: the SUT makes calls to a logger. We want to ensure that the DOC is getting called properly.
    • Procedural Behavior Verification. We want to capture the calls to the DOC during SUT execution and see what happens. This means installing a Test Spy. It receives the calls and records them; then afterwards we make assertions on what is recorded in the Spy. What to check indirect outputs? We say that that happens at an "observation point" (e.g., 137).
    • Expected Behavior. We install a Mock Object, and say in advance what we expect. If the Mock doesn't get what we expect, it fails the test.