Introduce unit testing to Vim plugin development with vim-vspec

2013-02-13T21:15:00+09:00 / tag:vim / Comments

Previously, I introduced how to "Use Travis CI for Vim plugin development". CI requires well-developed test suites to achieve advantages. But I did not describe much about how to write unit tests for Vim plugin development in the article. If you try to write unit tests for a Vim plugin, you will be faced with problems like the following:

It is hard to resolve each problem. So that there are many frameworks for unit testing. But all of them have one or more problems as follows:

Therefore I wrote another framework to resolve all problems. It is called vim-vspec. Though I wrote it 4 years ago, I did not talk much about it. Because the ecosystem including vim-vspec was not mature. But now the ecosystem has been mature and ready to use anyone. So let me describe how I write unit tests for Vim plugin development.

Requirements

The structure of directories

The structure of a test script

It often requires to describe detailed operations on Vim to test a Vim plugin. So that the best language to write unit tests for Vim plugins is — Vim script. (Note that the ecosystem allows writing tests in an arbitrary language, but it is not necessary in most cases.)

The structure of a test script is similar to RSpec. In short,

For example:

describe 'math#round_to_zero'
  it 'returns 0 as is'
    Expect math#round_to_zero(0) == 0
  end

  it 'returns a floor of a positive number'
    Expect math#round_to_zero(0.1) == 0
    Expect math#round_to_zero(1) == 1
    Expect math#round_to_zero(1.23) == 1
    Expect math#round_to_zero(123.456) == 123
  end

  it 'returns a ceiling of a negative number'
    Expect math#round_to_zero(-0.1) == 0
    Expect math#round_to_zero(-1) == -1
    Expect math#round_to_zero(-1.23) == -1
    Expect math#round_to_zero(-123.456) == -123
  end
end

How to write expectations

To compare actual results with expected results, use Expect. For example:

Expect foo#bar#baz() == 'qux'

How to run unit tests

It is fairly simple; just run rake test. You will see results of unit tests like the following:

$ rake test
bundle exec vim-flavor test
-------- Preparing dependencies
Checking versions...
  Use kana/vim-textobj-user ... 0.3.12
  Use kana/vim-vspec ... 1.1.0
Deploying plugins...
  kana/vim-textobj-user 0.3.12 ... skipped (already deployed)
  kana/vim-vspec 1.1.0 ... skipped (already deployed)
Completed.
-------- Testing a Vim plugin
Files=0, Tests=0,  0 wallclock secs ( 0.00 usr +  0.11 sys =  0.11 CPU)
Result: NOTESTS
t/c.vim .... ok
t/vim.vim .. ok
All tests successful.
Files=2, Tests=12,  2 wallclock secs ( 0.08 usr  0.72 sys +  0.21 cusr  0.97 csys =  1.98 CPU)
Result: PASS

If one or more unit tests are failed, you will see results like the following:

$ rake test
bundle exec vim-flavor test
-------- Preparing dependencies
Checking versions...
  Use kana/vim-textobj-user ... 0.3.12
  Use kana/vim-vspec ... 1.1.0
Deploying plugins...
  kana/vim-textobj-user 0.3.12 ... skipped (already deployed)
  kana/vim-vspec 1.1.0 ... skipped (already deployed)
Completed.
-------- Testing a Vim plugin
Files=0, Tests=0,  0 wallclock secs ( 0.00 usr +  0.00 sys =  0.00 CPU)
Result: NOTESTS
t/c.vim .... 1/?
not ok 1 - <Plug>(textobj-function-a) selects the next function if there is no function under the cursor
# Expected line("'<") == 3
#       Actual value: 2
#     Expected value: 3
t/c.vim .... Failed 1/6 subtests
t/vim.vim .. ok

Test Summary Report
-------------------
t/c.vim  (Wstat: 0 Tests: 6 Failed: 1)
  Failed test:  1
Files=2, Tests=12,  1 wallclock secs ( 0.03 usr  0.02 sys +  0.11 cusr  0.06 csys =  0.22 CPU)
Result: FAIL
rake aborted!
Command failed with status (1): [bundle exec vim-flavor test...]

Tasks: TOP => test
(See full trace by running task with --trace)

Note that actual output is colored.

Simplify common initializations and/or finalizations

Suppose that you define a new operator to edit text. To test its behavior, it is necessary to

Such initializations and finalizations can be simplified with before/after. before block will be executed before running each it block, and after block will be executed after running each it block. For example:

describe '...'
  before
    new
    put =[
    \   'foo',
    \   'bar',
    \   'baz',
    \   '...',
    \ ]
  end

  after
    close!
  end

  it '...'
    ...
  end

  it '...'
    ...
  end
end

Mark tests as "not implemented yet"

It takes a long time to write complete unit tests. So that it is common to

To indicate such "not implemented yet" tests, use TODO as follows:

it '...'
  TODO
end

TODO tests are always treated as failed ones, and they are highlighted like the following:

$ rake test
...
-------- Testing a Vim plugin
Files=0, Tests=0,  0 wallclock secs ( 0.00 usr +  0.04 sys =  0.04 CPU)
Result: NOTESTS
t/c.vim .... 1/?
not ok 1 - # TODO <Plug>(textobj-function-a) selects the next function if there is no function under the cursor
t/c.vim .... ok
t/vim.vim .. ok
All tests successful.
Files=2, Tests=12,  1 wallclock secs ( 0.05 usr  0.46 sys +  0.13 cusr  0.40 csys =  1.04 CPU)
Result: PASS

Skip tests for a specific environment

Sometimes you have to write tests for a specific environment. Such tests should be skipped for other environments. You can use SKIP to indicate such tests:

it '...'
  if executable('git') < 1
    SKIP 'Git is not available.'
  endif

  ...
end

SKIP tests are always treated as passed ones, and they are highlighted like the following:

$ rake test
...
-------- Testing a Vim plugin
Files=0, Tests=0,  0 wallclock secs ( 0.00 usr +  0.00 sys =  0.00 CPU)
Result: NOTESTS
t/c.vim .... 1/?
ok 1 - # SKIP <Plug>(textobj-function-a) selects the next function if there is no function under the cursor - 'Git is not available.'
t/c.vim .... ok
t/vim.vim .. ok
All tests successful.
Files=2, Tests=12,  0 wallclock secs ( 0.03 usr  0.01 sys +  0.08 cusr  0.02 csys =  0.14 CPU)
Result: PASS

Further reading

It is also possible to: