Differences between revisions 12 and 32 (spanning 20 versions)
Revision 12 as of 2013-03-13 16:37:00
Size: 6546
Editor: ?markus tornow
Comment: adding BR between Case [number]: ; to make it more clear
Revision 32 as of 2020-02-04 11:29:48
Size: 12185
Editor: Praveen A
Comment:
Deletions are marked like this. Additions are marked like this.
Line 9: Line 9:
= ruby tests =
Note: temporary content, copied from piratepad (to have the page here and make sure it won't be overlooked) If it is good it will stay.

Note: Probably a better place for this is Ruby/Packaging/Tests as it is common for any ruby gem.
Line 16: Line 11:
Most of the ruby libraries contain test code which uses the functionality provided by the library and compare results with known inputs. Commonly used testing framework is rspec. Look in your gem to see if there is a 'spec' directory. Other test frameworks may use a 'test' directory.

= Running tests =

If the gem has tests defined (see if it has 'test or 'spec' directories), make sure you run them successfully. By default gem2deb creates 'debian/ruby-tests.yaml', but it may not work always. If your test or spec directory contains files with extentions other than .rb, this is most certainly going to fail. You can try removing non-ruby files from ruby-test-files.yaml and rebuild it. If it still does not work, try the other two methods described below.

    You can try ruby-tests.rb, see ruby-warden for a sample file http://ur1.ca/ao6o8 It allows you to add custom directories to ruby library load path and you may need it if your tests fail to find files already present in 'lib' or 'test' directories.


Note: If you don't find ruby-tests.rb in debian directory, you have to create it from the sample file given above. You have to remove ruby-test-files.yaml too.

Uncomment this line in ruby-tests.rb to add custom library paths.
Most Ruby libraries contain code to test the functionality provided by the library and compare results with known inputs. One commonly used testing framework is RSpec, indicated by a `spec` directory in your gem's root folder. Other test frameworks may use a `test` directory.

= Running tests =

If the gem has tests defined (usually has a `test` or `spec` directory), make sure you run them successfully. Your package should contain one (and only one) of `debian/ruby-test-file.yaml`, `ruby-tests.rake`, or `ruby-tests.rb`.

1. By default, gem2deb creates `debian/`'''ruby-test-files.yaml'''. However, this may not always work. If your `test` or `spec` directory contains files with extentions other than .rb, this is most certainly going to fail. You can try removing non-ruby files from ruby-test-files.yaml and rebuild it. If it still does not work, try the other two methods described below.

2. You may also try '''ruby-tests.rake''' if the gem uses rspec to run the tests (in this case you have a spec/ directory with *_spec.rb files). See [[http://ur1.ca/ao6ok|ruby-dataobjects-sqlite3]] for an example.

3. You can try '''ruby-tests.rb''', see [[http://ur1.ca/ao6o8|ruby-warden]] for an example. It allows you to add custom directories to ruby library load path and you may need it if your tests fail to find files already present in 'lib' or 'test' directories.

    Note: If you don't find ruby-tests.rb in debian directory, you have to create it from the sample file given above. You have to remove ruby-test-files.yaml too.

    Uncomment this line in ruby-tests.rb to add custom library paths.
Line 33: Line 31:
If it could be done with not much effort, consider instead patching the tests not to override default $LOADPATH (aka $:) variable.

    You may also try ruby-tests.rake if the gem uses rspec to run the tests (in this case you have a spec/ directory with *_spec.rb files). See ruby-dataobjects-sqlite3 for an example http://ur1.ca/ao6ok
    If it could be done with not much effort, consider instead patching the tests not to override default $LOADPATH (aka $:) variable.
Line 39: Line 35:
1. Gem provide a test runner (ruby-vegas) - It has test_vegas_runner.rb, so just use it from ruby-tests.rb Gem provide a test runner (ruby-vegas) - It has test_vegas_runner.rb, so just use it from ruby-tests.rb
Line 44: Line 40:
<<BR>>

= fixing errors =

Tips to fixing test errors:

    Check for build-depencies on rubygems.org: they are listed as development (see the commented line in debian/control produced by gem2deb). You may not need all the build dependencies listed there, so the only way to be sure is to see the build errors. Normally, you need also the dependencies during the build, if you run the tests (since your library will be used).
<<BR>>

== Case one: a required library is missing ==

    If you see a file is missing in the error message, it may be provided by a gem not yet installed in your system. For example if you see


== Dealing with Bundler usage ==
Tips for rails testapp/dummy app or when you can't easily remove bundler usage by patching. Some libraries meant to work with rails create a dummy rails app to test their functionality with a rails app. "[[Teams/Ruby/Packaging/Tests#Case_six:_bundle_install | case six: bundle install]]" tells you how to avoid bundler completely. If that fails, try the steps below.

 1. Remove Gemfile.lock in `debian/clean` or remove from source tarball as we can't expect debian environment to match exact versions of libraries used by the upstream developers.
 1. Add build dependencies on ruby-bundler and rubygems-integration
 1. Patch out Gemfile/gemspec/metadata.yml if bundler insist on a specific version of library and debian already has a newer version. Relax the dependency as ">=" or "~> x.y"

= Common errors =

First, check for build-depencies on rubygems.org--they are listed as development (see also the commented line in debian/control produced by gem2deb). You may not need all the build dependencies listed there; the only way to be sure is to see the build errors. Many of these dependencies are required during the tests.

== Case one: missing library/gem ==

If you see a file is missing in the error message, it may be provided by a gem not yet installed in your system. For example, if you see
Line 61: Line 61:
search for shoulda ie, search for shoulda, ie,
Line 73: Line 73:

Note: Remember to add the build dependency you just found to Build-Depends: in debian/control

    If you cannot find the package via apt, try if it is a gem file not yet packaged, then run
Note: Remember to add the build dependency you just found to `Build-Depends` in debian/control (edit this file)

Change

{{{
Build-Depends: debhelper (>= 7.0.50~), gem2deb (>= 0.3.0~)
}}}

to

{{{
Build-Depends: debhelper (>= 7.0.50~), gem2deb (>= 0.3.0~), ruby-shoulda
}}}

Note: If you see something like `no such file to load -- rack/test (LoadError)`, look for ''rack-test'' as the gem name.

If you cannot find the package via apt, it may be a gem that has not yet been packaged for Debian. In this case, run:
Line 82: Line 95:
If you find it in the remote gem list, then you need to package it first :) So start from step 1 for this gem and come back to your current gem once your package enters debian!

<<BR>>

== Case two: lib or testor spec is not in library load path ==

    If you see spec_helper or test_helper is missing, you may need to add spec or test directory to library load path.
If you find it in the remote gem list, then you need to package it first :)

Note: There are some libraries which we can safely ignore. Use quilt for commenting these out. See [[Diaspora/Packaging/quilt|basic quilt]] for help in using quilt.

 1. `appraisal` - just uncomment `require 'appraisal'` lines as it is used for creating a specific environment using rubygems and bundler and will conflict with debian environment
 1. `simplecov` - used for code coverage analysis. Comment out `require 'simplecov'` and its other usages. [[http://anonscm.debian.org/gitweb/?p=pkg-ruby-extras/ruby-oauth2.git;a=blob;f=debian/patches/remove-simplecov.patch;h=1814b8605ce82e6f4fe4f20fdb63aa3b05b57417;hb=HEAD | ruby-oauth2 has an example patch]]
 1. `ammeter` - just comment out
 1. `coveralls` - code coverage, just comment out
 1. `rcov` - code coverage, just comment out
 1. `rubygems` - can conflict with apt
 1. `bundler` - can be difficult to remove sometimes. See "[[Teams/Ruby/Packaging/Tests#Case_six:_bundle_install | case six: bundle install]]" for tips.

== Case two: tests are not in library load path ==
If you see spec_helper or test_helper is missing, you may need to add spec or test directory to library load path.
Line 94: Line 114:
    Uncomment the following line in ruby-tests.rb and modify it accordingly. === Using ruby-tests.rb in debian folder ===
Uncomment the following line in ruby-tests.rb and modify it accordingly.
Line 106: Line 127:
=== Using ruby-tests.rake in debian folder ===
Add
{{{
t.libs << 'test' << '.'
}}}

just after
{{{
Gem2Deb::Rake::TestTask.new do |t|
}}}
Line 107: Line 138:

<<BR>>
Line 123: Line 152:
<<BR>>
Line 126: Line 154:
Error:
gem2deb couldn't create a spec or test folder, as it is missing in the gem. At upstream, usually github, it is existent.

Solution:
create a directory with the ruby-gem name and make it an empty git repo
{{{
mkdir ruby-subexec
cd !$
git init
}}}

Download the upstream tarball from github. Add tags to the homepage URL, so you will get a tar.gz file:
https://github.com/nulayer/subexec/tags
You can download it like usual from there (or wget if you prefer)

import the upstream version to git with
{{{
git-import-orig --pristine-tar ~/Downloads/subexec-0.2.2.tar.gz
}}}
You can now copy the complete debian directory gem2deb created for you and then go on like usual:
{{{
cp -a path/to/gem2deb_build/ruby-subexec/debian path/to/actual_build/ruby-subexec
}}}

<<BR>>

== Case five: Tests failure due to interactions ==
Sometimes, The tests may pass perfectly when they are run separately and fail when we are building it because then the tests are running together and they may interact with each other. To see if the tests run okay individually run the following commands inside the test directory.

Sometimes, gems are not packaged with their test folders. Often, these are present at upstream, e.g., at github.

Download the tarball from the upstream source (you may need to convert it to .tar.gz yourself), then run `dh-make-ruby <gem-name>.tar.gz`.

Then cd to ruby-<gem-name> and run `dpkg-buildpackage`.

Look for any lines saying "no test suites found" and proceed further accordingly
ignore the key signing and other warnings for now..

Note: Request for upstream to include tests in future version of the gem itself. See https://github.com/maxwell/rack-piwik/issues/6 for an example.

== Case five: tests failure due to interactions ==
Sometimes, tests may pass perfectly when they are run separately and fail when we are building it because then the tests are running together and interact with each other. To see if the tests pass individually, run the following commands inside the test directory.
Line 166: Line 180:

== Case six: bundle install ==

If you get an error like this asking you to run bundle install for any gem,

{{{
Could not find gem 'rake (>= 0) ruby' in the gems available on this machine.
Run `bundle install` to install missing gems.
}}}

Then it means the package is using bundler to satisfy dependencies. Building a deb package should not require use of bundler to install dependencies, but instead all dependencies should be provided in debian/control file as "Build-Depends:" and "Depends:"

You'll have to create a patch to un-comment use of bundler or rubygems like this. (example from ruby-rack-rewrite). Use quilt to create the patch. See [[Diaspora/Packaging/quilt|basic quilt]] for help in using quilt.

{{{
--- a/test/test_helper.rb
+++ b/test/test_helper.rb
@@ -1,13 +1,18 @@
-require 'rubygems'
-require 'bundler/setup'
+# We'll add dependencies in debian/control
+#require 'rubygems'
+#require 'bundler/setup'

-Bundler.require :default, :development
+#Bundler.require :default, :development

 require 'test/unit'
}}}

See "Case seven" below for a solution to errors caused by this.

<<BR>>

== Case seven: NoMethodError ==

If you get an error similar to this,

{{{
./test/rack-rewrite_test.rb:47: undefined method `context' for RackRewriteTest:Class (NoMethodError)
}}}

It usually means the library is using bundler to require other libraries and which are missing, see "Case six" above for details. We can add those dependencies manually by patching test_helper.rb or spec_helper.rb as is the case.

{{{
--- a/test/test_helper.rb
+++ b/test/test_helper.rb
@@ -1,13 +1,18 @@

 require 'test/unit'
+require 'shoulda'
+require 'mocha'
+require 'rack'
+require 'rack/rewrite'

 class Test::Unit::TestCase
 end

 def supported_status_codes
   [:r301, :r302, :r303, :r307]
-end
\ No newline at end of file
+end
}}}

In the above example, the dependency is found from Gemfile, and corresponding gemspec

{{{
source "http://rubygems.org"

gemspec

group :development do
  gem 'rake'
end
}}}

Dependency section of rack-rewrite.gemspec is given below

{{{
  s.add_development_dependency 'bundler'
  s.add_development_dependency 'shoulda', '~> 2.10.2'
  s.add_development_dependency 'mocha', '~> 0.9.7'
  s.add_development_dependency 'rack'
}}}

Using these as reference, we add those as require lines in the patch. If you cannot solve case six and seven, see "[[Teams/Ruby/Packaging/Tests#Dealing_with_Bundler_usage|Dealing with Bundler usage]]"

== Case eight: autopkgtest failure ==

If the tests run correctly during build, but fail in autopkgtest, it is most likely because of using relative paths in tests and gem2deb-test-runner will remove lib directory from source tree during tests. You can look at [[https://salsa.debian.org/ruby-team/ruby-fog-rackspace/blob/master/debian/patches/use-installed-lib.patch|ruby-fog-rackspace]] as an example.


== Case nine: tests missing from the upstream tarball ==

By default, gem2deb will pull an upstream tarball from rubygems.org. Some Ruby upstreams consider Rubygems to be some kind of "binary" repository, so they don't upload the full source, which usually means not including the test files (test/*, spec/*, etc). The solution to this is usually updating debian/watch to pull source tarballs from github, and re-running dh-make-ruby so that it creates the file under debian/ (usually debian/ruby-tests.rake) that will drive the test execution, both during the build, and under autopkgtest.

== Case ten: autopkgtest failure with multiple gemspec files ==

When there are multiple gemspec files in the source package, manually check the dependency using Test-Command field in debian/tests/control (ruby -e "gem 'gemname'"). Check ruby-flipper as an example.

Translation(s): none


Introduction to software testing

Most Ruby libraries contain code to test the functionality provided by the library and compare results with known inputs. One commonly used testing framework is RSpec, indicated by a spec directory in your gem's root folder. Other test frameworks may use a test directory.

Running tests

If the gem has tests defined (usually has a test or spec directory), make sure you run them successfully. Your package should contain one (and only one) of debian/ruby-test-file.yaml, ruby-tests.rake, or ruby-tests.rb.

1. By default, gem2deb creates debian/ruby-test-files.yaml. However, this may not always work. If your test or spec directory contains files with extentions other than .rb, this is most certainly going to fail. You can try removing non-ruby files from ruby-test-files.yaml and rebuild it. If it still does not work, try the other two methods described below.

2. You may also try ruby-tests.rake if the gem uses rspec to run the tests (in this case you have a spec/ directory with *_spec.rb files). See ruby-dataobjects-sqlite3 for an example.

3. You can try ruby-tests.rb, see ruby-warden for an example. It allows you to add custom directories to ruby library load path and you may need it if your tests fail to find files already present in 'lib' or 'test' directories.

  • Note: If you don't find ruby-tests.rb in debian directory, you have to create it from the sample file given above. You have to remove ruby-test-files.yaml too. Uncomment this line in ruby-tests.rb to add custom library paths.

$: << 'lib' << '.'
  • If it could be done with not much effort, consider instead patching the tests not to override default $LOADPATH (aka $:) variable.

Examples

Gem provide a test runner (ruby-vegas) - It has test_vegas_runner.rb, so just use it from ruby-tests.rb

 require 'test/test_vegas_runner.rb'

Dealing with Bundler usage

Tips for rails testapp/dummy app or when you can't easily remove bundler usage by patching. Some libraries meant to work with rails create a dummy rails app to test their functionality with a rails app. "case six: bundle install" tells you how to avoid bundler completely. If that fails, try the steps below.

  1. Remove Gemfile.lock in debian/clean or remove from source tarball as we can't expect debian environment to match exact versions of libraries used by the upstream developers.

  2. Add build dependencies on ruby-bundler and rubygems-integration
  3. Patch out Gemfile/gemspec/metadata.yml if bundler insist on a specific version of library and debian already has a newer version. Relax the dependency as ">=" or "~> x.y"

Common errors

First, check for build-depencies on rubygems.org--they are listed as development (see also the commented line in debian/control produced by gem2deb). You may not need all the build dependencies listed there; the only way to be sure is to see the build errors. Many of these dependencies are required during the tests.

Case one: missing library/gem

If you see a file is missing in the error message, it may be provided by a gem not yet installed in your system. For example, if you see

no such file to load -- shoulda (LoadError), 

search for shoulda, ie,

$ apt-cache search shoulda

If you find it is already packaged, install it with

# apt-get install ruby-shoulda

Note: Remember to add the build dependency you just found to Build-Depends in debian/control (edit this file)

Change

Build-Depends: debhelper (>= 7.0.50~), gem2deb (>= 0.3.0~)

to

Build-Depends: debhelper (>= 7.0.50~), gem2deb (>= 0.3.0~), ruby-shoulda

Note: If you see something like no such file to load -- rack/test (LoadError), look for rack-test as the gem name.

If you cannot find the package via apt, it may be a gem that has not yet been packaged for Debian. In this case, run:

gem list -r shoulda

If you find it in the remote gem list, then you need to package it first :)

Note: There are some libraries which we can safely ignore. Use quilt for commenting these out. See basic quilt for help in using quilt.

  1. appraisal - just uncomment require 'appraisal' lines as it is used for creating a specific environment using rubygems and bundler and will conflict with debian environment

  2. simplecov - used for code coverage analysis. Comment out require 'simplecov' and its other usages. ruby-oauth2 has an example patch

  3. ammeter - just comment out

  4. coveralls - code coverage, just comment out

  5. rcov - code coverage, just comment out

  6. rubygems - can conflict with apt

  7. bundler - can be difficult to remove sometimes. See "case six: bundle install" for tips.

Case two: tests are not in library load path

If you see spec_helper or test_helper is missing, you may need to add spec or test directory to library load path.

  in `require': no such file to load -- test_helper (LoadError)

Using ruby-tests.rb in debian folder

Uncomment the following line in ruby-tests.rb and modify it accordingly.

$: << 'lib' << '.' 

to

$: << 'test'  << '.' 

Using ruby-tests.rake in debian folder

Add

t.libs << 'test' << '.'

just after

Gem2Deb::Rake::TestTask.new do |t|

This adds test directory to load path and your tests will be able to find test_helper.

Case three: Argument Error

Example error statement : 'ensure_shared_example_group_name_not_taken'

This happens due to some namespacing error. Refer to this stackoverflow thread for more details.

Change this:

    Dir['{spec}/**/*.rb'].each { |f| require f }

to:

    Dir['{spec}/**/*_spec.rb'].each { |f| require f }

Case four: missing test folder from rubygems org

Sometimes, gems are not packaged with their test folders. Often, these are present at upstream, e.g., at github.

Download the tarball from the upstream source (you may need to convert it to .tar.gz yourself), then run dh-make-ruby <gem-name>.tar.gz.

Then cd to ruby-<gem-name> and run dpkg-buildpackage.

Look for any lines saying "no test suites found" and proceed further accordingly ignore the key signing and other warnings for now..

Note: Request for upstream to include tests in future version of the gem itself. See https://github.com/maxwell/rack-piwik/issues/6 for an example.

Case five: tests failure due to interactions

Sometimes, tests may pass perfectly when they are run separately and fail when we are building it because then the tests are running together and interact with each other. To see if the tests pass individually, run the following commands inside the test directory.

RUBY=ruby1.8 ./run_tests.sh
RUBY=ruby1.9.1 ./run_tests.sh

If the tests are passing now, you may want to figure out a way and edit the debian/ruby-tests.rb file by which the tests can run without interacting with each other. One such fix made to the ruby-tests.rb file is as follows.

Dir['test/*_test.rb'].each do |f|
  puts f        
  system("#{ENV['RUBY_TEST_BIN']} -Idebian/ruby-timecop/usr/lib/ruby/vendor_ruby:./test  #{f}") or raise        
end

The new ruby-tests.rb has a loop: for every test file, call ruby with this test file, and raise an exception if the test fails this way, all the tests are run separately.

Case six: bundle install

If you get an error like this asking you to run bundle install for any gem,

Could not find gem 'rake (>= 0) ruby' in the gems available on this machine.
Run `bundle install` to install missing gems.

Then it means the package is using bundler to satisfy dependencies. Building a deb package should not require use of bundler to install dependencies, but instead all dependencies should be provided in debian/control file as "Build-Depends:" and "Depends:"

You'll have to create a patch to un-comment use of bundler or rubygems like this. (example from ruby-rack-rewrite). Use quilt to create the patch. See basic quilt for help in using quilt.

--- a/test/test_helper.rb
+++ b/test/test_helper.rb
@@ -1,13 +1,18 @@
-require 'rubygems'
-require 'bundler/setup'
+# We'll add dependencies in debian/control
+#require 'rubygems'
+#require 'bundler/setup'

-Bundler.require :default, :development
+#Bundler.require :default, :development

 require 'test/unit'                               

See "Case seven" below for a solution to errors caused by this.


Case seven: NoMethodError

If you get an error similar to this,

./test/rack-rewrite_test.rb:47: undefined method `context' for RackRewriteTest:Class (NoMethodError)

It usually means the library is using bundler to require other libraries and which are missing, see "Case six" above for details. We can add those dependencies manually by patching test_helper.rb or spec_helper.rb as is the case.

--- a/test/test_helper.rb
+++ b/test/test_helper.rb
@@ -1,13 +1,18 @@

 require 'test/unit'
+require 'shoulda'
+require 'mocha'
+require 'rack'
+require 'rack/rewrite'

 class Test::Unit::TestCase
 end

 def supported_status_codes
   [:r301, :r302, :r303, :r307]
-end
\ No newline at end of file
+end

In the above example, the dependency is found from Gemfile, and corresponding gemspec

source "http://rubygems.org"

gemspec

group :development do
  gem 'rake'
end

Dependency section of rack-rewrite.gemspec is given below

  s.add_development_dependency 'bundler'
  s.add_development_dependency 'shoulda', '~> 2.10.2'
  s.add_development_dependency 'mocha', '~> 0.9.7'
  s.add_development_dependency 'rack'

Using these as reference, we add those as require lines in the patch. If you cannot solve case six and seven, see "Dealing with Bundler usage"

Case eight: autopkgtest failure

If the tests run correctly during build, but fail in autopkgtest, it is most likely because of using relative paths in tests and gem2deb-test-runner will remove lib directory from source tree during tests. You can look at ruby-fog-rackspace as an example.

Case nine: tests missing from the upstream tarball

By default, gem2deb will pull an upstream tarball from rubygems.org. Some Ruby upstreams consider Rubygems to be some kind of "binary" repository, so they don't upload the full source, which usually means not including the test files (test/*, spec/*, etc). The solution to this is usually updating debian/watch to pull source tarballs from github, and re-running dh-make-ruby so that it creates the file under debian/ (usually debian/ruby-tests.rake) that will drive the test execution, both during the build, and under autopkgtest.

Case ten: autopkgtest failure with multiple gemspec files

When there are multiple gemspec files in the source package, manually check the dependency using Test-Command field in debian/tests/control (ruby -e "gem 'gemname'"). Check ruby-flipper as an example.