[test] self-registration facilities
Hi, This message should serve as a temporary source of information regarding self registration faculties of Boost.Test. I will include the complete docs in next update. Here is the list of all available macros: ---------------------------------------------------- BOOST_AUTO_TEST_CASE( test_name ) ---------------------------------------------------- Simple free function like test case. Use like this: BOOST_AUTO_TEST_CASE( test ) { // test body } ---------------------------------------------------- BOOST_FIXTURE_TEST_CASE( test_name, F ) ---------------------------------------------------- Simple free function like test case. Employs struct F as a fixture. Use like this: struct F { F() { /* setup here */} ~F() { /* teardown here */} int i; }; BOOST_FIXTURE_TEST_CASE( test, F ) { // test body // i could be referenced here } ---------------------------------------------------- BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES( test_name, n ) ---------------------------------------------------- Use this to specify number of expected failures in test case. Use like this BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES( test1, 1 ) BOOST_AUTO_TEST_CASE( test1 ) { BOOST_CHECK( 2 == 1 ); } ---------------------------------------------------- BOOST_AUTO_TEST_CASE_TEMPLATE( test_name, type_name, type_list ) ---------------------------------------------------- Free function test case template for generating a test case for each type in type list. Use like this: typedef boost::mpl::list<char,int,float,double> test_types; BOOST_AUTO_TEST_CASE_TEMPLATE( test, T, test_types ) { // Body here; T is type under test } ---------------------------------------------------- BOOST_AUTO_TEST_SUITE( suite_name ) BOOST_AUTO_TEST_SUITE_END() ---------------------------------------------------- By default all test cases are registered in master test suite. There macros are used for constructing hierarchical test tree like this: BOOST_AUTO_TEST_SUITE( suite1 ); // this test case belongs to suite1 test suite BOOST_AUTO_TEST_CASE( test1 ) { // body here } BOOST_AUTO_TEST_SUITE_END(); ---------------------------------------------------- BOOST_FIXTURE_TEST_SUITE( suite_name, F ) ---------------------------------------------------- Similar to the BOOST_AUTO_TEST_SUITE, but force all the test cases within this test suite to use struct F as a fixture. BOOST_FIXTURE_TEST_SUITE( s, F ) // this test case use F as a fixture BOOST_AUTO_TEST_CASE( test1 ) { // body here } BOOST_AUTO_TEST_SUITE_END() ---------------------------------------------------- BOOST_GLOBAL_FIXTURE( F ) ---------------------------------------------------- Facilitate an ability to do global setup/teardown. Setup is invoked before any testing is started. Teardown - when all is done. Use like this: struct MyConfig { MyConfig() { std::cout << "global setup\n"; } ~MyConfig() { std::cout << "global teardown\n"; } }; BOOST_GLOBAL_FIXTURE( MyConfig ) You may have as many global fixtures as you want. To automatically setup init function define BOOST_TEST_MAIN in you main test module (in only one module in multi module test program) Alternatively you could define it explicitly. For example you could use it to rename a master test suite. Regards, Gennadiy
Gennadiy Rozental wrote:
Hi,
This message should serve as a temporary source of information regarding self registration faculties of Boost.Test. I will include the complete docs in next update.
1.34?
Here is the list of all available macros:
[snip]
---------------------------------------------------- BOOST_FIXTURE_TEST_CASE( test_name, F ) ----------------------------------------------------
Simple free function like test case. Employs struct F as a fixture. Use like this:
struct F { F() { /* setup here */} ~F() { /* teardown here */}
int i; };
BOOST_FIXTURE_TEST_CASE( test, F ) { // test body // i could be referenced here }
I haven't used Boost.Test previously, as I was missing some of the automatic test case registration and fixtures features from CppUnit. However, as I always use Boost in my projects I'd be happy to get rid of the CppUnit dependency. One of my pet peeves about CppUnit (and most of the xUnit frameworks) is that there's usually no controllable way of indicating a fixture setup failure. I'm running hardware dependent tests in a couple of my projects (e.g. serial port communications). I need a _supported_ way of indicating a setup error, leading to no further tests being executed in the same suite. Is this possible using Boost.Test? My main point is that the framework should be prepared for an exception to be thrown from the fixture setup (in this case, during F's ctor), and report it as an setup failure (including the exceptions' "what()" information). And, for the BOOST_FIXTURE_TEST_SUITE, don't run any tests within the suite when setup fails. [snip]
---------------------------------------------------- BOOST_FIXTURE_TEST_SUITE( suite_name, F ) ----------------------------------------------------
Similar to the BOOST_AUTO_TEST_SUITE, but force all the test cases within this test suite to use struct F as a fixture.
BOOST_FIXTURE_TEST_SUITE( s, F )
// this test case use F as a fixture BOOST_AUTO_TEST_CASE( test1 ) { // body here }
BOOST_AUTO_TEST_SUITE_END()
Do all the test cases within the BOOST_FIXTURE_TEST_SUITE and BOOST_AUTO_TEST_SUITE_END macros share the same instance of F (I'd personally prefer that)? Also, how about having a BOOST_FIXTURE_TEST_SUITE_END macro for symmetry (simply resolving to BOOST_AUTO_TEST_SUITE_END)?
---------------------------------------------------- BOOST_GLOBAL_FIXTURE( F ) ----------------------------------------------------
Facilitate an ability to do global setup/teardown. Setup is invoked before any testing is started. Teardown - when all is done.
Great! [snip rest] Another question, is there any possibility to implement hierarchical auto test suite registrations, e.g.: <i/o test suite> - "io" <ip test suite> - "ip" <http test suite> - "http" <file test suite> - "file" ... etc ... With reference to the test runner, it would be great to be able to specify e.g.: <runner> --run-tests "io" run all tests from "io","ip","http","file" test suites <runner> --run_tests "io/ip" run all tests from "ip","http" test suites Sorry for the noise if this feature already exists and I've missed it. // Johan
"Johan Nilsson" <r.johan.nilsson@gmail.com> wrote in message news:dnre20$vv1$1@sea.gmane.org...
Gennadiy Rozental wrote:
Hi,
This message should serve as a temporary source of information regarding self registration faculties of Boost.Test. I will include the complete docs in next update.
1.34?
I plan at least 3 updates before 1.34. Depends though how soon it will occur
One of my pet peeves about CppUnit (and most of the xUnit frameworks) is that there's usually no controllable way of indicating a fixture setup failure. I'm running hardware dependent tests in a couple of my projects (e.g. serial port communications). I need a _supported_ way of indicating a setup error, leading to no further tests being executed in the same suite. Is this possible using Boost.Test?
If I gather correctly you need one per suite fixture. It's last kind of fixture not yet supported and It's in my to do list for update 3. Once it's done you could just throw an exception to indicate init failure
My main point is that the framework should be prepared for an exception to be thrown from the fixture setup (in this case, during F's ctor), and report it as an setup failure (including the exceptions' "what()" information). And, for the BOOST_FIXTURE_TEST_SUITE, don't run any tests within the suite when setup fails.
BOOST_FIXTURE_TEST_SUITE doing different job - The fixture is per test case.
[snip]
---------------------------------------------------- BOOST_FIXTURE_TEST_SUITE( suite_name, F ) ----------------------------------------------------
Similar to the BOOST_AUTO_TEST_SUITE, but force all the test cases within this test suite to use struct F as a fixture.
BOOST_FIXTURE_TEST_SUITE( s, F )
// this test case use F as a fixture BOOST_AUTO_TEST_CASE( test1 ) { // body here }
BOOST_AUTO_TEST_SUITE_END()
Do all the test cases within the BOOST_FIXTURE_TEST_SUITE and BOOST_AUTO_TEST_SUITE_END macros share the same instance of F (I'd personally prefer that)?
No. I think that would defeat the purpose. You could always mimic what you want with plain Static instance: struct F{} f; TEST_CASE( t1 ) { f.i ... } TEST_CASE( t2 ) { f.i ... }
Also, how about having a BOOST_FIXTURE_TEST_SUITE_END macro for symmetry (simply resolving to BOOST_AUTO_TEST_SUITE_END)?
I don't see a need for this. BOOST_AUTO_TEST_SUITE_END ends any test suite.
Another question, is there any possibility to implement hierarchical auto test suite registrations, e.g.:
<i/o test suite> - "io" <ip test suite> - "ip" <http test suite> - "http" <file test suite> - "file"
... etc ...
Yes. BOOST_AUTO_TEST_SUITE could be put inside one another. Check examples in libs/example directory.
With reference to the test runner, it would be great to be able to specify e.g.:
<runner> --run-tests "io" run all tests from "io","ip","http","file" test suites <runner> --run_tests "io/ip" run all tests from "ip","http" test suites
"Run test by name" is scheduled for update 3. Gennadiy
Gennadiy Rozental wrote:
"Johan Nilsson" <r.johan.nilsson@gmail.com> wrote in message news:dnre20$vv1$1@sea.gmane.org...
Gennadiy Rozental wrote:
Hi,
This message should serve as a temporary source of information regarding self registration faculties of Boost.Test. I will include the complete docs in next update.
[snip]
One of my pet peeves about CppUnit (and most of the xUnit frameworks) is that there's usually no controllable way of indicating a fixture setup failure. I'm running hardware dependent tests in a couple of my projects (e.g. serial port communications). I need a _supported_ way of indicating a setup error, leading to no further tests being executed in the same suite. Is this possible using Boost.Test?
If I gather correctly you need one per suite fixture. It's last kind of fixture not yet supported and It's in my to do list for update 3. Once it's done you could just throw an exception to indicate init failure
That sounds about correct. I'll just have to wait and see.
My main point is that the framework should be prepared for an exception to be thrown from the fixture setup (in this case, during F's ctor), and report it as an setup failure (including the exceptions' "what()" information). And, for the BOOST_FIXTURE_TEST_SUITE, don't run any tests within the suite when setup fails.
BOOST_FIXTURE_TEST_SUITE doing different job - The fixture is per test case.
Even so (suite or not), if a fixture throws from the ctor, the framework should be prepared for that. I tried the following (1.33.1): --- namespace { struct Fixture { Fixture() { throw std::runtime_error("oops!"); } }; } BOOST_FIXTURE_TEST_CASE(Failure, Fixture) {} --- And the output was: --- unknown location(0): fatal error in "Failure": std::runtime_error: oops! --- Ideally, I'd like the framework to be able to differentiate between a Fixture construction failure (i.e. xUnit::TestXxx::setUp) and an error originating from e.g. BOOST_CHECK_xxx. Perhaps having a boost::test::setup_failure exception class that one could use for this purpose?
[snip]
---------------------------------------------------- BOOST_FIXTURE_TEST_SUITE( suite_name, F ) ----------------------------------------------------
Similar to the BOOST_AUTO_TEST_SUITE, but force all the test cases within this test suite to use struct F as a fixture.
BOOST_FIXTURE_TEST_SUITE( s, F )
// this test case use F as a fixture BOOST_AUTO_TEST_CASE( test1 ) { // body here }
BOOST_AUTO_TEST_SUITE_END()
Do all the test cases within the BOOST_FIXTURE_TEST_SUITE and BOOST_AUTO_TEST_SUITE_END macros share the same instance of F (I'd personally prefer that)?
No. I think that would defeat the purpose. You could always mimic what you want with plain Static instance:
struct F{} f;
TEST_CASE( t1 ) { f.i ... }
TEST_CASE( t2 ) { f.i ... }
And could you do this while using suites / fixtures - or is this part of the new functionality?
Also, how about having a BOOST_FIXTURE_TEST_SUITE_END macro for symmetry (simply resolving to BOOST_AUTO_TEST_SUITE_END)?
I don't see a need for this. BOOST_AUTO_TEST_SUITE_END ends any test suite.
I understood that (as you can see from my writing above). It's just that there exists BOOST_AUTO_TEST_SUITE and BOOST_AUTO_TEST_SUITE_END (which is symmetric and nice), but then there's also BOOST_FIXTURE_TEST_SUITE and it's not very intuitive to use BOOST_ *AUTO* _TEST_SUITE_END to end that suite. Perhaps use a plain BOOST_TEST_SUITE_END instead?
Another question, is there any possibility to implement hierarchical auto test suite registrations, e.g.:
<i/o test suite> - "io" <ip test suite> - "ip" <http test suite> - "http" <file test suite> - "file"
... etc ...
Yes. BOOST_AUTO_TEST_SUITE could be put inside one another. Check examples in libs/example directory.
I've searched, but couldn't find any examples with nested test suites. I only found one example with multiple test suites within the same file. Could you give me a pointer? Even if this is possible, does the current implementation require the suites to be defined within the same compilation unit? To be able to scale using this approach (I'm talking scaling within the context of maintaining large sets of unit tests), this should not be necessary.
With reference to the test runner, it would be great to be able to specify e.g.:
<runner> --run-tests "io" run all tests from "io","ip","http","file" test suites <runner> --run_tests "io/ip" run all tests from "ip","http" test suites
"Run test by name" is scheduled for update 3.
That's great - should that work with nested test suites as in my example above? // Johan
Even so (suite or not), if a fixture throws from the ctor, the framework should be prepared for that. I tried the following (1.33.1):
--- namespace { struct Fixture { Fixture() { throw std::runtime_error("oops!"); } }; }
BOOST_FIXTURE_TEST_CASE(Failure, Fixture) {} ---
And the output was:
--- unknown location(0): fatal error in "Failure": std::runtime_error: oops! ---
Ideally, I'd like the framework to be able to differentiate between a Fixture construction failure (i.e. xUnit::TestXxx::setUp) and an error originating from e.g. BOOST_CHECK_xxx. Perhaps having a boost::test::setup_failure exception class that one could use for this purpose?
With current design fixture is a natural part of test case. I recommend to use specific exception to report init failures.
struct F{} f;
TEST_CASE( t1 ) { f.i ... }
TEST_CASE( t2 ) { f.i ... }
And could you do this while using suites / fixtures - or is this part of the new functionality?
Why not? You were always able to do this even with explecetly registerred tet cases.
I don't see a need for this. BOOST_AUTO_TEST_SUITE_END ends any test suite.
I understood that (as you can see from my writing above). It's just that there exists BOOST_AUTO_TEST_SUITE and BOOST_AUTO_TEST_SUITE_END (which is symmetric and nice), but then there's also BOOST_FIXTURE_TEST_SUITE and it's not very intuitive to use BOOST_ *AUTO* _TEST_SUITE_END to end that suite.
Perhaps use a plain BOOST_TEST_SUITE_END instead?
I do not want to interfere with BOOST_TEST_SUITE macro that is used for explicit suite creation.
Another question, is there any possibility to implement hierarchical auto test suite registrations, e.g.:
<i/o test suite> - "io" <ip test suite> - "ip" <http test suite> - "http" <file test suite> - "file"
... etc ...
Yes. BOOST_AUTO_TEST_SUITE could be put inside one another. Check examples in libs/example directory.
I've searched, but couldn't find any examples with nested test suites. I only found one example with multiple test suites within the same file. Could you give me a pointer?
unit_test_example_04.cpp should give you introduction into suites usage. Indeed it does not show multilevel hierahy, but I will address this.
Even if this is possible, does the current implementation require the suites to be defined within the same compilation unit? To be able to scale using this approach (I'm talking scaling within the context of maintaining large sets of unit tests), this should not be necessary.
I will need to check this.
With reference to the test runner, it would be great to be able to specify e.g.:
<runner> --run-tests "io" run all tests from "io","ip","http","file" test suites <runner> --run_tests "io/ip" run all tests from "ip","http" test suites
"Run test by name" is scheduled for update 3.
That's great - should that work with nested test suites as in my example above?
Will see. I think so. Gennadiy
Gennadiy Rozental wrote:
Hi,
This message should serve as a temporary source of information regarding self registration faculties of Boost.Test. I will include the complete docs in next update.
Forgot this question in my previous posting: Is it possible to create static libraries containing self-registering tests, and still have the tests executed when the library is linked to form an executable / a test runner? For CppUnit, this wasn't possible - at least not portably - as the linker discarded the global registrar objects (as they weren't "used") that were responsible for registering the test suites. // Johan
Is it possible to create static libraries containing self-registering tests, and still have the tests executed when the library is linked to form an executable / a test runner?
You should be able to. Though, since I never tried, I couldn't promise 100%. Give it a try and let me know an outcome. Gennadiy
participants (13)
-
Gennadiy Rozental -
Johan Nilsson