Re: boost.test covered in testing frameworks rundown
Gennadiy Rozental wrote:
Boost.Test does support fixtures. At least in a sense author using this term. Noel found proper thread in mailing list to refer, but made completely wrong conclusion from it. To implement fixtures one needs to use constructor and destructor *for fixture*.
Thanks for pointing that out, Gennadiy. After so many years of using NUnit setup/teardown, I was missing the obvious way of doing the same thing in C++. I've updated the article to reflect that fixtures are perfectly possible with Boost.Test. http://www.gamesfromwithin.com/articles/0412/000061.html With that, Boot.Test definitely makes it to my list of finalists. The only two drawbacks I see are the reliance of some of the other Boost libraries (it would be ideal if something this low level could be fully self-contained), and the verbosity needed to create suites. Cheers. --Noel Games from Within http://www.gamesfromwithin.com
With that, Boot.Test definitely makes it to my list of finalists. The only two drawbacks I see are the reliance of some of the other Boost libraries (it would be ideal if something this low level could be fully self-contained),
Actually Boost.Test doesn't impose that many dependencies (in other case it wouldn't be able to use it for boost internal testing). It does depend config subsystem and some other core part of boost (like smart_ptr for example), which are mostly targeted for TR1. But namely due to this dependencies (not in spite of) Boost.Test achieve it's portability.
and the verbosity needed to create suites.
See below.
Cheers.
--Noel
And here some comments to the article:
On the downside, it's annoying to have to refer to the variables in the fixture through the object name. It would be great if they could somehow magically appear in the same scope as the test case itself. Also, it would have been a bit cleaner if the fixture could have been setup on the stack by the BOOST_AUTO_UNIT_TEST macro instead of having to explicitly put it on the stack for every test case.
Probably, but it's not exactly trivial to change. At least the default output is IDE friendly. I suspect I would need to dig deeper into the unit_test_log_formatter, but I certainly didn't see a variety of preset output types that I could just
Here is what you could do: #define TEST( F, test_name ) \ struct my_test ## __LINE__ : public F { \ void test_name(); \ }; \ \ void test_invoker ## __LINE__() \ { \ my_test mt; \ mt.test_name(); \ } \ \ static boost::unit_test::ut_detail::auto_unit_test_registrar \ BOOST_JOIN( test_registrar, __LINE__) \ ( boost::unit_test::create_test_case( test_invoker ## __LINE__ , #test_name ) ); \ \ void my_test ## __LINE__::test_name() \ /**/ And you got everything you asked for: struct MyFixture { MyFixture() { someValue = 2.0; str = "Hello"; } float someValue; std::string str; MyTestClass myObject; }; TEST ( MyFixture, TestCase1) { BOOST_CHECK_CLOSE ( someValue, 2.0f, 0.005f); } If you find it generally useful I could add this helper macro into framework. plug in. Boost.Test supply 2 predefined log formats (human readable and XML, which you could select by command-line) and unit_test_log_formatter interface you could implement for you own formats. What else do you need?
Supports suites. Yes, but with a big catch. Unless I'm missing something (which is very possible at this point--if so make sure to let me know), creating a suite requires a bunch of fairly verbose statements
Adding test_suites usually means only something like this: test_suite* test = BOOST_TEST_SUITE( "Master test suite" ); test_suite* ct = BOOST_TEST_SUITE( "Constructors test" ); test->add( ct ); It's not obvious how to add auto registration for suites, but if you have ant specific proposition let us hear it.
and also requires modifying the test runner itself in main.
I don't think that's true, or I do not understand what it means
Have a look at the example below. Couldn't that have been simplified to the extreme? It's not a big deal as this is my least-wanted requirement, but I wish I could label all the test cases in one file as part of a suite with a simple macro at the beginning of the file.
You could achieve something like this: #define SUITE( name ) \ static boost::unit_test::test_suite* suite = BOOST_TEST_SUITE( name ); \ static boost::unit_test::ut_detail::auto_unit_test_registrar \ BOOST_JOIN( test_registrar, __LINE__)( suite ); \ /**/ struct suite_registrar { suite_registrar( test_case* tc ) { suite->add( tc ); } }; #define TEST( name ) \ static void func_name(); \ suite_registrar \ BOOST_JOIN( suite_registrar, __LINE__) \ ( BOOST_TEST_CASE( func_name ) ); \ static void func_name() \ /**/
If you're going to be working mostly on the PC, ...., Boost.Test could be an excellent choice.
Did you really mean that Boost.Test is only PC oriented? Or it's actually something different? Like: "If you aren't doing embedded development..." Regards, Gennadiy
I've been using Boost.Test from the beggining of my project and never had any kind of issue with it. For me it "Just worked". Good thing too as I would probably have given up otherwise. Prior to this I had never used a unit testing setup of any kind. I only used the most rudimentary facilities - basically wrapping my test program withe execution test monitor and using the test macros. This article was helpful to me in understanding how the other aspects of it might be used. It also seems that the writer of the article who has quite a familiarity with such systems didn't see all the relevant aspects of the test system either. All this suggests to me that the test system could benefit from the addition of a more "tutorial" type documentation. I know you're going to point out the relevant part of the documentation that covers this or that point. In fact, now that I 've read Noel's article and these posts, I could probably go back and read the Boost Test documentation and it would all be clear to me. And that's my point. Robert Ramey
I've tried out the suggestions Gennadiy had for streamlining the use of Boost.Test fixtures and suites, and I'm extremely pleased with the results. This is what I ended up with (compiled and tested with gcc under Linux). Feel free to adapt anything into Boost.Test: #define BOOST_TEST_SUITE_MODULE( name ) \ namespace { \ static boost::unit_test::test_suite* suite = BOOST_TEST_SUITE( #name ); \ static boost::unit_test::ut_detail::auto_unit_test_registrar \ BOOST_JOIN( test_registrar, __LINE__)( suite ); \ \ struct suite_registrar { \ suite_registrar( boost::unit_test::test_case* tc ) { suite->add( tc ); } \ }; \ } \ /**/ #define BOOST_AUTO_UNIT_TEST_WITH_FIXTURE_AND_SUITE( fixture, test_name ) \ struct fixture ## test_name : public fixture { \ void test_name(); \ }; \ \ void fixture ## test_name ## _invoker () \ { \ fixture ## test_name mt; \ mt.test_name(); \ } \ \ static suite_registrar \ BOOST_JOIN( suite_registrar, __LINE__) \ ( boost::unit_test::create_test_case( fixture ## test_name ## _invoker, #test_name " fixture " #fixture) ); \ void fixture ## test_name::test_name() \ /**/ #define BOOST_AUTO_UNIT_TEST_WITH_SUITE(test_name) \ static void test_name(); \ static suite_registrar \ BOOST_JOIN( suite_registrar, __LINE__) \ ( BOOST_TEST_CASE( test_name ) ); \ static void test_name() \ /**/ // Shortcuts to avoid lots of typing #define TEST_F(fixture,test_name) BOOST_AUTO_UNIT_TEST_WITH_FIXTURE_AND_SUITE(fixture,test_name) #define TEST(test_name) BOOST_AUTO_UNIT_TEST_WITH_SUITE(test_name) Now, if we can only have a unit_test_log_formatter that uses DebugOutputString, that would be fantastic. I'd volunteer to look into it, but I don't use Windows for development at home anymore. A couple of questions: What was the reason behind the choice of using the function init_unit_test_suite() as the initialization point? Why not let the user create main() and use a very simple object to accomplish the same thing. Something along these lines: int main ( int /* argc */, char* /* argv */ [] ) { test_suite * testsuite = BOOST_TEST_SUITE("Master test suite"); testrunner runner(testsuite); return runner.run(); } That way we can do whatever we want before and after the test runs much more easily. Also, if I wanted to add timings around each test, where would be the best place to hook them up? I'm also interested in the timings around the whole set of tests, but that's something I could accomplish easily if we had an exposed main() function. I think that with these changes, Boost.Test has tipped the balance as the favorite. I'm going to have to run a few more tests, but if all continues to look good, I'm going to put up another article on my web site with everything I've learned about Boost.Test Thanks. --Noel
Noel Llopis wrote:
Now, if we can only have a unit_test_log_formatter that uses DebugOutputString, that would be fantastic. I'd volunteer to look into it, but I don't use Windows for development at home anymore.
I'm not sure if this is what you're referring to but ... I added all my tests to the post-build stop according to instructions in the VC 7.1 IDE. Each time I rebuild, the tests are run - and the messages all show up in the "Outpu Window" of the IDE. Also anything added to the "Tools" dropdown menu can be flaged to send its output to the IDE output window. If this is not what you're referring to, I'd be curious what you mean. Robert Ramey
Robert Ramey <ramey <at> rrsd.com> writes:
I'm not sure if this is what you're referring to but ... I added all my tests to the post-build stop according to instructions in the VC 7.1 IDE. Each time I rebuild, the tests are run - and the messages all show up in the "Outpu Window" of the IDE.
Hmm... I'm trying to think why I needed that in the past. Yes, redirecting the output to the output page will work fine as long as you're working with a "console" app. Unfortunately there are times that to initialize some of the libraries we need to create a DirectX surface, or at least a Window instance. In that case, we have to create a Win32 app, and std::out is not piped correctly to the output. I'll have to double-check this. One thing is for sure though, having control over main() is very important. Otherwise it might not even be possible to create anything other than a regular console app. I thought it would be relatively trivial to take over main, but then I realized that main() is part of the library right now, so the ony way to provide your own would be modifying the source code of the library and recompiling. Any thoughts on that? --Noel
Yes, redirecting the output to the output page will work fine as long as you're working with a "console" app. Unfortunately there are times that to initialize some of the libraries we need to create a DirectX surface, or at least a Window instance. In that case, we have to create a Win32 app, and std::out is not piped correctly to the output. I'll have to double-check this.
Where is it piped to then?
One thing is for sure though, having control over main() is very important. Otherwise it might not even be possible to create anything other than a regular console app. I thought it would be relatively trivial to take over main, but then I realized that main() is part of the library right now, so the ony way to provide your own would be modifying the source code of the library and recompiling.
Yes. Right now it's not possible to use framework as it is with WinMain (at lease that what I think). We may need to create another component usable for such cases. Gennadiy
Gennadiy Rozental wrote:
Yes, redirecting the output to the output page will work fine as long as you're working with a "console" app. Unfortunately there are times that to initialize some of the libraries we need to create a DirectX surface, or at least a Window instance. In that case, we have to create a Win32 app, and std::out is not piped correctly to the output. I'll have to double-check this.
Where is it piped to then?
I am not sure where it is piped to, but it is possible to get the output. Given: // winapp.exe WinMain(...) { std::cout << "Hello World!\n"; } On the console: winapp produces no output, while winapp | cat produces: Hello World! Regards, Reece
"Noel Llopis" <nll_pub@convexhull.com> wrote in message news:loom.20050111T201553-503@post.gmane.org...
I've tried out the suggestions Gennadiy had for streamlining the use of Boost.Test fixtures and suites, and I'm extremely pleased with the results.
This is what I ended up with (compiled and tested with gcc under Linux). Feel free to adapt anything into Boost.Test:
[...] I will add something along these lines.
Now, if we can only have a unit_test_log_formatter that uses DebugOutputString, that would be fantastic. I'd volunteer to look into it, but I don't use Windows for development at home anymore.
Actually I realized that I may mislead you a bit. You do not need custom formatter to direct output using DebugOutputString (unless of course you really need different format). What you need is custom std::ostream that do that. unit_test_log has an interface to reset output stream: unit_test_log::instance().set_log_stream( debug_output );
A couple of questions:
What was the reason behind the choice of using the function init_unit_test_suite() as the initialization point? Why not let the user create main() and use a very simple object to accomplish the same thing. Something along these lines:
int main ( int /* argc */, char* /* argv */ [] ) { test_suite * testsuite = BOOST_TEST_SUITE("Master test suite"); testrunner runner(testsuite); return runner.run(); }
1. If you take a look into unit_test_main.cpp you notice that there is quire a lot of things that needs to be done before test is initialized and run 2. We want to enforce some predefined set of result codes. User may return worng value by mistake 3. Any other solution would lead to extra work for test preparation (at least 2 lines like you have above) 4. There may be other things in framework housekeeping that may needs to be done pre/post test execution 5. This is common scheme used for all Boost.Test components
That way we can do whatever we want before and after the test runs much more easily.
Anything you want to do pre/post your test is part of fixure, isn't it?
Also, if I wanted to add timings around each test, where would be the best place to hook them up?
Including timing.
I'm also interested in the timings around the whole set of tests, but that's something I could accomplish easily if we had an exposed main() function.
I may give you wrong result since it is including a housekeeping time. And you could easily get this time from outside just be timing program execution Gennadiy
On Tuesday 11 January 2005 14:03, Gennadiy Rozental wrote:
What was the reason behind the choice of using the function init_unit_test_suite() as the initialization point? Why not let the user create main() and use a very simple object to accomplish the same thing.
Something along these lines:
int main ( int /* argc */, char* /* argv */ [] ) { test_suite * testsuite = BOOST_TEST_SUITE("Master test suite"); testrunner runner(testsuite); return runner.run(); }
1. If you take a look into unit_test_main.cpp you notice that there is quire a lot of things that needs to be done before test is initialized and run
Nothing that can't be easily wrapped in an object like the example above.
4. There may be other things in framework housekeeping that may needs to be done pre/post test execution
Right, but you can wrap all that inside a separate function instead of taking over main().
Anything you want to do pre/post your test is part of fixure, isn't it?
No. In this case I was thinking of tasks that I might have to do around the test run myself. For example, I don't think I could use Boost.Test with a Win32 application (as opposed to a DOS box). Also, I might want to set up some things conditionally depending on whether the debugger is attached or not. In general, it seems like a much better idea to let the user run the tests from anywhere they want as opposed to taking over main. For example, sometimes people like to embed the tests themselves into the program they're building, and running the tests if they pass a command-line parameter. This setup would allow them to do that without any problem. The ideal situation would be to wrap the contents of the current main in an object, and then provide a macro that gives you a main function calling that object. Those of us who need more control, we can write our own main and pass all the necessary parameters. That's one of the areas that I think CppUnit did a really good job. This is how one possible main looks for them: int main( int argc, char* argv[] ) { // Create the event manager and test controller CPPUNIT_NS::TestResult controller; // Add a listener that colllects test result CPPUNIT_NS::TestResultCollector result; controller.addListener( &result ); // Add the top suite to the test runner CPPUNIT_NS::TestRunner runner; runner.addTest( CPPUNIT_NS::TestFactoryRegistry::getRegistry().makeTest() ); runner.run( controller ); // Print test in a compiler compatible format. CPPUNIT_NS::CompilerOutputter outputter( &result, std::cerr ); outputter.write(); return result.wasSuccessful() ? 0 : 1; } It's very configurable, and it can be called from anywhere. It might be a bit long, but you're only going to write it once per test project, so it's not a big deal at all. --Noel
At 05:57 PM 1/11/2005, Noel Llopis wrote:
That's one of the areas that I think CppUnit did a really good job. This is how one possible main looks for them:
int main( int argc, char* argv[] ) { // Create the event manager and test controller CPPUNIT_NS::TestResult controller;
// Add a listener that colllects test result CPPUNIT_NS::TestResultCollector result; controller.addListener( &result );
// Add the top suite to the test runner CPPUNIT_NS::TestRunner runner; runner.addTest( CPPUNIT_NS::TestFactoryRegistry::getRegistry().makeTest() ); runner.run( controller );
// Print test in a compiler compatible format. CPPUNIT_NS::CompilerOutputter outputter( &result, std::cerr ); outputter.write();
return result.wasSuccessful() ? 0 : 1; }
It's very configurable, and it can be called from anywhere. It might be a
bit long, but you're only going to write it once per test project, so it's not a big deal at all.
I beg to differ. Having to supply all that boilerplate would be a huge disincentive. One of the really nice things about Boost.Test is how easy it is to set up a new test project. It really encourages setting up test programs not just for large important projects, but also for every little project that come along. The low entry cost of starting a Boost.Test project is also an incentive when trying to convince more programmers to get into the habit of always writing tests. If Gennadiy can provide addition functionality making it easier for those needing more startup/shutdown control, great. But it shouldn't come at any cost to those who don't need that functionality. --Beman
On Tuesday 11 January 2005 16:32, Beman Dawes wrote:
I beg to differ. Having to supply all that boilerplate would be a huge disincentive.
If that functionality were exposed separately, putting it together in one macro with all the options you want to would be absolutely trivial. Then all you would write is BOOST_TEST_MY_MAIN And if that's not good enough, then you get your hands dirty and write main.
If Gennadiy can provide addition functionality making it easier for those needing more startup/shutdown control, great. But it shouldn't come at any cost to those who don't need that functionality.
From my point of view, it's more important to provide that functionality than to keep the ease of use. Otherwise it might simply come a time that you can't use it (combining it with another library that also provides its own custom main). Fortunately, once that functionality is provided, wrapping it in your own macro with the exact options you want is quite trivial.
The more I think about it though, the more I see that the problem lies in the fact that Boost.Test is trying to do double-duty. On one hand, it's a unit-test framework to efficiently create and run unit tests. On the other hand, it's a part of the Boost.Test library, and tries to conform to certain outputs and command-line arguments, and that's why main is tightly controlled. Those two aspects should really be separated. The Boost.Test component can provide a main function that parses the command-line in a certain way, and calls the unit test framework accordingly. But people should also be free to use the unit test framework without being tied to that. For example, I might want to set different options (like what log to use) based on a command-line parameter, or set exception-handling on/off based on whether the debugger is present. Another way of separating those two aspects would be simply to create another library with what's right now in test_main.cpp. If you link to that library, you get the standard main. Otherwise you need to provide your own. Does that make sense? --Noel Games from Within http://www.gamesfromwithin.com
participants (7)
-
Beman Dawes -
Gennadiy Rozental -
Noel Llopis -
Reece Dunn -
Robert Ramey