Test automation frameworks
Recently I researched many testing frameworks in order to choose one to use at work. I spent few days practically testing the following frameworks:
I chose Robot Framework. It is not perfect, but promising, extensible, and actively developed by the open source community. The following features made me choose this framework over others:
- Keyword driven
- Ability to use re-usable keywords
- Large collection of built-in keywords
- Test cases and suites are HTML documents
- RobotIDE test case editor
- Extendable in Python and Jython
- Reporting using HTML and XML files
All this gives a solid base to extend and wrap around in order to create state-of-the art testing framework. Robot framework is easily installable. Examples and documentation are more than enough to get automation engineers and testers started. It requires some learning and getting used to (especially RobotIDE) but once grasped, creating test cases is easy. Additionally, new keywords can be simply programmed in Python or Jython, or created with existing keywords (macros). Tests are started with a command line tool and can be very simply scheduled using cron.
FitNesse framework is widely used and popular, but is not as extendable as robot framework. It’s a testing wiki. Test cases are created as tables in wiki documents and are started by clicking a button on a page. The results show up immediately, but there is no way to store them. Each test table must have a code fixture written in Java. Other languages can be used with a help of additional Fit servers. Being a wiki, FitNesse cannot be used for automatic and non-interactive testing. Automation can be achieved by using command line runner or ant tasks.
Cucumber is a testing environment that allows testers to write tests in a a domain-specific language based on a spoken language. It’s behavior-driven, but all used sentences must be mapped in underlying programming language (Ruby by default). It’s easy to transcribe users stories into test cases, but it requires automation engineer to work with testers writing every test case. On the other hand, it’s a very interesting and amazing way to write unit tests.