Extension talk:Selenium.php

Selenium is a browser testing environment for testing the various aspects of a sites features and functionality from a users perspective. We see this as an extremely useful tool for use in the MediaWiki environment, for testing MediaWiki installations and installed extensions. Such tests could be maintained along with extension code and documentation by the community responsible for that extension.

Selenium is available for many programming environments including PHP. The tests are written in a simple language called Seleniumese which can be contained in many common textual structures such as XML or HTML. Essentially tests are just lists of Selenium function calls so if tests were maintained as articles in the wiki, a bullet list would be most appropriate. There are also test suites which are in a similar format to the tests but refer to a list of tests rather than a list of functions. We have some example tests which act on a wiki here.

The idea of this extension is to bundle selenium up along with the MediaWiki extension and provide a simple means of incorporating tests along with extensions which can be maintained as both wiki articles, and a special page used to manually run the tests. Also we'd like the ability to run tests on a regular schedule and have an email or talk-page based failure notification system. This would allow us to know immediately when changes to code affect extension version functionality when running in various MediaWiki versions.

Taking testing to a further level of organisation would allow us to package up wiki-based organisational systems which exactly specify their functionality and exhibit corresponding tests for every item specified. This would allow us to move into a higher level of service automation and would form the basis of a more robust SLA-based service architecture.

In-wiki test format

Rather than dividing the tests into two kinds of article (tests and test-suites) I think it may be more convenient to have just a single article which defines all the tests for a given extension or functionality set. And rather than using HTML table format, simple bullet lists would be more convenient. So each test article would be a suite of tests where each individual test is in a separate heading section, and then the actual functions composing the test are in a bullet list within its section. This simplifies things by allowing all the tests for a given extension to be in just one article.

Test article naming

The name of the article should be in the "test", "selenium" or maybe "spec" namespace and have the same title as the extension it is designed to test. This way, the Selenium extension can offer an action button to the tests if there is one corresponding to the current page.

Comments

Each section is a test for some specific functionality which should be described clearly in plain english below the heading before the bullet list of test functions. This way each extension's test article becomes a complete specification of the extended functionality which is both human readable and machine testable. For this reason the "spec" namespace may be more appropriate than "test".