Extension talk:Selenium.php
|
The idea of this extension is to bundle selenium up along with the MediaWiki extension and provide a simple means of incorporating tests along with extensions which can be maintained as both wiki articles, and a special page used to manually run the tests. Also we'd like the ability to run tests on a regular schedule and have an email or talk-page based failure notification system. This would allow us to know immediately when changes to code affect extension version functionality when running in various MediaWiki versions.
Taking testing to a further level of organisation would allow us to package up wiki-based organisational systems which exactly specify their functionality and exhibit corresponding tests for every item specified. This would allow us to move into a higher level of service automation and would form the basis of a more robust SLA-based service architecture.
Contents
Installation
The extension consists of a single script which should be in a directory called Selenium in your extensions directory. The extension is then included from LocalSettings as usual: <php> include("$IP/extensions/Selenium/Selenium.php"); </php> The selenium core needs to also be installed on the system somewhere, and the default location is in a directory called selenium-core along side the extension script in the Selenium directory. The Selenium core can be downloaded from here as a zip file, or can be maintained via SVN as follows:
cd Selenium svn co https://svn.openqa.org/svn/selenium-core/trunk/src/main/resources selenium-core
Road Map
The overall plan for the Selenium extension is to work with Extension:Packages.php to create packages of tests and the articles required by the tests. Each package can test a particular set of functionality such as testing each feature that a particular extension claims to add to the environment, testing that examples in documentation aren't broken, or a general test of MediaWiki functionality etc.
This structured system of tests can then be integrated into our organisational system so that it can be easily implemented on other client servers and kept up to date. The tests can then be run on schedule on all of our test wikis so that automatic reports of failures on various environments and versions can be delivered in email or recent changes.
This testing environment will then become an important addition to our offered services which come with our general infrastructure that we install and maintain for clients and associated organisations. The current state of development of this infrastructure is in Wiki Organisation.
In-wiki test format
|
If each test was a separate article, then they can be categorized etc. Utilizing existing Selenium IDE output HTML table format would be convenient. Each test article would be a suite of tests where each individual test is in a separate heading section, and then the actual functions composing the test is just a table within its section. This simplifies things by allowing all the tests for a given extension to be in just one article. It is quickest to maintain HTML tables as these can be imported quickly from the Selenium IDE as is. The only redundant tags are the <html>...</html> wrappers.
It appears that the table tags <thead></thead><tbody></tbody> are not currently defined by the wiki parser, see W:Help:HTML_in_wikitext#Permitted_HTML.
- Either use a <selenium> or just add tag-hooks for thead and tbody which return null - probly easiest --Nad 09:19, 7 December 2007 (NZDT)
I guess for simplicity the test environment should utilize the current way selenium works taking you to a URL which shows the selenium frames and the tests it is about to perform where you select the article to test from the Special page.
The extension should come with a test that examines the selenium extension, checks it is installed, what version it is and usage.
Specialpage
The url http:///www.organicdesign.co.nz/Specialpage:Selenium should allow users to select an article that they want to test from a pull down list, when the test is run to selenium the suite html will be created on the fly by selecting the url - http://www.organicdesign.co.nz/Special:Selenium/Test article. As the article is made up of individual tests, the special page would also create the test html dynamically, e.g. <bash>
- Test article
http://www.organicdesign.co.nz/Special:Selenium/Test_article http://www.organicdesign.co.nz/Special:Selenium/Test1 http://www.organicdesign.co.nz/Special:Selenium/Test2
http://www.organicdesign.co.nz/Special:Selenium/Test_article#third_selenium_tag_header
http://www.organicdesign.co.nz/Special:Selenium/Test4 ...
- Extension location
$IP/extensions/Selenium/Selenium.php
- Selenium core location
$IP/extensions/Selenium/selenium-core-0 $IP/extensions/Selenium/selenium -> $IP/extensions/Selenium/selenium-core-0
</bash>
The expected html of a selenium test is currently a test Suite file which links to many tests. An example structure is
- Test suite wrapper file
<html><head><meta content="text/html; charset=ISO-8859-1" http-equiv="content-type"> <title>Test Suite</title> </head><body><table cellpadding="1" cellspacing="1" border="1"><tbody> <tr><td><b>Flouzo Test Suite</b></td></tr> <tr><td><a href="/selenium/Wikitests/Logout.html">Logout.html</a></td></tr> </tbody></table></body></html>
- A selenium test - /selenium/Wikitests/Logout.htm
<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> <title>New Test</title> </head> <body> <table cellpadding="1" cellspacing="1" border="1"> <thead> <tr><td rowspan="1" colspan="3">New Test</td></tr> </thead><tbody> <tr> <td>open</td> <td>/mediawiki-1.11.0/index.php?title=Special:Userlogout</td> <td></td> </tr> </tbody></table> </body> </html>
Test article naming
The name of the article should be in the "test", "selenium" or maybe "spec" namespace and have the same title as the extension it is designed to test. This way, the Selenium extension can offer an action button to the tests if there is one corresponding to the current page. Whatever the name, there should be a unique namespace, then these articles could be transcluded into a master article allowing template variable embedding, however we want to suppress certain wikitext though.
Specification
Each section is a test for some specific functionality which should be described clearly in plain english below the heading before the bullet list/HTML table of test functions. This way each extension's test article becomes a complete specification of the extended functionality which is both human readable and machine testable. For this reason the "spec" namespace may be more appropriate than "test".
Test Execution
The special page allows all the known tests to be listed, and specific tests to be executed by launching the Test Runner. The test runner requires the tests to be in a simple HTML table format, and requires them to be available from a URL relative to the Selenium code. The special page must therefore be able to accept for its parameter the name of the tests to run and be able to return them in the appropriate format for Selenium.
Test Automation
We need to include as part of the extension (maybe in an additional PERL script) the ability to run the tests automatically from crontab and report to us via talk page or a test log article any problems which crop up due to code or environment changes.
- Automation of the selenium-core suite can be done by adding the query string parameter &auto=true (see details). An additional url can be provided &resultsUrl = [relative path] which creates an html output file. See Continuous integration section for details. By default the querystring parameter resultsUrl posts results to /postResults, which is in the path
$IP/extensions/Selenium/selenium-core/core.
By creating a quick script called results.php in a temporary directory in $IP/extensions/Selenium/ with contents;
and redirecting resultsUrl=../../tmp/results.php.
an example of the posted parameters sent are as follows;
It appears that the important posted parameters are;
- result => passed
- totalTime => 5
- numTestPasses => 1
- numTestFailures => 0
- numCommandPasses => 0
- numCommandFailures => 0
- numCommandErrors => 0
I think its probly best to keep all its files in its own dir (ie extensions/selenium) same as other extensions like searchlog which generate other files it should also be able to be executed from shell and run selected tests eg from cron etc.
How does this complement SubVersion (SVN)?
I guess SVN testing can only check server side PHP for errors in core (and extension) MediaWiki functionality, whereas Selenium is testing the output environment from the end user clients perspective.
See also
- MediaWiki development policy
- Selenium
- Extension:CategoryHook.php -Categorization of <selenium> tags
- extractSections() in Parser.php
MediaWiki:Common.css
The following CSS rules can be added to enhance table rendering;
/** CSS placed here will be applied to all skins */
div.selenium table {
border:1px solid #CCCCCC;
border-collapse:collapse;
font-family:Verdana,Arial,sans-serif;
font-size:12px;
}
Cross site scripting
- Pseudo code
