Extension talk:Selenium.php
|
Selenium Core is a test tool for web applications. Selenium Core tests run directly in a browser, just as real users do. And they run in Internet Explorer, Mozilla and Firefox on Windows, Linux, and Macintosh. No other test tool covers such a wide array of platforms. The idea of this extension is to bundle the Selenium Core functionality up as a MediaWiki extension to provide a simple means of incorporating tests along with extensions so that all their aspects of functionality can be tested for any given wiki installation, and failure reports be delivered automatically whenever functionality breaks.
Selenium tests are defined in an HTML table format which can be initially created in the Selenium IDE which runs in your browser. The process of creating and running tests in the Selenium IDE is very similar to recording and playing macros in Open Office or other popular office suite applications. The functionality of the recorded test text in HTML format which can be further refined in a text editor if necessary.
In the MediaWiki environment, articles can be created which contain suites of tests instead of having them in HTML files. A test-suite article is divided into sectons, one heading for each test, then the HTML table of each test is pasted into each section. For tests or parts of tests which are re-used across a number of suites or tests, templates can be transcluded the usual way to make up the content of the test-suite article. See the Example Selenium test suite for details.
The Selenium special page is then used to select test suites from a list which can be launched in the Selenium IDE to be run. For the test suite article to be available from the select list in the special page, it must be categorised into Category:Selenium. Tests can also be set to run on a schedule along with actions to be taken should any of them fail, this is described in more details below.
Taking testing to a further level of organisation would allow us to package up wiki-based organisational systems which exactly specify their functionality and exhibit corresponding tests for every item specified. This would allow us to move into a higher level of service automation and would form the basis of a more robust SLA-based service architecture.
Contents
Installation
The extension consists of a single script which should be in a directory called Selenium in your extensions directory. The extension is then included from LocalSettings as usual: <php> include("$IP/extensions/Selenium/Selenium.php"); </php> The selenium core needs to also be installed on the system somewhere, and the default location is in a directory called selenium-core along side the extension script in the Selenium directory. The Selenium core can be downloaded from here as a zip file, or can be maintained via SVN as follows:
cd Selenium svn co https://svn.openqa.org/svn/selenium-core/trunk/src/main/resources selenium-core
The Selenium IDE
Selenium has both a server and a client side, the client side is Selenium IDE which is a Firefox browser plugin. Selenium IDE is an integrated development environment for Selenium tests. It is implemented as a Firefox extension, and allows you to record, edit, and debug tests. Selenium IDE includes the entire Selenium Core, allowing you to easily and quickly record and play back tests in the actual client browser environment that they will run.
Selenium IDE is not only recording tool: it is a complete IDE. You can choose to use its recording capability, or you may edit your scripts by hand. With autocomplete support and the ability to move commands around quickly, Selenium IDE is the ideal environment for creating Selenium tests no matter what style of tests you prefer.
It is also recommended to install the firefox extension XPath Checker which provides a right click option View XPath over any part of a webpage which provides the XPath expression.
Creating tests
The easiest way to create tests is to record them in firefox after enabling the Selenium IDE from the tools pull down menu. As you fill in fields within forms, hitting tab will guarantee that the value is recorded by the Selenium IDE. If a recorded field is incorrect, just edit the html table source removing the table row error and continue the recording process. After recording a test, the source html can be modified and refined by hand. In general long form URLs should be used to minimize ambiguity for portability of tests as some wiki's use Friendly URLs.
Running tests on a schedule
One of the main requirements that we have for the Selenium testing environment is to have it report to us whenever any functionality breaks from the development we do on our extensions. For example we like to schedule a suite of tests which cover all the functionality of the SimpleSecurity extension and test all that functionality on all the different MediaWiki code-bases we have installed in our wikia.
Test suite articles
|
If each test was a separate article, then they can be categorized etc. Utilizing existing Selenium IDE output HTML table format would be convenient. Each test article would be a suite of tests where each individual test is in a separate heading section, and then the actual functions composing the test is just a table within its section. This simplifies things by allowing all the tests for a given extension to be in just one article. It is quickest to maintain HTML tables as these can be imported quickly from the Selenium IDE as is. The only redundant tags are the <html>...</html> wrappers.
It appears that the table tags <thead></thead><tbody></tbody> are not currently defined by the wiki parser, see W:Help:HTML_in_wikitext#Permitted_HTML.
- Either use a <selenium> or just add tag-hooks for thead and tbody which return null - probly easiest --Nad 09:19, 7 December 2007 (NZDT)
I guess for simplicity the test environment should utilize the current way selenium works taking you to a URL which shows the selenium frames and the tests it is about to perform where you select the article to test from the Special page.
The extension should come with a test that examines the selenium extension, checks it is installed, what version it is and usage.
Specialpage
The url http:///www.organicdesign.co.nz/Specialpage:Selenium should allow users to select an article that they want to test from a pull down list, when the test is run to selenium the suite html will be created on the fly by selecting the url - http://www.organicdesign.co.nz/Special:Selenium/Test article. As the article is made up of individual tests, the special page would also create the test html dynamically, e.g. <bash>
- Test article
http://www.organicdesign.co.nz/Special:Selenium/Test_article http://www.organicdesign.co.nz/Special:Selenium/Test1 http://www.organicdesign.co.nz/Special:Selenium/Test2
http://www.organicdesign.co.nz/Special:Selenium/Test_article#third_selenium_tag_header
http://www.organicdesign.co.nz/Special:Selenium/Test4 ...
- Extension location
$IP/extensions/Selenium/Selenium.php
- Selenium core location
$IP/extensions/Selenium/selenium-core-0 $IP/extensions/Selenium/selenium -> $IP/extensions/Selenium/selenium-core-0
</bash>
The expected html of a selenium test is currently a test Suite file which links to many tests. An example structure is
- Test suite wrapper file
<html><head><meta content="text/html; charset=ISO-8859-1" http-equiv="content-type"> <title>Test Suite</title> </head><body><table cellpadding="1" cellspacing="1" border="1"><tbody> <tr><td><b>Flouzo Test Suite</b></td></tr> <tr><td><a href="/selenium/Wikitests/Logout.html">Logout.html</a></td></tr> </tbody></table></body></html>
- A selenium test - /selenium/Wikitests/Logout.htm
<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> <title>New Test</title> </head> <body> <table cellpadding="1" cellspacing="1" border="1"> <thead> <tr><td rowspan="1" colspan="3">New Test</td></tr> </thead><tbody> <tr> <td>open</td> <td>/mediawiki-1.11.0/index.php?title=Special:Userlogout</td> <td></td> </tr> </tbody></table> </body> </html>
Test article naming
The name of the article should be in the "test", "selenium" or maybe "spec" namespace and have the same title as the extension it is designed to test. This way, the Selenium extension can offer an action button to the tests if there is one corresponding to the current page. Whatever the name, there should be a unique namespace, then these articles could be transcluded into a master article allowing template variable embedding, however we want to suppress certain wikitext though.
Specification
Each section is a test for some specific functionality which should be described clearly in plain english below the heading before the bullet list/HTML table of test functions. This way each extension's test article becomes a complete specification of the extended functionality which is both human readable and machine testable. For this reason the "spec" namespace may be more appropriate than "test".
Test Execution
The special page allows all the known tests to be listed, and specific tests to be executed by launching the Test Runner. The test runner requires the tests to be in a simple HTML table format, and requires them to be available from a URL relative to the Selenium code. The special page must therefore be able to accept for its parameter the name of the tests to run and be able to return them in the appropriate format for Selenium.
Test Automation
We need to include as part of the extension (maybe in an additional PERL script) the ability to run the tests automatically from crontab and report to us via talk page or a test log article any problems which crop up due to code or environment changes.
- Automation of the selenium-core suite can be done by adding the query string parameter &auto=true (see details). An additional url can be provided &resultsUrl = [relative path] which creates an html output file. See Continuous integration section for details. By default the querystring parameter resultsUrl posts results to /postResults, which is in the path
$IP/extensions/Selenium/selenium-core/core.
By creating a quick script called results.php in a temporary directory in $IP/extensions/Selenium/ with contents;
and redirecting resultsUrl=../../tmp/results.php.
an example of the posted parameters sent are as follows;
It appears that the important posted parameters are;
- result => passed
- totalTime => 5
- numTestPasses => 1
- numTestFailures => 0
- numCommandPasses => 0
- numCommandFailures => 0
- numCommandErrors => 0
I think its probly best to keep all its files in its own dir (ie extensions/selenium) same as other extensions like searchlog which generate other files it should also be able to be executed from shell and run selected tests eg from cron etc.
How does this complement SubVersion (SVN)?
I guess SVN testing can only check server side PHP for errors in core (and extension) MediaWiki functionality, whereas Selenium is testing the output environment from the end user clients perspective.
See also
- MediaWiki development policy
- Selenium
- Extension:CategoryHook.php -Categorization of <selenium> tags
- extractSections() in Parser.php
MediaWiki:Common.css
The following CSS rules can be added to enhance table rendering;
/** CSS placed here will be applied to all skins */
div.selenium table {
border:1px solid #CCCCCC;
border-collapse:collapse;
font-family:Verdana,Arial,sans-serif;
font-size:12px;
}
Cross site scripting
- Pseudo code
