Extension talk:Selenium.php

This talk page pertains specifically to the development of this extension. For more general discussion about bugs and usage etc, please refer to the mediawiki.org talk page at MW:Extension talk:Selenium

Selenium Core is a test tool for web applications. Selenium Core tests run directly in a browser, just as real users do. And they run in Internet Explorer, Mozilla and Firefox on Windows, Linux, and Macintosh. No other test tool covers such a wide array of platforms. The idea of this extension is to bundle the Selenium Core functionality up as a MediaWiki extension to provide a simple means of incorporating tests along with extensions so that all their aspects of functionality can be tested for any given wiki installation, and failure reports be delivered automatically whenever functionality breaks.

Selenium tests are defined in an HTML table format which can be initially created in the Selenium IDE which runs in your browser. The process of creating and running tests in the Selenium IDE is very similar to recording and playing macros in Open Office or other popular office suite applications. A recorded test can be further refined in a text editor if necessary.

In the MediaWiki environment, test-suites don't need to be a specific format themselves, they are just articles which contain more than one test. For tests or parts of tests which are re-used across a number of suites, templates can be transcluded the to collate tests into a test-suite article. See the Example Selenium test suite for details.

The Selenium special page is then used to select test suites from a list which can be launched in the Selenium IDE to be run. Articles containing tests are automatically categorised into Category:Selenium which is how the special page generates the selection list. Tests can also be set to run on a schedule along with actions to be taken should any of them fail, this is described in more details below.

Taking testing to a further level of organisation would allow us to package up wiki-based organisational systems which exactly specify their functionality and exhibit corresponding tests for every item specified. This would allow us to move into a higher level of service automation and would form the basis of a more robust SLA-based service architecture.

Installation

The extension consists of a directory called Selenium containing a few four php scripts. Copy the directory into your extensions directory and include the main script from LocalSettings; <php> include("$IP/extensions/Selenium/Selenium.php"); </php> The selenium core environment is an independent project which needs to be downloaded and placed into the extensions/Selenium directory. The Selenium core can be downloaded from here as a zip file, it should be unpacked and it's name changed to selenium-core. Alternatively the extensions/Selenium/selenium-core directory can be maintained directly from their subversion repository as follows:

cd path-to-wiki/extensions/Selenium
svn co https://svn.openqa.org/svn/selenium-core/trunk/src/main/resources selenium-core

Selenium server installation (Debian Linux)

The Selenium Remote Control (SRC) is used to allow scheduling of automated test execution. The SRC uses a java service called the Selenium Server to launch and control browser instances to run the Selenium tests in and report results back. The SRC can be downloaded from here, it comes as a zip file containing directories for the Selenium Server and for each of the programming languages the remote control API is available for. I doesn't matter where the Selenium Server is unpacked to, but I prefer to take only the Server folder, rename it to selenium-server put it in the Selenium extension directory along with selenium-core.

It's in this selenium-server directory that the main java server program resides, it's called selenium-server.jar. To be able to run it, youll need to install the Java framework if you don't already have it, which you can do on Debian or Ubuntu with the following package install:

apt-get install sun-java5-jdk

The next technical difficulty to overcome is that the Selenium Server actually runs real browser instances which open a window in the desktop, but the problem is that we need this to run on our remote web server which has no screen or desktop environment installed. So to get around this we must first install a minimal set of X-server components and a browser. On Debian servers this can be done with the following package installations. We've chosen firefox as our browser, but you can use any browser which uses a standard browser-DOM such as Opera, Konquerer or you could even use Internet Explorer through Wine.

apt-get install xvfb xserver-xorg xfs xfonts-base firefox

To test that the browser can run, we'll first create a virtual screen, launch the browser in it and then take a snapshot to see what it looks like. Best to do the following from a root terminal.

nohup Xvfb :1 -screen 0 1024x768x24 2>&1 > /var/log/xvfb.log &
DISPLAY=:1 firefox
DISPLAY=:1 import -window root screenshot.png

If all is working correctly, you'll see a normal browser window in the generated file screenshot.png. If not, then you'll need to begin tracking down the problem starting with the information logged to /var/log/xvfb.log.

The Selenium IDE

Selenium IDE

Selenium has both a server and a client side, the client side is Selenium IDE which is a Firefox browser plugin. Selenium IDE is an integrated development environment for Selenium tests. It is implemented as a Firefox extension, and allows you to record, edit, and debug tests. Selenium IDE includes the entire Selenium Core, allowing you to easily and quickly record and play back tests in the actual client browser environment that they will run.

Selenium IDE is not only recording tool: it is a complete IDE. You can choose to use its recording capability, or you may edit your scripts by hand. With auto-complete support and the ability to move commands around quickly, Selenium IDE is the ideal environment for creating Selenium tests no matter what style of tests you prefer.

It is also recommended to install the firefox extension XPath Checker, which provides a right click option View XPath over any part of a web page which provides the XPath expression.

Creating tests

The easiest way to create tests is to record them in firefox after enabling the Selenium IDE from the tools pull down menu. As you populate form fields with text, hitting tab will guarantee that the value is recorded by the Selenium IDE. If a recorded field is incorrect, just edit the html table source removing the table row error and continue the recording process. After recording a test, the source html can be modified and refined by hand. In general long form URLs should be used to minimise ambiguity for portability of tests as some wiki's use Friendly URLs.

Running tests on a schedule

One of the main requirements that we have for the Selenium testing environment is to have it report to us whenever any functionality breaks from the development we do on our extensions. For example we like to schedule a suite of tests which cover all the functionality of the SimpleSecurity extension, and test all that functionality on all the different MediaWiki code-bases we have installed in our wikia.

The extension uses a single global variable to contain the information about what test should be periodically run. The $wgSeleniumSchedule global contains a list of times (items with numeric keys) each in the format of hh:mm that determine at what times each day the tests should be run. The same global also contains key:value pairs. These determine what test suites are currently scheduled to run. Each key:value pair has a test-suite name for a key, and an array of domains that the test should be run on. This allows wikis that do not have the Selenium extension installed to be tested. However, the Selenium core environment must still exist in the extensions directory of the domain being tested. here's an example of filling in the $wgSeleniumSchedule global. <php> $egSeleniumSchdeule = array( "10:30", "14:00", "23:55", "Example Selenium test suite" => array($wgServer, 'http://od.localhost') ); </php> All the information about test run and failures are appended to $wgSeleniumLog which is set by default to MediaWiki:Selenium log. It's been done as an article edit so that the failure shows up in the recent changes, and so that any interested parties can watch the article and thereby be notified of failures over email if they choose.

Selenium Remote Control (SRC)

Running the tests automatically on a schedule is not quite as simple as initially expected. The Selenium IDE is a complex JavaScript application and so must run within a full browser DOM. It cannot be run from a simple perl module user agent like lib-www. But this is the job of the Selenium Remote Control enrionment which is designed to instantiate a local browser, run the tests in it and capture the results. It has interfaces to the most popular programming languages so that the functionality of the Selenium IDE can be placed under automated control.

The Selenium Remote Control is a daemon written in Java that launches browser instances and runs the Selenium IDE in them. One of the options allows it to launch a browser and then run the specified test suite for the given domain.

java -jar selenium-server.jar "*custom /usr/bin/firefox" domain suite.html results.txt

There are a number of difficulties with this which require workarounds:

• The Selenium core is expected to reside at a default path of /selenium-server/core
• The tests are expected to exist in /selenium-server/tests
• The results are POSTed back to '/selenium-server/postResults
• The script doesn't close after running the test and -timeout nnnn' option must be added
• The results.txt file is not updated (probably because it doesn't close properly)
• The firefox browser asks about starting a new session or restoring which the SRC can't answer inassisted

I've been able to get the system working by creating a symlink to ensure that the path /selenium-server exists and points to the Selenium core in /var/www/extensions/Selenium/selenium-core. And I created a rewrite rule to map the non-existent postResults script onto postResults.php which appends the relavent passes and fails to the results.txt file.

A further difficulty is that a minimal installation of a browser must be set up on the server which does not require a screen or the massive XWindow layer to work (this example and this one may help with that).

Test suite articles

The tests produced by the Selenium IDE are text encoded into a specific HTML format. Each test is in the form of an HTML table, but unfortunately the table's generated HTML syntax contains some tags which are not allowed in wikitext. Here's an example of the logout test: <xml>

Logout
open	/wiki/index.php?title=Main_page
clickAndWait	link=Log out
verifyHtmlSource	glob:*You are now logged out.*

</xml> As you can see, the test comtain a number of illegal tags, and the whole test is surrounded by an html tag which is reserved by MediaWiki, so we need to surround Selenium tests in selenium tags so that we can remove these unwanted tags before parsing the content normally. We still want to parse the content because we need transclusion and parser functions to work as usual, and also we'd like the table to render properly and we don't want to send the content back to the client as raw HTML because of XSS concerns.

So the test above would be changed slightly to the following - the whole test is wrapped in the selenium tag and the path to the Main Page article is done using the localurl parser function. <xml> <selenium>

   
        
            
            
        
        
            

                

                    
Logout
                
                

                    

                        
open
                        
{(localurl:Main Page}}
                        

                    
                    

                        
clickAndWait
                        
link=Log out
                        

                    
                    

                        
verifyHtmlSource
                        
glob:*You are now logged out.*
                        

                    
                
            
        
    

</selenium> </xml>

Table style

The body tag replaces itself with a div of class "selenium" allowing the tests to have nicer styling applied to them from CSS, for example, we added the following rules to our own MediaWiki:Common.css. <css> .selenium table {

   border: 2px solid #dde;
   background: white;
   font-family: Verdana, Arial, sans-serif;
   font-size: 12px;
   font-weight: bold;
   text-align: center;

} .selenium td {

   border: none;
   padding: 4px;
   margin: 0;

} .selenium tr {

   background: #dde;

} .selenium tr + tr {

   font-weight: normal;
   background: #eef;
   text-align: left;

} </css>

Test Automation

We need to include as part of the extension (maybe in an additional PERL script) the ability to run the tests automatically from crontab and report to us via talk page or a test log article any problems which crop up due to code or environment changes.

Automation of the selenium-core suite can be done by adding the query string parameter &auto=true (see details). An additional url can be provided &resultsUrl = [relative path] which creates an html output file. See Continuous integration section for details. By default the querystring parameter resultsUrl posts results to /postResults, which is in the path $IP/extensions/Selenium/selenium-core/core.

By creating a quick script called results.php in a temporary directory in $IP/extensions/Selenium/ with contents;

<?php

foreach($_POST as $key=>$value) {
print "$key => $value <br />";
}

?>

and redirecting resultsUrl=../../tmp/results.php.

An example of the posted parameters sent are as follows;

selenium_version => 0.8.3 selenium_revision => 1879 result => passed totalTime => 5 numTestPasses => 1 numTestFailures => 0 numCommandPasses => 0 numCommandFailures => 0 numCommandErrors => 0 testTable_1 => <thead> </thead><tbody> </tbody> New Test open /wiki/index.php/Main_Page clickAndWait link=Special pages clickAndWait link=Categories clickAndWait link=Selenium numTestTotal => 1 suite => <tbody> </tbody> Foo <a href="/wiki/index.php?title=Special:Selenium&suite=Foo&test=Test">Test</a> log => info: Starting test /wiki/index.php info: Executing:

It appears that the important posted parameters are;

• result => passed
• totalTime => 5
• numTestPasses => 1
• numTestFailures => 0
• numCommandPasses => 0
• numCommandFailures => 0
• numCommandErrors => 0

I think its probly best to keep all its files in its own dir (ie extensions/selenium) same as other extensions like searchlog which generate other files it should also be able to be executed from shell and run selected tests eg from cron etc.

See also

• MediaWiki development policy
• Selenium