Difference between revisions of "Elastica and CirrusSearch extensions"

From Organic Design wiki
m (Accessing Elastic documents using Curl)
(Accessing Elastic documents from PHP)
Line 62: Line 62:
  
 
== Accessing Elastic documents from PHP ==
 
== Accessing Elastic documents from PHP ==
 +
Here's how we can do a bulk query using the Elastica class:
 +
<source lang="php">
 +
$search = MediaWiki\MediaWikiServices::getInstance()->getSearchEngineFactory()->create( 'CirrusSearch' );
 +
$conn = $search->getConnection();
 +
$index = '{!DBname!}-{!DBprefix!}_general';
 +
$bulk = new \Elastica\Bulk( $conn->getClient() );
 +
$bulk->setType( "$index/page" );
 +
$doc = new \Elastica\Document( {!PageID!}, ['{!FIELD!}' => '{!VALUE!}'], 'page', $index );
 +
$bulk->addData( $doc, 'update' ); # call this to add each doc
 +
$res = $bulk->send();
 +
</source>
 +
 +
A better way, however, is to use the CirrusSearch's ''DataSender'' class because all the graceful exception handling is done:
 +
<source lang="php">
 +
$index = '{!DBname!}-{!DBprefix!}_general';
 +
$docs = [
 +
    new \Elastica\Document( {!PAGEID1!}, ['{!FIELD1!}' => '{!VALUE1!}'], 'page', $index ),
 +
    new \Elastica\Document( {!PAGEID2!}, ['{!FIELD2!}' => '{!VALUE2!}'], 'page', $index )
 +
];
 +
$search = MediaWiki\MediaWikiServices::getInstance()->getSearchEngineFactory()->create( 'CirrusSearch' );
 +
$conn = $search->getConnection();
 +
$conf = MediaWiki\MediaWikiServices::getInstance()->getConfigFactory()->makeConfig( 'CirrusSearch' );
 +
$sender = new \CirrusSearch\DataSender( $conn, $conf );
 +
$sender->sendData( 'page', $docs );
 +
</source>
  
 
== Modifying the default Elastic document structure ==
 
== Modifying the default Elastic document structure ==

Revision as of 16:20, 24 September 2019

ElasticSearch is a distributed, RESTful search and analytics engine capable of addressing a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data so you can discover the expected and uncover the unexpected. PHP projects can use the Elastica library to integrate with Elastic Search in an efficient, scalable and well-structured manner.

The Elastica MediaWiki extension is used to integrate MediaWiki with a local ElasticSearch back-end suing the Elastica library and integrating tightly with MediaWiki's object structure. The CirrusSearch extension uses the interface provided by the Elastica extension to provide MediaWiki with an alternative to the default SQL-based search engine. All the Wikimedia projects including Wikipedia use Cirrus Search and Elastica.

Learning the Elastic document structure used by MediaWiki, and the syntax for integrating with it is not well documented, so this page provides some examples of commonly needed processes. This article assumes that all the components are already installed and searching the wiki using the CirrusSearch engine is already working.

CirrusSearch maintenance scripts

There are some maintenance scripts in the maintenance directory of the CirrusSearch extension which are used to initialise or rebuild the Elastic documents that compose the text search index of your wikis content.

To clear the entire index, deleting all Elastic docuemnts for the wiki:

php updateSearchIndexConfig.php --startOver

Rebuilding the indexes takes to passes, one to build from the rendered content and one to index links from the source code. If you have a large wiki with thousands of pages, it's best to include the queue and maxJobs parameters otherwise the process may not index all the articles.

php forceSearchIndex.php --skipLinks --indexOnSkip --queue --maxJobs=100
php forceSearchIndex.php --skipParse --queue --maxJobs=100

Accessing Elastic documents using Curl

Communicating with ElasticSearch is done over a standard HTTP connection which is on port 9200 by default. Here's some simple queries that can be done by connecting directly to the service on the command line using Curl.

Performing a basic search query:

curl 'localhost:9200/_search?q=searchterm&pretty'


Get info on a specific document including all its fields:

curl 'localhost:9200/DBname-DBprefix_general/page/PAGEID?pretty'


Delete a field from a document:

curl -H "Content-Type: application/json" -XPOST 'localhost:9200/DBname-DBprefix_general/page/PAGEID/_update' -d '{"script" : "ctx._source.remove(\"FIELDNAME\")"}'


Create or update a field:

curl  -H "Content-Type: application/json" -XPOST 'localhost:9200/DBname-DBprefix_general/page/PAGEID/_update' -d '{"doc":{"FIELDNAME":"VALUE"}}'


To create or update bulk fields, first create a file called data containing information to update in the following format:

{"update":{"_id":"PAGEID1"}}
{"doc":{"FIELDNAME1":"VALUE1"}}
{"update":{"_id":"PAGEID2"}}
{"doc":{"FIELDNAME2":"VALUE2"}}
{"update":{"_id":"PAGEID3"}}
{"doc":{"FIELDNAME3":"VALUE3"}}

Then perform the bulk request as follows:

curl -H "Content-Type: application/json" -XPOST 'localhost:9200/DBname-DBprefix_general/page/_bulk' --data-binary "@data"

Accessing Elastic documents from PHP

Here's how we can do a bulk query using the Elastica class:

$search = MediaWiki\MediaWikiServices::getInstance()->getSearchEngineFactory()->create( 'CirrusSearch' );
$conn = $search->getConnection();
$index = 'DBname-DBprefix_general';
$bulk = new \Elastica\Bulk( $conn->getClient() );
$bulk->setType( "$index/page" );
$doc = new \Elastica\Document( PageID, ['FIELD' => 'VALUE'], 'page', $index );
$bulk->addData( $doc, 'update' ); # call this to add each doc
$res = $bulk->send();

A better way, however, is to use the CirrusSearch's DataSender class because all the graceful exception handling is done:

$index = 'DBname-DBprefix_general';
$docs = [
    new \Elastica\Document( PAGEID1, ['FIELD1' => 'VALUE1'], 'page', $index ),
    new \Elastica\Document( PAGEID2, ['FIELD2' => 'VALUE2'], 'page', $index )
];
$search = MediaWiki\MediaWikiServices::getInstance()->getSearchEngineFactory()->create( 'CirrusSearch' );
$conn = $search->getConnection();
$conf = MediaWiki\MediaWikiServices::getInstance()->getConfigFactory()->makeConfig( 'CirrusSearch' );
$sender = new \CirrusSearch\DataSender( $conn, $conf );
$sender->sendData( 'page', $docs );

Modifying the default Elastic document structure

See also