Elastica and CirrusSearch extensions

From Organic Design wiki
Revision as of 16:07, 24 September 2019 by Nad (talk | contribs) (CirrusSearch maintenance scripts)

ElasticSearch is a distributed, RESTful search and analytics engine capable of addressing a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data so you can discover the expected and uncover the unexpected. PHP projects can use the Elastica library to integrate with Elastic Search in an efficient, scalable and well-structured manner.

The Elastica MediaWiki extension is used to integrate MediaWiki with a local ElasticSearch back-end suing the Elastica library and integrating tightly with MediaWiki's object structure. The CirrusSearch extension uses the interface provided by the Elastica extension to provide MediaWiki with an alternative to the default SQL-based search engine. All the Wikimedia projects including Wikipedia use Cirrus Search and Elastica.

Learning the Elastic document structure used by MediaWiki, and the syntax for integrating with it is not well documented, so this page provides some examples of commonly needed processes. This article assumes that all the components are already installed and searching the wiki using the CirrusSearch engine is already working.

CirrusSearch maintenance scripts

There are some maintenance scripts in the maintenance directory of the CirrusSearch extension which are used to initialise or rebuild the Elastic documents that compose the text search index of your wikis content.

To clear the entire index, deleting all Elastic docuemnts for the wiki:

php updateSearchIndexConfig.php --startOver

Rebuilding the indexes takes to passes, one to build from the rendered content and one to index links from the source code. If you have a large wiki with thousands of pages, it's best to include the queue and maxJobs parameters otherwise the process may not index all the articles.

php forceSearchIndex.php --skipLinks --indexOnSkip --queue --maxJobs=100
php forceSearchIndex.php --skipParse --queue --maxJobs=100

Accessing Elastic documents using Curl

Performing a basic search query:

curl localhost:9200/_search?q=searchterm&pretty


Get info on a specific document including all its fields:

curl 'localhost:9200/DBname-DBprefix_general/page/PAGEID?pretty'


Delete a field from a document:

curl -H "Content-Type: application/json" -XPOST 'localhost:9200/DBname-DBprefix_general/page/PAGEID/_update' -d '{"script" : "ctx._source.remove(\"FIELDNAME\")"}'


Create or update a field:

curl  -H "Content-Type: application/json" -XPOST 'localhost:9200/DBname-DBprefix_general/page/PAGEID/_update' -d '{"doc":{"FIELDNAME":"VALUE"}}'


To create or update bulk fields, first create a file called data containing information to update in the following format:

{"update":{"_id":"PAGEID1"}}
{"doc":{"FIELDNAME1":"VALUE1"}}
{"update":{"_id":"PAGEID2"}}
{"doc":{"FIELDNAME2":"VALUE2"}}
{"update":{"_id":"PAGEID3"}}
{"doc":{"FIELDNAME3":"VALUE3"}}

Then perform the bulk request as follows:

curl -H "Content-Type: application/json" -XPOST 'localhost:9200/DBname-DBprefix_general/page/_bulk' --data-binary "@data"

Accessing Elastic documents from PHP

Modifying the default Elastic document structure

See also