Recoll indexing can be performed along two different modes:

The choice between the two methods is mostly a matter of preference, and they can be combined by setting up multiple indexes (ie: use periodic indexing on a big documentation directory, and real time indexing on a small home directory). Monitoring a big file system tree can consume significant system resources.

The choice of method and the parameters used can be configured from the recoll GUI: Preferences → Indexing schedule

The File menu also has entries to start or stop the current indexing operation. Stopping indexing is performed by killing the recollindex process, which will checkpoint its state and exit. A later restart of indexing will mostly resume from where things stopped (the file tree walk has to be restarted from the beginning).

When the real time indexer is running, only a stop operation is available from the menu. When no indexing is running, you have a choice of updating the index or rebuilding it (the first choice only processes changed files, the second one zeroes the index before starting so that all files are processed).

The parameters describing what is to be indexed and local preferences are defined in text files contained in a configuration directory.

All parameters have defaults, defined in system-wide files.

Without further configuration, Recoll will index all appropriate files from your home directory, with a reasonable set of defaults.

A default personal configuration directory ($HOME/.recoll/) is created when a Recoll program is first executed. It is possible to create other configuration directories, and use them by setting the RECOLL_CONFDIR environment variable, or giving the -c option to any of the Recoll commands.

In some cases, it may be interesting to index different areas of the file system to separate databases. You can do this by using multiple configuration directories, each indexing a file system area to a specific database. Typically, this would be done to separate personal and shared indexes, or to take advantage of the organization of your data to improve search precision.

The generated indexes can be queried concurrently in a transparent manner.

For index generation, multiple configurations are totally independant from each other. When multiple indexes need to be used for a single search, some parameters should be consistent among the configurations.

Recoll knows about quite a few different document types. The parameters for document types recognition and processing are set in configuration files.

Most file types, like HTML or word processing files, only hold one document. Some file types, like email folders or zip archives, can hold many individually indexed documents, which may themselves be compound ones. Such hierarchies can go quite deep, and Recoll can process, for example, a LibreOffice document stored as an attachment to an email message inside an email folder archived in a zip file...

Recoll indexing processes plain text, HTML, OpenDocument (Open/LibreOffice), email formats, and a few others internally.

Other file types (ie: postscript, pdf, ms-word, rtf ...) need external applications for preprocessing. The list is in the installation section. After every indexing operation, Recoll updates a list of commands that would be needed for indexing existing files types. This list can be displayed by selecting the menu option File → Show Missing Helpers in the recoll GUI. It is stored in the missing text file inside the configuration directory.

By default, Recoll will try to index any file type that it has a way to read. This is sometimes not desirable, and there are ways to either exclude some types, or on the contrary to define a positive list of types to be indexed. In the latter case, any type not in the list will be ignored.

Excluding types can be done by adding wildcard name patterns to the skippedNames list, which can be done from the GUI Index configuration menu. For versions 1.20 and later, you can alternatively set the excludedmimetypes list in the configuration file. This can be redefined for subdirectories.

You can also define an exclusive list of MIME types to be indexed (no others will be indexed), by settting the indexedmimetypes configuration variable. Example:

indexedmimetypes = text/html application/pdf
          

It is possible to redefine this parameter for subdirectories. Example:

[/path/to/my/dir]
indexedmimetypes = application/pdf
          

(When using sections like this, don't forget that they remain in effect until the end of the file or another section indicator).

excludedmimetypes or indexedmimetypes, can be set either by editing the main configuration file (recoll.conf), or from the GUI index configuration tool.

Indexing may fail for some documents, for a number of reasons: a helper program may be missing, the document may be corrupt, we may fail to uncompress a file because no file system space is available, etc.

Recoll versions prior to 1.21 always retried to index files which had previously caused an error. This guaranteed that anything that may have become indexable (for example because a helper had been installed) would be indexed. However this was bad for performance because some indexing failures may be quite costly (for example failing to uncompress a big file because of insufficient disk space).

The indexer in Recoll versions 1.21 and later does not retry failed file by default. Retrying will only occur if an explicit option (-k) is set on the recollindex command line, or if a script executed when recollindex starts up says so. The script is defined by a configuration variable (checkneedretryindexscript), and makes a rather lame attempt at deciding if a helper command may have been installed, by checking if any of the common bin directories have changed.

In the rare case where the index becomes corrupted (which can signal itself by weird search results or crashes), the index files need to be erased before restarting a clean indexing pass. Just delete the xapiandb directory (see next section), or, alternatively, start the next recollindex with the -z option, which will reset the database before indexing.

Xapian versions usually support several formats for index storage. A given major Xapian version will have a current format, used to create new indexes, and will also support the format from the previous major version.

Xapian will not convert automatically an existing index from the older format to the newer one. If you want to upgrade to the new format, or if a very old index needs to be converted because its format is not supported any more, you will have to explicitly delete the old index, then run a normal indexing process.

Using the -z option to recollindex is not sufficient to change the format, you will have to delete all files inside the index directory (typically ~/.recoll/xapiandb) before starting the indexing.

The Recoll index does not hold copies of the indexed documents. But it does hold enough data to allow for an almost complete reconstruction. If confidential data is indexed, access to the database directory should be restricted.

Recoll will create the configuration directory with a mode of 0700 (access by owner only). As the index data directory is by default a sub-directory of the configuration directory, this should result in appropriate protection.

If you use another setup, you should think of the kind of protection you need for your index, set the directory and files access modes appropriately, and also maybe adjust the umask used during index updates.

Multiple Recoll indexes can be created by using several configuration directories which are usually set to index different areas of the file system. A specific index can be selected for updating or searching, using the RECOLL_CONFDIR environment variable or the -c option to recoll and recollindex.

When working with the recoll index configuration GUI, the configuration directory for which parameters are modified is the one which was selected by RECOLL_CONFDIR or the -c parameter, and there is no way to switch configurations within the GUI.

Additional configuration directory (beyond ~/.recoll) must be created by hand (mkdir or such), the GUI will not do it. This is to avoid mistakenly creating additional directories when an argument is mistyped.

A typical usage scenario for the multiple index feature would be for a system administrator to set up a central index for shared data, that you choose to search or not in addition to your personal data. Of course, there are other possibilities. There are many cases where you know the subset of files that should be searched, and where narrowing the search can improve the results. You can achieve approximately the same effect with the directory filter in advanced search, but multiple indexes will have much better performance and may be worth the trouble.

A recollindex program instance can only update one specific index.

The main index (defined by RECOLL_CONFDIR or -c) is always active. If this is undesirable, you can set up your base configuration to index an empty directory.

The different search interfaces (GUI, command line, ...) have different methods to define the set of indexes to be used, see the appropriate section.

If a set of multiple indexes are to be used together for searches, some configuration parameters must be consistent among the set. These are parameters which need to be the same when indexing and searching. As the parameters come from the main configuration when searching, they need to be compatible with what was set when creating the other indexes (which came from their respective configuration directories).

Most importantly, all indexes to be queried concurrently must have the same option concerning character case and diacritics stripping, but there are other constraints. Most of the relevant parameters are described in the linked section.

As of Recoll version 1.18 you have a choice of building an index with terms stripped of character case and diacritics, or one with raw terms. For a source term of Résumé, the former will store resume, the latter Résumé.

Each type of index allows performing searches insensitive to case and diacritics: with a raw index, the user entry will be expanded to match all case and diacritics variations present in the index. With a stripped index, the search term will be stripped before searching.

A raw index allows for another possibility which a stripped index cannot offer: using case and diacritics to discriminate between terms, returning different results when searching for US and us or resume and résumé. Read the section about search case and diacritics sensitivity for more details.

The type of index to be created is controlled by the indexStripChars configuration variable which can only be changed by editing the configuration file. Any change implies an index reset (not automated by Recoll), and all indexes in a search must be set in the same way (again, not checked by Recoll).

If the indexStripChars is not set, Recoll 1.18 creates a stripped index by default, for compatibility with previous versions.

As a cost for added capability, a raw index will be slightly bigger than a stripped one (around 10%). Also, searches will be more complex, so probably slightly slower, and the feature is still young, so that a certain amount of weirdness cannot be excluded.

One of the most adverse consequence of using a raw index is that some phrase and proximity searches may become impossible: because each term needs to be expanded, and all combinations searched for, the multiplicative expansion may become unmanageable.

The Recoll indexing process recollindex can use multiple threads to speed up indexing on multiprocessor systems. The work done to index files is divided in several stages and some of the stages can be executed by multiple threads. The stages are:

You can also read a longer document about the transformation of Recoll indexing to multithreading.

The threads configuration is controlled by two configuration file parameters.

The following example would use three queues (of depth 2), and 4 threads for converting source documents, 2 for processing their text, and one to update the index. This was tested to be the best configuration on the test system (quadri-processor with multiple disks).

thrQSizes = 2 2 2
thrTCounts =  4 2 1

The following example would use a single queue, and the complete processing for each document would be performed by a single thread (several documents will still be processed in parallel in most cases). The threads will use mutual exclusion when entering the index update stage. In practise the performance would be close to the precedent case in general, but worse in certain cases (e.g. a Zip archive would be performed purely sequentially), so the previous approach is preferred. YMMV... The 2 last values for thrTCounts are ignored.

thrQSizes = 2 -1 -1
thrTCounts =  6 1 1

The following example would disable multithreading. Indexing will be performed by a single thread.

thrQSizes = -1 -1 -1

Most parameters for a given index configuration can be set from a recoll GUI running on this configuration (either as default, or by setting RECOLL_CONFDIR or the -c option.)

The interface is started from the Preferences → Index Configuration menu entry. It is divided in four tabs, Global parameters, Local parameters, Web history (which is explained in the next section) and Search parameters.

The Global parameters tab allows setting global variables, like the lists of top directories, skipped paths, or stemming languages.

The Local parameters tab allows setting variables that can be redefined for subdirectories. This second tab has an initially empty list of customisation directories, to which you can add. The variables are then set for the currently selected directory (or at the top level if the empty line is selected).

The Search parameters section defines parameters which are used at query time, but are global to an index and affect all search tools, not only the GUI.

The meaning for most entries in the interface is self-evident and documented by a ToolTip popup on the text label. For more detail, you will need to refer to the configuration section of this guide.

The configuration tool normally respects the comments and most of the formatting inside the configuration file, so that it is quite possible to use it on hand-edited files, which you might nevertheless want to backup first...

If both tesseract and pdftoppm (generally from the poppler-utils package) are installed, the PDF handler may attempt OCR on PDF files with no text content. This is controlled by the pdfocr configuration variable, which is false by default because OCR is very slow.

The choice of language is very important for successfull OCR. Recoll has currently no way to determine this from the document itself. You can set the language to use through the contents of a .ocrpdflang text file in the same directory as the PDF document, or through the RECOLL_TESSERACT_LANG environment variable, or through the contents of an ocrpdf text file inside the configuration directory. If none of the above are used, Recoll will try to guess the language from the NLS environment.

The rclpdf.py script in Recoll version 1.23.2 and later can extract XMP metadata fields by executing the pdfinfo command (usually found with poppler-utils). This is controlled by the pdfextrameta configuration variable, which specifies which tags to extract and, possibly, how to rename them.

The pdfextrametafix variable can be used to designate a file with Python code to edit the metadata fields (available for Recoll 1.23.3 and later. 1.23.2 has equivalent code inside the handler script). Example:

import sys
import re

class MetaFixer(object):
    def __init__(self):
        pass

    def metafix(self, nm, txt):
        if nm == 'bibtex:pages':
            txt = re.sub(r'--', '-', txt)
        elif nm == 'someothername':
            # do something else
            pass
        elif nm == 'stillanother':
            # etc.
            pass
    
        return txt
    def wrapup(self, metaheaders):
        pass
        

If the 'metafix()' method is defined, it is called for each metadata field. A new MetaFixer object is created for each PDF document (so the object can keep state for, for example, eliminating duplicate values). If the 'wrapup()' method is defined, it is called at the end of XMP fields processing with the whole metadata as parameter, as an array of '(nm, val)' pairs, allowing an alternate approach for editing or adding/deleting fields.

If pdftk is installed, and if the the pdfattach configuration variable is set, the PDF input handler will try to extract PDF attachements for indexing as sub-documents of the PDF file. This is disabled by default, because it slows down PDF indexing a bit even if not one attachment is ever found (PDF attachments are uncommon in my experience).

Indexing is always performed by the recollindex program, which can be started either from the command line or from the File menu in the recoll GUI program. When started from the GUI, the indexing will run on the same configuration recoll was started on. When started from the command line, recollindex will use the RECOLL_CONFDIR variable or accept a -c confdir option to specify a non-default configuration directory.

If the recoll program finds no index when it starts, it will automatically start indexing (except if canceled).

The recollindex indexing process can be interrupted by sending an interrupt (Ctrl-C, SIGINT) or terminate (SIGTERM) signal. Some time may elapse before the process exits, because it needs to properly flush and close the index. This can also be done from the recoll GUI File → Stop Indexing menu entry.

After such an interruption, the index will be somewhat inconsistent because some operations which are normally performed at the end of the indexing pass will have been skipped (for example, the stemming and spelling databases will be inexistant or out of date). You just need to restart indexing at a later time to restore consistency. The indexing will restart at the interruption point (the full file tree will be traversed, but files that were indexed up to the interruption and for which the index is still up to date will not need to be reindexed).

recollindex has a number of other options which are described in its man page. Only a few will be described here.

Option -z will reset the index when starting. This is almost the same as destroying the index files (the nuance is that the Xapian format version will not be changed).

Option -Z will force the update of all documents without resetting the index first. This will not have the "clean start" aspect of -z, but the advantage is that the index will remain available for querying while it is rebuilt, which can be a significant advantage if it is very big (some installations need days for a full index rebuild).

Option -k will force retrying files which previously failed to be indexed, for example because of a missing helper program.

Of special interest also, maybe, are the -i and -f options. -i allows indexing an explicit list of files (given as command line parameters or read on stdin). -f tells recollindex to ignore file selection parameters from the configuration. Together, these options allow building a custom file selection process for some area of the file system, by adding the top directory to the skippedPaths list and using an appropriate file selection method to build the file list to be fed to recollindex -if. Trivial example:

            find . -name indexable.txt -print | recollindex -if
          

recollindex -i will not descend into subdirectories specified as parameters, but just add them as index entries. It is up to the external file selection method to build the complete file list.

The most common way to set up indexing is to have a cron task execute it every night. For example the following crontab entry would do it every day at 3:30AM (supposing recollindex is in your PATH):

30 3 * * * recollindex > /some/tmp/dir/recolltrace 2>&1

Or, using anacron:

1  15  su mylogin -c "recollindex recollindex > /tmp/rcltraceme 2>&1"

As of version 1.17 the Recoll GUI has dialogs to manage crontab entries for recollindex. You can reach them from the Preferences → Indexing Schedule menu. They only work with the good old cron, and do not give access to all features of cron scheduling.

The usual command to edit your crontab is crontab -e (which will usually start the vi editor to edit the file). You may have more sophisticated tools available on your system.

Please be aware that there may be differences between your usual interactive command line environment and the one seen by crontab commands. Especially the PATH variable may be of concern. Please check the crontab manual pages about possible issues.

When using the real time monitor, it may happen that some files need to be indexed, but change so often that they impose an excessive load for the system.

Recoll provides a configuration option to specify the minimum time before which a file, specified by a wildcard pattern, cannot be reindexed. See the mondelaypatterns parameter in the configuration section.

The initial default search mode is Query language. Without special directives, this will look for documents containing all of the search terms (the ones with more terms will get better scores), just like the All terms mode which will ignore such directives. Any term will search for documents where at least one of the terms appear.

The Query Language features are described in a separate section.

All search modes allow wildcards inside terms (*, ?, []). You may want to have a look at the section about wildcards for more information about this.

File name will specifically look for file names. The point of having a separate file name search is that wild card expansion can be performed more efficiently on a small subset of the index (allowing wild cards on the left of terms without excessive penality). Things to know:

You can search for exact phrases (adjacent words in a given order) by enclosing the input inside double quotes. Ex: "virtual reality".

When using a stripped index, character case has no influence on search, except that you can disable stem expansion for any term by capitalizing it. Ie: a search for floor will also normally look for flooring, floored, etc., but a search for Floor will only look for floor, in any character case. Stemming can also be disabled globally in the preferences. When using a raw index, the rules are a bit more complicated.

Recoll remembers the last few searches that you performed. You can use the simple search text entry widget (a combobox) to recall them (click on the thing at the right of the text field). Please note, however, that only the search texts are remembered, not the mode (all/any/file name).

Typing Esc Space while entering a word in the simple search entry will open a window with possible completions for the word. The completions are extracted from the database.

Double-clicking on a word in the result list or a preview window will insert it into the simple search entry field.

You can cut and paste any text into an All terms or Any term search field, punctuation, newlines and all - except for wildcard characters (single ? characters are ok). Recoll will process it and produce a meaningful search. This is what most differentiates this mode from the Query Language mode, where you have to care about the syntax.

You can use the Tools → Advanced search dialog for more complex searches.

After starting a search, a list of results will instantly be displayed in the main list window.

By default, the document list is presented in order of relevance (how well the system estimates that the document matches the query). You can sort the result by ascending or descending date by using the vertical arrows in the toolbar.

Clicking on the Preview link for an entry will open an internal preview window for the document. Further Preview clicks for the same search will open tabs in the existing preview window. You can use Shift+Click to force the creation of another preview window, which may be useful to view the documents side by side. (You can also browse successive results in a single preview window by typing Shift+ArrowUp/Down in the window).

Clicking the Open link will start an external viewer for the document. By default, Recoll lets the desktop choose the appropriate application for most document types (there is a short list of exceptions, see further). If you prefer to completely customize the choice of applications, you can uncheck the Use desktop preferences option in the GUI preferences dialog, and click the Choose editor applications button to adjust the predefined Recoll choices. The tool accepts multiple selections of MIME types (e.g. to set up the editor for the dozens of office file types).

Even when Use desktop preferences is checked, there is a small list of exceptions, for MIME types where the Recoll choice should override the desktop one. These are applications which are well integrated with Recoll, especially evince for viewing PDF and Postscript files because of its support for opening the document at a specific page and passing a search string as an argument. Of course, you can edit the list (in the GUI preferences) if you would prefer to lose the functionality and use the standard desktop tool.

You may also change the choice of applications by editing the mimeview configuration file if you find this more convenient.

Each result entry also has a right-click menu with an Open With entry. This lets you choose an application from the list of those which registered with the desktop for the document MIME type.

The Preview and Open edit links may not be present for all entries, meaning that Recoll has no configured way to preview a given file type (which was indexed by name only), or no configured external editor for the file type. This can sometimes be adjusted simply by tweaking the mimemap and mimeview configuration files (the latter can be modified with the user preferences dialog).

The format of the result list entries is entirely configurable by using the preference dialog to edit an HTML fragment.

You can click on the Query details link at the top of the results page to see the query actually performed, after stem expansion and other processing.

Double-clicking on any word inside the result list or a preview window will insert it into the simple search text.

The result list is divided into pages (the size of which you can change in the preferences). Use the arrow buttons in the toolbar or the links at the bottom of the page to browse the results.

In Recoll 1.15 and newer, the results can be displayed in spreadsheet-like fashion. You can switch to this presentation by clicking the table-like icon in the toolbar (this is a toggle, click again to restore the list).

Clicking on the column headers will allow sorting by the values in the column. You can click again to invert the order, and use the header right-click menu to reset sorting to the default relevance order (you can also use the sort-by-date arrows to do this).

Both the list and the table display the same underlying results. The sort order set from the table is still active if you switch back to the list mode. You can click twice on a date sort arrow to reset it from there.

The header right-click menu allows adding or deleting columns. The columns can be resized, and their order can be changed (by dragging). All the changes are recorded when you quit recoll

Hovering over a table row will update the detail area at the bottom of the window with the corresponding values. You can click the row to freeze the display. The bottom area is equivalent to a result list paragraph, with links for starting a preview or a native application, and an equivalent right-click menu. Typing Esc (the Escape key) will unfreeze the display.

Apart from the Open and Open With operations, which allow starting an application on a result document (or a temporary copy), based on its MIME type, it is also possible to run arbitrary commands on results which are top-level files, using the Run Script entry in the results pop-up menu.

The commands which will appear in the Run Script submenu must be defined by .desktop files inside the scripts subdirectory of the current configuration directory.

Here follows an example of a .desktop file, which could be named for example, ~/.recoll/scripts/myscript.desktop (the exact file name inside the directory is irrelevant):

[Desktop Entry]
Type=Application
Name=MyFirstScript
Exec=/home/me/bin/tryscript %F
MimeType=*/*
      

The Name attribute defines the label which will appear inside the Run Script menu. The Exec attribute defines the program to be run, which does not need to actually be a script, of course. The MimeType attribute is not used, but needs to exist.

The commands defined this way can also be used from links inside the result paragraph.

As an example, it might make sense to write a script which would move the document to the trash and purge it from the Recoll index.

The default format for the result list entries and the detail area of the result table display an icon for each result document. The icon is either a generic one determined from the MIME type, or a thumbnail of the document appearance. Thumbnails are only displayed if found in the standard freedesktop location, where they would typically have been created by a file manager.

Recoll has no capability to create thumbnails. A relatively simple trick is to use the Open parent document/folder entry in the result list popup menu. This should open a file manager window on the containing directory, which should in turn create the thumbnails (depending on your settings). Restarting the search should then display the thumbnails.

There are also some pointers about thumbnail generation on the Recoll wiki.

The preview window opens when you first click a Preview link inside the result list.

Subsequent preview requests for a given search open new tabs in the existing window (except if you hold the Shift key while clicking which will open a new window for side by side viewing).

Starting another search and requesting a preview will create a new preview window. The old one stays open until you close it.

You can close a preview tab by typing Ctrl-W (Ctrl + W) in the window. Closing the last tab for a window will also close the window.

Of course you can also close a preview window by using the window manager button in the top of the frame.

You can display successive or previous documents from the result list inside a preview tab by typing Shift+Down or Shift+Up (Down and Up are the arrow keys).

A right-click menu in the text area allows switching between displaying the main text or the contents of fields associated to the document (ie: author, abtract, etc.). This is especially useful in cases where the term match did not occur in the main text but in one of the fields. In the case of images, you can switch between three displays: the image itself, the image metadata as extracted by exiftool and the fields, which is the metadata stored in the index.

You can print the current preview window contents by typing Ctrl-P (Ctrl + P) in the window text.

Selecting the Tools → Query Fragments menu entry will open a window with radio- and check-buttons which can be used to activate query language fragments for filtering the current query. This can be useful if you have frequent reusable selectors, for example, filtering on alternate directories, or searching just one category of files, not covered by the standard category selectors.

The contents of the window are entirely customizable, and defined by the contents of the fragbuts.xml file inside the configuration directory. The sample file distributed with Recoll (which you should be able to find under /usr/share/recoll/examples/fragbuts.xml), contains an example which filters the results from the WEB history.

Here follows an example:

<?xml version="1.0" encoding="UTF-8"?>

<fragbuts version="1.0">

  <radiobuttons>

    <fragbut>
      <label>Include Web Results</label>
      <frag></frag>
    </fragbut>

    <fragbut>
      <label>Exclude Web Results</label>
      <frag>-rclbes:BGL</frag>
    </fragbut>

    <fragbut>
      <label>Only Web Results</label>
      <frag>rclbes:BGL</frag>
    </fragbut>

  </radiobuttons>

  <buttons>

    <fragbut>
      <label>Year 2010</label>
      <frag>date:2010-01-01/2010-12-31</frag>
    </fragbut>

    <fragbut>
      <label>My Great Directory Only</label>
      <frag>dir:/my/great/directory</frag>
    </fragbut>

  </buttons>
</fragbuts>

Each radiobuttons or buttons section defines a line of checkbuttons or radiobuttons inside the window. Any number of buttons can be selected, but the radiobuttons in a line are exclusive.

Each fragbut section defines the label for a button, and the Query Language fragment which will be added (as an AND filter) before performing the query if the button is active.

This feature is new in Recoll 1.20, and will probably be refined depending on user feedback.

The advanced search dialog helps you build more complex queries without memorizing the search language constructs. It can be opened through the Tools menu or through the main toolbar.

Recoll keeps a history of searches. See Advanced search history.

The dialog has two tabs:

Click on the Start Search button in the advanced search dialog, or type Enter in any text field to start the search. The button in the main window always performs a simple search.

Click on the Show query details link at the top of the result page to see the query expansion.

Recoll automatically manages the expansion of search terms to their derivatives (ie: plural/singular, verb inflections). But there are other cases where the exact search term is not known. For example, you may not remember the exact spelling, or only know the beginning of the name.

The search will only propose replacement terms with spelling variations when no matching document were found. In some cases, both proper spellings and mispellings are present in the index, and it may be interesting to look for them explicitely.

The term explorer tool (started from the toolbar icon or from the Term explorer entry of the Tools menu) can be used to search the full index terms list. It has three modes of operations:

Note that in cases where Recoll does not know the beginning of the string to search for (ie a wildcard expression like *coll), the expansion can take quite a long time because the full index term list will have to be processed. The expansion is currently limited at 10000 results for wildcards and regular expressions. It is possible to change the limit in the configuration file.

Double-clicking on a term in the result list will insert it into the simple search entry field. You can also cut/paste between the result list and any entry field (the end of lines will be taken care of).

See the section describing the use of multiple indexes for generalities. Only the aspects concerning the recoll GUI are described here.

A recoll program instance is always associated with a specific index, which is the one to be updated when requested from the File menu, but it can use any number of Recoll indexes for searching. The external indexes can be selected through the external indexes tab in the preferences dialog.

Index selection is performed in two phases. A set of all usable indexes must first be defined, and then the subset of indexes to be used for searching. These parameters are retained across program executions (there are kept separately for each Recoll configuration). The set of all indexes is usually quite stable, while the active ones might typically be adjusted quite frequently.

The main index (defined by RECOLL_CONFDIR) is always active. If this is undesirable, you can set up your base configuration to index an empty directory.

When adding a new index to the set, you can select either a Recoll configuration directory, or directly a Xapian index directory. In the first case, the Xapian index directory will be obtained from the selected configuration.

As building the set of all indexes can be a little tedious when done through the user interface, you can use the RECOLL_EXTRA_DBS environment variable to provide an initial set. This might typically be set up by a system administrator so that every user does not have to do it. The variable should define a colon-separated list of index directories, ie:

export RECOLL_EXTRA_DBS=/some/place/xapiandb:/some/other/db

Another environment variable, RECOLL_ACTIVE_EXTRA_DBS allows adding to the active list of indexes. This variable was suggested and implemented by a Recoll user. It is mostly useful if you use scripts to mount external volumes with Recoll indexes. By using RECOLL_EXTRA_DBS and RECOLL_ACTIVE_EXTRA_DBS, you can add and activate the index for the mounted volume when starting recoll.

RECOLL_ACTIVE_EXTRA_DBS is available for Recoll versions 1.17.2 and later. A change was made in the same update so that recoll will automatically deactivate unreachable indexes when starting up.

Documents that you actually view (with the internal preview or an external tool) are entered into the document history, which is remembered.

You can display the history list by using the Tools/Doc History menu entry.

You can erase the document history by using the Erase document history entry in the File menu.

The documents in a result list are normally sorted in order of relevance. It is possible to specify a different sort order, either by using the vertical arrows in the GUI toolbox to sort by date, or switching to the result table display and clicking on any header. The sort order chosen inside the result table remains active if you switch back to the result list, until you click one of the vertical arrows, until both are unchecked (you are back to sort by relevance).

Sort parameters are remembered between program invocations, but result sorting is normally always inactive when the program starts. It is possible to keep the sorting activation state between program invocations by checking the Remember sort activation state option in the preferences.

It is also possible to hide duplicate entries inside the result list (documents with the exact same contents as the displayed one). The test of identity is based on an MD5 hash of the document container, not only of the text contents (so that ie, a text document with an image added will not be a duplicate of the text only). Duplicates hiding is controlled by an entry in the GUI configuration dialog, and is off by default.

As of release 1.19, when a result document does have undisplayed duplicates, a Dups link will be shown with the result list entry. Clicking the link will display the paths (URLs + ipaths) for the duplicate entries.

Both simple and advanced query dialogs save recent history, but the amount is limited: old queries will eventually be forgotten. Also, important queries may be difficult to find among others. This is why both types of queries can also be explicitely saved to files, from the GUI menus: File → Save last query / Load last query

The default location for saved queries is a subdirectory of the current configuration directory, but saved queries are ordinary files and can be written or moved anywhere.

Some of the saved query parameters are part of the preferences (e.g. autophrase or the active external indexes), and may differ when the query is loaded from the time it was saved. In this case, Recoll will warn of the differences, but will not change the user preferences.

You can customize some aspects of the search interface by using the GUI configuration entry in the Preferences menu.

There are several tabs in the dialog, dealing with the interface itself, the parameters used for searching and returning results, and what indexes are searched.

User interface parameters: 

Result list parameters: 

Search parameters: 

External indexes: This panel will let you browse for additional indexes that you may want to search. External indexes are designated by their database directory (ie: /home/someothergui/.recoll/xapiandb, /usr/local/recollglobal/xapiandb).

Once entered, the indexes will appear in the External indexes list, and you can chose which ones you want to use at any moment by checking or unchecking their entries.

Your main database (the one the current configuration indexes to), is always implicitly active. If this is not desirable, you can set up your configuration so that it indexes, for example, an empty directory. An alternative indexer may also need to implement a way of purging the index from stale data,

<a href="F%N">%P</a>

    "<table class=\"respar\">\n"
    "<tr>\n"
    "<td><a href='%U'><img src='%I' width='64'></a></td>\n"
    "<td>%L &nbsp;<i>%S</i> &nbsp;&nbsp;<b>%T</b><br>\n"
    "<span style='white-space:nowrap'><i>%M</i>&nbsp;%D</span>&nbsp;&nbsp;&nbsp; <i>%U</i>&nbsp;%i<br>\n"
    "%A %K</td>\n"
    "</tr></table>\n"

<u><b><a href="P%N">%T</a></b></u><br>
%A<font color=#008000>%U - %S</font> - %L

<img src="%I" align="left">%L <font color="#900000">%R</font>
&nbsp;&nbsp;<b>%T&</b><br>%S&nbsp;
<font color="#808080"><i>%U</i></font>
<table bgcolor="#e0e0e0">
<tr><td><div>%A</div></td></tr>
</table>%K

The Recoll KIO slave allows performing a Recoll search by entering an appropriate URL in a KDE open dialog, or with an HTML-based interface displayed in Konqueror.

The HTML-based interface is similar to the Qt-based interface, but slightly less powerful for now. Its advantage is that you can perform your search while staying fully within the KDE framework: drag and drop from the result list works normally and you have your normal choice of applications for opening files.

The alternative interface uses a directory view of search results. Due to limitations in the current KIO slave interface, it is currently not obviously useful (to me).

The interface is described in more detail inside a help file which you can access by entering recoll:/ inside the konqueror URL line (this works only if the recoll KIO slave has been previously installed).

The instructions for building this module are located in the source tree. See: kde/kio/recoll/00README.txt. Some Linux distributions do package the kio-recoll module, so check before diving into the build process, maybe it's already out there ready for one-click installation.

As a sample application, the Recoll KIO slave could allow preparing a set of HTML documents (for example a manual) so that they become their own search interface inside konqueror.

This can be done by either explicitly inserting <a href="recoll://..."> links around some document areas, or automatically by adding a very small javascript program to the documents, like the following example, which would initiate a search by double-clicking any term:

<script language="JavaScript">
    function recollsearch() {
        var t = document.getSelection();
        window.location.href = 'recoll://search/query?qtp=a&p=0&q=' +
            encodeURIComponent(t);
    }
</script>
 ....
<body ondblclick="recollsearch()">

Some characters are recognized as search modifiers when found immediately after the closing double quote of a phrase, as in "some term"modifierchars. The actual "phrase" can be a single term of course. Supported modifiers:

All words entered in Recoll search fields will be processed for wildcard expansion before the request is finally executed.

The wildcard characters are:

You should be aware of a few things when using wildcards.

Two characters are used to specify that a search hit should occur at the beginning or at the end of the text. ^ at the beginning of a term or phrase constrains the search to happen at the start, $ at the end force it to happen at the end.

As this function is implemented as a phrase search it is possible to specify a maximum distance at which the hit should occur, either through the controls of the advanced search panel, or using the query language, for example, as in:

"^someterm"o10

which would force someterm to be found within 10 terms of the start of the text. This can be combined with a field search as in somefield:"^someterm"o10 or somefield:someterm$.

This feature can also be used with an actual phrase search, but in this case, the distance applies to the whole phrase and anchor, so that, for example, bla bla my unexpected term at the beginning of the text would be a match for "^my term"o5.

Anchored searches can be very useful for searches inside somewhat structured documents like scientific articles, in case explicit metadata has not been supplied (a most frequent case), for example for looking for matches inside the abstract or the list of authors (which occur at the top of the document).

It is surprisingly convenient to be able to show or hide the Recoll GUI with a single keystroke. Recoll comes with a small Python script, based on the libwnck window manager interface library, which will allow you to do just this. The detailed instructions are on this wiki page.

This is probably obsolete now. Anyway:

The Recoll source tree contains the source code to the recoll_applet, a small application derived from the find_applet. This can be used to add a small Recoll launcher to the KDE panel.

The applet is not automatically built with the main Recoll programs, nor is it included with the main source distribution (because the KDE build boilerplate makes it relatively big). You can download its source from the recoll.org download page. Use the omnipotent configure;make;make install incantation to build and install.

You can then add the applet to the panel by right-clicking the panel and choosing the Add applet entry.

The recoll_applet has a small text window where you can type a Recoll query (in query language form), and an icon which can be used to restrict the search to certain types of files. It is quite primitive, and launches a new recoll GUI instance every time (even if it is already running). You may find it useful anyway.

Recoll simple handlers are usually shell-scripts, but this is in no way necessary. Extracting the text from the native format is the difficult part. Outputting the format expected by Recoll is trivial. Happily enough, most document formats have translators or text extractors which can be called from the handler. In some cases the output of the translating program is completely appropriate, and no intermediate shell-script is needed.

Input handlers are called with a single argument which is the source file name. They should output the result to stdout.

When writing a handler, you should decide if it will output plain text or HTML. Plain text is simpler, but you will not be able to add metadata or vary the output character encoding (this will be defined in a configuration file). Additionally, some formatting may be easier to preserve when previewing HTML. Actually the deciding factor is metadata: Recoll has a way to extract metadata from the HTML header and use it for field searches..

The RECOLL_FILTER_FORPREVIEW environment variable (values yes, no) tells the handler if the operation is for indexing or previewing. Some handlers use this to output a slightly different format, for example stripping uninteresting repeated keywords (ie: Subject: for email) when indexing. This is not essential.

You should look at one of the simple handlers, for example rclps for a starting point.

Don't forget to make your handler executable before testing !

If you can program and want to write an execm handler, it should not be too difficult to make sense of one of the existing modules. There is a sample one with many comments, not actually used by Recoll, which would index a text file as one document per line. Look for rcltxtlines.py in the src/filters directory in the Recoll BitBucket repository (the sample not in the distributed release at the moment).

You can also have a look at the slightly more complex rclzip which uses Zip file paths as identifiers (ipath).

execm handlers sometimes need to make a choice for the nature of the ipath elements that they use in communication with the indexer. Here are a few guidelines:

In any case, the main goal is that it should be easy for the handler to extract the target document, given the file name and the ipath element.

execm handlers will also produce a document with a null ipath element. Depending on the type of document, this may have some associated data (e.g. the body of an email message), or none (typical for an archive file). If it is empty, this document will be useful anyway for some operations, as the parent of the actual data documents.

There are two elements that link a file to the handler which should process it: the association of file to MIME type and the association of a MIME type with a handler.

The association of files to MIME types is mostly based on name suffixes. The types are defined inside the mimemap file. Example:

.doc = application/msword

If no suffix association is found for the file name, Recoll will try to execute a system command (typically file -i or xdg-mime) to determine a MIME type.

The second element is the association of MIME types to handlers in the mimeconf file. A sample will probably be better than a long explanation:

[index]
application/msword = exec antiword -t -i 1 -m UTF-8;\
     mimetype = text/plain ; charset=utf-8

application/ogg = exec rclogg

text/rtf = exec unrtf --nopict --html; charset=iso-8859-1; mimetype=text/html

application/x-chm = execm rclchm

The fragment specifies that:

Both the simple and persistent input handlers can return any MIME type to Recoll, which will further process the data according to the MIME configuration.

Most input filters filters produce either text/plain or text/html data. There are exceptions, for example, filters which process archive file (zip, tar, etc.) will usually return the documents as they are found, without processing them further.

There is nothing to say about text/plain output, except that its character encoding should be consistent with what is specified in the mimeconf file.

For filters producing HTML, the output could be very minimal like the following example:

<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html;charset=UTF-8">
  </head>
  <body>
   Some text content
  </body>
</html>
          

You should take care to escape some characters inside the text by transforming them into appropriate entities. At the very minimum, "&" should be transformed into "&", "<" should be transformed into "<". This is not always properly done by external helper programs which output HTML, and of course never by those which output plain text.

When encapsulating plain text in an HTML body, the display of a preview may be improved by enclosing the text inside <pre> tags.

The character set needs to be specified in the header. It does not need to be UTF-8 (Recoll will take care of translating it), but it must be accurate for good results.

Recoll will process meta tags inside the header as possible document fields candidates. Documents fields can be processed by the indexer in different ways, for searching or displaying inside query results. This is described in a following section.

By default, the indexer will process the standard header fields if they are present: title, meta/description, and meta/keywords are both indexed and stored for query-time display.

A predefined non-standard meta tag will also be processed by Recoll without further configuration: if a date tag is present and has the right format, it will be used as the document date (for display and sorting), in preference to the file modification date. The date format should be as follows:

<meta name="date" content="YYYY-mm-dd HH:MM:SS">
or
<meta name="date" content="YYYY-mm-ddTHH:MM:SS">
          

Example:

<meta name="date" content="2013-02-24 17:50:00">
          

Input handlers also have the possibility to "invent" field names. This should also be output as meta tags:

<meta name="somefield" content="Some textual data" />

You can embed HTML markup inside the content of custom fields, for improving the display inside result lists. In this case, add a (wildly non-standard) markup attribute to tell Recoll that the value is HTML and should not be escaped for display.

<meta name="somefield" markup="html" content="Some <i>textual</i> data" />

As written above, the processing of fields is described in a further section.

Persistent filters can use another, probably simpler, method to produce metadata, by calling the setfield() helper method. This avoids the necessity to produce HTML, and any issue with HTML quoting. See, for example, rclaudio in Recoll 1.23 and later for an example of handler which outputs text/plain and uses setfield() to produce metadata.

The indexer will interpret ^L characters in the handler output as indicating page breaks, and will record them. At query time, this allows starting a viewer on the right page for a hit or a snippet. Currently, only the PDF, Postscript and DVI handlers generate page breaks.

Recoll versions after 1.11 define a Python programming interface, both for searching and creating/updating an index.

The search interface is used in the Recoll Ubuntu Unity Lens and the Recoll Web UI. It can run queries on any Recoll configuration.

The index update section of the API may be used to create and update Recoll indexes on specific configurations (separate from the ones created by recollindex). The resulting databases can be queried alone, or in conjunction with regular ones, through the GUI or any of the query interfaces.

The search API is modeled along the Python database API specification. There were two major changes along Recoll versions:

We will describe the new API and package structure here. A paragraph at the end of this section will explain a few differences and ways to write code compatible with both versions.

The Python interface can be found in the source package, under python/recoll.

The python/recoll/ directory contains the usual setup.py. After configuring the main Recoll code, you can use the script to build and install the Python module:

            cd recoll-xxx/python/recoll
            python setup.py build
            python setup.py install
          

As of Recoll 1.19, the module can be compiled for Python3.

The normal Recoll installer installs the Python2 API along with the main code. The Python3 version must be explicitely built and installed.

When installing from a repository, and depending on the distribution, the Python API can sometimes be found in a separate package.

As an introduction, the following small sample will run a query and list the title and url for each of the results. It would work with Recoll 1.19 and later. The python/samples source directory contains several examples of Python programming with Recoll, exercising the extension more completely, and especially its data extraction features.

#!/usr/bin/env python

from recoll import recoll

db = recoll.connect()
query = db.query()
nres = query.execute("some query")
results = query.fetchmany(20)
for doc in results:
    print(doc.url, doc.title)

A few elements in the interface are specific and and need an explanation.

The update API can be used to create an index from data which is not accessible to the regular Recoll indexer, or structured to present difficulties to the Recoll input handlers.

An indexer created using this API will be have equivalent work to do as the the Recoll file system indexer: look for modified documents, extract their text, call the API for indexing it, take care of purging the index out of data from documents which do not exist in the document store any more.

The data for such an external indexer should be stored in an index separate from any used by the Recoll internal file system indexer. The reason is that the main document indexer purge pass (removal of deleted documents) would also remove all the documents belonging to the external indexer, as they were not seen during the filesystem walk. The main indexer documents would also probably be a problem for the external indexer own purge operation.

While there would be ways to enable multiple foreign indexers to cooperate on a single index, it is just simpler to use separate ones, and use the multiple index access capabilities of the query interface, if needed.

There are two parts in the update interface:

[MBOX]
fetch = /path/to/recoll/src/python/samples/rclmbox.py fetch
makesig = path/to/recoll/src/python/samples/rclmbox.py makesig
        

The following code fragments can be used to ensure that code can run with both the old and the new API (as long as it does not use the new abilities of the new API of course).

Adapting to the new package structure:

try:
    from recoll import recoll
    from recoll import rclextract
    hasextract = True
except:
    import recoll
    hasextract = False

      

Adapting to the change of nature of the next Query member. The same test can be used to choose to use the scroll() method (new) or set the next value (old).

       rownum = query.next if type(query.next) == int else \
                 query.rownumber

      

If you can install any or all of the following through the package manager for your system, all the better. Especially Qt is a very big piece of software, but you will most probably be able to find a binary package.

If you are building for an exotic or older system, it may be useful to note that functional improvements in Recoll have been relatively marginal in recent versions, and that you may make your life easier by using an older release, without losing major function.

The shopping list:

Check the Recoll download page for up to date version information.

Recoll has been built on Linux, FreeBSD, Mac OS X, and Solaris, most versions after 2005 should be ok, maybe some older ones too (Solaris 8 is ok). If you build on another system, and need to modify things, I would very much welcome patches.

Configure options: 

Normal procedure (for source extracted from a tar distribution):

        cd recoll-xxx
        ./configure
        make
        (practices usual hardship-repelling invocations)
      

When building from source cloned from the BitBucket repository, you also need to install autoconf, automake, and libtool and you must execute sh autogen.sh in the top source directory before running configure.

Use make install in the root of the source tree. This will copy the commands to prefix/bin and the sample configuration files, scripts and other shared data to prefix/share/recoll.

This file contains information about dynamic fields handling in Recoll. Some very basic fields have hard-wired behaviour, and, mostly, you should not change the original data inside the fields file. But you can create custom fields fitting your data and handle them just like they were native ones.

The fields file has several sections, which each define an aspect of fields processing. Quite often, you'll have to modify several sections to obtain the desired behaviour.

We will only give a short description here, you should refer to the comments inside the default file for more detailed information.

Field names should be lowercase alphabetic ASCII.

Here follows a small example of a personal fields file. This would extract a specific email header and use it as a searchable field, with data displayable inside result lists. (Side note: as the email handler does no decoding on the values, only plain ascii headers can be indexed, and only the first occurrence will be used for headers that occur several times).

[prefixes]
# Index mailmytag contents (with the given prefix)
mailmytag = XMTAG

[stored]
# Store mailmytag inside the document data record (so that it can be
# displayed - as %(mailmytag) - in result lists).
mailmytag = 

[queryaliases]
filename = fn
containerfilename = cfn

[mail]
# Extract the X-My-Tag mail header, and use it internally with the
# mailmytag field name
x-my-tag = mailmytag

mimemap specifies the file name extension to MIME type mappings.

For file names without an extension, or with an unknown one, a system command (file -i, or xdg-mime) will be executed to determine the MIME type (this can be switched off, or the command changed inside the main configuration file).

All extension values in mimemap must be entered in lower case. File names extensions are lower-cased for comparison during indexing, meaning that an upper case mimemap entry will never be matched.

The mappings can be specified on a per-subtree basis, which may be useful in some cases. Example: okular notes have a .xml extension but should be handled specially, which is possible because they are usually all located in one place. Example:

[~/.kde/share/apps/okular/docdata]
.xml = application/x-okular-notes

The recoll_noindex mimemap variable has been moved to recoll.conf and renamed to noContentSuffixes, while keeping the same function, as of Recoll version 1.21. For older Recoll versions, see the documentation for noContentSuffixes but use recoll_noindex in mimemap.

mimeconf specifies how the different MIME types are handled for indexing, and which icons are displayed in the recoll result lists.

Changing the parameters in the [index] section is probably not a good idea except if you are a Recoll developer.

The [icons] section allows you to change the icons which are displayed by recoll in the result lists (the values are the basenames of the png images inside the iconsdir directory (specified in recoll.conf).

mimeview specifies which programs are started when you click on an Open link in a result list. Ie: HTML is normally displayed using firefox, but you may prefer Konqueror, your openoffice.org program might be named oofice instead of openoffice etc.

Changes to this file can be done by direct editing, or through the recoll GUI preferences dialog.

If Use desktop preferences to choose document editor is checked in the Recoll GUI preferences, all mimeview entries will be ignored except the one labelled application/x-all (which is set to use xdg-open by default).

In this case, the xallexcepts top level variable defines a list of MIME type exceptions which will be processed according to the local entries instead of being passed to the desktop. This is so that specific Recoll options such as a page number or a search string can be passed to applications that support them, such as the evince viewer.

As for the other configuration files, the normal usage is to have a mimeview inside your own configuration directory, with just the non-default entries, which will override those from the central configuration file.

All viewer definition entries must be placed under a [view] section.

The keys in the file are normally MIME types. You can add an application tag to specialize the choice for an area of the filesystem (using a localfields specification in mimeconf). The syntax for the key is mimetype|tag

The nouncompforviewmts entry, (placed at the top level, outside of the [view] section), holds a list of MIME types that should not be uncompressed before starting the viewer (if they are found compressed, ie: mydoc.doc.gz).

The right side of each assignment holds a command to be executed for opening the file. The following substitutions are performed:

In addition to the predefined values above, all strings like %(fieldname) will be replaced by the value of the field named fieldname for the document. This could be used in combination with field customisation to help with opening the document.

ptrans specifies query-time path translations. These can be useful in multiple cases.

The file has a section for any index which needs translations, either the main one or additional query indexes. The sections are named with the Xapian index directory names. No slash character should exist at the end of the paths (all comparisons are textual). An exemple should make things sufficiently clear

          [/home/me/.recoll/xapiandb]
          /this/directory/moved = /to/this/place

          [/path/to/additional/xapiandb]
          /server/volume1/docdir = /net/server/volume1/docdir
          /server/volume2/docdir = /net/server/volume2/docdir