recoll / Code / Diff of /src/README

Diff of /src/README [79119c] .. [504155]

Switch to unified view


...

  Jean-Francois Dockes

   <jfd@recoll.org>

   Copyright (c) 2005-2014 Jean-Francois Dockes

   Permission is granted to copy, distribute and/or modify this document
   under the terms of the GNU Free Documentation License, Version 1.3 or any
   later version published by the Free Software Foundation; with no Invariant
   Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the
   license can be found at the following location: GNU web site.

   This document introduces full text search notions and describes the
   installation and use of the Recoll application. It currently describes
   Recoll 1.20.

     ----------------------------------------------------------------------

   Table of Contents

...

                             5.4.6. The ptrans file

                             5.4.7. Examples of configuration adjustments

                            Chapter 1. Introduction

1.1. Giving it a try

   If you do not like reading manuals (who does?) and would like to give
   Recoll a try, just install the application and start the recoll graphical
...
   options to help you find what you are looking for. However, there are
   other ways to perform Recoll searches: mostly a command line interface, a
   Python programming interface, a KDE KIO slave module, and a Ubuntu Unity
   Lens module.

                              Chapter 2. Indexing

2.1. Introduction

   Indexing is the process by which the set of documents is analyzed and the
   data entered into the database. Recoll indexing is normally incremental:
...

  2.1.1. Indexing modes

   Recoll indexing can be performed along two different modes:

     * Periodic (or batch) indexing: indexing takes place at discrete times,
       by executing the recollindex command. The typical usage is to have a
       nightly indexing run programmed into your cron file.

     * Real time indexing: indexing takes place as soon as a file is created
       or changed. recollindex runs as a daemon and uses a file system
       alteration monitor such as inotify, Fam or Gamin to detect file
       changes.

   The choice between the two methods is mostly a matter of preference, and
...

   The default location for the index data is the xapiandb subdirectory of
   the Recoll configuration directory, typically $HOME/.recoll/xapiandb/.
   This can be changed via two different methods (with different purposes):

     * You can specify a different configuration directory by setting the
       RECOLL_CONFDIR environment variable, or using the -c option to the
       Recoll commands. This method would typically be used to index
       different areas of the file system to different indexes. For example,
       if you were to issue the following commands:

...

       Using multiple configuration directories and configuration options
       allows you to tailor multiple configurations and indexes to handle
       whatever subset of the available data you wish to make searchable.

     * For a given configuration directory, you can specify a non-default
       storage location for the index by setting the dbdir parameter in the
       configuration file (see the configuration section). This method would
       mainly be of use if you wanted to keep the configuration directory in
       its default location, but desired another location for the index,
       typically out of disk occupation concerns.
...

   Recoll provides a configuration option to specify the minimum time before
   which a file, specified by a wildcard pattern, cannot be reindexed. See
   the mondelaypatterns parameter in the configuration section.

                              Chapter 3. Searching

3.1. Searching with the Qt graphical user interface

   The recoll program provides the main user interface for searching. It is
   based on the Qt library.

   recoll has two search modes:

     * Simple search (the default, on the main screen) has a single entry
       field where you can enter multiple words.

     * Advanced search (a panel accessed through the Tools menu or the
       toolbox bar icon) has multiple entry fields, which you may use to
       build a logical condition, with additional filtering on file type,
       location in the file system, modification date, and size.

   In most cases, you can enter the terms as you think them, even if they
...
   File name will specifically look for file names. The point of having a
   separate file name search is that wild card expansion can be performed
   more efficiently on a small subset of the index (allowing wild cards on
   the left of terms without excessive penality). Things to know:

     * White space in the entry should match white space in the file name,
       and is not treated specially.

     * The search is insensitive to character case and accents, independantly
       of the type of index.

     * An entry without any wild card character and not capitalized will be
       prepended and appended with '*' (ie: etc -> *etc*, but Etc -> etc).

     * If you have a big index (many files), excessively generic fragments
       may result in inefficient searches.

   You can search for exact phrases (adjacent words in a given order) by
   enclosing the input inside double quotes. Ex: "virtual reality".

...
   standard desktop tool.

   You may also change the choice of applications by editing the mimeview
   configuration file if you find this more convenient.

   Each result entry also has a right-click menu with an Open With entry.
   This lets you choose an application from the list of those which
   registered with the desktop for the document MIME type.

   The Preview and Open edit links may not be present for all entries,
   meaning that Recoll has no configured way to preview a given file type
   (which was indexed by name only), or no configured external editor for the
   file type. This can sometimes be adjusted simply by tweaking the mimemap
   and mimeview configuration files (the latter can be modified with the user
...

   Apart from the preview and edit links, you can display a pop-up menu by
   right-clicking over a paragraph in the result list. This menu has the
   following entries:

     * Preview

     * Open

     * Copy File Name

     * Copy Url

     * Save to File

     * Find similar

     * Preview Parent document

     * Open Parent document

     * Open Snippets Window

   The Preview and Open entries do the same thing as the corresponding links.

   The Copy File Name and Copy Url copy the relevant data to the clipboard,
   for later pasting.
...

   This part of the dialog lets you constructc a query by combining multiple
   clauses of different types. Each entry field is configurable for the
   following modes:

     * All terms.

     * Any term.

     * None of the terms.

     * Phrase (exact terms in order within an adjustable window).

     * Proximity (terms in any order within an adjustable window).

     * Filename search.

   Additional entry fields can be created by clicking the Add clause button.

   When searching, the non-empty clauses will be combined either with an AND
   or an OR conjunction, depending on the choice made on the left (All
...
    3.1.6.2. Avanced search: the "filter" tab

   This part of the dialog has several sections which allow filtering the
   results of a search according to a number of criteria

     * The first section allows filtering by dates of last modification. You
       can specify both a minimum and a maximum date. The initial values are
       set according to the oldest and newest documents found in the index.

     * The next section allows filtering the results by file size. There are
       two entries for minimum and maximum size. Enter decimal numbers. You
       can use suffix multipliers: k/K, m/M, g/G, t/T for 1E3, 1E6, 1E9, 1E12
       respectively.

     * The next section allows filtering the results by their MIME types, or
       MIME categories (ie: media/text/message/etc.).

       You can transfer the types between two boxes, to define which will be
       included or excluded by the search.

       The state of the file type selection can be saved as the default (the
       file type filter will not be activated at program start-up, but the
       lists will be in the restored state).

     * The bottom section allows restricting the search results to a sub-tree
       of the indexed area. You can use the Invert checkbox to search for
       files not in the sub-tree instead. If you use directory filtering
       often and on big subsets of the file system, you may think of setting
       up multiple indexes instead, as the performance may be better.

...
   which will let you adjust what columns are displayed. You can drag the
   column headers to adjust their order. You can click them to sort by the
   field displayed in the column. You can also save the result list in CSV
   format.

   Changing the GUI geometry. It is possible to configure the GUI in wide
   form factor by dragging the toolbars to one of the sides (their location
   is remembered between sessions), and moving the category filters to a menu
   (can be set in the Preferences -> GUI configuration -> User interface
   panel).

   Query explanation. You can get an exact description of what the query
   looked for, including stem expansion, and Boolean operators used, by
   clicking on the result list header.

   Advanced search history. As of Recoll 1.18, you can display any of the
...
   the parameters used for searching and returning results, and what indexes
   are searched.

   User interface parameters: 

     * Highlight color for query terms: Terms from the user query are
       highlighted in the result list samples and the preview window. The
       color can be chosen here. Any Qt color string should work (ie red,
       #ff0000). The default is blue.

     * Style sheet: The name of a Qt style sheet text file which is applied
       to the whole Recoll application on startup. The default value is
       empty, but there is a skeleton style sheet (recoll.qss) inside the
       /usr/share/recoll/examples directory. Using a style sheet, you can
       change most recoll graphical parameters: colors, fonts, etc. See the
       sample file for a few simple examples.
...
       set the foreground to a light color and the background to a dark one
       in the desktop preferences, but only the background is set inside the
       Recoll style sheet, and it is light too, then text will appear
       light-on-light inside the Recoll GUI.

     * Maximum text size highlighted for preview Inserting highlights on
       search term inside the text before inserting it in the preview window
       involves quite a lot of processing, and can be disabled over the given
       text size to speed up loading.

     * Prefer HTML to plain text for preview if set, Recoll will display HTML
       as such inside the preview window. If this causes problems with the Qt
       HTML display, you can uncheck it to display the plain text version
       instead.

     * Plain text to HTML line style: when displaying plain text inside the
       preview window, Recoll tries to preserve some of the original text
       line breaks and indentation. It can either use PRE HTML tags, which
       will well preserve the indentation but will force horizontal scrolling
       for long lines, or use BR tags to break at the original line breaks,
       which will let the editor introduce other line breaks according to the
       window width, but will lose some of the original indentation. The
       third option has been available in recent releases and is probably now
       the best one: use PRE tags with line wrapping.

     * Use desktop preferences to choose document editor: if this is checked,
       the xdg-open utility will be used to open files when you click the
       Open link in the result list, instead of the application defined in
       mimeview. xdg-open will in term use your desktop preferences to choose
       an appropriate application.

     * Exceptions: when using the desktop preferences for opening documents,
       these are MIME types that will still be opened according to Recoll
       preferences. This is useful for passing parameters like page numbers
       or search strings to applications that support them (e.g. evince).
       This cannot be done with xdg-open which only supports passing one
       parameter.

     * Choose editor applications this will let you choose the command
       started by the Open links inside the result list, for specific
       document types.

     * Display category filter as toolbar... this will let you choose if the
       document categories are displayed as a list or a set of buttons.

     * Auto-start simple search on white space entry: if this is checked, a
       search will be executed each time you enter a space in the simple
       search input field. This lets you look at the result list as you enter
       new terms. This is off by default, you may like it or not...

     * Start with advanced search dialog open : If you use this dialog
       frequently, checking the entries will get it to open when recoll
       starts.

     * Remember sort activation state if set, Recoll will remember the sort
       tool stat between invocations. It normally starts with sorting
       disabled.

   Result list parameters: 

     * Number of results in a result page

     * Result list font: There is quite a lot of information shown in the
       result list, and you may want to customize the font and/or font size.
       The rest of the fonts used by Recoll are determined by your generic Qt
       config (try the qtconfig command).

     * Edit result list paragraph format string: allows you to change the
       presentation of each result list entry. See the result list
       customisation section.

     * Edit result page HTML header insert: allows you to define text
       inserted at the end of the result page HTML header. More detail in the
       result list customisation section.

     * Date format: allows specifying the format used for displaying dates
       inside the result list. This should be specified as an strftime()
       string (man strftime).

     * Abstract snippet separator: for synthetic abstracts built from index
       data, which are usually made of several snippets from different parts
       of the document, this defines the snippet separator, an ellipsis by
       default.

   Search parameters: 

     * Hide duplicate results: decides if result list entries are shown for
       identical documents found in different places.

     * Stemming language: stemming obviously depends on the document's
       language. This listbox will let you chose among the stemming databases
       which were built during indexing (this is set in the main
       configuration file), or later added with recollindex -s (See the
       recollindex manual). Stemming languages which are dynamically added
       will be deleted at the next indexing pass unless they are also added
       in the configuration file.

     * Automatically add phrase to simple searches: a phrase will be
       automatically built and added to simple searches when looking for Any
       terms. This will give a relevance boost to the results where the
       search terms appear as a phrase (consecutive and in order).

     * Autophrase term frequency threshold percentage: very frequent terms
       should not be included in automatic phrase searches for performance
       reasons. The parameter defines the cutoff percentage (percentage of
       the documents where the term appears).

     * Replace abstracts from documents: this decides if we should synthesize
       and display an abstract in place of an explicit abstract found within
       the document itself.

     * Dynamically build abstracts: this decides if Recoll tries to build
       document abstracts (lists of snippets) when displaying the result
       list. Abstracts are constructed by taking context from the document
       information, around the search terms.

     * Synthetic abstract size: adjust to taste...

     * Synthetic abstract context words: how many words should be displayed
       around each term occurrence.

     * Query language magic file name suffixes: a list of words which
       automatically get turned into ext:xxx file name suffix clauses when
       starting a query language query (ie: doc xls xlsx...). This will save
       some typing for people who use file types a lot when querying.

   External indexes: This panel will let you browse for additional indexes
...
    3.1.12.1. The result list format

   The result list presentation can be exhaustively customized by adjusting
   two elements:

     * The paragraph format

     * HTML code inside the header section

   These can be edited from the Result list tab of the GUI configuration.

   Newer versions of Recoll (from 1.17) use a WebKit HTML object by default
   (this may be disabled at build time), and total customisation is possible
...
      The paragraph format

   This is an arbitrary HTML string where the following printf-like %
   substitutions will be performed:

     * %A. Abstract

     * %D. Date

     * %I. Icon image name. This is normally determined from the MIME type.
       The associations are defined inside the mimeconf configuration file.
       If a thumbnail for the file is found at the standard Freedesktop
       location, this will be displayed instead.

     * %K. Keywords (if any)

     * %L. Precooked Preview, Edit, and possibly Snippets links

     * %M. MIME type

     * %N. result Number inside the result page

     * %P. Parent folder Url. In the case of an embedded document, this is
       the parent folder for the top level container file.

     * %R. Relevance percentage

     * %S. Size information

     * %T. Title or Filename if not set.

     * %t. Title or Filename if not set.

     * %U. Url

   The format of the Preview, Edit, and Snippets links is <a href="P%N">, <a
   href="E%N"> and <a href="A%N"> where docnum (%N) expands to the document
   number inside the result page).

   It is also possible to use a "F%N" value as a link target. This will open
   the document corresponding to the %P parent folder expansion, usually
   creating a file manager window on the folder where the container file
   resides. E.g.:

 <a href="F%N">%P</a>

   In addition to the predefined values above, all strings like %(fieldname)
   will be replaced by the value of the field named fieldname for this
   document. Only stored fields can be accessed in this way, the value of
   indexed but not stored fields is not known at this point in the search
...
3.3. Searching on the command line

   There are several ways to obtain search results as a text stream, without
   a graphical interface:

     * By passing option -t to the recoll program.

     * By using the recollq program.

     * By writing a custom Python program, using the Recoll Python API.

   The first two methods work in the same way and accept/need the same
   arguments (except for the additional -t to recoll). The query to be
   executed is specified as command line arguments.

...

   In some cases, the document paths stored inside the index do not match the
   actual ones, so that document previews and accesses will fail. This can
   occur in a number of circumstances:

     * When using multiple indexes it is a relatively common occurrence that
       some will actually reside on a remote volume, for exemple mounted via
       NFS. In this case, the paths used to access the documents on the local
       machine are not necessarily the same than the ones used while indexing
       on the remote machine. For example, /home/me may have been used as a
       topdirs elements while indexing, but the directory might be mounted as
       /net/server/home/me on the local machine.

     * The case may also occur with removable disks. It is perfectly possible
       to configure an index to live with the documents on the removable
       disk, but it may happen that the disk is not mounted at the same place
       so that the documents paths from the index are invalid.

     * As a last exemple, one could imagine that a big directory has been
       moved, but that it is currently inconvenient to run the indexer.

   More generally, the path translation facility may be useful whenever the
   documents paths seen by the indexer are not the same as the ones which
   should be used at query time.
...

   As usual, words inside quotes define a phrase (the order of words is
   significant), so that title:"prejudice pride" is not the same as
   title:prejudice title:pride, and is unlikely to find a result.

   To save you some typing, recent Recoll versions (1.20 and later) interpret
   a comma-separated list of terms as an AND list inside the field. Use slash
   characters ('/') for an OR list. No white space is allowed. So

 author:john,lennon

   will search for documents with john and lennon inside the author field (in
   any order), and

 author:john/ringo

   would search for john or ringo.

   Modifiers can be set on a phrase clause, for example to specify a
   proximity search (unordered). See the modifier section.

   Recoll currently manages the following default fields:

     * title, subject or caption are synonyms which specify data to be
       searched for in the document title or subject.

     * author or from for searching the documents originators.

     * recipient or to for searching the documents recipients.

     * keyword for searching the document-specified keywords (few documents
       actually have any).

     * filename for the document's file name. This is not necessarily set for
       all documents: internal documents contained inside a compound one (for
       example an EPUB section) do not inherit the container file name any
       more, this was replaced by an explicit field (see next). Sub-documents
       can still have a specific filename, if it is implied by the document
       format, for example the attachment file name for an email attachment.

     * containerfilename. This is set for all documents, both top-level and
       contained sub-documents, and is always the name of the filesystem
       directory entry which contains the data. The terms from this field can
       only be matched by an explicit field specification (as opposed to
       terms from filename which are also indexed as general document
       content). This avoids getting matches for all the sub-documents when
       searching for the container file name.

     * ext specifies the file name extension (Ex: ext:html)

   Recoll 1.20 and later have a way to specify aliases for the field names,
   which will save typing, for example by aliasing filename to fn or
   containerfilename to cfn. See the section about the fields file

   The field syntax also supports a few field-like, but special, criteria:

     * dir for filtering the results on file location (Ex:
       dir:/home/me/somedir). -dir also works to find results not in the
       specified directory (release >= 1.15.8). Tilde expansion will be
       performed as usual (except for a bug in versions 1.19 to 1.19.11p1).
       Wildcards will be expanded, but please have a look at an important
       limitation of wildcards in path filters.
...
       and are best avoided.

       You need to use double-quotes around the path value if it contains
       space characters.

     * size for filtering the results on file size. Example: size<10000. You
       can use <, > or = as operators. You can specify a range like the
       following: size>100 size<1000. The usual k/K, m/M, g/G, t/T can be
       used as (decimal) multipliers. Ex: size>1k to search for files bigger
       than 1000 bytes.

     * date for searching or filtering on dates. The syntax for the argument
       is based on the ISO8601 standard for dates and time intervals. Only
       dates are supported, no times. The general syntax is 2 elements
       separated by a / character. Each element can be a date or a period of
       time. Periods are specified as PnYnMnD. The n numbers are the
       respective numbers of years, months or days, any of which may be
       missing. Dates are specified as YYYY-MM-DD. The days and months parts
       may be missing. If the / is present but an element is missing, the
       missing element is interpreted as the lowest or highest date in the
       index. Examples:

          * 2001-03-01/2002-05-01 the basic syntax for an interval of dates.

          * 2001-03-01/P1Y2M the same specified with a period.

          * 2001/ from the beginning of 2001 to the latest date in the index.

          * 2001 the whole year of 2001

          * P2D/ means 2 days ago up to now if there are no documents with
            dates in the future.

          * /2003 all documents from 2003 or older.

       Periods can also be specified with small letters (ie: p2y).

     * mime or format for specifying the MIME type. This one is quite special
       because you can specify several values which will be OR'ed (the normal
       default for the language is AND). Ex: mime:text/plain mime:text/html.
       Specifying an explicit boolean operator before a mime specification is
       not supported and will produce strange results. You can filter out
       certain types by using negation (-mime:some/type), and you can use
       wildcards in the value (mime:text/*). Note that mime is the ONLY field
       with an OR default. You do need to use OR with ext terms for example.

     * type or rclcat for specifying the category (as in
       text/media/presentation/etc.). The classification of MIME types in
       categories is defined in the Recoll configuration (mimeconf), and can
       be modified or extended. The default category names are those which
       permit filtering results in the main GUI screen. Categories are OR'ed
       like MIME types above. This can't be negated with - either.
...
   Some characters are recognized as search modifiers when found immediately
   after the closing double quote of a phrase, as in "some
   term"modifierchars. The actual "phrase" can be a single term of course.
   Supported modifiers:

     * l can be used to turn off stemming (mostly makes sense with p because
       stemming is off by default for phrases).

     * o can be used to specify a "slack" for phrase and proximity searches:
       the number of additional terms that may be found between the specified
       ones. If o is followed by an integer number, this is the slack, else
       the default is 10.

     * p can be used to turn the default phrase search into a proximity one
       (unordered). Example:"order any in"p

     * C will turn on case sensitivity (if the index supports it).

     * D will turn on diacritics sensitivity (if the index supports it).

     * A weight can be specified for a query element by specifying a decimal
       value at the start of the modifiers. Example: "Important"2.5.

3.6. Search case and diacritics sensitivity

   For Recoll versions 1.18 and later, and when working with a raw index (not
...
   All words entered in Recoll search fields will be processed for wildcard
   expansion before the request is finally executed.

   The wildcard characters are:

     * * which matches 0 or more characters.

     * ? which matches a single character.

     * [] which allow defining sets of characters to be matched (ex: [abc]
       matches a single character which may be 'a' or 'b' or 'c', [0-9]
       matches any number.

   You should be aware of a few things when using wildcards.

     * Using a wildcard character at the beginning of a word can make for a
       slow search because Recoll will have to scan the whole index term list
       to find the matches. However, this is much less a problem for field
       searches, and queries like author:*@domain.com can sometimes be very
       useful.

     * For Recoll version 18 only, when working with a raw index (preserving
       character case and diacritics), the literal part of a wildcard
       expression will be matched exactly for case and diacritics. This is
       not true any more for versions 19 and later.

     * Using a * at the end of a word can produce more matches than you would
       think, and strange search results. You can use the term explorer tool
       to check what completions exist for a given term. You can also see
       exactly what search was performed by clicking on the link at the top
       of the result list. In general, for natural language terms, stem
       expansion will produce better results than an ending * (stem expansion
...
3.8. Desktop integration

   Being independant of the desktop type has its drawbacks: Recoll desktop
   integration is minimal. However there are a few tools available:

     * The KDE KIO Slave was described in a previous section.

     * If you use a recent version of Ubuntu Linux, you may find the Ubuntu
       Unity Lens module useful.

     * There is also an independantly developed Krunner plugin.

   Here follow a few other things that may help.

  3.8.1. Hotkeying recoll

...
   query (in query language form), and an icon which can be used to restrict
   the search to certain types of files. It is quite primitive, and launches
   a new recoll GUI instance every time (even if it is already running). You
   may find it useful anyway.

                        Chapter 4. Programming interface

   Recoll has an Application Programming Interface, usable both for indexing
   and searching, currently accessible from the Python language.

   Another less radical way to extend the application is to write input
...
   kind will not be described here.

   There are currently (1.18 and since 1.13) two kinds of external executable
   input handlers:

     * Simple exec handlers run once and exit. They can be bare programs like
       antiword, or scripts using other programs. They are very simple to
       write, because they just need to print the converted document to the
       standard output. Their output can be plain text or HTML. HTML is
       usually preferred because it can store metadata fields and it allows
       preserving some of the formatting for the GUI preview.

     * Multiple execm handlers can process multiple files (sparing the
       process startup time which can be very significant), or multiple
       documents per file (e.g.: for zip or chm files). They communicate with
       the indexer through a simple protocol, but are nevertheless a bit more
       complicated than the older kind. Most of new handlers are written in
       Python, using a common module to handle the protocol. There is an
...

   execm handlers sometimes need to make a choice for the nature of the ipath
   elements that they use in communication with the indexer. Here are a few
   guidelines:

     * Use ASCII or UTF-8 (if the identifier is an integer print it, for
       example, like printf %d would do).

     * If at all possible, the data should make some kind of sense when
       printed to a log file to help with debugging.

     * Recoll uses a colon (:) as a separator to store a complex path
       internally (for deeper embedding). Colons inside the ipath elements
       output by a handler will be escaped, but would be a bad choice as a
       handler-specific separator (mostly, again, for debugging issues).

   In any case, the main goal is that it should be easy for the handler to
...

 application/x-chm = execm rclchm

   The fragment specifies that:

     * application/msword files are processed by executing the antiword
       program, which outputs text/plain encoded in utf-8.

     * application/ogg files are processed by the rclogg script, with default
       output type (text/html, with encoding specified in the header, or
       utf-8 by default).

     * text/rtf is processed by unrtf, which outputs text/html. The
       iso-8859-1 encoding is specified because it is not the utf-8 default,
       and not output by unrtf in the HTML header section.

     * application/x-chm is processed by a persistant handler. This is
       determined by the execm keyword.

  4.1.4. Input handler HTML output

   The output HTML could be very minimal like the following example:
...
   Recoll defines a number of default fields. Additional ones can be output
   by handlers, and described in the fields configuration file.

   Fields can be:

     * indexed, meaning that their terms are separately stored in inverted
       lists (with a specific prefix), and that a field-specific search is
       possible.

     * stored, meaning that their value is recorded in the index data record
       for the document, and can be returned and displayed with search
       results.

   A field can be either or both indexed and stored. This and other aspects
   of fields handling is defined inside the fields configuration file.

   The sequence of events for field processing is as follows:

     * During indexing, recollindex scans all meta fields in HTML documents
       (most document types are transformed into HTML at some point). It
       compares the name for each element to the configuration defining what
       should be done with fields (the fields file)

     * If the name for the meta element matches one for a field that should
       be indexed, the contents are processed and the terms are entered into
       the index with the prefix defined in the fields file.

     * If the name for the meta element matches one for a field that should
       be stored, the content of the element is stored with the document data
       record, from which it can be extracted and displayed at query time.

     * At query time, if a field search is performed, the index prefix is
       computed and the match is only performed against appropriately
       prefixed terms in the index.

     * At query time, the field can be displayed inside the result list by
       using the appropriate directive in the definition of the result list
       paragraph format. All fields are displayed on the fields screen of the
       preview window (which you can reach through the right-click menu).
       This is independant of the fact that the search which produced the
       results used the field or not.
...
   searching one is used in the Recoll Ubuntu Unity Lens and Recoll Web UI.

   The API is inspired by the Python database API specification. There were
   two major changes in recent Recoll versions:

     * The basis for the Recoll API changed from Python database API version
       1.0 (Recoll versions up to 1.18.1), to version 2.0 (Recoll 1.18.2 and
       later).
     * The recoll module became a package (with an internal recoll module) as
       of Recoll version 1.19, in order to add more functions. For existing
       code, this only changes the way the interface must be imported.

   We will mostly describe the new API and package structure here. A
   paragraph at the end of this section will explain a few differences and
...

    4.3.2.2. Recoll package

   The recoll package contains two modules:

     * The recoll module contains functions and classes used to query (or
       update) the index.

     * The rclextract module contains functions and classes used to access
       document data.

    4.3.2.3. The recoll module

      Functions

   connect(confdir=None, extra_dbs=None, writable = False)
           The connect() function connects to one or several Recoll index(es)
           and returns a Db object.
              * confdir may specify a configuration directory. The usual
                defaults apply.
              * extra_dbs is a list of additional indexes (Xapian
                directories).
              * writable decides if we can index new data through this
                connection.
           This call initializes the recoll module, and it should always be
           performed before any other call or object creation.

      Classes
...

        rownum = query.next if type(query.next) == int else \
                  query.rownumber


                   Chapter 5. Installation and configuration

5.1. Installing a binary copy

   There are three types of binary Recoll installations:

     * Through your system normal software distribution framework (ie,
       Debian/Ubuntu apt, FreeBSD ports, etc.).

     * From a package downloaded from the Recoll web site.

     * From a prebuilt tree downloaded from the Recoll web site.

   In all cases, the strict software dependancies (ie on Xapian or iconv)
   will be automatically satisfied, you should not have to worry about them.

   You will only have to check or install supporting applications for the
...
   by ad hoc handler code now use the xsltproc command, which usually comes
   with libxslt. These are: abiword, fb2 (ebooks), kword, openoffice, svg.

   Now for the list:

     * Openoffice files need unzip and xsltproc.

     * PDF files need pdftotext which is part of the Xpdf or Poppler
       packages.

     * Postscript files need pstotext. The original version has an issue with
       shell character in file names, which is corrected in recent packages.
       See http://www.recoll.org/features.html for more detail.

     * MS Word needs antiword. It is also useful to have wvWare installed as
       it may be be used as a fallback for some files which antiword does not
       handle.

     * MS Excel and PowerPoint are processed by internal Python handlers.

     * MS Open XML (docx) needs xsltproc.

     * Wordperfect files need wpd2html from the libwpd (or libwpd-tools on
       Ubuntu) package.

     * RTF files need unrtf, which, in its standard version, has much trouble
       with non-western character sets. Check
       http://www.recoll.org/features.html.

     * TeX files need untex or detex. Check
       http://www.recoll.org/features.html for sources if it's not packaged
       for your distribution.

     * dvi files need dvips.

     * djvu files need djvutxt and djvused from the DjVuLibre package.

     * Audio files: Recoll releases 1.14 and later use a single Python
       handler based on mutagen for all audio file types.

     * Pictures: Recoll uses the Exiftool Perl package to extract tag
       information. Most image file formats are supported. Note that there
       may not be much interest in indexing the technical tags (image size,
       aperture, etc.). This is only of interest if you store personal tags
       or textual descriptions inside the image files.

     * chm: files in Microsoft help format need Python and the pychm module
       (which needs chmlib).

     * ICS: up to Recoll 1.13, iCalendar files need Python and the icalendar
       module. icalendar is not needed for newer versions, which use internal
       code.

     * Zip archives need Python (and the standard zipfile module).

     * Rar archives need Python, the rarfile Python module and the unrar
       utility.

     * Midi karaoke files need Python and the Midi module

     * Konqueror webarchive format with Python (uses the Tarfile module).

     * Mimehtml web archive format (support based on the email handler, which
       introduces some mild weirdness, but still usable).

   Text, HTML, email folders, and Scribus files are processed internally. Lyx
   is used to index Lyx files. Many handlers need iconv and the standard sed
   and awk.
...

   You may have to compile Xapian but this is easy.

   The shopping list:

     * C++ compiler. Up to Recoll version 1.13.04, its absence can manifest
       itself by strange messages about a missing iconv_open.

     * Development files for Xapian core.

  Important

       If you are building Xapian for an older CPU (before Pentium 4 or
       Athlon 64), you need to add the --disable-sse flag to the configure
       command. Else all Xapian application will crash with an illegal
       instruction error.

     * Development files for Qt 4 . Recoll has not been tested with Qt 5 yet.
       Recoll 1.15.9 was the last version to support Qt 3. If you do not want
       to install or build the Qt Webkit module, Recoll has a configuration
       option to disable its use (see further).

     * Development files for X11 and zlib.

     * You may also need libiconv. On Linux systems, the iconv interface is
       part of libc and you should not need to do anything special.

   Check the Recoll download page for up to date version information.

  5.3.2. Building
...
   ok). If you build on another system, and need to modify things, I would
   very much welcome patches.

   Configure options: 

     * --without-aspell will disable the code for phonetic matching of search
       terms.

     * --with-fam or --with-inotify will enable the code for real time
       indexing. Inotify support is enabled by default on recent Linux
       systems.

     * --with-qzeitgeist will enable sending Zeitgeist events about the
       visited search results, and needs the qzeitgeist package.

     * --disable-webkit is available from version 1.17 to implement the
       result list with a Qt QTextBrowser instead of a WebKit widget if you
       do not or can't depend on the latter.

     * --disable-idxthreads is available from version 1.19 to suppress
       multithreading inside the indexing process. You can also use the
       run-time configuration to restrict recollindex to using a single
       thread, but the compile-time option may disable a few more unused
       locks. This only applies to the use of multithreading for the core
       index processing (data input). The Recoll monitor mode always uses at
       least two threads of execution.

     * --disable-python-module will avoid building the Python module.

     * --disable-xattr will prevent fetching data from file extended
       attributes. Beyond a few standard attributes, fetching extended
       attributes data can only be useful is some application stores data in
       there, and also needs some simple configuration (see comments in the
       fields configuration file).

     * --enable-camelcase will enable splitting camelCase words. This is not
       enabled by default as it has the unfortunate side-effect of making
       some phrase searches quite confusing: ie, "MySQL manual" would be
       matched by "MySQL manual" and "my sql manual" but not "mysql manual"
       (only inside phrase searches).

     * --with-file-command Specify the version of the 'file' command to use
       (ie: --with-file-command=/usr/local/bin/file). Can be useful to enable
       the gnu version on systems where the native one is bad.

     * --disable-qtgui Disable the Qt interface. Will allow building the
       indexer and the command line search program in absence of a Qt
       environment.

     * --disable-x11mon Disable X11 connection monitoring inside recollindex.
       Together with --disable-qtgui, this allows building recoll without Qt
       and X11.

     * --disable-pic will compile Recoll with position-dependant code. This
       is incompatible with building the KIO or the Python or PHP extensions,
       but might yield very marginally faster code.

     * Of course the usual autoconf configure options, like --prefix apply.

   Normal procedure:

         cd recoll-xxx
         configure
...
         defaultcharset = utf-8
        

   There are three kinds of lines:

     * Comment (starts with #) or empty.

     * Parameter affectation (name = value).

     * Section definition ([somedirname]).

   Depending on the type of configuration file, section definitions either
   separate groups of parameters or allow redefining some parameters for a
   directory sub-tree. They stay in effect until another section definition,
   or the end of file, is encountered. Some of the parameters used for
...
   embedded spaces can be quoted using double-quotes.

   Encoding issues. Most of the configuration parameters are plain ASCII. Two
   particular sets of values may cause encoding issues:

     * File path parameters may contain non-ascii characters and should use
       the exact same byte values as found in the file system directory.
       Usually, this means that the configuration file should use the system
       default locale encoding.

     * The unac_except_trans parameter should be encoded in UTF-8. If your
       system locale is not UTF-8, and you need to also specify non-ascii
       file paths, this poses a difficulty because common text editors cannot
       handle multiple encodings in a single file. In this relatively
       unlikely case, you can edit the configuration file as two separate
       text files with appropriate encodings, and concatenate them to create
...
           indexing, or for all files inside the selected subtrees,
           independently of MIME type.

   usesystemfilecommand

           Decide if we execute a system command (file -i by default) as a
           final step for determining the MIME type for a file (the main
           procedure uses suffix associations as defined in the mimemap
           file). This can be useful for files with suffix-less names, but it
           will also cause the indexing of many bogus "text" files.

   systemfilecommand

           Command to use for mime for mime type determination if
           usesystefilecommand is set. Recent versions of xdg-mime sometimes
           work better than file.

   processwebqueue

           If this is set, process the directory where Web browser plugins
           copy visited pages for indexing.
...
   The fields file has several sections, which each define an aspect of
   fields processing. Quite often, you'll have to modify several sections to
   obtain the desired behaviour.

   We will only give a short description here, you should refer to the
   comments inside the default file for more detailed information.

   Field names should be lowercase alphabetic ASCII.

   [prefixes]

...

   [aliases]

           This section defines lists of synonyms for the canonical names
           used inside the [prefixes] and [stored] sections

   [queryaliases]

           This section also defines aliases for the canonic field names,
           with the difference that the substitution will only be used at
           query time, avoiding any possibility that the value would pick-up
           random metadata from documents.

   handler-specific sections

           Some input handlers may need specific configuration for handling
           fields. Only the email message handler currently has such a
...

 [stored]
 # Store mailmytag inside the document data record (so that it can be
 # displayed - as %(mailmytag) - in result lists).
 mailmytag =

 [queryaliases]
 filename = fn
 containerfilename = cfn

 [mail]
 # Extract the X-My-Tag mail header, and use it internally with the
 # mailmytag field name
 x-my-tag = mailmytag
...
   mydoc.doc.gz).

   The right side of each assignment holds a command to be executed for
   opening the file. The following substitutions are performed:

     * %D. Document date

     * %f. File name. This may be the name of a temporary file if it was
       necessary to create one (ie: to extract a subdocument from a
       container).



     * %i. Internal path, for subdocuments of containers. The format depends
       on the container type. If this appears in the command line, Recoll
       will not create a temporary file to extract the subdocument, expecting
       the called application (possibly a script) to be able to handle it.

     * %M. MIME type

     * %p. Page index. Only significant for a subset of document types,
       currently only PDF, Postscript and DVI files. Can be used to start the
       editor at the right page for a match or snippet.

     * %s. Search term. The value will only be set for documents with indexed
       page numbers (ie: PDF). The value will be one of the matched search
       terms. It would allow pre-setting the value in the "Find" entry inside
       Evince for example, for easy highlighting of the term.

     * %u. Url.

   In addition to the predefined values above, all strings like %(fieldname)
   will be replaced by the value of the field named fieldname for the
   document. This could be used in combination with field customisation to
   help with opening the document.
...
   the result list (when found by file name). The file names end in .blob and
   can be displayed by application blobviewer.

   You need two entries in the configuration files for this to work:

     * In $RECOLL_CONFDIR/mimemap (typically ~/.recoll/mimemap), add the
       following line:

 .blob = application/x-blobapp

       Note that the MIME type is made up here, and you could call it
       diesel/oil just the same.

     * In $RECOLL_CONFDIR/mimeview under the [view] section, add:

 application/x-blobapp = blobviewer %f

       We are supposing that blobviewer wants a file name parameter here, you
       would use %u if it liked URLs better.
...
   text and that you know how to extract it with a command line program.
   Getting Recoll to index the files is easy. You need to perform the above
   alteration, and also to add data to the mimeconf file (typically in
   ~/.recoll/mimeconf):

     * Under the [index] section, add the following line (more about the
       rclblob indexing script later):

 application/x-blobapp = exec rclblob

     * Under the [icons] section, you should choose an icon to be displayed
       for the files inside the result lists. Icons are normally 64x64 pixels
       PNG files which live in /usr/[local/]share/recoll/images.

     * Under the [categories] section, you should add the MIME type where it
       makes sense (you can also create a category). Categories may be used
       for filtering in advanced search.

   The rclblob handler should be an executable program or script which exists
   inside /usr/[local/]share/recoll/filters. It will be given a file name as