Open Source Projects Europe

                               Recoll user manual

   This document introduces full text search notions and describes the

   the installation and use of the Recoll application.

     ----------------------------------------------------------------------

                             4.2.1. Installing through a package system

     ----------------------------------------------------------------------

                            Chapter 1. Introduction

   If you do not like reading manuals (who does?) and would like to give

   give Recoll a try, just perform installation and start the recoll

   Recoll a try, just perform installation and start the recoll user

   user interface, which will index your home directory and let you

   interface, which will index your home directory and let you search it

   search it right after.

   Do not do this if your home has a huge number of documents and you do not

   do not want to wait or are very short on disk space. In this case,

   want to wait or are very short on disk space. In this case, you may want

   you may want to edit the configuration file first to restrict the

   to edit the configuration file first to restrict the indexed area.

   Also be aware that you will need to install the appropriate supporting

   supporting applications for document types that need them (for

   applications for document types that need them (for example antiword for

     ----------------------------------------------------------------------

   Recoll is a full text search application. Full text search applications

   matching documents, ordered so that the most relevant documents

   return a list of matching documents, ordered so that the most relevant

   documents will appear first.

   You do not need to remember in what file or email message you stored a

   stored a given piece of information. You just ask for related

   given piece of information. You just ask for related terms, and the tool

   This mode of operation has been made very familiar by internet search

   search engines.

   application can only try a guess. The quality of this guess is probably

   probably the most important element for a search application.

   In many cases, you are looking for all the forms of a word, not for a

   for a specific form or spelling. These different forms may include

   specific form or spelling. These different forms may include plurals,

   plurals, different tenses for a verb, or terms derived from the

   different tenses for a verb, or terms derived from the same root or stem

     ----------------------------------------------------------------------

   Recoll uses the Xapian information retrieval library as its storage and

   storage and retrieval engine. Xapian is a very mature package

   retrieval engine. Xapian is a very mature package using a sophisticated

   In practice, Xapian works by remembering where terms appear in your

   your document files. The acquisition process is called indexation.

   The resulting database can be big (roughly the size of the original

   original document set), but it is not a document archive. Recoll

   document set), but it is not a document archive. Recoll can only display

   Recoll stores all internal data in Unicode UTF-8 format, and it can index

   can index files with different character sets, encodings, and

   files with different character sets, encodings, and languages into the

   Stemming depends on the document language. Recoll stores the unstemmed

   unstemmed versions of terms and uses auxiliary databases for term

   versions of terms and uses auxiliary databases for term expansion. It can

   currently makes no attempt at automatic language recognition.

   Recoll has many parameters which define exactly what to index, and how to

   how to classify and decode the source documents. These are kept in

   classify and decode the source documents. These are kept in a

   a configuration file. A default configuration is copied into a

   configuration file. A default configuration is copied into a standard

   by default in the .recoll subdirectory of your home directory. The default

   configuration will index your home directory with default parameters and

   Indexation is started automatically the first time you execute the recoll

   recoll search graphical user interface, or by executing the

   search graphical user interface, or by executing the recollindex command.

   Searches are performed inside the recoll program, which has many options

   options to help you find what you are looking for.

     ----------------------------------------------------------------------

                             Chapter 2. Indexation

   Indexation is the process by which the set of documents is analyzed and

   analyzed and the data entered into the database. Recoll indexation

   the data entered into the database. Recoll indexation is normally

   is normally incremental: documents will only be processed if they

   incremental: documents will only be processed if they have been modified.

   Recoll indexation takes place at discrete times. There is currently no

   +------------------------------------------------------------------------+

   | Side note: there is nothing in Recoll and Xapian that would prevent    |

   | -i can be used to add individual files to the index if you     |

   | modified. recollindex -i can be used to add individual files to the    |

   | want to play with this, see the manual page.                   |

   | index if you want to play with this, see the manual page.              |

   +------------------------------------------------------------------------+

   Recoll knows about quite a few different document types. The parameters

   parameters for document types recognition and processing are set

   for document types recognition and processing are set in configuration

   Recoll indexation processes plain text, HTML, openoffice and e-mail files

   Without further configuration, Recoll will index all appropriate files

   files from your home directory, with a reasonable set of defaults.

     ----------------------------------------------------------------------

   /usr/[local/]share/recoll/examples/recoll.conf) can be overriden by those

   by those set in the personal one, named $HOME/.recoll/recoll.conf

   set in the personal one, named $HOME/.recoll/recoll.conf by default or

   by default or $RECOLL_CONFDIR/recoll.conf if RECOLL_CONFDIR is

   $RECOLL_CONFDIR/recoll.conf if RECOLL_CONFDIR is set.

   The most accurate documentation for editing the file is given by comments

   comments inside the central one. If you want to adjust the

   inside the central one. If you want to adjust the configuration before

   The configuration is also documented inside the installation chapter of

   chapter of this document, or in the recoll.conf(5) man page.

     ----------------------------------------------------------------------

   Indexation is performed either by the recollindex program, or by the

   the indexation thread inside the recoll program (use the File

   indexation thread inside the recoll program (use the File menu).

   It is best to avoid interrupting the indexation process, as this may

   may sometimes leave the database in a bad state. This is not a

   sometimes leave the database in a bad state. This is not a serious

   serious problem, as you then just need to clear everything and

   problem, as you then just need to clear everything and restart the

   restart the indexation: the database files are normally stored in

   indexation: the database files are normally stored in the

   the $HOME/.recoll/xapiandb directory, which you can just delete if

   $HOME/.recoll/xapiandb directory, which you can just delete if needed.

   needed. Alternatively, you can start recollindex -z, which will

   Alternatively, you can start recollindex -z, which will reset the database

     ----------------------------------------------------------------------

   The most common way to set up indexation is to have a cron task execute it

   execute it every night. For example the following crontab entry

   every night. For example the following crontab entry would do it every day

   would do it every day at 3:30AM (supposing recollindex is in your

   at 3:30AM (supposing recollindex is in your PATH):

   The usual command to edit your crontab is crontab -e (which will usually

   usually start the vi editor to edit the file). You may have more

   start the vi editor to edit the file). You may have more sophisticated

   sophisticated tools available on your system.

     ----------------------------------------------------------------------

                               Chapter 3. Search

   The recoll program provides the user interface for searching. It is based

   is based on the QT library.

     ----------------------------------------------------------------------

    2. Possibly choose a search mode: Any term or All terms or File name.

    3. Enter search term(s) in the text field at the top of the window.

    4. Click the Search button or hit the Enter key to start the search.

   The initial default search mode is Any term. This will look for documents

   documents with any of the search terms (the ones with more terms

   with any of the search terms (the ones with more terms will get better

   You can use the Tools / Advanced search dialog for more complex searches.

   After starting a search, a list of results will instantly be displayed in

   displayed in the main list window. Clicking on the Preview link

   the main list window. Clicking on the Preview link for an entry will open

   By default, the document list is presented in order of relevance (how well

   (how well the system estimates that the document matches the

   the system estimates that the document matches the query). You can specify

   The Preview and Edit edit links may not be present for all entries,

   entries, meaning that Recoll has no configured way to preview a

   meaning that Recoll has no configured way to preview a given file type

   You can click on the Query details link at the top of the results page to

   page to see the query actually performed, after stem expansion and

   see the query actually performed, after stem expansion and other

   other processing.

     ----------------------------------------------------------------------

   The advanced search dialog has fields that will allow a more refined

   refined search, looking for documents with all given words, a

   search, looking for documents with all given words, a given exact phrase,

   It will let you search for documents of specific mime types (ie: only

   only text/plain, or text/html or application/pdf etc...)

   It will let you restrict the search results to a subtree of the indexed

   Click on the Start Search button in the advanced search dialog to start

   start the search. The button in the main window always performs a

   the search. The button in the main window always performs a simple search.

   Click on the Show query details link at the top of the result page to see

   to see the query expansion.

     ----------------------------------------------------------------------

   Documents that you actually view (with the internal preview or an external

   external tool) are entered into the document history, which is

   tool) are entered into the document history, which is remembered. You can

     ----------------------------------------------------------------------

   The documents in a result list are normally sorted in order of relevance.

   relevance. It is possible to specify different sort parameters by

   It is possible to specify different sort parameters by using the Sort

   using the Sort parameters dialog (located in the Tools menu).

   The tool sorts a specified number of the most relevant documents in the

   in the result list, according to specified criteria. The currently

   result list, according to specified criteria. The currently available

   available criteria are date and mime type.

   The sort parameters stay in effect until they are explicitely reset, or

   reset, or the program exits. An activated sort is indicated in the

   the program exits. An activated sort is indicated in the result list

     ----------------------------------------------------------------------

   Disabling stem expansion. Entering a capitalized word in any search field

   search field will prevent stem expansion (no search for gardening

   will prevent stem expansion (no search for gardening if you enter Garden

   Phrases. A phrase can be looked for by enclosing it in double quotes.

   quotes. Example: "user manual" will look only for occurrences of

   Example: "user manual" will look only for occurrences of user immediately

   phrase field of the advanced search dialog to the same effect.

   Query explanation. You can get an exact description of what the query

   query looked for, including stem expansion, and boolean operators

   looked for, including stem expansion, and boolean operators used, by

   used, by clicking on the result list header.

   File names. All file name elements (the broken up file path) are entered

   entered as terms during indexation, and you can specify them as

   as terms during indexation, and you can specify them as ordinary terms in

   Closing previews. Entering ^W in a preview tab will close it (and, for the

   for the last tab, close the preview window).

     ----------------------------------------------------------------------

   It is possible to customise some aspects of the search interface by using

   by using Query configuration entry in the Preferences menu.

   There are two tabs in the dialog, dealing with the interface itself, and

   itself, and with the parameters used for searching and returning

   with the parameters used for searching and returning results.

     * Result list font: There is quite a lot of information shown in the

       the result list, and you may want to customise the font and/or

       result list, and you may want to customise the font and/or font size.

       font size. The rest of the fonts used by Recoll are determined

       The rest of the fonts used by Recoll are determined by your generic QT

       by your generic QT config (try the qtconfig command.

     * Html help browser: this will let you chose your the preferred browser

       browser which will be started from the Help menu to read the

       which will be started from the Help menu to read the user manual. You

     * Show document type icons in result list: icons in the result list can

       list can be turned off. They take quite a lot of space and

       be turned off. They take quite a lot of space and convey relatively

       convey relatively little useful information.

     * Stemming language: stemming obviously depends on the document's

       document's language. This listbox will let you chose among the

       language. This listbox will let you chose among the stemming databases

       next indexation pass unless they are also added in the

       will be deleted at the next indexation pass unless they are also added

       in the configuration file.

     * Dynamically build abstracts: this decides if Recoll tries to build

       build document abstracts when displaying the result list.

       document abstracts when displaying the result list. Abstracts are

       Abstracts are constructed by taking context from the document

       constructed by taking context from the document information, around

     * Replace abstracts from documents: this decides if we should synthetize

       synthetize and display an abstract in place of an explicit

       and display an abstract in place of an explicit abstract found within

     ----------------------------------------------------------------------

                            Chapter 4. Installation

   At the very least, you will need to download and install the xapian core

   xapian core package (Recoll currently uses version 0.9.2), and the

   package (Recoll currently uses version 0.9.2), and the qt runtime and

   You will most probably be able to find a binary package for qt for your

   your system. You may have to compile Xapian but this is not

   system. You may have to compile Xapian but this is not difficult (if you

   difficult (if you are using FreeBSD, there is a port).

   You may also need libiconv. Recoll currently uses version 1.9 (this should

   External file types. Recoll uses external applications to index some file

   some file types. You need to install them for the file types that

   types. You need to install them for the file types that you wish to have

     * MP3: Recoll will use the id3info command from the id3lib package to

       package to extract tag information. Without it, only the

       extract tag information. Without it, only the filenames will be

   Text, Html, mail folders and Openoffice files are processed internally.

     ----------------------------------------------------------------------

   Recoll has been built on Linux (redhat7.3, mandriva 2005, Fedora Core 3),

   Core 3), FreeBSD and Solaris 8. If you build on another system, I

   FreeBSD and Solaris 8. If you build on another system, I would very much

   Depending on the qt configuration on your system, you may have to set the

   set the QTDIR and QMAKESPECS variables in your environment:

     * QTDIR should point to the directory above the one that holds the qt

       the qt include files (ie: qt.h).

   On many Linux systems, QTDIR is set by the login scripts, and QMAKESPECS

   QMAKESPECS is not needed because there is a default link in

   is not needed because there is a default link in mkspecs/.

   The Recoll configure script does a better job of checking these variables

   variables after release 1.1.1. Before this, unexplained errors

   after release 1.1.1. Before this, unexplained errors will occur during

   There little autoconfiguration. The configure script will mainly link one

   link one of the system-specific files in the mk directory to

   of the system-specific files in the mk directory to mk/sysconf. If your

     ----------------------------------------------------------------------

   Either type make install or execute recollinstall prefix, in the root of

   root of the source tree. This will copy the commands to prefix/bin

   the source tree. This will copy the commands to prefix/bin and the sample

     ----------------------------------------------------------------------

   If you are lucky enough to be using a port system or a prebuilt package

   package (RPM or other), just follow the usual procedure, and have

   (RPM or other), just follow the usual procedure, and have a look at the

   a look at the configuration section.

     ----------------------------------------------------------------------

   The unpackaged binary versions are just compressed tar files of a build

   build tree, where only the useful parts were kept (executables and

   tree, where only the useful parts were kept (executables and sample

   sample configuration).

   The executable binary files are built with a static link to libxapian and

   After extracting the tar file, you can proceed with installation as if you

   as if you had built the package from source.

     ----------------------------------------------------------------------

   There are two sets of configuration files. The system-wide files are kept

   the system. A parallel set of files exists in the .recoll

   default values for the system. A parallel set of files exists in the

   .recoll directory in your home (this can be changed with the

   RECOLL_CONFDIR environment variable. The database is also kept in .recoll

   .recoll by default, (this can be changed by a configuration

   by default, (this can be changed by a configuration parameter).

   If the .recoll directory does not exist when recoll or recollindex are

   are started, it will be created with a set of empty configuration

   started, it will be created with a set of empty configuration files.

   files. recoll will give you a chance to edit the configuration

   recoll will give you a chance to edit the configuration file before

   file before starting indexation. recollindex will proceed

   starting indexation. recollindex will proceed immediately.

   Most of the parameters specific to the recoll GUI are set through the

   the Preferences menu and stored in the standard QT place

   Preferences menu and stored in the standard QT place ($HOME/.qt/recollrc).

   ($HOME/.qt/recollrc). You probably do not want to edit this by

   You probably do not want to edit this by hand.

   For other options, Recoll uses text configuration files. You will have to

   have to edit them by hand for now (there is still some hope for a

   edit them by hand for now (there is still some hope for a GUI

   GUI configuration tool in the future). The most accurate

   configuration tool in the future). The most accurate documentation for the

   All configuration files share the same format. For exemple, a short

   short extract of the main configuration file might look as

   extract of the main configuration file might look as follows:

   Section lines allow redefining some parameters for a directory subtree.

   subtree. Some of the parameters used for indexation are looked up

   Some of the parameters used for indexation are looked up hierarchically

   The tilde character (~) is expanded in file names to the name of the

   the user's home directory.

   White space is used for separation inside lists. Elements with embedded

   embedded spaces can be quoted using double-quotes.

     ----------------------------------------------------------------------

   recoll.conf is the main configuration file. It defines things like what to

   what to index (top directories and things to ignore), and the

   index (top directories and things to ignore), and the default character

   The default configuration will index your home directory. If this is not

   is not appropriate, use recoll to copy the sample configuration,

   appropriate, use recoll to copy the sample configuration, click Cancel,

           directories that should be completely ignored. The list defined in

           defined in the default file is:

           The list can be redefined for subdirectories, but is only actually

           actually changed for the top level ones in topdirs.

           The top-level directories are not affected by this list (that is,

           (that is, a directory in topdirs might match and would

           a directory in topdirs might match and would still be indexed).

           The list in the default configuration does not exclude hidden

           hidden directories (names beginning with a dot), which

           directories (names beginning with a dot), which means that it may

           solution is to have .* in skippedNames, and add things like

           Verbosity level for recoll and recollindex. A value of 4 lists

           lists quite a lot of debug/information messages. 2 only

           quite a lot of debug/information messages. 2 only lists errors.

           Where should the messages go. 'stderr' can be used as a special

           special value.

           A directory to search for the external filter scripts used to

           to index some types of files. The value should not be

           index some types of files. The value should not be changed, except

           scripts. The value can be redefined for any subdirectory.

           A list of languages for which the stem expansion databases will be

           will be built. See recollindex(1) for possible values. You

           built. See recollindex(1) for possible values. You can add a stem

           can add a stem expansion database for a different language

           expansion database for a different language by using recollindex

           languages listed in the configuration file are permanent.

           The name of the directory where recoll result list icons are

           are stored. You can change this if you want different

           stored. You can change this if you want different images.

           The name of the Xapian database directory. It will be created if

           created if needed when the database is initialized.

           The name of the character set used for files that do not contain a

           contain a character set definition (ie: plain text files).

           character set definition (ie: plain text files). This can be

           This can be redefined for any subdirectory. If it is not

           redefined for any subdirectory. If it is not set at all, the

           internal value is available (ie: for plain text files). This does

           This does not work well in general, and should probably

           not work well in general, and should probably not be used.

           Decide if we use the file -i system command as a final step for

           step for determining the mime type for a file (the main

           determining the mime type for a file (the main procedure uses

           suffixless names, but it will also cause the indexation of

           useful for files with suffixless names, but it will also cause the

           indexation of many bogus "text" files.

           Recoll indexes file names in a special section of the database to

           database to allow specific file names searches using wild

           allow specific file names searches using wild cards. This

           cards. This parameter decides if file name indexing is

           parameter decides if file name indexing is performed only for

     ----------------------------------------------------------------------

   For file names without an extension, or with an unknown one, the system's

   system's file -i command will be executed to determine the mime

   file -i command will be executed to determine the mime type (this can be

   type (this can be switched off inside the main configuration

   switched off inside the main configuration file).

   mimemap also has a list of extensions which should be ignored totally (to

   totally (to avoid losing time by executing file for things that

   avoid losing time by executing file for things that certainly should not

   The mappings can be specified on a per-subtree basis, which may be useful

   useful in some cases. Example: gaim logs have a .txt extension but

   in some cases. Example: gaim logs have a .txt extension but should be

   should be handled specially, which is possible because they are

   handled specially, which is possible because they are usually all located

   mimemap also has a recoll_noindex variable which is a list of suffixes.

   suffixes. Matching files will be skipped (avoids unnecessary

   Matching files will be skipped (avoids unnecessary decompressions or file

     ----------------------------------------------------------------------

   Changing the indexation parameters is probably not a good idea except if

   except if you are a Recoll developper.

   You may want to adjust the external viewers defined in (ie: html is either

   is either previewed internally or displayed using firefox, but you

   previewed internally or displayed using firefox, but you may prefer

   may prefer mozilla, your openoffice.org program might be named

   mozilla, your openoffice.org program might be named oofice instead of

   oofice instead of openoffice ...). Look for the [view] section.

   You can also change the icons which are displayed by recoll in the result

   result lists (the values are the basenames of the png images

   lists (the values are the basenames of the png images inside the iconsdir

   inside the iconsdir directory (specified in recoll.conf).

     ----------------------------------------------------------------------