Open Source Projects Europe

  Jean-Francois Dockes

  Jean-Francois Dockes

   <jfd@recoll.org>

   <jfd@recoll.org>

   Permission is granted to copy, distribute and/or modify this document

   Permission is granted to copy, distribute and/or modify this document

   under the terms of the GNU Free Documentation License, Version 1.3 or any

   under the terms of the GNU Free Documentation License, Version 1.3 or any

   later version published by the Free Software Foundation; with no Invariant

   later version published by the Free Software Foundation; with no Invariant

   Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the

   Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the

   license can be found at the following location: GNU web site.

   license can be found at the following location: GNU web site.

   This document introduces full text search notions and describes the

   This document introduces full text search notions and describes the

     ----------------------------------------------------------------------

     ----------------------------------------------------------------------

   Table of Contents

   Table of Contents

                             2.1.2. Configurations, multiple indexes

                             2.1.2. Configurations, multiple indexes

                             2.1.3. Document types

                             2.1.3. Document types

                2.2. Index storage

                2.2. Index storage

                             2.2.1. Xapian index formats

                             2.2.1. Xapian index formats

                             3.1.12. Sorting search results and collapsing

                             3.1.12. Sorting search results and collapsing

                             duplicates

                             duplicates

                             3.1.13. Search tips, shortcuts

                             3.1.13. Search tips, shortcuts

                3.2. Searching with the KDE KIO slave

                3.2. Searching with the KDE KIO slave

                             3.2.1. What's this

                             3.2.1. What's this

   5. Installation and configuration

   5. Installation and configuration

                5.1. Installing a binary copy

                5.1. Installing a binary copy

                5.2. Supporting packages

                5.2. Supporting packages

                5.3. Building from source

                5.3. Building from source

                             5.3.1. Prerequisites

                             5.3.1. Prerequisites

                             5.3.3. Installation

                             5.3.3. Installation

                5.4. Configuration overview

                5.4. Configuration overview

Chapter 1. Introduction

Chapter 1. Introduction

1.1. Giving it a try

1.1. Giving it a try

   documents will only be processed if they have been modified since the last

   documents will only be processed if they have been modified since the last

   run. On the first execution, all documents will need processing. A full

   run. On the first execution, all documents will need processing. A full

   index build can be forced later by specifying an option to the indexing

   index build can be forced later by specifying an option to the indexing

   command (recollindex -z or -Z).

   command (recollindex -z or -Z).

   The following sections give an overview of different aspects of the

   The following sections give an overview of different aspects of the

   indexing processes and configuration, with links to detailed sections.

   indexing processes and configuration, with links to detailed sections.

  2.1.1. Indexing modes

  2.1.1. Indexing modes

   Recoll indexing can be performed along two different modes:

   Recoll indexing can be performed along two different modes:

   excludedmimetypes or indexedmimetypes, can be set either by editing the

   excludedmimetypes or indexedmimetypes, can be set either by editing the

   main configuration file (recoll.conf), or from the GUI index configuration

   main configuration file (recoll.conf), or from the GUI index configuration

   tool.

   tool.

   In the rare case where the index becomes corrupted (which can signal

   In the rare case where the index becomes corrupted (which can signal

   itself by weird search results or crashes), the index files need to be

   itself by weird search results or crashes), the index files need to be

   erased before restarting a clean indexing pass. Just delete the xapiandb

   erased before restarting a clean indexing pass. Just delete the xapiandb

   directory (see next section), or, alternatively, start the next

   directory (see next section), or, alternatively, start the next

   index first. This will not have the "clean start" aspect of -z, but the

   index first. This will not have the "clean start" aspect of -z, but the

   advantage is that the index will remain available for querying while it is

   advantage is that the index will remain available for querying while it is

   rebuilt, which can be a significant advantage if it is very big (some

   rebuilt, which can be a significant advantage if it is very big (some

   installations need days for a full index rebuild).

   installations need days for a full index rebuild).

   Of special interest also, maybe, are the -i and -f options. -i allows

   Of special interest also, maybe, are the -i and -f options. -i allows

   indexing an explicit list of files (given as command line parameters or

   indexing an explicit list of files (given as command line parameters or

   read on stdin). -f tells recollindex to ignore file selection parameters

   read on stdin). -f tells recollindex to ignore file selection parameters

   from the configuration. Together, these options allow building a custom

   from the configuration. Together, these options allow building a custom

   file selection process for some area of the file system, by adding the top

   file selection process for some area of the file system, by adding the top

   If you use the daemon completely out of an X11 session, you need to add

   If you use the daemon completely out of an X11 session, you need to add

   option -x to disable X11 session monitoring (else the daemon will not

   option -x to disable X11 session monitoring (else the daemon will not

   start).

   start).

   depending on the log level.

   When building Recoll, the real time indexing support can be customised

   When building Recoll, the real time indexing support can be customised

   during package configuration with the --with[out]-fam or

   during package configuration with the --with[out]-fam or

   --with[out]-inotify options. The default is currently to include inotify

   --with[out]-inotify options. The default is currently to include inotify

   monitoring on systems that support it, and, as of Recoll 1.17, gamin

   monitoring on systems that support it, and, as of Recoll 1.17, gamin

   printed is for east-asian languages (Chinese, Japanese, Korean). Words

   printed is for east-asian languages (Chinese, Japanese, Korean). Words

   composed of single or multiple characters should be entered separated by

   composed of single or multiple characters should be entered separated by

   white space in this case (they would typically be printed without white

   white space in this case (they would typically be printed without white

   space).

   space).

  3.1.1. Simple search

  3.1.1. Simple search

    1. Start the recoll program.

    1. Start the recoll program.

    2. Possibly choose a search mode: Any term, All terms, File name or Query

    2. Possibly choose a search mode: Any term, All terms, File name or Query

  3.1.8. Complex/advanced search

  3.1.8. Complex/advanced search

   The advanced search dialog helps you build more complex queries without

   The advanced search dialog helps you build more complex queries without

   memorizing the search language constructs. It can be opened through the

   memorizing the search language constructs. It can be opened through the

   Tools menu or through the main toolbar.

   Tools menu or through the main toolbar.

   The dialog has two tabs:

   The dialog has two tabs:

    1. The first tab lets you specify terms to search for, and permits

    1. The first tab lets you specify terms to search for, and permits

       specifying multiple clauses which are combined to build the search.

       specifying multiple clauses which are combined to build the search.

   Printing previews. Entering Ctrl-P in a preview window will print the

   Printing previews. Entering Ctrl-P in a preview window will print the

   currently displayed text.

   currently displayed text.

   Quitting. Entering Ctrl-Q almost anywhere will close the application.

   Quitting. Entering Ctrl-Q almost anywhere will close the application.

   You can customize some aspects of the search interface by using the GUI

   You can customize some aspects of the search interface by using the GUI

   configuration entry in the Preferences menu.

   configuration entry in the Preferences menu.

   There are several tabs in the dialog, dealing with the interface itself,

   There are several tabs in the dialog, dealing with the interface itself,

   always implicitly active. If this is not desirable, you can set up your

   always implicitly active. If this is not desirable, you can set up your

   configuration so that it indexes, for example, an empty directory. An

   configuration so that it indexes, for example, an empty directory. An

   alternative indexer may also need to implement a way of purging the index

   alternative indexer may also need to implement a way of purging the index

   from stale data,

   from stale data,

   The result list presentation can be exhaustively customized by adjusting

   The result list presentation can be exhaustively customized by adjusting

   two elements:

   two elements:

     o The paragraph format

     o The paragraph format

      The paragraph format

      The paragraph format

   This is an arbitrary HTML string where the following printf-like %

   This is an arbitrary HTML string where the following printf-like %

   substitutions will be performed:

   substitutions will be performed:

   example candidate would be the recipient field which is generated by the

   example candidate would be the recipient field which is generated by the

   message input handlers.

   message input handlers.

   The default value for the paragraph format string is:

   The default value for the paragraph format string is:

   You may, for example, try the following for a more web-like experience:

   You may, for example, try the following for a more web-like experience:

 <u><b><a href="P%N">%T</a></b></u><br>

 <u><b><a href="P%N">%T</a></b></u><br>

 %A<font color=#008000>%U - %S</font> - %L

 %A<font color=#008000>%U - %S</font> - %L

   or lennon and either live or unplugged but not potatoes (in any part of

   or lennon and either live or unplugged but not potatoes (in any part of

   the document).

   the document).

   An element is composed of an optional field specification, and a value,

   An element is composed of an optional field specification, and a value,

   separated by a colon (the field separator is the last colon in the

   separated by a colon (the field separator is the last colon in the

   element). Example: Eugenie, author:balzac, dc:title:grandet

   The colon, if present, means "contains". Xesam defines other relations,

   The colon, if present, means "contains". Xesam defines other relations,

   which are mostly unsupported for now (except in special cases, described

   which are mostly unsupported for now (except in special cases, described

   further down).

   further down).

   Beatles OR Lennon. The OR must be entered literally (capitals), and it has

   Beatles OR Lennon. The OR must be entered literally (capitals), and it has

   priority over the AND associations: word1 word2 OR word3 means word1 AND

   priority over the AND associations: word1 word2 OR word3 means word1 AND

   (word2 OR word3) not (word1 AND word2) OR word3. Explicit parenthesis are

   (word2 OR word3) not (word1 AND word2) OR word3. Explicit parenthesis are

   not supported.

   not supported.

   An element preceded by a - specifies a term that should not appear.

   As usual, words inside quotes define a phrase (the order of words is

   As usual, words inside quotes define a phrase (the order of words is

   significant), so that title:"prejudice pride" is not the same as

   significant), so that title:"prejudice pride" is not the same as

   title:prejudice title:pride, and is unlikely to find a result.

   title:prejudice title:pride, and is unlikely to find a result.

   To save you some typing, recent Recoll versions (1.20 and later) interpret

   To save you some typing, recent Recoll versions (1.20 and later) interpret

   a comma-separated list of terms as an AND list inside the field. Use slash

   a comma-separated list of terms as an AND list inside the field. Use slash

   characters ('/') for an OR list. No white space is allowed. So

   characters ('/') for an OR list. No white space is allowed. So

 author:john,lennon

 author:john,lennon

 author:john/ringo

 author:john/ringo

   would search for john or ringo.

   would search for john or ringo.

   proximity search (unordered). See the modifier section.

   Recoll currently manages the following default fields:

   Recoll currently manages the following default fields:

     o title, subject or caption are synonyms which specify data to be

     o title, subject or caption are synonyms which specify data to be

       searched for in the document title or subject.

       searched for in the document title or subject.

       text/media/presentation/etc.). The classification of MIME types in

       text/media/presentation/etc.). The classification of MIME types in

       categories is defined in the Recoll configuration (mimeconf), and can

       categories is defined in the Recoll configuration (mimeconf), and can

       be modified or extended. The default category names are those which

       be modified or extended. The default category names are those which

       permit filtering results in the main GUI screen. Categories are OR'ed

       permit filtering results in the main GUI screen. Categories are OR'ed

       like MIME types above. This can't be negated with - either.

       like MIME types above. This can't be negated with - either.

   The document input handlers used while indexing have the possibility to

   The document input handlers used while indexing have the possibility to

   create other fields with arbitrary names, and aliases may be defined in

   create other fields with arbitrary names, and aliases may be defined in

   the configuration, so that the exact field search possibilities may be

   the configuration, so that the exact field search possibilities may be

   different for you if someone took care of the customisation.

   different for you if someone took care of the customisation.

Chapter 5. Installation and configuration

Chapter 5. Installation and configuration

5.1. Installing a binary copy

5.1. Installing a binary copy

   by Recoll (text, HTML, email files, and a few others).

   You should also maybe have a look at the configuration section (but this

   You should also maybe have a look at the configuration section (but this

   may not be necessary for a quick test with default parameters). Most

   may not be necessary for a quick test with default parameters). Most

   parameters can be more conveniently set from the GUI interface.

   parameters can be more conveniently set from the GUI interface.

5.2. Supporting packages

5.2. Supporting packages

   Recoll uses external applications to index some file types. You need to

   Recoll uses external applications to index some file types. You need to

   install them for the file types that you wish to have indexed (these are

   install them for the file types that you wish to have indexed (these are

     o Of course the usual autoconf configure options, like --prefix apply.

     o Of course the usual autoconf configure options, like --prefix apply.

   Normal procedure:

   Normal procedure:

         cd recoll-xxx

         cd recoll-xxx

         configure

         make

         make

         (practices usual hardship-repelling invocations)

         (practices usual hardship-repelling invocations)

   There is little auto-configuration. The configure script will mainly link

   There is little auto-configuration. The configure script will mainly link

       handle multiple encodings in a single file. In this relatively

       handle multiple encodings in a single file. In this relatively

       unlikely case, you can edit the configuration file as two separate

       unlikely case, you can edit the configuration file as two separate

       text files with appropriate encodings, and concatenate them to create

       text files with appropriate encodings, and concatenate them to create

       the complete configuration.

       the complete configuration.

   recoll.conf is the main configuration file. It defines things like what to

   recoll.conf is the main configuration file. It defines things like what to

   index (top directories and things to ignore), and the default character

   index (top directories and things to ignore), and the default character

   set to use for document types which do not specify it internally.

   set to use for document types which do not specify it internally.

   Most of the following parameters can be changed from the Index

   Most of the following parameters can be changed from the Index

   Configuration menu in the recoll interface. Some can only be set by

   Configuration menu in the recoll interface. Some can only be set by

   editing the configuration file.

   editing the configuration file.

   topdirs

   topdirs

           Specifies the list of directories or files to index (recursively

           Specifies the list of directories or files to index (recursively

           for directories). You can use symbolic links as elements of this

           for directories). You can use symbolic links as elements of this

           hidden directories, and you probably want this indexed. One

           hidden directories, and you probably want this indexed. One

           possible solution is to have .* in skippedNames, and add things

           possible solution is to have .* in skippedNames, and add things

           like ~/.thunderbird or ~/.evolution in topdirs.

           like ~/.thunderbird or ~/.evolution in topdirs.

           Not even the file names are indexed for patterns in this list. See

           Not even the file names are indexed for patterns in this list. See

           indexes the file names.

   skippedPaths and daemSkippedPaths

   skippedPaths and daemSkippedPaths

           A space-separated list of patterns for paths of files or

           A space-separated list of patterns for paths of files or

           directories that should be skipped. There is no default in the

           directories that should be skipped. There is no default in the

           The path to the web indexing queue. This is hard-coded in the

           The path to the web indexing queue. This is hard-coded in the

           Firefox plugin as ~/.recollweb/ToIndex so there should be no need

           Firefox plugin as ~/.recollweb/ToIndex so there should be no need

           to change it.

           to change it.

   Changing some of these parameters will imply a full reindex. Also, when

   Changing some of these parameters will imply a full reindex. Also, when

   using multiple indexes, it may not make sense to search indexes that don't

   using multiple indexes, it may not make sense to search indexes that don't

   share the values for these parameters, because they usually affect both

   share the values for these parameters, because they usually affect both

   search and index operations.

   search and index operations.

 field2 = value for field2

 field2 = value for field2

           field1 and field2 will be set inside the document metadata.

           field1 and field2 will be set inside the document metadata.

   dbdir

   dbdir

           The name of the Xapian data directory. It will be created if

           The name of the Xapian data directory. It will be created if

           needed when the index is initialized. If this is not an absolute

           needed when the index is initialized. If this is not an absolute

           usage also depends on average document size. The default value is

           usage also depends on average document size. The default value is

           10, and it is probably a bit low. If your system usually has free

           10, and it is probably a bit low. If your system usually has free

           memory, you can try higher values between 20 and 80. In my

           memory, you can try higher values between 20 and 80. In my

           experience, values beyond 100 are always counterproductive.

           experience, values beyond 100 are always counterproductive.

   The Recoll indexing process recollindex can use multiple threads to speed

   The Recoll indexing process recollindex can use multiple threads to speed

   up indexing on multiprocessor systems. The work done to index files is

   up indexing on multiprocessor systems. The work done to index files is

   divided in several stages and some of the stages can be executed by

   divided in several stages and some of the stages can be executed by

   multiple threads. The stages are:

   multiple threads. The stages are:

   The following example would disable multithreading. Indexing will be

   The following example would disable multithreading. Indexing will be

   performed by a single thread.

   performed by a single thread.

 thrQSizes = -1 -1 -1

 thrQSizes = -1 -1 -1

   autodiacsens

   autodiacsens

           IF the index is not stripped, decide if we automatically trigger

           IF the index is not stripped, decide if we automatically trigger

           diacritics sensitivity if the search term has accented characters

           diacritics sensitivity if the search term has accented characters

   logfilename, daemlogfilename

   logfilename, daemlogfilename

           Where the messages should go. 'stderr' can be used as a special

           Where the messages should go. 'stderr' can be used as a special

           value, and is the default. The daemversion is specific to the

           value, and is the default. The daemversion is specific to the

           indexing monitor daemon.

           indexing monitor daemon.

   mondelaypatterns

   mondelaypatterns

           This allows specify wildcard path patterns (processed with

           This allows specify wildcard path patterns (processed with

           fnmatch(3) with 0 flag), to match files which change too often and

           fnmatch(3) with 0 flag), to match files which change too often and

           This allows definining location-related quirks for the mailbox

           This allows definining location-related quirks for the mailbox

           handler. Currently only the tbird flag is defined, and it should

           handler. Currently only the tbird flag is defined, and it should

           be set for directories which hold Thunderbird data, as their

           be set for directories which hold Thunderbird data, as their

           folder format is weird.

           folder format is weird.

   This file contains information about dynamic fields handling in Recoll.

   This file contains information about dynamic fields handling in Recoll.

   Some very basic fields have hard-wired behaviour, and, mostly, you should

   Some very basic fields have hard-wired behaviour, and, mostly, you should

   not change the original data inside the fields file. But you can create

   not change the original data inside the fields file. But you can create

   custom fields fitting your data and handle them just like they were native

   custom fields fitting your data and handle them just like they were native

 [mail]

 [mail]

 # Extract the X-My-Tag mail header, and use it internally with the

 # Extract the X-My-Tag mail header, and use it internally with the

 # mailmytag field name

 # mailmytag field name

 x-my-tag = mailmytag

 x-my-tag = mailmytag

   Recoll versions 1.19 and later process user extended file attributes as

   Recoll versions 1.19 and later process user extended file attributes as

   documents fields by default.

   documents fields by default.

   Attributes are processed as fields of the same name, after removing the

   Attributes are processed as fields of the same name, after removing the

   The [xattrtofields] section of the fields file allows specifying

   The [xattrtofields] section of the fields file allows specifying

   translations from extended attributes names to Recoll field names. An

   translations from extended attributes names to Recoll field names. An

   empty translation disables use of the corresponding attribute data.

   empty translation disables use of the corresponding attribute data.

   mimemap specifies the file name extension to MIME type mappings.

   mimemap specifies the file name extension to MIME type mappings.

   For file names without an extension, or with an unknown one, the system's

   For file names without an extension, or with an unknown one, the system's

   file -i command will be executed to determine the MIME type (this can be

   file -i command will be executed to determine the MIME type (this can be

   The mappings can be specified on a per-subtree basis, which may be useful

   The mappings can be specified on a per-subtree basis, which may be useful

   in some cases. Example: gaim logs have a .txt extension but should be

   in some cases. Example: gaim logs have a .txt extension but should be

   handled specially, which is possible because they are usually all located

   handled specially, which is possible because they are usually all located

   in one place.

   in one place.

   mimeconf specifies how the different MIME types are handled for indexing,

   mimeconf specifies how the different MIME types are handled for indexing,

   and which icons are displayed in the recoll result lists.

   and which icons are displayed in the recoll result lists.

   Changing the parameters in the [index] section is probably not a good idea

   Changing the parameters in the [index] section is probably not a good idea

   The [icons] section allows you to change the icons which are displayed by

   The [icons] section allows you to change the icons which are displayed by

   recoll in the result lists (the values are the basenames of the png images

   recoll in the result lists (the values are the basenames of the png images

   inside the iconsdir directory (specified in recoll.conf).

   inside the iconsdir directory (specified in recoll.conf).

   mimeview specifies which programs are started when you click on an Open

   mimeview specifies which programs are started when you click on an Open

   link in a result list. Ie: HTML is normally displayed using firefox, but

   link in a result list. Ie: HTML is normally displayed using firefox, but

   you may prefer Konqueror, your openoffice.org program might be named

   you may prefer Konqueror, your openoffice.org program might be named

   oofice instead of openoffice etc.

   oofice instead of openoffice etc.

   In addition to the predefined values above, all strings like %(fieldname)

   In addition to the predefined values above, all strings like %(fieldname)

   will be replaced by the value of the field named fieldname for the

   will be replaced by the value of the field named fieldname for the

   document. This could be used in combination with field customisation to

   document. This could be used in combination with field customisation to

   help with opening the document.

   help with opening the document.

   ptrans specifies query-time path translations. These can be useful in

   ptrans specifies query-time path translations. These can be useful in

   multiple cases.

   multiple cases.

   The file has a section for any index which needs translations, either the

   The file has a section for any index which needs translations, either the

           [/path/to/additional/xapiandb]

           [/path/to/additional/xapiandb]

           /server/volume1/docdir = /net/server/volume1/docdir

           /server/volume1/docdir = /net/server/volume1/docdir

           /server/volume2/docdir = /net/server/volume2/docdir

           /server/volume2/docdir = /net/server/volume2/docdir

   Imagine that you have some kind of file which does not have indexable

   Imagine that you have some kind of file which does not have indexable

   content, but for which you would like to have a functional Open link in

   content, but for which you would like to have a functional Open link in

   the result list (when found by file name). The file names end in .blob and

   the result list (when found by file name). The file names end in .blob and

   can be displayed by application blobviewer.

   can be displayed by application blobviewer.

   MIME type which it already knows, you would just need to edit mimeview.

   MIME type which it already knows, you would just need to edit mimeview.

   The entries you add in your personal file override those in the central

   The entries you add in your personal file override those in the central

   configuration, which you do not need to alter. mimeview can also be

   configuration, which you do not need to alter. mimeview can also be

   modified from the Gui.

   modified from the Gui.

   Let us now imagine that the above .blob files actually contain indexable

   Let us now imagine that the above .blob files actually contain indexable

   text and that you know how to extract it with a command line program.

   text and that you know how to extract it with a command line program.

   Getting Recoll to index the files is easy. You need to perform the above

   Getting Recoll to index the files is easy. You need to perform the above

   alteration, and also to add data to the mimeconf file (typically in

   alteration, and also to add data to the mimeconf file (typically in