recoll / Code / Diff of /src/README

Diff of /src/README [6d5b04] .. [9238d6]

Switch to unified view

-a/src/README
+b/src/README
 ...
 .4.2. Using cron to automate indexing
 .5. Real time indexing
-. Search
+. Searching
 .1. Simple search
 .2. The result list
 .2.1. The result list right-click menu
 .3. The preview window
+.4. The query language
-.4. Complex/advanced search
+.5. Complex/advanced search
-.5. The term explorer tool
+.6. The term explorer tool
+.7. More about wildcards
-.6. Multiple databases
+.8. Multiple databases
-.7. Document history
+.9. Document history
-.8. Sorting search results
+.10. Sorting search results
-.9. Search tips, shortcuts
+.11. Search tips, shortcuts
-.10. Customizing the search interface
+.12. Customizing the search interface
 . Installation
 .1. Installing a prebuilt copy
 ...
 .4.2. The mimemap file
 .4.3. The mimeconf file
 .4.4. The mimeview file
+.4.5. Examples of configuration adjustments
      ----------------------------------------------------------------------
                             Chapter 1. Introduction
 ...
    Indexing is the process by which the set of documents is analyzed and the
    data entered into the database. Recoll indexing is normally incremental:
    documents will only be processed if they have been modified. On the first
    execution, of course, all documents will need processing. A full index
-   build can be forced later on by specifying an option to the indexing
+   build can be forced later by specifying an option to the indexing command
-   command (recollindex -z).
+   (recollindex -z).
    Recoll indexing can be performed with two different methods:
      * Periodic indexing: indexing takes place at discrete times, by
        executing the recollindex command. The typical usage is to have a
 ...
    email folders change. You probably do not want to enable it if your system
    is short on resources. Periodic indexing is adequate in most cases.
      ----------------------------------------------------------------------
-                               Chapter 3. Search
+                              Chapter 3. Searching
    The recoll program provides the user interface for searching. It is based
    on the QT library.
      ----------------------------------------------------------------------
 ...
 . Enter search term(s) in the text field at the top of the window.
 . Click the Search button or hit the Enter key to start the search.
-   The initial default search mode is Any term. This will look for documents
+   The initial default search mode is All terms. This will look for documents
-   with any of the search terms (the ones with more terms will get better
+   containing all of the search terms (the ones with more terms will get
-   scores). All terms will ensure that only documents with all the terms will
+   better scores). Any term will search for documents where at least one of
-   be returned. File name will specifically look for file names, and allows
+   the terms appear. File name will specifically look for file names.
-   using wildcards (*, ? , []).
+   The fourth entry (Query Language) is described in its own section.
+   All search modes allow wildcards inside terms (*, ?, []). You may want to
+   have a look at the section about wildcards for more information about
+   this.
    You can search for exact phrases (adjacent words in a given order) by
    enclosing the input inside double quotes. Ex: "virtual reality".
    Character case has no influence on search, except that you can disable
 ...
    Recoll remembers the last few searches that you performed. You can use the
    simple search text entry widget (a combobox) to recall them (click on the
    thing at the right of the text field). Please note, however, that only the
    search texts are remembered, not the mode (all/any/file name).
-   Typing Esc Space) while entering a word in the simple search entry will
+   Typing Esc Space while entering a word in the simple search entry will
    open a window with possible completions for the word. The completions are
    extracted from the database.
    Double-clicking on a word in the result list or a preview window will
    insert it into the simple search entry field.
+   Note that, apart from wildcard characters (single ? characters are ok),
+   you can cut and paste any text into an All terms or Any term search field,
+   punctuation, newlines and all. Recoll will process it and produce a
+   meaningful search. This is what most differentiates this mode from the
+   Query Language mode, where you have to care about the syntax.
    You can use the Tools / Advanced search dialog for more complex searches.
      ----------------------------------------------------------------------
 ...
    Clicking on the Preview link for an entry will open an internal preview
    window for the document. Further Preview clicks for the same search will
    open tabs in the existing preview window. You can use Shift+Click to force
    the creation of another preview window, which may be useful to view the
-   documents side by side.
+   documents side by side. (You can also browse successive results in a
+   single preview window by typing Shift+ArrowUp/Down in the window).
    Clicking the Edit link will attempt to start an external viewer. The
    viewers can be configured through the user preferences dialog, or by
    editing the mimeview configuration file.
 ...
      * Find similar
      * Parent document
    The Preview and Edit entries do the same thing as the corresponding links.
-   The two following entries will copy either an URL or the file path to the
-   clipboard, for pasting into another application.
+   The Copy File Name and Copy Url copy the relevant data to the clipboard,
+   for later pasting.
    The Find similar entry will select a number of relevant term from the
    current document and enter them into the simple search field. You can then
    start a simple search, with a good chance of finding documents related to
    the current result.
-   The Copy File Name and Copy Url copy the relevant data to the clipboard,
-   for later pasting.
    The Parent document entry will appear for documents which are not actually
    files but are part of, or attached to, a higher level document. This entry
    is mainly useful for email attachments and permits viewing the message to
    which the document is attached. Note that the entry will also appear for
 ...
    The preview window opens when you first click a Preview link inside the
    result list.
    Subsequent preview requests for a given search open new tabs in the
-   existing window.
+   existing window (except if you hold the Shift key while clicking which
+   will open a new window for side by side viewing).
    Starting another search and requesting a preview will create a new preview
    window. The old one stays open until you close it.
    You can close a preview tab by typing ^W (Ctrl + W) in the window. Closing
 ...
    string is found, the cursor will be positioned at the first occurrence of
    the search string.
      ----------------------------------------------------------------------
+.4. The query language
+   The query language processor is activated on the simple search entry when
+   the search mode selector is set to Query Language.
+   Here follows a sample request that we are going to explain:
+           mime:message/rfc822 author:"john doe" Beatles OR Lennon Live OR Unplugged -potatoes
+   This would search for all email messages with John Doe appearing as a
+   phrase in the From: header, and containing either beatles or lennon and
+   either live or unplugged but not potatoes.
+   The first element, mime:message/rfc822 is a special switch that restricts
+   the results to be email messages. There could be several such switches,
+   which would form a list of allowed types.
+   The second element author:"john doe" is a phrase search limited to a
+   specific field. Phrase searches are specified as usual by enclosing the
+   words in double quotes. The field specification appears before the colon.
+   Recoll currently manages the following fields:
+     * title, subject or caption are synonyms which specify data to be
+       searched for in the document title or subject.
+     * author or from for searching the documents originators.
+     * keyword for searching the document specified keywords (few documents
+       actually have any).
+   The query language is currently the only way to use the Recoll field
+   search capability.
+   All elements in the search entry are normally combined with an implicit
+   AND. It is possible to specify that elements be OR'ed instead, as in
+   Beatles OR Lennon. The OR must be entered literally (capitals), and it has
+   priority over the AND associations: word1 word2 OR word3 means word1 AND
+   (word2 OR word3) not (word1 AND word2) OR word3. Do not enter explicit
+   parenthesis, they are not supported for now.
+   An entry preceded by a - specifies a term that should not appear.
+   Words inside phrases and capitalized words are not stem-expanded.
+   Wildcards may be used anywhere.
+   You can use the show query link at the top of the result list to check the
+   exact query which was finally executed by Xapian.
+     ----------------------------------------------------------------------
-.4. Complex/advanced search
+.5. Complex/advanced search
-   The advanced search dialog has fields that will allow a more refined
+   The advanced search dialog has a number of fields that will allow a more
-   search. It has a number of entry fields, each of which is configurable for
+   refined search. Each entry field is configurable for the following modes:
-   the following modes:
      * All terms.
      * Any term.
 ...
      * Filename search with wildcards.
    Additional entry fields can be created by clicking the Add clause button.
-   All relevant fields will be combined by an implicit AND or OR conjunction.
+   You can choose that all relevant fields will be combined by either an AND
-   All types of clauses except "phrase" and "near" can accept a mix of single
+   or an OR conjunction. All types of clauses except "phrase" and "near" can
-   words and phrases enclosed in double quotes. Stemming expansion will be
+   accept a mix of single words and phrases enclosed in double quotes.
-   performed for all terms not beginning with a capital letter, except for
+   Stemming expansion will be performed for all terms not beginning with a
-   "phrase" clauses.
+   capital letter, except for terms inside "phrase" clauses. Wildcards will
+   be processed everywhere.
    Advanced search will also let you search for documents of specific mime
    types (ie: only text/plain, or text/HTML or application/pdf etc...). The
    state of the file type selection can be saved as the default (the file
    type filter will not be activated at program start-up, but the lists will
 ...
    Click on the Show query details link at the top of the result page to see
    the query expansion.
      ----------------------------------------------------------------------
-.5. The term explorer tool
+.6. The term explorer tool
    Recoll automatically manages the expansion of search terms to their
    derivatives (ie: plural/singular, verb inflections). But there are other
    cases where the exact search term is not known. For example, you may not
    remember the exact spelling, or only know the beginning of the name.
 ...
    terms list. It has three modes of operations:
    Wildcard
            In this mode of operation, you can enter a search string with
-           shell-like wildcards (*, ?). ie: xapi* .
+           shell-like wildcards (*, ?, []). ie: xapi* would display all index
+           terms beginning with xapi. (More about wildcards here).
    Regular expression
            This mode will accept a regular expression as input. Example:
-           word[0-9]+ . The regular expression is anchored by enclosing in ^
+           word[0-9]+. The expression is implicitely anchored at the
-           and $ before execution.
+           beginning. Ie: press will match pression but not expression. You
+           can use .*press to match the latter, but be aware that this will
+           cause a full index term list scan, which can be quite long.
    Stem expansion
            This mode will perform the usual stem expansion normally done as
            part user input processing. As such it is probably mostly useful
 ...
    simple search entry field. You can also cut/paste between the result list
    and any entry field (the end of lines will be taken care of).
      ----------------------------------------------------------------------
+.7. More about wildcards
+   All words entered in Recoll search fields will be processed for wildcard
+   expansion before the request is finally executed.
+   The wildcard characters are:
+     * * which matches 0 or more characters.
+     * ? which matches a single character.
+     * [] which allow defining sets of characters to be matched (ex: [abc]
+       matches a single character which may be 'a' or 'b' or 'c', [0-9]
+       matches any number.
+   You should be aware of a few things before using wildcards.
+     * Using a wildcard character at the beginning of a word can make for a
+       slow search because Recoll will have to scan the whole index term list
+       to find the matches.
+     * Using a * at the end of a word can produce more matches than you would
+       think, and strange search results. You can use the term explorer tool
+       to check what completions exist for a given term. You can also see
+       exactly what search was performed by clicking on the link at the top
+       of the result list. In general, for natural language terms, stem
+       expansion will produce better results than an ending * (stem expansion
+       is turned off when any wildcard character appears in the term).
+     ----------------------------------------------------------------------
-.6. Multiple databases
+.8. Multiple databases
    Multiple Recoll databases or indexes can be created by using several
    configuration directories which are usually set to index different areas
    of the file system. A specific index can be selected for updating or
    searching, using the RECOLL_CONFDIR environment variable or the -c option
 ...
  export RECOLL_EXTRA_DBS=/some/place/xapiandb:/some/other/db
    A typical usage scenario for the multiple index feature would be for a
    system administrator to set up a central index for shared data, that you
-   may choose to search, or not, in addition to your personal data. Of
+   choose to search or not in addition to your personal data. Of course,
-   course, there are other possibilities. There are many cases where you know
+   there are other possibilities. There are many cases where you know the
-   the subset of files that you want to be searched for a given query, and
+   subset of files that should be searched, and where narrowing the search
-   where restricting the query will much improve the precision of the
+   can improve the results. You can achieve approximately the same effect
-   results. This can also be performed with the directory filter in advanced
+   with the directory filter in advanced search, but multiple indexes will
-   search, but multiple indexes will have much better performance and may be
+   have much better performance and may be worth the trouble.
-   worth the trouble.
      ----------------------------------------------------------------------
-.7. Document history
+.9. Document history
    Documents that you actually view (with the internal preview or an external
    tool) are entered into the document history, which is remembered. You can
    display the history list by using the Tools/Doc History menu entry.
      ----------------------------------------------------------------------
-.8. Sorting search results
+.10. Sorting search results
    The documents in a result list are normally sorted in order of relevance.
    It is possible to specify different sort parameters by using the Sort
    parameters dialog (located in the Tools menu).
 ...
    The sort parameters stay in effect until they are explicitly reset, or the
    program exits. An activated sort is indicated in the result list header.
      ----------------------------------------------------------------------
-.9. Search tips, shortcuts
+.11. Search tips, shortcuts
    Term completion. Typing Esc Space in the simple search entry field while
    entering a word will either complete the current word if its beginning
    matches a unique term in the index, or open a window to propose a list of
    completions.
 ...
    Quitting. Entering ^Q almost anywhere will close the application.
      ----------------------------------------------------------------------
-.10. Customizing the search interface
+.12. Customizing the search interface
    It is possible to customize some aspects of the search interface by using
    Query configuration entry in the Preferences menu.
    There are two tabs in the dialog, dealing with the interface itself, and
 ...
      * Auto-start simple search on white space entry: if this is checked, a
        search will be executed each time you enter a space in the simple
        search input field. This lets you look at the result list as you enter
        new terms. This is off by default, you may like it or not...
+     * Start with advanced search dialog open and Start with sort dialog
+       open: If you use these dialogs all the time, checking these entries
+       will get them to open when recoll starts.
+     * Use desktop preferences to choose document editor: if this is checked,
+       the xdg-open utility will be used to open files when you click the
+       Edit link in the result list, instead of the application defined in
+       mimeview. xdg-open will in term use your desktop preferences to choose
+       an appropriate application.
    Search parameters:
      * Stemming language: stemming obviously depends on the document's
        language. This listbox will let you chose among the stemming databases
 ...
    External indexes: This panel will let you browse for additional indexes
    that you may want to search. External indexes are designated by their
    database directory (ie: /home/someothergui/.recoll/xapiandb,
    /usr/local/recollglobal/xapiandb).
-   Once entered, the indexes will appear in the All indexes list, and you can
+   Once entered, the indexes will appear in the External indexes list, and
-   chose which ones you want to use at any moment by transferring them
+   you can chose which ones you want to use at any moment by checking or
-   to/from the Active indexes list.
+   unchecking their entries.
    Your main database (the one the current configuration indexes to), is
    always implicitly active. If this is not desirable, you can set up your
    configuration so that it indexes, for example, an empty directory.
 ...
      * MP3: Recoll will use the id3info command from the id3lib package to
        extract tag information. Without it, only the file names will be
        indexed.
-   Text, HTML, mail folders and Openoffice files are processed internally.
+   Text, HTML, mail folders Openoffice and Scribus files are processed
+   internally. Lyx is used to index Lyx files. Many filters need sed and awk.
      ----------------------------------------------------------------------
 .3. Building from source
 ...
    recoll and recollindex.
    If the .recoll directory does not exist when recoll or recollindex are
    started, it will be created with a set of empty configuration files.
    recoll will give you a chance to edit the configuration file before
-   starting indexing. recollindex will proceed immediately.
+   starting indexing. recollindex will proceed immediately. To avoid
+   mistakes, the automatic directory creation will only occur for the default
+   location, not if -c or RECOLL_CONFDIR were used (in the latter cases, you
+   will have to create the directory).
    All configuration files share the same format. For example, a short
    extract of the main configuration file might look as follows:
          # Space-separated list of directories to index.
 ...
    in the next section.
    The tilde character (~) is expanded in file names to the name of the
    user's home directory.
-   White space is used for separation inside lists. Elements with embedded
+   White space is used for separation inside lists. List elements with
-   spaces can be quoted using double-quotes.
+   embedded spaces can be quoted using double-quotes.
      ----------------------------------------------------------------------
 .4.1. Main configuration file
 ...
    dbdir
            The name of the Xapian data directory. It will be created if
            needed when the index is initialized. If this is not an absolute
            path, it will be interpreted relative to the configuration
-           directory.
+           directory. The value can have embedded spaces but starting or
+           trailing spaces will be trimmed. You cannot use quotes here.
    skippedNames
            A space-separated list of patterns for names of files or
            directories that should be completely ignored. The list defined in
            the default file is:
- *~ #* bin CVS  Cache caughtspam  tmp
+ skippedNames = #* bin CVS  Cache cache* caughtspam  tmp .thumbnails .svn \
+          *~ recollrc
            The list can be redefined for sub-directories, but is only
            actually changed for the top level ones in topdirs.
            The top-level directories are not affected by this list (that is,
 ...
            index quite a few things that you do not want. On the other hand,
            mail user agents like thunderbird usually store messages in hidden
            directories, and you probably want this indexed. One possible
            solution is to have .* in skippedNames, and add things like
            ~/.thunderbird or ~/.evolution in topdirs.
+   skippedPaths and daemSkippedPaths
+           A space-separated list of patterns for paths of files or
+           directories that should be skipped. There is no default in the
+           sample configuration file, but the code always adds the
+           configuration and database directories in there.
+           skippedPaths is used both by batch and real time indexing.
+           daemSkippedPaths can be used to specify things that should be
+           indexed at startup, but not monitored.
+           Example of use for skipping text files only in a specific
+           directory:
+ skippedPaths = ~/somedir/*.txt
    loglevel,daemloglevel
            Verbosity level for recoll and recollindex. A value of 4 lists
            quite a lot of debug/information messages. 2 only lists errors.
 ...
    non-default entries, which will override those from the central
    configuration file.
    Please note that these entries must be placed under a [view] section.
+   If Use desktop preferences to choose document editor is checked in the
+   user preferences, all mimeview entries will be ignored except the one
+   labelled application/x-all (which is set to use xdg-open by default).
      ----------------------------------------------------------------------
+.4.5. Examples of configuration adjustments
+.4.5.1. Adding an external viewer for an non-indexed type
+   Imagine that you have some kind of file which does not have indexable
+   content, but for which you would like to have a functional Edit link in
+   the result list (when found by file name). The file names end in .blob and
+   can be displayed by application blobviewer.
+   You need two entries in the configuration files for this to work:
+     * In $RECOLL_CONFDIR/mimemap (typically ~/.recoll/mimemap), add the
+       following line:
+              application/x-blobapp = .blob
+       Note that the mime type is made up here, and you could call it
+       diesel/oil just the same.
+     * In $RECOLL_CONFDIR/mimeview under the [view] section:
+                  application/x-blobapp = blobviewer %f
+       We are supposing that blobviewer wants a file name parameter here, you
+       would use %u if it liked URLs better.
+   If you just wanted to change the application used by Recoll to display a
+   mime type which it already knows, you would just need to edit mimeview.
+   The entries you add in your personal file override those in the central
+   configuration, which you do not need to alter
+     ----------------------------------------------------------------------
+.4.5.2. Adding indexing support for a new file type
+   Let us now imagine that the above .blob files actually contain indexable
+   text and that you know how to extract it with a command line program.
+   Getting Recoll to index the files is easy. You need to perform the above
+   alteration, and also to add data to the mimeconf file (typically in
+   ~/.recoll/mimeconf):
+     * Under the [index] section, add the following line (more about the
+       rclblob indexing script later):
+                  application/x-blobapp = exec rclblob
+     * Under the [icons] section, you should choose an icon to be displayed
+       for the files inside the result lists. Icons are normally 64x64 pixels
+       PNG files which live in /usr/[local/]share/recoll/images.
+     * Under the [categories] section, you should add the mime type where it
+       makes sense (you can also create a category). Categories may be used
+       for filtering in advanced search.
+   The rclblob filter should be an executable program or script which exists
+   inside /usr/[local/]share/recoll/filters. It will be given a file name as
+   argument and should output the text contents in html format on the
+   standard output.
+   The html could be very minimal like the following example:
+ <html><head>
+ <meta http-equiv="Content-Type" content="text/html;charset=UTF-8">
+ </head>
+ <body>some text content</body></html>
+   You should take care to escape some characters inside the text by
+   transforming them into appropriate entities. "&" should be transformed
+   into "&amp;", "<" should be transformed into "&lt;".
+   The character set needs to be specified in the header. It does not need to
+   be UTF-8 (Recoll will take care of translating it), but it must be
+   accurate for good results.
+   Recoll will also make use of other header fields if they are present:
+   title, description, keywords.
+   The easiest way to write a new filter is probably to start from an
+   existing one.
+     ----------------------------------------------------------------------