recoll / Code / Diff of /src/doc/user/usermanual.xml

Diff of /src/doc/user/usermanual.xml [3de5b5] .. [c897a0]

Switch to unified view


...
          web site</ulink>.</literal></para>

      <para>This document introduces full text search notions
      and describes the installation and use of the &RCL;
      application. It currently describes &RCL; &RCLVERSION;.</para>



    </abstract>


  </bookinfo>
  
...
        punctuation and capitalization are lost).</para>

      <para>&RCL; stores all internal data in <application>Unicode
      UTF-8</application> format, and it can index files with
      different character sets, encodings, and languages into the same
      index. It has can process many document types.</para>
      
      <para>Stemming is the process by which &RCL; reduces words to
        their radicals so that searching does not depend, for example, on a
        word being singular or plural (floor, floors), or on a verb tense
        (flooring, floored). Because the mechanisms used for stemming
...
          be ignored.</para>
        <para>Excluding types can be done by adding wildcard name
          patterns to the <literal>skippedNames</literal> list, which
          can be done from the GUI Index configuration menu. It is
          also possible to exclude a mime type independantly of the
          file name by associating it with the
          <filename>rclnull</filename> input handler. This can be done
          by editing the <link linkend="RCL.INSTALL.CONFIG.MIMECONF">
            <filename>mimeconf</filename> configuration
            file</link>.</para>

        <para>In order to define a positive list, You need to edit the 
          <link linkend="RCL.INSTALL.CONFIG.RECOLLCONF">main
...
        stored by default, apart from the values above
        (only <literal>author</literal>
        and <literal>filename</literal>), so this feature will need
        some custom local configuration to be useful. An example
        candidate would be the <literal>recipient</literal> field
        which is generated by the message input handlers.</para>

        <para>The default value for the paragraph format string is:
        <screen><![CDATA[
<img src="%I" align="left">%R %S %L &nbsp;&nbsp;<b>%T</b><br>
%M&nbsp;%D&nbsp;&nbsp;&nbsp;<i>%U</i>&nbsp;%i<br>
...
        slow search (or even an incorrect one if the expansion is
        truncated because of excessive size). Also see 
        <link linkend="RCL.SEARCH.WILDCARDS">
          More about wildcards</link>.</para>

      <para>The document input handlers used while indexing have the
        possibility to create other fields with arbitrary names, and
        aliases may be defined in the configuration, so that the exact
        field search possibilities may be different for you if someone
        took care of the customisation.</para>

...
      <para>&RCL; has an Application Programming Interface, usable both
        for indexing and searching, currently accessible from the
        <application>Python</application> language.</para>

      <para>Another less radical way to extend the application is to
        write input handlers for new types of documents.</para>

      <para>The processing of metadata attributes for documents
        (<literal>fields</literal>) is highly configurable.</para>



      <sect1 id="RCL.PROGRAM.FILTERS">
        <title>Writing a document input handler</title>
        
        <note><title>Terminology</title>The small programs or pieces
        of code which handle the processing of the different document
        types for &RCL; used to be called <literal>filters</literal>,
        which is still reflected in the name of the directory which
        holds them and many configuration variables. They were named
        this way because one of their primary functions is to filter
        out the formatting directives and keep the text
        content. However these modules may have other behaviours, and
        the term <literal>input handler</literal> is now progressively
        substituted in the documentation. <literal>filter</literal> is
        still used in many places though.</note>

        <para>&RCL; input handlers cooperate to translate from the multitude
        of input document formats, simple ones
        as <application>opendocument</application>, 
          <application>acrobat</application>), or compound ones such
          as <application>Zip</application>
          or <application>Email</application>, into the final &RCL;
          indexing input format, which is plain text.
          Most input handlers are executable

          programs or scripts. A few handlers are coded in C++ and live
          inside <command>recollindex</command>. This latter kind will not
          be described here.</para>

        <para>There are currently (1.18 and since 1.13) two kinds of
        external executable input handlers:
          <itemizedlist>
        <listitem><para>Simple <literal>exec</literal> handlers
            run once and exit. They can be bare programs like
            <command>antiword</command>, or scripts using other
            programs. They are very simple to write, because they just
            need to print the converted document to the standard
            output. Their output can be plain text or HTML. HTML is
            usually preferred because it can store metadata fields and
            it allows preserving some of the formatting for the GUI
            preview.</para>
        </listitem>
        <listitem><para>Multiple <literal>execm</literal> handlers
      can process multiple files (sparing the process startup
      time which can be very significant), or multiple documents
      per file (e.g.: for <application>zip</application> or
      <application>chm</application> files). They communicate
      with the indexer through a simple protocol, but are
      nevertheless a bit more complicated than the older
      kind. Most of new handlers are written in


        <application>Python</application>, using a common module
        to handle the protocol. There is an exception,
        <command>rclimg</command> which is written in Perl. The
      subdocuments output by these handlers can be directly
      indexable (text or HTML), or they can be other simple or
      compound documents that will need to be processed by
      another handler.</para>
        </listitem>
      </itemizedlist>
        </para>

        <para>In both cases, handlers deal with regular file system
          files, and can process either a single document, or a
          linear list of documents in each file. &RCL; is responsible
          for performing up to date checks, deal with more complex
          embedding and other upper level issues.</para>

        <para>A simple handler returning a
          document in <literal>text/plain</literal> format, can transfer

          no metadata to the indexer. Generic metadata, like document
          size or modification date, will be gathered and stored by
          the indexer.</para>

        <para>Handlers that produce  <literal>text/html</literal>
          format can return an arbitrary amount of metadata inside HTML
          <literal>meta</literal> tags. These will be processed
          according to the directives found in 
          the <link linkend="RCL.PROGRAM.FIELDS">
            <filename>fields</filename> configuration
            file</link>.</para>

        <para>The handlers that can handle multiple documents per file
          return a single piece of data to identify each document inside
          the file. This piece of data, called
          an <literal>ipath element</literal> will be sent back by
          &RCL; to extract the document at query time, for previewing,
          or for creating a temporary file to be opened by a
          viewer.</para>  

        <para>The following section describes the simple
          handlers, and the next one gives a few explanations about
          the <literal>execm</literal> ones. You could conceivably
          write a simple handler with only the elements in the
          manual. This will not be the case for the other ones, for
          which you will have to look at the code.</para>

      <sect2 id="RCL.PROGRAM.FILTERS.SIMPLE">
        <title>Simple input handlers</title>

      <para>&RCL; simple handlers are usually shell-scripts, but this is in
        no way necessary. Extracting the text from the native format is the
        difficult part. Outputting the format expected by &RCL; is
        trivial. Happily enough, most document formats have translators or
        text extractors which can be called from the handler. In some cases
        the output of the translating program is completely appropriate,
        and no intermediate shell-script is needed.</para>

        <para>Input handlers are called with a single argument which is the
        source file name. They should output the result to stdout.</para>

      <para>When writing a handler, you should decide if it will output
      plain text or HTML. Plain text is simpler, but you will not be able
      to add metadata or vary the output character encoding (this will be
      defined in a configuration file). Additionally, some formatting may
      be easier to preserve when previewing HTML. Actually the deciding factor
      is metadata: &RCL; has a way to <link linkend="RCL.PROGRAM.FILTERS.HTML">
      extract metadata from the HTML header and use it for field 
      searches.</link>.</para>

      <para>The <envar>RECOLL_FILTER_FORPREVIEW</envar> environment
        variable (values <literal>yes</literal>, <literal>no</literal>)
        tells the handler if the operation is for indexing or
        previewing. Some handlers use this to output a slightly different
        format, for example stripping uninteresting repeated keywords (ie:
        <literal>Subject:</literal> for email) when indexing. This is not
        essential.</para>

      <para>You should look at one of the simple handlers, for example
        <command>rclps</command> for a starting point.</para>

        <para>Don't forget to make your handler executable before 
         testing !</para>

      </sect2>

      <sect2 id="RCL.PROGRAM.FILTERS.MULTIPLE">
        <title>"Multiple" handlers</title>

        <para>If you can program and want to write
          an <literal>execm</literal> handler, it should not be too
          difficult to make sense of one of the existing modules. For
          example, look at <command>rclzip</command> which uses Zip
          file paths as identifiers (<literal>ipath</literal>),
          and <command>rclics</command>, which uses an integer
          index. Also have a look at the comments inside
          the <filename>internfile/mh_execm.h</filename> file and
          possibly at the corresponding module.</para>

        <para><literal>execm</literal> handlers sometimes need to make
          a choice for the nature of the <literal>ipath</literal>
          elements that they use in communication with the
          indexer. Here are a few guidelines:
          <itemizedlist>
            <listitem><para>Use ASCII or UTF-8 (if the identifier is an
...
                debugging.</para></listitem>
            <listitem><para>&RCL; uses a colon (<literal>:</literal>) as a
                separator to store a complex path internally (for
                deeper embedding). Colons inside
                the <literal>ipath</literal> elements output by a
                handler will be escaped, but would be a bad choice as a
                handler-specific separator (mostly, again, for
                debugging issues).</para></listitem>
          </itemizedlist>
          In any case, the main goal is that it should
          be easy for the handler to extract the target document, given
          the file name and the <literal>ipath</literal>
          element.</para>

        <para><literal>execm</literal> handlers will also produce
          a document with a null <literal>ipath</literal>
          element. Depending on the type of document, this may have
          some associated data (e.g. the body of an email message), or
          none (typical for an archive file). If it is empty, this
          document will be useful anyway for some operations, as the
          parent of the actual data documents.</para>
      </sect2>

      <sect2 id="RCL.PROGRAM.FILTERS.ASSOCIATION">
        <title>Telling &RCL; about the handler</title>

      <para>There are two elements that link a file to the handler which
      should process it: the association of file to mime type and the
      association of a mime type with a handler.</para>

      <para>The association of files to mime types is mostly based on
        name suffixes. The types are defined inside the
        <link linkend="RCL.INSTALL.CONFIG.MIMEMAP">
        <filename>mimemap</filename> file</link>. Example:
...
</programlisting>
       If no suffix association is found for the file name, &RCL; will try
       to execute the <command>file -i</command> command to determine a
       mime type.</para>

      <para>The association of file types to handlers is performed in
      the <link linkend="RCL.INSTALL.CONFIG.MIMECONF">
      <filename>mimeconf</filename> file</link>. A sample will probably be
      of better help than a long explanation:</para>
<programlisting>

...
            <literal>iso-8859-1</literal> encoding is specified because it
            is not the <literal>utf-8</literal> default, and not output by
            <command>unrtf</command> in the HTML header section.</para>
      </listitem>
      <listitem><para><literal>application/x-chm</literal> is processed
          by a persistant handler. This is determined by the
          <literal>execm</literal> keyword.</para>
      </listitem>
    </itemizedlist>
       </para> 

      </sect2>

    <sect2 id="RCL.PROGRAM.FILTERS.HTML">
        <title>Input handler HTML output</title>

        <para>The output HTML could be very minimal like the following
        example:
          <programlisting>
&lt;html>
...
          <programlisting>
&lt;meta name="date" content="2013-02-24 17:50:00">
          </programlisting>
        </para>

        <para>Input handlers also have the possibility to "invent" field
          names. This should also be output as meta tags:</para>

        <programlisting>
&lt;meta name="somefield" content="Some textual data" /&gt;
</programlisting>
...

    <sect2 id="RCL.PROGRAM.FILTERS.PAGES">
        <title>Page numbers</title>

        <para>The indexer will interpret <literal>^L</literal> characters
          in the handler output as indicating page breaks, and will record
          them. At query time, this allows starting a viewer on the right
          page for a hit or a snippet. Currently, only the PDF, Postscript
          and DVI handlers generate page breaks.</para>

      </sect2>

    </sect1>

...
      <para><literal>Fields</literal> are named pieces of information
      in or about documents, like <literal>title</literal>,
      <literal>author</literal>, <literal>abstract</literal>.</para> 

      <para>The field values for documents can appear in several ways
      during indexing: either output by input handlers
      as <literal>meta</literal> fields in the HTML header section, or
      extracted from file extended attributes, or added as attributes
      of the <literal>Doc</literal> object when using the API, or
      again synthetized internally by &RCL;.</para>

      <para>The &RCL; query language allows searching for text in a
      specific field.</para>

      <para>&RCL; defines a number of default fields. Additional
      ones can be output by handlers, and described in the
      <filename>fields</filename> configuration file.</para>

      <para>Fields can be:</para>
      <itemizedlist>

...
        
        <sect5 id="RCL.PROGRAM.PYTHON.RECOLL.CLASSES.DB">
          <title>The Db class</title>

          <para>A Db object is created by
            a <literal>connect()</literal> call and holds a 
            connection to a Recoll index.</para>
          <variablelist>
            <title>Methods</title>
            <varlistentry>
              <term>Db.close()</term>
...
    <guilabel>File</guilabel> menu. The list is stored in the
    <filename>missing</filename> text file inside the configuration
    directory.</para>

      <para>A list of common file types which need external
        commands follows. Many of the handlers need the
        <command>iconv</command> command, which is not always listed as a
        dependancy.</para> 

      <para>Please note that, due to the relatively dynamic nature of this
    information, the most up to date version is now kept on &RCLAPPS;
...
        are sometimes outdated, or not the best version for &RCL;, so you
        should take a look at &RCLAPPS; if a file
        type is important to you.</para>

      <para>As of &RCL; release 1.14, a number of XML-based formats that
        were handled by ad hoc handler code now use the
        <command>xsltproc</command> command, which usually comes with  
      <application>libxslt</application>. These are: abiword, fb2
      (ebooks), kword, openoffice, svg.</para> 

      <para>Now for the list:</para>
...
        <command>antiword</command>. It is also useful to have
        <command>wvWare</command> installed as it may be 
        be used as a fallback for some files which
        <command>antiword</command> does not handle.</para></listitem>

        <listitem><para>MS Excel and PowerPoint are processed by
        internal <command>Python</command> handlers.</para></listitem>

        <listitem><para>MS Open XML (docx) needs <command>
         xsltproc</command>.</para></listitem>

        <listitem><para>Wordperfect files need <command>wpd2html</command>
...

        <listitem><para>djvu files need <command>djvutxt</command> and
        <command>djvused</command> from the
        <application>DjVuLibre</application> package.</para></listitem>
          
        <listitem><para>Audio files: &RCL; releases 1.14 and later use





        a single <application>Python</application> handler based 
        on <application>mutagen</application> for all audio file
        types.</para>
    </listitem>

        <listitem><para>Pictures: &RCL; uses the 
         <application>Exiftool</application>
         <application>Perl</application> package to extract tag
...
         there may not be much interest in indexing the technical tags
         (image size, aperture, etc.). This is only of interest if you
         store personal tags or textual descriptions inside the image
         files.</para></listitem>

    <listitem><para>chm: files in Microsoft help format need Python and
          the <application>pychm</application> module (which needs 
          <application>chmlib</application>).</para></listitem>

    <listitem><para>ICS: up to &RCL; 1.13, iCalendar files need 
        <application>Python</application>
...
        </listitem>

        <listitem><para>Konqueror webarchive format with Python (uses the
        Tarfile module).</para></listitem>

        <listitem><para>Mimehtml web archive format (support based on
        the email handler, which introduces some mild weirdness, but
        still usable).</para></listitem>

        </itemizedlist>

        <para>Text, HTML, email folders, and Scribus files are
        processed internally. <application>Lyx</application> is used to
        index Lyx files. Many handlers need <command>iconv</command> and the
        standard <command>sed</command> and <command>awk</command>.
        </para>

    </sect1>

...
      <varlistentry id="RCL.INSTALL.CONFIG.RECOLLCONF.ZIPSKIPPEDNAMES">
        <term><varname>zipSkippedNames</varname></term>
        <listitem><para>A space-separated list of patterns for
               names of files or directories that should be ignored
               inside zip archives. This is used directly by the zip
               handler, and has a function similar to skippedNames, but
               works independantly. Can be redefined for filesystem
               subdirectories. For versions up to 1.19, you will need
               to update the Zip handler and install a supplementary
               Python module. The details are
               described <ulink url="https://bitbucket.org/medoc/recoll/wiki/Filtering%20out%20Zip%20archive%20members">on
          the &RCL; wiki</ulink>.
        </para></listitem>
      </varlistentry>
...
             indexer (default class 3, no data).</para>
              </listitem>
            </varlistentry>

            <varlistentry><term><varname>filtermaxseconds</varname></term>
              <listitem><para>Maximum handler execution time, after which it
            is aborted. Some postscript programs just loop...</para> 
              </listitem>
            </varlistentry>
          <varlistentry><term><varname>filtersdir</varname></term>
            <listitem><para>A directory to search for the external
            input handler scripts used to index some types of files. The
            value should not be changed, except if you want to modify
            one of the default scripts. The value can be redefined for
            any sub-directory. </para>
            </listitem>
          </varlistentry>
...
            canonical names used inside the <literal>[prefixes]</literal>
            and <literal>[stored]</literal> sections</para>
            </listitem>
          </varlistentry>
          <varlistentry>
            <term>handler-specific sections</term>
            <listitem><para>Some input handlers may need specific
            configuration for handling fields. Only the email message handler
            currently has such a section (named
            <literal>[mail]</literal>). It allows indexing arbitrary email
            headers in addition to the ones indexed by default. Other such
            sections may appear in the future.</para>
            </listitem>
...

      <para>Here follows a small example of a personal
      <filename>fields</filename> 
      file. This would extract a specific email header and
      use it as a searchable field, with data displayable inside result
      lists. (Side note: as the email handler does no decoding on the values,
      only plain ascii headers can be indexed, and only the
      first occurrence will be used for headers that occur several times).

<programlisting>[prefixes]
# Index mailmytag contents (with the given prefix)
...
        (you can also create a category). Categories may be used
        for filtering in advanced search.</para>
            </listitem>
          </itemizedlist>

          <para>The <replaceable>rclblob</replaceable> handler should
            be an executable program or script which exists inside
            <filename>/usr/[local/]share/recoll/filters</filename>. It
            will be given a file name as argument and should output the
            text or html contents on the standard output.</para>

          <para>The <link linkend="RCL.PROGRAM.FILTERS">filter
              programming</link> section describes in more detail how
              to write an input handler.</para> 

        </sect3>

      </sect2>
