Open Source Projects Europe

== Character case and diacritic marks (2), user interface

In a link:ZDevCaseAndDiacritics1.html[previous document], we discussed some

of the problems which arise when mixing case/diacritics sensitivity and

stemming.

As of version 1.18, Recoll can create two types of indexes:

* _Dumb_ indexes contain terms which are lowercased and stripped of

  diacritics. Searches using such an index are naturally case- and

  diacritics- insensitive: search terms are stripped before processing.

* _Raw_ indexes contain terms which are just like they were found in the

  source document. Searching such an index is naturally sensitive to case

  and diacritics, and can be made insensitive by further processing.

The following explains how users can control these Recoll features.

=== Controlling the type of index we create: stripped or raw

The kind of index that recoll creates is determined by:

 * A build-time *configure* switch: _--enable-stripchars_. If this is

   set, the code for case and diacritics sensitivity is not compiled in and

   recoll will work like the previous versions: unaccented and casefolded

   index, no runtime options for case or diacritics sensitivity

 * An indexing configuration switch (in recoll.conf): if Recoll was built

   with _--disable-stripchars_, this will provide a dynamic way to return

   to the "traditional" index. The case and diacritics code will be present

   but inactive. Normally, a recoll installation with this switch set

   should behave exactly like one built with _--enable-stripchars_. When

   using multiple indexes, this switch MUST be consistent between

   indexes. There is no support whatsoever for mixing raw and dumb indexes.

   The option is named _indexStripChars_, and it is not settable from the

   GUI to avoid errors. This is something that would typically be set once

   and for all for a given installation. We need to decide what the default

   value will be for 1.18

 * A number of query time switches. Using these it is also possible to

   perform a search insensitive to case and diacritics on a raw index. Note

   however, that, given the complexity of the issues involved, I give no

   guaranty at this time that this will yield exactly the same results as

   searching a dumb index. Details about query time behaviour follow.

=== Controlling stem, case and diacritics expansion: user query interface 

Recoll versions up to 1.17 were insensitive to case and diacritics. We only

needed to give the user a way to control stem expansion. This was done in

three ways:

 * Globally, by setting a menu option.

 * Globally, by setting the stemming language value to empty.

 * On a term by term basis by Capitalizing the term, or, in query language

   mode only, by using an 'l' clause modifier (_"term"l_).

After switching to an unstripped index, capable of case and diacritic

sensitivity, we need ways to control what processing is performed among:

 * Case expansion.

 * Diacritics expansion.

 * Stem expansion.

The default mode will be compatible with the previous version, because

this is is most generally what we want to do: ignore case and diacritics,

expand stems.

There are two easy approaches for controlling the parameters:

 * Global options set in the GUI menus or as *recollq* command line

   switches. 

 * Per-clause options set by modifiers in the query language.

We would like, however to let the user entry automatically override the

defaults in a sensible way. For example:

 * If a term is entered with diacritics, diacritic sensitivity is turned on

   (for this term only).

 * If a term is entered with upper-case characters, case sensitivity is

   turned on. In this case, we turn off stem expansion, because it makes

   really no sense with case sensitivity.

With this method we are stuck with 3 problems (only if the global mode is

set to insensitive, and we're not using the query language):

 * Turning off stemming without turning on case sensitivity.

 * Searching for an all lower-case term in case-sensitive mode.

 * Searching for a term without diacritics in diacritic-sensitive mode.

The two latter issues are relatively marginal and can be worked around easily

by switching to query language mode or using negative clauses in the

advanced search. 

However, we need to be able to turn stemming off while remaining

insensitive to case, and we need to stay reasonably compatible with the

previous versions. This means that a term which has a capital first letter

but is otherwise lowercase will turn stemming off, but not case sensitivity

on. 

So we're left with how to search for such a term in a case-sensitive way,

and for this, you'll have to use global options or the query language.

The modified method is:

 * If a term is entered with diacritics, diacritic sensitivity is turned on

   (for this term only).

 * If the first letter in a term is upper-case and the rest is lower-case,

   we turn stem expansion off, but we do not become case-sensitive

 * If any letter in a term except the first is upper-case, case sensitivity

   is turned on. Stem expansion is also turned-off (even if the first

   letter is lower-case), because it makes really no sense with case

   sensitivity.

 * To search for an all lower-case or capitalized term in a case-sensitive

   way, use the query language: "Capitalized"C, "lowercase"C

 * Use the query language and the "D" modifier to turn on diacritics

   sensitivity.

It can be noted that some combinations of choices do not make sense and

they are not allowed by Recoll: for example, diacritics or case sensitivity

do not make sense with stem expansion (which cannot preserve diacritics in

any meaningful general way).

The [[ZDevCaseAndDiacritics3.wiki|next page]] describes the actual

implementation in Recoll 1.18.