Robust Natural Language Processing For Information Retrieval.pdf

Our information retrieval system consists of a traditional statistical backbone augmented with various natural language processing components. The system processes the database text with a fast syntactic parser, extracts certain types of phrases, and uses them as compound indexing terms. The extracted phrases are statistically analyzed to discover similarity links between smaller subphrases and words occurring in them. A further filtering process maps these similarity links onto semantic relations, which are used to transform the user's request into a search query. The user's natural language request is also parsed, and all indexing terms occurring in it are identified. Certain highly ambiguous terms are dropped, and other terms may be added via admissible similarity relations. The final query is constructed, and the database search follows, returning a ranked list of documents. All processing steps are fully automated, with no human intervention or manual encoding required.

TIP, a full grammar parser based on the Linguistic String Grammar, produces regularized parse tree representations for each sentence. It has a built-in timer that regulates the amount of time allowed for parsing any one sentence. If a parse is not returned before the allotted time elapses, the parser enters the skip-and-fit mode, attempting to "fit" the parse by forcibly reducing incomplete constituents and skipping portions of input. The parser is equipped with a powerful skip-and-fit recovery mechanism that allows it to operate effectively in the face of ill-formed input or under severe time pressure.

The parser's speed averaged between 0.45 and 0.5 seconds per sentence, or up to 2600 words per minute, on a 21 MIPS SparcStation ELC. The parser is a full grammar parser that initially attempts to generate a complete analysis for each sentence. However, it has a built-in timer that regulates the amount of time allowed for parsing any one sentence. If a parse is not returned before the allotted time elapses, the parser enters the skip-and-fit mode, attempting to "fit" the parse by forcibly reducing incomplete constituents and skipping portions of input.

A stochastic parts of speech tagger is used to preprocess the text, removing most of the lexical level ambiguity from the input text prior to parsing. This allows the parser to operate effectively in the face of ill-formed input or under severe time pressure. The parser's speed averaged between 0.45 and 0.5 seconds per sentence, or up to 2600 words per minute, on a 21 MIPS SparcStation ELC.

The system uses a conservative dictionary-assisted suffix trimmer to reduce inflected word forms to their root forms. The suffix trimmer performs two tasks: reducing inflected word forms to their root forms as specified in the dictionary, and converting nominalized verb forms to the root forms of corresponding verbs. This is accomplished by removing a standard suffix, replacing it with a standard root ending, and checking the newly created word against the dictionary. The system uses Oxford Advanced Learner's Dictionary (OALD) MRD to perform these tasks.