Orthographic Errors In Web Pages: Cleaning Web Corpora.pdf

The article discusses orthographic errors in web pages and their impact on corpus linguistics, where the Web serves as a significant public repository of natural language texts. The authors address several key points:

1. Prevalence of Errors: Web pages often contain spelling, grammatical, and formatting errors due to non-native speakers, inadequate character sets, or optical character recognition (OCR). These errors can negatively affect applications like speech recognition, language generation, and computer-assisted language learning.

2. Experiment and Findings: Researchers conducted experiments using large English and German web corpora from general and specific thematic areas, including HTML and PDF documents. They identified four main types of orthographic errors: typing errors, spelling errors, encoding issues, and garbled words from OCR.

3. Frequency Analysis: The study investigates the frequency of different error types and how many web pages exceed a given error level (percentage of erroneous tokens). Results vary based on language, thematic area, genre, and document format.

4. Detection and Marking Methods: The article explores methods to automatically detect and mark orthographic errors in web pages, aiming to reduce the number of errors in linguistic knowledge bases derived from web corpora.