SRI's Real-Time Spoken-Language System.pdf

Evaluation of SRI's NL, SLS, and speech recognition technologies showed a weighted error rate of 31.0% for NL and 45.4% for SLS for ATIS class A and D utterances combined in February-92. SRI's SPREC word error rate for all ATIS utterances was 11.0% in the same period. For the speaker-independent, closed 5K vocabulary CSR task, SRI achieved a 16.6% word error rate in the verbalized punctuation test and a 17.1% word error rate in the non-verbalized-punctuation test using the standard bigram language model.

Encouraging initial results were obtained in a study of verbal repairs in the ATIS MADCOW corpus, including analysis and classification of the repairs in the corpus as well as a method using both pattern matching techniques and acoustic evidence to detect and correct the repairs. RASTA filtering was implemented to improve channel robustness.

Experiments showed differences between spontaneous and read speech and the need for including spontaneous speech in training, development, and test sets. Development of infrastructure necessary for dealing with very large vocabularies and language models was undertaken.

Improvements were made to the GEMINI natural-language processing system's linguistic coverage of a 2100-utterance ATIS training corpus to 92% syntactic coverage and 77% semantic coverage. Algorithms for tracking evolving discourse structure were developed, with these algorithms correctly determining the discourse context with greater than 90% accuracy on the ATIS corpus development set.

The template matcher was improved with new templates, improved portability, more linguistic knowledge incorporated, reduced overgeneration, improved context handling, and improved coverage of the database retrieval component, all without significantly increasing response time, which is still less than a second.

Analysis of data on human-machine problem solving using the SLS ATIS system was undertaken, including the effect on user satisfaction and system performance of system errors, user experience, and instructions to users. Trade-offs of time vs. speed and user satisfaction were also explored.

Data collection using the SLS ATIS system included 72 speakers, 2301 utterances, and 134 scenarios. Improvements were made to the SLS ATIS demo system, including better paraphrasing of the system's understanding, easier to read displays, improved system error messages, and simpler control of the context mechanism.

Development with MIT of a method for end-to-end evaluation that takes into account the whole interaction was undertaken, with implementation of the method in progress. Plans for the coming year include fully integrating GEMINI into SRI's ATIS system, exploring tighter integration of speech and NL processing for better overall SLS performance, and improving speech recognition computational and word-accuracy performance for the ATIS and CSR domains with better models of the channel, speaker, and spontaneous speech effects.