A striking feature of human syntactic pro-
cessing is that it is context-dependent, that
is, it seems to take into account seman-
tic information from the discourse con-
text and world knowledge. In this paper,
we attempt to use this insight to bridge
the gap between SRL results from gold
parses and from automatically-generated
parses.
To do this, we jointly perform
parsing and semantic role labeling, using
a probabilistic SRL system to rerank the
results of a probabilistic parser. Our cur-
rent results are negative, because a locally-
trained SRL model can return inaccurate
probability estimates.
Although much effort has gone into developing
statistical parsing models and they have improved
steadily over the years, in many applications that
use parse trees errors made by the parser are a ma-
jor source of errors in the final output. A promising
approach to this problem is to perform both pars-
ing and the higher-level task in a single, joint prob-
abilistic model.
This not only allows uncertainty
about the parser output to be carried upward, such
as through an k-best list, but also allows informa-
tion from higher-level processing to improve pars-
ing. For example, Miller et al. (2000) showed that
performing parsing and information extraction in a
joint model improves performance on both tasks. In
particular, one suspects that attachment decisions,
which are both notoriously hard and extremely im-
portant for semantic analysis, could benefit greatly
from input from higher-level semantic analysis.
The recent interest in semantic role labeling pro-
vides an opportunity to explore how higher-level se-
mantic information can inform syntactic parsing. In
previous work, it has been shown that SRL systems
that use full parse information perform better than
those that use shallow parse information, but that
machine-generated parses still perform much worse
than human-corrected gold parses.
The goal of this investigation is to narrow the gap
between SRL results from gold parses and from au-
tomatic parses. We aim to do this by jointly perform-
ing parsing and semantic role labeling in a single
probabilistic model. In both parsing and SRL, state-
of-the-art systems are probabilistic; therefore, their
predictions can be combined in a principled way by
multiplying probabilities.
In this paper, we rerank the k-best parse trees
from a probabilistic parser using an SRL system. We
compare two reranking approaches, one that linearly
weights the log probabilities, and the other that learns
a reranker over parse trees and SRL frames in the
manner of Collins (2000).
Our approach to joint parsing and SRL begins with
a base SRL system, which uses a standard architec-
ture from the literature. Our base SRL system is a
cascade of maximum-entropy classifiers which se-
lect the semantic argument label for each constituent
of a full parse tree.
The base SRL system is a cascade of maximum-
entropy classifiers which select the semantic argu-
ment label for each constituent of a full parse tree.
We use three stages: pruning, identification, and classifica-
tion. First, in pruning, we use a deterministic pre-
processing procedure introduced by Xue and Palmer
(2004) to prune many constituents which are almost
certainly not arguments.
Second, in identification, a binary MaxEnt classifier
is used to prune remaining constituents which are pre-
dicted to be null with high probability.
Finally, in classification, a multi-class MaxEnt classifier
is used to predict the argument type of the remaining con-
stituents.
This classifier also has the option to output NULL.
It can happen that the returned semantic argu-
ments overlap, because the local classifiers take no
global constraints into account.
We resolve overlaps using a simple recursive algorithm.
For each parent node that overlaps with one of its
descendents, we check which predicted probability is
greater: that the parent has its locally-predicted argu-
ment label and all its descen-dents are null, or that the
descendants have their optimal labeling, and the parent
is null.
This algorithm returns the non-overlapping assignment
with globally highest confidence.
Overlaps are uncommon, however; they occurred only
68 times on the 1346 sentences in the development set.
We train the classifiers on PropBank sections 02–
21.
If a true semantic argument fails to match any
bracketing in the parse tree, then it is ignored.
Both the identification and classification models are
trained using gold parse trees.
All of our features are standard features for this task
that have been used in previous work, and are listed
in Tables 1 and 2.
We use the maximum-entropy implementation in
the Mallet toolkit (McCallum, 2002) with a Gaussian
prior on parameters.
Here we give the general framework for the rerank-
ing methods that we present in the next section.
We write a joint probability model over semantic
frames F and parse trees t given a sentence x as
p(F, t|x) = p(F|t, x)p(t|x),
where p(t|x) is given by a standard probabilistic
parsing model, and p(F|t, x) is given by the base-
line SRL model described previously.
The k-best list of parses is generated using Dan
Bikel’s (2004) implementation of Michael Collins’
parsing model.
The parser is trained on sections 2–21 of the WSJ
Treebank, which does not overlap with the development
or test sets.
The k-best list is generated in Bikel’s implementation
by essentially turning off dynamic programming and
doing very aggressive beam search.
We gather a maximum of 500 best parses, but the
limit is not usually reached using feasible beam
widths.
The mean number of parses per sentence is 176.
In this section we present results on several rerank-
ing methods for joint parsing and semantic role la-
beling.
Table 3 compares F1 on the development set
of our different reranking methods.
The first four rows in Table 3 are baseline systems.
We present baselines using gold parse trees.
Parse Trees Used
SRL F1
Gold
77.1
1-best
63.9
Reranked by gold parse F1
68.1
Reranked by gold frame F1
74.2
Simple SRL combination (α = 0.5)
56.9
Chosen using trained reranker
63.6
Table 3: Comparison of Overall SRL F1 on devel-
opment set by the type of parse trees used.
