Knowledge Graph Embeddings Evaluation.pdf

Knowledge graph embedding models are biased towards popular entities and relations, leading to overestimation of their performance. Current evaluation metrics, such as hits@k and mrr, are flawed as they calculate accuracy proportionally to entity and relation frequency, under-emphasizing predictions about unpopular entities and relations.

To address this issue, two new evaluation metrics, strat-hits@k and strat-mrr, are proposed, which take into account the popularity of entities and relations. These metrics provide a more accurate estimation of model performance by exposing the popularity bias in embedding models.

Experiments on benchmark datasets show that the performance of embedding models degrades as the popularity of entities and relations decreases. The proposed metrics can help identify models that perform well on both popular and long-tail items, which is essential for effective knowledge graph completion.

The popularity bias in knowledge graphs is attributed to the fact that most graphs are automatically constructed from online sources with intrinsic biases. The proposed metrics can be used to evaluate models on knowledge graphs with any level of correlation between entity and relation popularity.

Overall, the new metrics provide a more nuanced understanding of model performance, allowing for the development of more effective embedding models that can propagate information to unfamiliar entities and relations, and infer new knowledge that cannot be easily extracted by online text mining or simple graphical models.