Questionable Reproducibility In Deep Learning: Environmental Effects On Published Results.pdf

The author argues that while source code sharing is a step in the right direction, it's insufficient to ensure reproducibility due to unreported environmental factors that can significantly impact model performance.

Key Findings:

Controllable environmental effects: The paper identifies several unreported variables (e.g., hardware specifications, optimizer settings, hyperparameters) that can dramatically alter model effectiveness. These factors are often crucial for achieving published results but remain hidden in the literature.
Impact on reproducibility: Due to the non-convex nature of optimization landscapes, even minor changes in computation can cause models to converge to different local minima. This raises questions about the validity of reported improvements and associated claims of progress.
Generalizability: The effects observed are not specific to a particular model or task; they apply broadly to all neural network-based research.
Call for better reporting practices: The paper emphasizes the need for researchers to report detailed experimental setups, including all environmental factors, to ensure true reproducibility and allow for further validation of results.

Specific Examples:

The study uses a simplified question answering model based on a Siamese architecture (Sequiera et al., 2017) and tests its performance against the TrecQA dataset. They demonstrate that variations in:

Hardware: Different GPUs can lead to varying training times and model performance.
Optimizer settings: Choices like learning rate, momentum, and weight decay have a significant impact on convergence and final accuracy.
* Hyperparameters: Parameters related to the network architecture (e.g., embedding size, hidden layer sizes) directly influence the effectiveness of the model.

Conclusion:

The paper argues that true progress in scientific research requires not only sharing code but also openly documenting and replicating experimental setups. This transparency allows for community scrutiny, reproducibility checks, and more robust advancement of the field.