Home › Categories › Technology› Download

Download.pdf

differential-data-quality-verification-for-partitioned-data.pdf

🔗 Source: assets.amazon.science

📊 Size: 358 KB

📄 Pages: 6 pages

⬇️ Downloads: 338

Summary

Modern companies rely on data to guide decisions, but missing or incorrect information can compromise this process. Deequ, a Spark-based library, automates data quality verification at scale, but its previous computational model is not flexible enough for many scenarios in modern data pipelines. The authors present a differential generalization of Deequ's computational model, using algebraic states with monoid properties, to efficiently evaluate dataset-level quality constraints after individual partition updates, without re-reading already processed partitions. This approach enables the verification of data quality on select combinations of partitions and optimizes workloads to minimize passes over the data. The authors demonstrate the effectiveness of their approach through empirical validation, showing decreased runtimes for updating data metrics under data changes and for different combinations of partitions.

Download File

Description

Modern companies rely on data to guide decisions, but missing or incorrect information can compromise this process.

Technical Information

File Format: PDF
File Size: 358 KB
Pages: 6
Language: EN
Total Downloads: 338
Last Updated: 1 month ago

Document Overview

This PDF document about Download provides comprehensive information and guidance. Whether you're a beginner or advanced user, this resource offers valuable insights into Download.