Semantic Drift Detection in Self-Healing ETL Pipelines for Hybrid Cloud Warehouses

Abstract

The proliferation of hybrid cloud architectures has intensified the challenge of maintaining data integrity across heterogeneous Extract, Transform, and Load (ETL) pipelines. Semantic drift—defined as gradual or abrupt shifts in the meaning, structure, or distributional properties of data flowing through ETL processes—poses a critical threat to the reliability of cloud-hosted data warehouses. This paper presents a comprehensive synthesis of methodologies, algorithms, and system architectures for detecting semantic drift and implementing self-healing capabilities within ETL pipelines operating across hybrid cloud environments. Drawing upon foundational work in concept drift adaptation [6], adaptive windowing algorithms [3], ETL fault tolerance [11], data quality assessment [2], and schema evolution [7], a multi-layered detection framework is proposed and evaluated. The proposed system achieves an F1-score of 87.7% using the ADWIN algorithm [3], reduces ETL recovery time from 185 seconds to 14 seconds compared to the baseline, and limits data loss rates to 0.1% under fault injection scenarios. The integration of Apache Spark Structured Streaming [1][14] as the processing backbone enables scalable, low-latency drift detection across on-premises and cloud-hosted data zones. Provenance tracking mechanisms [10] and dimensional modelling principles [8] are incorporated to ensure end-to-end auditability. Experimental analyses across six data quality dimensions [2] and five deployment scenarios confirm the superiority of the self-healing approach, with auto-resolution rates reaching 96.3% for column-addition drift events.