What is Shift Left?

Historically data engineering teams have relied on Extract-Transform-Load (ETL) pipelines with finite storage and limited compute to take raw datasets from on-premises databases, transform it into the most relevant and load it into expensive data warehouses. An exponential growth in the number of systems (both on-prem and on the cloud) brought about multiple point-to-point ETL processes across the organization creating a giant mess — complicated, expensive, and risky.

In an attempt to alleviate the mess, the next generation of cloud-enabled tools tried to “Shift Right” and centralize all their data into one place before processing for further consumption, often in a multi-hop medallion architecture. Cloud-based data tools enabled plentiful, on-demand, and cheap storage while computers became powerful and supposedly cheaper. The pendulum swung to Extract-Load-Transform (ELT) workloads that ingest a variety of data as-is into a cloud data warehouse or data lakehouse for in-place transformation.

Data continued to explode in volume, variety, and velocity with more and more applications, systems, and teams across the organization requiring access to data for reporting or analytics. The ostensibly cheaper and faster alternative using cloud data warehouses, lakehouses, and ELT pipelines, that enabled accessibility and usage by multiple data teams, ironically turned these systems into data swamps: massive and ungoverned messes with high latency and numerous lower-value workloads dramatically increasing costs. Worse yet, organizations see the need for their data in multiple places by multiple systems and stakeholders, and trip all over themselves rebuilding the same cleaning logic in multiple places using “reverse ETL” pipelines to feed data from lakehouses and data warehouses back to operational systems. 

So-called ‘modern’ data stacks of today continue to be built on complex, time-intensive, expensive ELT and batch-based multi-hop data architectures that shift processing downstream to the right, dramatically increasing costs, and limiting innovation and growth.

• Stale and inconsistent data: Batch processing increases latency and creates low fidelity snapshots leading to poor business decisions made on outdated inputs

• Redundant processing: Most organizations have multiple downstream systems, including different warehouses, applications or AI platforms. Teams often do the same processing in multiple places or create copies of data to apply processing logic to to meet various use case demands. In addition, when data arrives in micro-batches, additional processing is required to stitch together the incremental changes. 

• Bad data and manual break-fix: The rapid pace of development, frequent changes upstream, and distributed teams often means that data producers and consumers are not communicating effectively, increasing the chances of bad data being generated and proliferated. In a traditional setup, data producers are not involved in what happens downstream and data consumers have very few or no ways to influence the data being produced upstream.

• Pipeline sprawl: Lack of good documentation on lineage leads to teams adding new “similar-yet-different” pipelines for new use cases, further worsening pipeline sprawl and increasing operational overheads, where expensive resources are spent on managing ETL jobs instead of focusing on innovation.

Organizations that continue to run data processing and governance processes including large, complex, and inefficient batch-based transformations downstream in a data life cycle face:

• Increased costs from redundant and intensive processing, time spent on low-value tasks, data downtime, and data proliferation

• Long time-to-value for data engineering teams running business initiatives from complicated batch-processing and redundant efforts

• Insufficient data adoption due to data integrity challenges and mistrust

• Sub-optimal innovation and growth: Resources are wasted on low-value activities like data cleaning, and pipeline maintenance instead of building better customer experiences

Shift Left ensures that data downstream is always fresh and up-to-date; that it’s trustworthy, reliable, discoverable and instantly usable so teams across the organizations can get data insights or build new applications more easily. 

Shifting Left eliminates data inconsistency challenges, reduces the duplication of processing and associated costs, prevents data quality issues before they become problematic downstream, and maximizes the ROI of the data warehouse and data lakes. 

Shift Left delivers the benefits from batch based legacy approaches without the drawbacks that come with these previous approaches by enabling:

• A single source of truth upstream, eliminating waste and reducing TCO

• Governed / bottled-at-the-source data with data contracts

• Re-usable, high-quality data products that unifies operational and analytical estates

• Real-time decision making with fresh, trustworthy data

With Shift Left, organizations can free up valuable time and resources and accelerate self-service data access and discovery to decrease costs of existing data infrastructure while enabling faster business decisions and fueling innovation and growth.

Confluent’s cloud-native, all-in-one Data Streaming Platform supports streaming, connecting, processing and governing all of an organization’s data in real-time and at scale.

Confluent provides the best ecosystem of zero code connectors for data streaming including various CDC connectors to synchronize changes data downstream. Once that data is in Confluent, the schema registry enables quality control, Flink stream processing enables data enrichment and transformation on in-flight data to create ready-to-use reusable data products that can be simultaneously consumed by your operational and analytical systems and applications. And with the simplification of operational data access in the analytical estate with Tableflow, Confluent brings a truly unified view of the data, as a stream or as a table — bringing about a true convergence of the operational and analytical data estates.

• Connect & Stream real-time data anywhere: Continuously capture and share real-time data everywhere — to your data warehouse, data lake, operational systems and applications.

• Deliver well-curated, reusable data products: Continuously process data in-stream with Flink the moment it’s created, to shape data into multiple contexts on-the-fly 

• Build trusted, high-quality data products at source: Reduce faulty data downstream by enforcing quality checks and controls in the pipeline with data contracts, schema registry, and data quality rules.

• Self-service data discovery with Data Portal: Enable anyone with the right access controls to effortlessly explore and use real-time data products for greater data autonomy

• View Stream Lineage: Understand data relationships and the data journey in visual graphs to ensure trustworthiness

• Read data products as stream or tables with Tableflow: Simplify representing your operational data as a ready-to-use Iceberg table for analytical systems in just one-click