data-engineering

---
name: data-engineering
description: Data engineering patterns for ETL pipelines, data warehousing, Apache Spark, and data quality validation
---

# Data Engineering

## ETL Pipeline Pattern

```python
from datetime import datetime
from dataclasses import dataclass

@dataclass
class PipelineResult:
    records_extracted: int
    records_transformed: int
    records_loaded: int
    errors: list[str]
    duration_seconds: float

class OrderPipeline:
    def __init__(self, source_db, warehouse_d