data_pipeline_architecture
Comprehensive guide to designing and implementing production data pipelines.
Content Preview
# Data Pipeline Architecture Comprehensive guide to designing and implementing production data pipelines. ## Table of Contents 1. [Architecture Patterns](#architecture-patterns) 2. [Batch Processing](#batch-processing) 3. [Stream Processing](#stream-processing) 4. [Exactly-Once Semantics](#exactly-once-semantics) 5. [Error Handling](#error-handling) 6. [Data Ingestion Patterns](#data-ingestion-patterns) 7. [Orchestration](#orchestration) --- ## Architecture Patterns ### Lambda Architecture
How to Use
Recommended: Install to project (local)
mkdir -p .claude/skills
curl -o .claude/skills/data_pipeline_architecture.md \
https://raw.githubusercontent.com/alirezarezvani/claude-skills/main/engineering-team/senior-data-engineer/references/data_pipeline_architecture.mdSkill is scoped to this project only. Add .claude/skills/ to your .gitignoreif you don't want to commit it.
Alternative: Clone full repo
git clone https://github.com/alirezarezvani/claude-skillsThen reference at engineering-team/senior-data-engineer/references/data_pipeline_architecture.md
Related Skills
cs-engineering-lead
Engineering Team Lead agent for coordinating QA, security, data engineering, ML, and frontend/backend teams. Orchestrates engineering-team skills for team-level technical decisions. Spawn when users need team coordination, tech stack evaluation, incident response, or cross-functional engineering work.
agentsengineeringleadagent
by alirezarezvani · alirezarezvani-claude-skills
data
Data engineering for Apache Airflow and Astronomer. Author DAGs with best practices, debug pipeline failures, trace data lineage, profile tables, migrate Airflow 2 to 3, and manage local and cloud deployments.
pluginpluginmarketplacedata
by Anthropic · anthropic-official-plugins
feature_engineering_patterns
World-class feature engineering patterns for senior data scientist.
engineering-teamfeatureengineeringpatterns
by alirezarezvani · alirezarezvani-claude-skills
data-engineering
Data engineering patterns for ETL pipelines, data warehousing, Apache Spark, and data quality validation
data-engineeringdataengineering
by rohitg00 · awesome-claude-toolkit