millitool
data-ai/

spark-optimization

Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.

sparkoptimization
Content Preview
---
name: spark-optimization
description: "Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines."
risk: unknown
source: community
date_added: "2026-02-27"
---

# Apache Spark Optimization

Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.

## Do not use this
How to Use

Recommended: Install to project (local)

mkdir -p .claude/skills
curl -o .claude/skills/spark-optimization.md \
  https://raw.githubusercontent.com/sickn33/antigravity-awesome-skills/main/skills/spark-optimization/SKILL.md

Skill is scoped to this project only. Add .claude/skills/ to your .gitignoreif you don't want to commit it.

Alternative: Clone full repo

git clone https://github.com/sickn33/antigravity-awesome-skills

Then reference at skills/spark-optimization/SKILL.md

Related Skills

data
Data engineering for Apache Airflow and Astronomer. Author DAGs with best practices, debug pipeline failures, trace data lineage, profile tables, migrate Airflow 2 to 3, and manage local and cloud deployments.
pluginpluginmarketplacedata

by Anthropic · anthropic-official-plugins

data-engineering-data-driven-feature
Build features guided by data insights, A/B testing, and continuous measurement using specialized agents for analysis, implementation, and experimentation.
data-aidataengineeringdriven

by sickn33 (Antigravity) · antigravity-awesome-skills

data-quality-frameworks
Implement data quality validation with Great Expectations, dbt tests, and data contracts. Use when building data quality pipelines, implementing validation rules, or establishing data contracts.
data-aidataqualityframeworks

by sickn33 (Antigravity) · antigravity-awesome-skills

data-scientist
Expert data scientist for advanced analytics, machine learning, and statistical modeling. Handles complex data analysis, predictive modeling, and business intelligence.
data-aidatascientist

by sickn33 (Antigravity) · antigravity-awesome-skills