Spark Optimization
技能 已验证 活跃Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.
Optimize Apache Spark jobs by providing expert patterns and configurations for partitioning, memory management, shuffle optimization, and caching.
功能
- Optimize Apache Spark jobs
- Improve Spark performance
- Debug slow Spark jobs
- Scale data processing pipelines
- Provide best practices for partitioning, caching, memory, and shuffle tuning
使用场景
- Optimizing slow Spark jobs
- Tuning memory and executor configuration
- Implementing efficient partitioning strategies
- Debugging Spark performance issues
- Scaling Spark pipelines for large datasets
非目标
- Running Spark jobs directly
- Managing Spark cluster infrastructure
- Providing a general-purpose Python coding assistant
Versioning
- info:Release ManagementWhile there is no explicit versioning in the skill's frontmatter or CHANGELOG, the installation method refers to 'HEAD' and the code itself is updated frequently.
安装
请先添加 Marketplace
/plugin marketplace add wshobson/agents/plugin install data-engineering@claude-code-workflows质量评分
已验证类似扩展
Spark Engineer
99Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads. Invoke to write DataFrame transformations, optimize Spark SQL queries, implement RDD pipelines, tune shuffle operations, configure executor memory, process .parquet files, handle data partitioning, or build structured streaming analytics.
Data Engineer
94Build scalable data pipelines, modern data warehouses, and real-time streaming architectures. Implements Apache Spark, dbt, Airflow, and cloud-native data platforms.
Performance Analysis
100Comprehensive performance analysis, bottleneck detection, and optimization recommendations for Claude Flow swarms
Oraclaw Solver
100为 AI 代理提供工业级的调度和资源优化。在几毫秒内通过能源匹配、预算分配和任何 LP/MIP 约束问题来解决任务调度。
Oraclaw Decide
100为 AI 代理提供决策智能。分析选项、使用 PageRank 映射决策依赖关系、检测信息源冲突，并找出最重要的选择。
MongoDB Connection Optimizer
100为任何支持的驱动程序语言优化 MongoDB 客户端连接配置（池、超时、模式）。在处理/更新/审查实例化或配置 MongoDB 客户端（例如，调用 `connect()` 时）、配置连接池、对连接错误（ECONNREFUSED、超时、池耗尽）进行故障排除、优化与连接相关的性能问题时，请使用此技能。这包括构建具有 MongoDB 的无服务器函数、创建使用 MongoDB 的 API 端点、优化高流量 MongoDB 应用程序、创建长期运行任务和并发性，或调试与连接相关的失败等场景。