Topic · Code and Developer
Data Engineering and Pipelines skills for Claude Code and Codex
Skills for data pipelines and ETL: Airflow DAGs on MWAA and Cloud Composer, dbt translation, Spark, Flink, Kafka and Kinesis streams, data lake ingestion, lineage, and dataframe engines such as Polars, Dask, cuDF and Ray Data. They help an agent build, debug and migrate the jobs that move data.
Highest-scoring data engineering skills in the October 22, 2026 snapshot: dbt-sf-to-bq-translator from google/skills (89/100). Scores come from the Suede Skill Check.
- 31 listed skills
- 7 repositories
- 2 with full pages
26 from official publishers. 3 bundle scripts. 1 connect to an outside service. Listed in Data, Analytics and Databases (19), DevOps and Cloud (5), Testing and QA (4), AI Agents and LLM Engineering (2), Frontend and Design (1).
All 31 data engineering skills
No matching skills. Try fewer words or clear your search.
| Skill | Repository | What it does |
|---|---|---|
| dbt-sf-to-bq-translator 89/100 · B | google/skills | Translates Snowflake dbt SQL models to BigQuery SQL, handling SQL compilation, Jinja macro masking, BigQuery Translation Service workflows, config transformations, type casting, and deduplication. |
| google-cloud-solution-agentic-analytics-spark-knowledge-catalog 85/100 · B | google/skills | Discovers requirements and designs a governed agentic analytics solution using Knowledge Catalog and Managed Service for Apache Spark, with metadata governance. |
| accelerated-computing-cudf 73/100 · C | NVIDIA/skills | Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV/Parquet I/O, nullable… |
| authoring-mwaa-workflow 76/100 · C | aws/agent-toolkit-for-aws | Authors and deploys MWAA workflow artifacts: Python Airflow DAGs for provisioned environments or YAML workflow files for Serverless. Covers operator… |
| aws-messaging-and-streaming 77/100 · C | aws/agent-toolkit-for-aws | Guides general use of AWS messaging and streaming services. Covers Amazon SQS, Amazon SNS, Amazon EventBridge, Amazon MQ, Amazon Kinesis Data… |
| bigquery-pipeline-audit 67/100 · D | github/awesome-copilot | Audits Python + BigQuery pipelines for cost safety, idempotency, and production readiness. Returns a structured report with exact patch locations. |
| creating-data-lake-table 82/100 · B | aws/agent-toolkit-for-aws | Create managed Iceberg tables using Amazon S3 Tables (s3tables API namespace) with automatic compaction and snapshot management. Sets up table… |
| dask 87/100 · B | K-Dense-AI/claude-scientific-skills | Scales pandas, NumPy, and custom Python research workflows beyond memory or across clusters with Dask. Covers DataFrames, Arrays, Bags, Futures… |
| datalineage-bigquery-asset-impact-analysis 85/100 · B | google/skills | Analyzes the downstream impact (blast radius) using data lineage on Google Cloud when a BigQuery table or view is broken, stale, or modified… |
| datalineage-summary 87/100 · B | google/skills | Summarizes data lineage graphs on Google Cloud to help users debug data quality issues and understand data provenance for BigQuery and Cloud… |
| debugging-mwaa-workflow 78/100 · C | aws/agent-toolkit-for-aws | Diagnoses and root-causes Amazon MWAA workflow failures across Provisioned (Python DAG) and Serverless (YAML workflow) environments. Provisioned… |
| developing-applications-on-managed-service-for-apache-flink 68/100 · D | aws/agent-toolkit-for-aws | MANDATORY for Flink or Amazon Managed Service for Apache Flink (MSF) questions. You MUST activate this skill BEFORE answering — do not answer from… |
| exploring-data-catalog 84/100 · B | aws/agent-toolkit-for-aws | Full inventory and audit of AWS Glue Data Catalog assets across S3 Tables, Redshift-federated, and remote Iceberg catalogs. Triggers on: inventory… |
| fabric-lakehouse 80/100 · B | github/awesome-copilot | Use this skill to get context about Fabric Lakehouse and its features for software systems and AI-powered functions. It offers descriptions of… |
| finding-data-lake-assets 84/100 · B | aws/agent-toolkit-for-aws | Resolve data lake and lakehouse asset references across Glue Data Catalog, S3, S3 Tables, and Redshift. Triggers on: find the table, where is our… |
| google-cloud-solution-agentic-ai-borderless-data-lakehouse 85/100 · B | google/skills | Discovers requirements and designs a borderless open data lakehouse using Lakehouse for Apache Iceberg and BigQuery data agents. Use when… |
| ingesting-into-data-lake 80/100 · B | aws/agent-toolkit-for-aws | Import data into the AWS data lake from S3 files, local uploads, JDBC databases (Oracle, SQL Server, PostgreSQL, MySQL, RDS, Aurora), Amazon… |
| managed-airflow-dag-authoring 84/100 · B | google/skills | Provides guidance for authoring Apache Airflow DAGs in Managed Service for Apache Airflow (MSAA; formerly Cloud Composer). Covers environment… |
| managed-airflow-dag-troubleshooting 84/100 · B | google/skills | Provides guidance for troubleshooting Apache Airflow DAGs (failed DAG runs and task instances) in Managed Service for Apache Airflow (MSAA; formerly… |
| managed-airflow-migrations 87/100 · B | google/skills | Provides guidance for migrating Apache Airflow DAGs in Managed Service for Apache Airflow (MSAA; formerly Cloud Composer). Covers migration to… |
| managing-amazon-kinesis-data-streams 79/100 · C | aws/agent-toolkit-for-aws | Operates Amazon Kinesis Data Streams (KDS). Covers streaming tables and Amazon S3 delivery - serverless, fully managed delivery from a KDS stream to… |
| managing-amazon-msk 82/100 · B | aws/agent-toolkit-for-aws | Operates Amazon MSK Provisioned clusters (Standard and Express brokers). Required for ANY MSK Provisioned task — training data conflates Standard… |
| migrate-to-msk 78/100 · C | aws/agent-toolkit-for-aws | Helps migrate self-managed Apache Kafka workloads to Amazon MSK Express. Inventories the source cluster (from IaC files, Kafka CLI output, or manual… |
| ml-pipeline 82/100 · B | jeffallan/claude-skills | Designs and implements production-grade ML pipeline infrastructure: configures experiment tracking with MLflow or Weights & Biases, creates Kubeflow… |
| paidf-orchestration-write-dag 78/100 · C | NVIDIA/skills | Use when a user describes a custom PAIDF Orchestration pipeline — a specific ordered combination of stages such as augmentation only, auto-labeling… |
| polars 79/100 · C | K-Dense-AI/claude-scientific-skills | High-performance DataFrame library for Python ETL, analytics, and pandas migration. It supports expression-based data manipulation with lazy query… |
| querying-data-lake 84/100 · B | aws/agent-toolkit-for-aws | Execute and manage Athena SQL queries across default and federated catalogs (Glue, S3 Tables, Redshift). Triggers on phrases like: query data, run… |
| ray-data 77/100 · C | Orchestra-Research/AI-Research-SKILLs | Scalable data processing for ML workloads. Streaming execution across CPU/GPU, supports Parquet/CSV/JSON/images. Integrates with Ray Train, PyTorch… |
| spark-engineer 84/100 · B | jeffallan/claude-skills | Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data… |
| testing-mwaa-workflow 78/100 · C | aws/agent-toolkit-for-aws | Tests Amazon MWAA workflow execution end-to-end: trigger a run and monitor it to completion for Provisioned (Python DAG, via Airflow REST API) and… |
| upgrading-mwaa-environments 77/100 · C | aws/agent-toolkit-for-aws | Upgrades an MWAA environment to a newer Airflow version — within 2.x, within 3.x, or across the 2.x-to-3.x boundary. Computes the version-jump path… |
How to choose data engineering skills
Start with the orchestrator you run. Google covers managed Airflow authoring, troubleshooting and migrations; AWS covers MWAA, Kinesis, MSK and data lake ingestion.
Pipeline skills touch production data and cloud resources. Give the agent a dev environment and a sample dataset, and review DAG and job changes before they deploy.
Dataframe skills (Polars, Dask, cuDF, Ray Data) help with the transform step itself; pick by data size and whether you have a GPU.
Ranked data engineering shortlists
Related topics
AWS 151Data Warehouses 29Google Cloud 146Data Analysis 27SQL and Databases 116
Find your next useful skill
Explore Rising Skills with quality scores and source reports, or search from inside your agent.
Get Rising Skills by email · Read Rising Skills · Install the finder · State of Agent Skills