Step 03 of 9 4-6 weeks· advanced
Step 3: Pipeline Framework Design
Build the metadata-driven framework before building any specific pipelines. Multiplier work — done right, every subsequent pipeline is faster.
Recommended prompts
Use one of these to do the work in your IDE
Open the template to read it in full. Click Copy prompt to grab it (with your stack values pre-filled where they apply) — then paste into Claude Code, Cursor, or wherever you build.
Recommended skills
Drop these into Claude Code for this phase
Skills auto-trigger on the right kind of request. Install once; they apply to every prompt that fits.
Recommended MCP configs
Wire these tools into Claude Code first
MCP servers give Claude Code direct access to external systems (Jira, browsers, databases). Configure once.
When you're done
Verify these in your own work before moving on
This is a checklist for you to mentally tick off in your repo and IDE — the site doesn't track it, you do.
- Metadata-driven framework deployed in dev
- Master orchestration pipeline working
- Sample child pipeline working with metadata
- Error-handling pattern proven
- Logging to meta table working
- Notifications integrated
- Repository structure documented
- Sample "onboarding a new source" runbook written
Common pitfalls
What goes wrong at this step
- Skipping metadata-driven design — building 50 nearly-identical pipelines is the most common ADF anti-pattern
- No error-handling baseline — pipelines without this are time bombs
- Hardcoded credentials — use Key Vault from day 1
- No logging — debugging at 3am without structured logs is brutal
- Building the framework without a real pipeline — validate on at least one real source before declaring it done