From
fundamentals
to production.

A structured path to become a Data Engineer. Master programming and SQL, build modern Azure data platforms, and finish by shipping a complete end-to-end system.

⌘6 Learning StagesStructured & practical

◆Azure-first StackIndustry-relevant

⌘End-to-end ProjectBuild real skills

▥Interview ReadyPractice key concepts

Start Learning →Explore the Stack ▷ View Project

⌁ Build Real Data SolutionsSame stack. Real-world approach.

MULTIPLE SOURCESstructured · unstructured

▣SOURCESQL · API · CSV

◇ADFINGEST

●ADLS GEN2RAW ZONE

▱DATABRICKSPYSPARK · DELTA

◇SYNAPSEANALYTICS

▮POWER BIINSIGHTS

BUSINESS INSIGHTSmetrics · dashboards

● INGEST● STORE● TRANSFORM● SERVE● VISUALIZE

STAGE 01 · FOUNDATION

SQL + Python

Build the two core skills used throughout the roadmap: querying data and writing reliable data-engineering code.

Start here

01A

Python Fundamentals

PROGRAMMING

Variables & data typesOperatorsConditionalsLoopsLists · tuples · sets · dictsStringsFunctionsExceptionsOOPFile handlingModules & packagesLambdaComprehensionsGeneratorsDecoratorsLoggingVirtual environmentsTruthy / Falsy

CHECKPOINTWrite clean Python scripts, handle failures, structure projects and explain OOP concepts.

01B

Python for Data

DATA TOOLING

PandasNumPyRequests APIJSON processingPython DB connectionPySpark basics

CHECKPOINTConnect Python to a database, consume an API, process JSON and manipulate tabular data.

01C

SQL Engineering

QUERYING

SELECT · WHERE · ORDER BYGROUP BY · HAVINGDISTINCTAll JOIN typesDDL · DML · DCL · TCLSubqueriesCTEsRecursive CTEWindow functionsIndexesExecution plansOptimizationViews & proceduresFunctions & triggers

CHECKPOINTSolve real SQL interview problems and reason about query correctness and performance.

01D

Database Concepts

DATA FOUNDATIONS

Relational databasesPrimary / foreign keysConstraints1NF · 2NF · 3NF · BCNFNoSQL conceptsMongoDBCassandraCosmos DB

CHECKPOINTChoose relational vs NoSQL storage based on workload and data-model requirements.

STAGE 02 · PLATFORM BASICS

Data Warehousing + Azure Data Factory

Understand analytical modeling and the mechanics of moving data through repeatable pipelines.

Model + ingest

02A

Data Warehousing

MODELING · OLAP

OLTP vs OLAPWarehouse architectureData martsStar schemaSnowflake schemaFact & dimension tablesSCD 0 · 1 · 2 · 3 · 4 · 6Surrogate keysGrainAggregationPartitioning

02B

ETL / ELT

PIPELINE PATTERNS

Extract · transform · loadData qualityValidationCleansingProfilingFull / incremental / deltaCDCSchedulingBatchReal-time

02C

Azure Fundamentals

CLOUD FOUNDATION

IaaS · PaaS · SaaSSubscriptionsResource groupsRegionsAvailability zonesVNet & subnetsEntra IDRBACManaged identity

02D

Azure Data Factory

ORCHESTRATION

PipelinesActivitiesDatasetsLinked servicesCopy activityData FlowLookupStored ProcedureWebIf / Switch / ForEach / UntilAzure IRSelf-hosted IRMonitoringRetries & alerts

CHECKPOINTBuild an ingestion pipeline with parameters, incremental loading, retries, monitoring and failure handling.

STAGE 03 · DISTRIBUTED PROCESSING

PySpark + Azure Databricks

Move from single-machine data manipulation to distributed transformation with Spark and Delta Lake.

Transform at scale

03A

PySpark

DISTRIBUTED COMPUTE

DataFramesTransformationsActionsLazy evaluationNarrow / wide transformationsPartitioningCachingBroadcast joinsShuffle

03B

Azure Databricks

SPARK + DELTA LAKE

WorkspaceClustersDriver / executorNotebooksDelta LakeACIDTime travelMERGE / upsertWorkflowsUnity Catalog

CHECKPOINTBuild a Bronze → Silver transformation and explain Spark execution, partitioning and Delta Lake guarantees.

STAGE 04 · ANALYTICS + BI

Synapse + Power BI

Serve curated data for analytics and build a semantic layer that turns the platform into business value.

Serve + visualize

04A

Azure Synapse

ANALYTICS WAREHOUSE

Synapse StudioDedicated SQL poolServerless SQL poolSpark poolSynapse pipelinesDistributionPartitioningMaterialized views

04B

Power BI

SEMANTIC LAYER

ImportDirectQueryRelationshipsStar schemaPower QueryDAXCALCULATEFILTERSUMXRANKXVisualization

CHECKPOINTPublish a governed semantic model and dashboard backed by curated Synapse or Databricks data.

04C

Stream Processing

REAL-TIME DATA

Event processingEvent HubsStream AnalyticsTumbling windowsSliding windowsSession windows

STAGE 05 · SHIP + GOVERN

Git + Azure DevOps

Turn working notebooks and pipelines into maintainable, testable and deployable engineering systems.

Operate

05A

Git + Azure DevOps

CI/CD · IAC

BranchingMergingPull requestsAzure ReposAzure PipelinesBuild / releaseCI/CDARM templatesTerraform basicsPipeline testingMonitoring & SLAs

05B

Security + Governance

TRUST

RBACLeast privilegeEncryption at restEncryption in transitKey VaultMaskingRLSLineageCatalogPurview

05C

Azure SQL + Cosmos DB

OPERATIONAL STORES

Azure SQL DatabaseManaged InstanceQuery StoreScalingTDEDynamic Data MaskingAlways EncryptedCosmos partition keysThroughputConsistencySQL / Mongo APIs

05D

Architecture + Airflow

SYSTEM DESIGN · SUPPORTING ORCHESTRATION

Bronze / Silver / GoldMedallionLambdaKappaAirflow DAGsSchedulingFailure modesTrade-offs

CHECKPOINTExplain architecture choices, secure the platform, version the code and deploy changes through CI/CD.

STAGE 06 · CAPSTONE

End-to-End Data Engineering Project

Build, deploy and showcase a complete Azure-first data platform from ingestion to business insight.

Build · deploy · showcase

06A

End-to-End Platform

CAPSTONE

Source ingestionADF orchestrationData Lake storageDatabricks transformationSynapse servingPower BI semantic modelCI/CDMonitoringSecurityDocumentation

CHECKPOINTShip a portfolio-ready system and explain every engineering decision from source to dashboard.