Data Engineer Roadmap

A step-by-step Data Engineer roadmap: the full stack, the skills to learn in order and the best places to learn each step.

The stack, end to end

  • Backend: Python, Lists, dictionaries and sets, Functions and modules, Virtual environments
  • Database: SQL, SELECT and WHERE, JOINs, GROUP BY and aggregates, INSERT, UPDATE, DELETE, Subqueries and CTEs, Window functions, Indexes, Query optimisation, Transactions, Stored procedures
  • Data: Data warehousing, Star schema, Dimensional modelling, ETL vs ELT, ETL pipelines, Apache Airflow, dbt, Data quality checks, Apache Spark, Apache Kafka, Streaming, Parquet
  • Cloud: AWS, EC2 and Lambda, S3, IAM, VPC
  • DevOps: Docker, Dockerfile, Docker Compose, Container networking
  • Get hired: Portfolio projects, Technical resume, Interview preparation

  1. Python

    Suggested pace: Start here

    Python is readable, popular and used for web, automation, data and AI. Learn the syntax, data structures, functions, files, errors, virtual environments and packages.

    Skills

    • Python
    • Lists, dictionaries and sets
    • Functions and modules
    • Virtual environments

    Tips

    • Automate something boring in your own life. It keeps motivation high.

    Where to learn

  2. SQL basics

    Suggested pace: Next

    SQL is how you ask a relational database questions. Learn SELECT, WHERE, ORDER BY, GROUP BY, aggregate functions, joins and how to create, insert, update and delete data.

    Skills

    • SQL
    • SELECT and WHERE
    • JOINs
    • GROUP BY and aggregates
    • INSERT, UPDATE, DELETE

    Tips

    • Practise on real-looking data. Write the query by hand before using any tool that writes it for you.

    Where to learn

  3. Advanced SQL and performance

    Suggested pace: Then

    Go beyond the basics: subqueries, common table expressions, window functions, indexes, reading execution plans, transactions and stored procedures. This is what separates a SQL developer from a beginner.

    Skills

    • Subqueries and CTEs
    • Window functions
    • Indexes
    • Query optimisation
    • Transactions
    • Stored procedures

    Tips

    • Learn to read an execution plan. It shows you why a query is slow.

    Where to learn

  4. Data modelling and warehousing

    Suggested pace: After that

    Learn how data is organised for reporting: facts and dimensions, star schemas, and the difference between a transactional database and a warehouse.

    Skills

    • Data warehousing
    • Star schema
    • Dimensional modelling
    • ETL vs ELT

    Tips

    • Model a small business (a shop, a school) as a star schema on paper.

    Where to learn

  5. Pipelines and orchestration

    Suggested pace: Later

    Pipelines move and transform data on a schedule. Learn to build reliable ones: extract, transform, load, handle failures and re-runs, and schedule them with a tool such as Apache Airflow.

    Skills

    • ETL pipelines
    • Apache Airflow
    • dbt
    • Data quality checks

    Tips

    • Design every step so it can be safely re-run.

    Where to learn

  6. Big data and streaming

    Suggested pace: Later

    When data no longer fits on one machine, you need distributed tools: Apache Spark for large-scale processing and Apache Kafka for streams of events.

    Skills

    • Apache Spark
    • Apache Kafka
    • Streaming
    • Parquet

    Tips

    • Learn why distributed processing is harder before reaching for it. Many jobs do not need it.

    Where to learn

  7. Amazon Web Services (AWS)

    Suggested pace: Later

    Learn the core services: compute (EC2, Lambda), storage (S3), databases (RDS, DynamoDB), networking (VPC), identity (IAM) and monitoring. Use the free tier and set a budget alert.

    Skills

    • AWS
    • EC2 and Lambda
    • S3
    • IAM
    • VPC

    Tips

    • Set a billing alert before you create anything, and delete what you are not using.

    Where to learn

  8. Docker and containers

    Suggested pace: Later

    A container packages an app with everything it needs so it runs the same everywhere. Learn images, containers, volumes, networks, Dockerfiles and Docker Compose.

    Skills

    • Docker
    • Dockerfile
    • Docker Compose
    • Container networking

    Tips

    • Containerise one of your own projects end to end.

    Where to learn

  9. Build your portfolio, resume and interview skills

    Suggested pace: When you are ready

    Employers want proof. Finish two or three projects you can show and explain, put them on GitHub, write a clear resume that matches the jobs you apply for, and practise the interview questions for the role.

    Skills

    • Portfolio projects
    • Technical resume
    • Interview preparation

    Tips

    • Describe what you built, why, and what you would improve. That is what interviewers ask about.

    Build your resume · Check your resume (ATS) · Interview questions · Salaries · Browse jobs

Frequently asked questions

Where should I start with Data Engineer?

Start at step 1 and work down in order, because later steps build on earlier ones. If you are signed in, the roadmap compares your resume with each step and marks where you could start.

Do I need a degree to work as a Data Engineer?

It depends on the employer and the country. Many employers accept proven skills and projects, while others ask for a degree. Read the adverts for the roles you want and ask the employer if it is unclear.

How do I know I am ready to apply?

When you can build a small project from start to finish without copying a tutorial, and explain why you made your choices. Apply while you keep learning; you do not need to know everything.