AZURE DATA ENGINEERING + FABRIC DATA ENGINEERING

Introduction to Data Engineering:

• Overview of Data Engineering
• Definition and importance
• Key roles and responsibilities
• Introduction to Data Pipelines
• What is a data pipeline?
• Components of a data pipeline
• Examples of data pipelines in the industry

Azure Data Factory

Azure Data Factory

• 1. Intro to Azure Data Factory:

  • Introduction to ADF
  • Different ways to work with ADF
  • Pipelines and Activities in ADF
  • Linked Services and Data Sets
  • Triggers in ADF
  • Schedule Trigger in ADF
  • Tumbling Window Trigger
  • Event Based Triggers
  • Integration Runtime
  • Azure Integration Runtime
  • Self Hosted Integration Runtime
  • Derived Column transformations

2. Data Flows and Transformations:

• Derived Column transformations in dataflows
• Exists Transformations
• Union Transformations
• Lookup Transformations
• Sort Transformations
• New Branch in Mapping Dataflows
• Select Transformations
• Pivot Transformations
• Unpivot Transformations
• Surrogate Key Transformations
• Window Transformations
• Alter Row Transformations
• Flatten Transformations
• Parameterize Mapping Dataflow
• Validate Schema Mapping Data flow
• Schema Drift Mapping Dataflow
• Wrangling Dataflow
• Merge Queries in wrangling Dataflow
• Groupby in wrangling dataflow

3. Azure Data Factory - Advanced:

• Author Modes
• Set Up Git hub Code repo
• Setup Azure devops Git code repo
• Use Azure Key vault secrets
• CI/CD in ADF
• How to Read JSON Output from 1 Activity to Another
• Annotations in ADF
• Templates Overview in ADF
• Global Params in ADF
• Rank Transformations in ADF
• Cache Sink and cached Lookup
• Session Log in copy activity
• Write cache sink to activity output
• Parse Transformation in mapping Dataflow
• Fail Activity
• Inline Dataset
• Stringify Transformation in Dataflows
• Assert Transformations
• Flowlets
• Script Activity
• UDFs in Dataflows
• Fuzzy Joins in Dataflows
• Parameterize Linked Services
• Cast Transformation in Dataflows
• Extract Data from table of website Pages
• Per pipeline Billing view
• Create Alert Rules
• Pipeline return value in set variable
• Copy activity pagination rules

4. ADF Real Time Scenarios - 20.

5. Python for Data Engineering:

• Python Fundamentals
• Variables and Data Types
• Basic Operators
• Control Flow (if-else, loops)
• Functions and Modules
• Defining and calling functions
• Importing and using modules
• Working with Data Structures
• Lists, Tuples, Dictionaries, and Sets
• File Handling
• Reading and writing files
• Working with CSV and JSON files
• Python Libraries for Data Engineering
• Overview of Pandas, NumPy, and Matplotlib

SQL for Data Engineering:

1. Introduction to SQL and Databases

• What is SQL?
• Types of SQL commands (DDL, DML, DCL, TCL)
• Tables, rows, and columns basics

2. Data Definition Language (DDL)

• Creating tables using CREATE TABLE
• Modifying tables using ALTER TABLE
• Deleting tables with DROP TABLE

3. Data Manipulation Language (DML)

• Inserting data using INSERT INTO
• Updating records with UPDATE
• Deleting records using DELETE

4. Data Retrieval using SELECT Statement

• Basic SELECT queries
• Using WHERE clause to filter data
• Sorting with ORDER BY, limiting with LIMIT or TOP

5. Filtering and Pattern Matching

• AND, OR, NOT conditions
• Using LIKE, IN, BETWEEN, IS NULL

6. Aggregate Functions and Grouping

• Functions like COUNT(), SUM(), AVG(), MAX(), MIN()
• Grouping data using GROUP BY
• Filtering groups with HAVING

7. Introduction to Databricks:

• Overview of Databricks
• What is Databricks?
• Databricks vs traditional data platforms
• Setting Up Databricks Environment
• Creating a Databricks account
• Navigating the Databricks workspace
• Databricks Notebooks
• Creating and managing notebooks
• Using markdown and code cells

8. Data Ingestions and Transformation with Databricks:

• Data Ingestion Techniques
• Reading data from various sources (CSV, JSON, Parquet)
• Connecting to databases
• Data Transformation
• Basic transformations using Databricks
• Using SQL in Databricks
• Handling Missing Data and Duplicates
• Techniques for dealing with missing values
• Removing and handling duplicates

9. Apache Spark and PySpark:

• Overview of Apache Spark
• What is Apache Spark?
• Spark ecosystem and components
• Introduction to PySpark
• Setting up PySpark in Databricks
• PySpark vs Pandas

10. PySpark Basics:

• PySpark DataFrames
• Creating DataFrames
• Performing basic operations on DataFrames
• DataFrame Transformations and Actions
• Common transformations (select, filter, groupBy, etc.)
• Actions (collect, show, count, etc.)

11. Advanced PySpark Concepts:

• Working with Spark SQL
• Using SQL queries in PySpark
• Integrating SQL and DataFrame API
• User Defined Functions (UDFs)
• Creating and using UDFs
• Performance considerations

12. Data Aggregation and Analysis with PySpark:

• Aggregation Functions
• Grouping and aggregating data
• Window functions
• Data Joins in PySpark
• Different types of joins (inner, outer, etc.)
• Best practices for joins

13. Optimizing and Managing Spark jobs:

• Performance Tuning
• Caching and persistence
• Partitioning and shuffling
• Spark Job Monitoring and Debugging
• Using Spark UI for monitoring
• Debugging common issues

14. Advanced Topics in Databricks and PySpark:

• Delta Lake and Databricks Delta
• Introduction to Delta Lake
• Implementing Delta Lake in Databricks
• RealTime Data Processing with Structured Streaming
• Basics of Structured Streaming
• Building and managing streaming pipelines

Microsoft Fabric for Data Engineering

Module 1: Introduction to Microsoft Fabric

• What is Microsoft Fabric? Overview of its capabilities
• Key Components: Data Engineering, Data Factory, Synapse, OneLake, etc.
• Comparing Microsoft Fabric with Azure Synapse and Databricks
• Understanding Fabric’s Unified Data Lake (OneLake)
• Setting up a Microsoft Fabric Workspace

Module 2: Data Ingestion in Fabric

• Data Ingestion Methods: Batch vs. Streaming
• Connecting to Data Sources: Azure Blob, ADLS, SQL, APIs
• Using Fabric Pipelines for ETL (Extract, Transform, Load)
• Working with Eventstreams for real-time data
• Handling structured and unstructured data ingestion

Module 3: Storage and Management with OneLake

• Introduction to OneLake Storage in Microsoft Fabric
• OneLake vs. ADLS (Azure Data Lake Storage)
• Creating and Managing Lakehouses in Fabric
• Delta Tables: Format, Transactions, and Versioning
• Data Security and Access Control in OneLake

Module 4: Data Processing with Spark and Notebooks

• Introduction to Apache Spark in Fabric
• Setting up and running Spark Notebooks
• Data Transformation using PySpark
• Optimizing Spark Performance in Fabric
• Managing Spark Jobs and Scheduling

Module 5: Data Transformation with Dataflows and Pipelines

• Introduction to Dataflows Gen2
• Creating and Managing Dataflows
• Data Transformation with Power Query and M Language
• Automating Data Pipelines using Fabric Data Factory
• Debugging and Monitoring Fabric Pipelines

Module 6: Data Modeling and SQL Analytics in Fabric

• Understanding Fabric’s Data Warehouse
• Writing SQL Queries for Data Analysis
• Performance Optimization in Fabric SQL Engine
• Implementing Slowly Changing Dimensions (SCDs)
• Materialized Views and Query Optimization

Module 7: Orchestration & Automation

• Understanding Microsoft Fabric Data Factory
• Creating and Scheduling Pipelines
• Integrating Data Factory with Synapse and Power BI
• Error Handling and Logging Mechanisms
• CI/CD Deployment in Microsoft Fabric