About Course
This course will equip students with the knowledge and skills to effectively use Azure Data Factory for data integration and management. They will learn how to connect to various data sources, automate data workflows, and process large datasets efficiently. By the end, students will be able to build scalable data pipelines, handle real-world data challenges, and contribute to data-driven decision-making within organizations. This hands-on learning experience will prepare them for roles in data engineering, analytics, and cloud computing, enhancing their career prospects in the growing field of data technology.
What Will You Learn?
- How to create and manage Azure Data Factory pipelines.
- Integrate and transform data from multiple sources.
- Monitor and troubleshoot data pipeline performance.
- Implement best practices for data security and governance.
- Understand data flow concepts in a practical, hands-on manner.
Course Content
SECTION 1: INTRODUCTION
-
Introduction
SECTION 2: ENVIRONMENT SETUP
-
Module Overview
-
Create Azure Account
-
Azure Portal Walkthrough
-
Create Storage Account
-
Create Azure DataLake Gen 2
-
Create Azure Data Factory
-
Create Azure SQL
-
Create Azure SQL Server VM (Virtual Machine)
SECTION 3: BUILDING BLOCKS OF ADF
-
Pipelines
-
Activities
-
Linked Service
-
Dataset
-
Supported File Formats
-
Interaction Between Different Components
SECTION 4: COPY DATA ACTIVITY
-
Copy Data From Blob ADLS(Azure DataLake Storage)
-
Copy Data Logging Settings
-
Copy Data – Multiple Files From a Folder & Other Settings
-
Copy Data – Last Modified Filter
-
Copy Data SQL DB to ADLS
-
Copy Data Fault Tolerance
-
Setup Self Hosted IR(Integration Runtime)
-
Copy Data On Prem DB to ADLS
-
Delete SQL Server VM
-
Ingest Data From API to ADLS
SECTION 5: PARAMETERS AND VARIABLES
-
Parameterized Components
-
Variables in ADF
-
System Variables
-
Set Variable Activity
SECTION 6: CONTROL FLOW ACTIVITY
-
Filter Activity
-
ForEach Activity
-
LookUp Activity
-
Real World Scenario 2#
-
Get Meta Data Activity
-
Real World Scenario
-
If Condition Activity
-
The Wait Activity
-
The Switch Activity
-
Validation Activity
SECTION 7: OTHER IMPORTANT PIPELINE ACTIVITIES
-
Stored Procedure Activity
-
Real World Scenario
-
Delete Activity
-
Execute Pipeline
-
Dataflow Activity
SECTION 8: MAPPING DATAFLOWS
-
Mapping Dataflows
-
Source Transformation
-
Sink Transformation
-
Select Transformations
-
Filter Transformation
-
Exist Transformation
-
Sort Transformation
-
Rank Transformation
-
Aggregate Transformation
-
Join Transformation
-
Pivot Transformation
-
LookUp Transformation
-
Derived Column
-
New Branch
-
Surrogate Key Transformation
-
Flatten Transformation
-
Alter Row Transformation
-
Asset Transformation
SECTION 9: ADVANCED CONCEPT
-
Expression & Function
-
Dynamic Queries
SECTION 10: TRIGGERS
-
Types of Trigger
-
Schedule Trigger
-
Storage Event Based Trigger
-
Tumbling Window Trigger
SECTION 11: MONITORING
-
Pipeline Monitoring
Student Ratings & Reviews
No Review Yet