Introduction to Azure Databricks-Welcome to the course
Introduction to the course
()
Course syllabus
How to be successful in this course
Introduction to Azure Databricks-Describe Azure Databricks
Explain Azure Databricks
()
Create an Azure Databricks workspace and cluster
Create and execute a notebook
Exercise: Work with Notebooks
Lesson summary
()
Introduction to Azure Databricks-Spark architecture fundamentals
Lesson introduction
()
Understand the architecture of Azure Databricks Spark cluster
()
Understand the architecture of spark job
()
Lesson summary
()
Read and write data in Azure Databricks-Use Azure Databricks to prepare the data for advanced analytics and machine learning operations
Lesson introduction
()
Read data in CSV format
Read data in JSON format
Read data in Parquet format
Read data stored in tables and views
Write data
Exercises: Read and write data
Lesson summary
()
Data processing in Azure Databricks-Work with DataFrames in Azure Databricks
Lesson introduction
()
Describe a DataFrame
Use common DataFrame methods
Use the display function
Exercise: Distinct articles
Lesson summary
()
Data processing in Azure Databricks-Describe lazy evaluation and other performance features in Azure Databricks
Lesson introduction
()
Describe the difference between eager and lazy execution
Describe the fundamentals of how the Catalyst Optimizer works
()
Define and identify actions and transformations
Describe performance enhancements enabled by shuffle operations and Tungsten
()
Lesson summary
()
Work with DataFrames in Azure Databricks-Work with DataFrames columns in Azure Databricks
Lesson introduction
()
Describe the column class
Work with column expressions
Exercise: Washingtons and Marthas
Lesson summary
()
Work with DataFrames in Azure Databricks-Work with DataFrames advanced methods in Azure Databricks
Lesson introduction
()
Perform date and time manipulation
Use aggregate functions
Exercise: Deduplication of data
Lesson summary
()
Platform architecture, security, and data protection in Azure Databricks-Describe platform architecture, security, and data protection in Azure Databricks
Lesson introduction
()
Create the required resources
Describe the Azure Databricks platform architecture
()
Perform data protection
()
Describe Azure key vault and Databricks security scopes
Secure access with Azure IAM and authentication
()
Describe security
()
Exercise: Access Azure Storage with key vault-backed secrets
Lesson summary
()
Further resources
Delta Lake-Build and query a Delta Lake
Describe the open source Delta Lake
()
Get started with Delta using Spark APIs
Exercise: Work with basic Delta Lake functionality
Describe how Azure Databricks manages Delta Lake
Exercise: Use the Delta Lake Time Machine and perform optimization
Lesson summary
()
Delta Lake-Describe Azure Databricks Delta Lake architecture
Lesson introduction
()
Describe bronze, silver, and gold architecture
()
Perform batch and stream processing
Lesson summary
()
Further resources
Analyze streaming data and create production workloads-Process streaming data with Azure Databricks structured streaming
Lesson introduction
()
Describe Azure Databricks structured streaming
()
Perform stream processing using structured streaming
Work with Time Windows
Process data from Event Hubs with structured streaming
Lesson summary
()
Analyze streaming data and create production workloads-Create production workloads on Azure Databricks with Azure Data Factory
Lesson introduction
()
Create the required resources
()
Schedule Databricks jobs in a Data Factory pipeline
Pass parameters into and out of Databricks jobs in Data Factory
Summary
()
Further resources
Create a data architecture-Implement CI/CD with Azure DevOps
Lesson introduction
()
Describe CI/CD
()
Create a CI/CD process with Azure DevOps
Lesson summary
()
Create a data architecture-Integrate Azure Databricks with other Azure services
Lesson summary
()
Set up Azure Synapse Analytics
Integrate with Azure Synapse Analytics
Lesson summary
()
Create a data architecture-Describe Azure Databricks best practices
Lesson introduction
()
Understand workspace administration best practices
()
List security best practices
()
Describe tools and integration best practices
()
Explain Databricks runtime best practices
()
Understand cluster best practices
Lesson summary
()
Further resources
Practice Exam on Data engineering with Azure Databricks-Course practice exam
Course recap
()
About the practice exam
Practice Exam on Data engineering with Azure Databricks-Course wrap up
Course summary
()
Next steps