The Agenda

If we want to achieve any data processing in Azure you need an umbrella service to manage, monitor and schedule your solution. For a long time when working on premises, the SQL Agent has been our go-to tool, combined with T-SQL and SSIS packages. It’s now time to upgrade our skills and start using cloud native services to achieve the same thing on the Microsoft Cloud Platform. Within a PaaS only Modern Data Platform, the primary component for delivering that orchestration is Azure Data Factory, combined with various other compute resources.

In this full day of training we’ll start with the basics and learn how to orchestrate your Azure Data Platform end to end. You will learn how to build our Azure ETL/ELT pipelines using all Data Factory has to offer. Plus, consider hybrid architectures, dynamic design patterns, think about lifting and shifting legacy packages, and explore complex bootstrapping to orchestrate everything within your solution.

• Module 1: Data Factory Fundamentals
  • What is it and why use it?
  • Resource Components
  • Common Activities
  • Execution Dependencies

• Module 2: Uploading Data to Azure
  • Integration Runtimes
  • Hosted IR Patterns
  • Private Endpoints

• Module 3: Using SSIS Packages in Azure
  • SSIS Integration Runtime
  • Packages Running on PaaS
  • Scaling Out Package Execution

• Module 4: Data Flows
  • Mapping Data Flows
  • Wrangling Data Flows
  • Configuration
  • Use Cases

• Module 5: Metadata Driven Pipelines
  • Expressions
  • Dynamic Pipelines
  • Orchestration Framework – ADFprocfwk.com

• Module 6: Monitoring Alerting Security
  • Logging – Kusto Queries
  • Alerting
  • Roll Based Access
  • Using Azure Key Vault

• Module 7: Pricing & Limitations
  • Data Integration Units
  • Data Flow Compute
  • Wider Platform Orchestration & Cost Control
  • Resource Limitations

• Module 8: CI/CD with Azure DevOps
  • Source Control vs Developer UI
  • ARM Template Deployments
  • PowerShell Deployments

• Module 9: Data Factory in Production
  • Testing
  • Bootstrapping
  • Best Practices

• Module 10: Wrap Up
  • Conclusions
  • Questions
  • Homework

If that’s not enough content for one day, you will also get access to a set of hands-on labs that you can work through at your own pace. Whether you are new to Azure Data Factory or have some experience, you will leave this workshop with new skills and ideas for your projects.