The Azure Data Factory Data Flow Join and Lookup transformations both provide controls for aggregating data across multiple data sources. Join Transformation There are two AzureSQL tables from previous posts: Buildings and Maintenance_Logs. The latter has a foreign key to… Read More ›
The Azure Data Factory Data Flow Pivot transformation can be used to reshape the data from the previous post at https://blog.westmorr.com/2024/01/09/azure-data-factory-data-flow-aggregate-and-sort/. In that post, the following data slice was generated to visualize the total number of maintenance tickets by type… Read More ›
Azure Data Factory Data Flow Aggregate and Sort transformations can be used similarly to SQL Groups and aggregates such as COUNT, SUM, MIN, etc. The Aggregate and Sort transformations are used against our existing Maintenance Tickets table to count the… Read More ›
The Azure Data Factory Exists Transformation, is demonstrated in a reusable automation pattern. The business requirement is to create a ticket for every maintenance log if that hasn’t been done yet. Configure the Exists Transformation Maintenance_Logs and Maintenance_Tickets are both… Read More ›
The Azure Data Factory Select schema and Filter row modifiers can be used together to transform the source schema and exclude unwanted rows during data loading. In this example, Maintenance Tickets will be created from Maintenance Logs. The Select modifier… Read More ›
A Lookup Activity provides an easy way to obtain an iterator over dataset records. When this activity is used to read a configuration dataset, for example, then it is easy to pass this configuration into our pipelines, linked services, data… Read More ›
Learn how to use the Azure Data Factory, Copy Data activity, for copying SQL into an Azure Data Lake CSV.. Setup SQL Data I’ve created a table with the following schema in Azure SQL: CREATE TABLE Building ( Building_Id VARCHAR(5)… Read More ›
This post provides an example of how to work with the Azure Data Factory Copy Data activity to copy multiple files from an Azure Blob to Azure Data Lake storage. Uploading Multiple Files Continue with the “Buildings” example by uploading… Read More ›
This post demonstrates the configuration and runtime results for Azure Data Factory Copy Data logging. Locating the Copy Data Logging Configuration The logging configuration for the “Copy data” activity is found in the “Settings” tab. Continuing with the Pipeline from… Read More ›
This post demonstrates how to use Azure Data Factory Copy Data activity to copy an Azure Blob file to Data Lake. The steps followed are to: Create Azure Blob Content To start, upload a CSV file to an Azure Blob… Read More ›