The Azure Data Factory Select schema and Filter row modifiers can be used together to transform the source schema and exclude unwanted rows during data loading.
In this example, Maintenance Tickets will be created from Maintenance Logs. The Select modifier is used to exclude columns which don’t map between Logs and Tickets. Afterwards, the Filter modifier is used to include rows which match a data flow input parameter.
Configure the Select Schema Modifier
First, the data flow source is configured to use a View which combines Building-and-Log data. See the screenshot of the source and sink tables below and differences in the table schemas:
Only Building_id, Log_Date, and Maintenance_Type should be moved from the Logs into the Tickets.
Next, the Select schema modifier is updated as shown below. Only matching columns remain, and the “Building_id” column from the source is renamed to “Building_Id” so that it matches the tickets table column name:
Configure the Filter Row Modifier
Afterwards, the data flow is configured with an input parameter “Log_Date”. The parameter is used to filter the source where Log_Date is equal to the parameter value.Also, the parameter is a string, so toDate() is used to apply a type conversion.
Run the Data Flow from Azure Data Factory Pipeline
The final step is to run the data flow and pass in the Log_Date value as a parameter. In the screenshot below, the value is hard-coded, but this could easily come from a Pipeline parameter, Lookup value, etc.
The new tickets are now visible:
Conclusion: Azure Data Factory Select and Filter
The Azure Data Factory Select and Filter modifiers can be combined together as shown above. In this way, one may cleanly alter and filter source schema before loading records into the sink dataset.