This post provides an example of how to work with the Azure Data Factory Copy Data activity to copy multiple files from an Azure Blob to Azure Data Lake storage.
Uploading Multiple Files
Continue with the “Buildings” example by uploading maintenance records to the Azure Blob storage container. To do this, follow the normal upload process, but expand the “Advanced” options and specify the folder name.
Notice that there are “3 file(s) selected”, and the “Upload to folder” configuration value is specified:
After completing the upload, a new subdirectory will appear under the container with all selected files:
Using Copy Data with Multiple Files
Create Data Sets
The process for copying multiple files from an Azure Blob into Azure Data Lake is nearly identical to copying a single file. To implement this, setup the data set to point to the root of the data directory instead of a specific file:
Next, create a data set to point to the correct Azure Data Lake container. The Copy Data activity will create the container if it doesn’t exist.
Configure Source and Sink
The key property to chose on the Source settings for Copy Data is “Recursively”. Read here for more details, https://learn.microsoft.com/en-us/azure/data-factory/connector-file-system?tabs=data-factory.The specified path for the dataset points to the root container. Selecting the recursive option will cause the iterator to include the specified directory contents as well as all folders-and-files within that container. Specify a wildcard filter if appropriate:
After this, configure the Sink to hold the Azure Data Lake container data set and that’s it.
Conclusion
Using Azure Data Factory to copy multiple files with the Copy Data activity is almost identical to copying a single file. The key steps are to:
- Configure your Source dataset to point to a container / folder root instead of a specific file.
- Select the “Recursively” option for the Source configuration in the Copy Data if you also need to include subdirectories
Categories: Azure Data Factory
Leave a Reply