ADF Mapping Data Flows: Read and Write Partitioned Files

Use ADF Mapping Data Flows to read and write partitioned folders and files from your Data Lake for Big Data Analytics in the Cloud.

#Azure #DataFactory #MappingDataFlows

[rohitjusl]

The partitioned sink removes the key columns from the final putput. Is there a way to keep them in? In your case basically, also have 2 columns for ReleaseYear and Month.

[bhaveshpatelaus]

Good content. What application is used to open parquet files?

[sunidhisahay]

I want the union of all csv files present in a single folder from DataLakeStorage Gen 1 using mapping data flow, can i do that? if my source points to that folder which contains all the csv, the next block being of union and the last block is of the sink destination, the merged csv file? Please help.

[abdulwahabo]

At 08:31 could see some additional files in the output folder (_commit, _SUCCESS etc.), Is there any way to prevent that ?. In my case I have a blob trigger attached to my output container, So its trigger for all files.

[hlt]

So, could I read multiple CSVs at once from one folder in Blob Storage?

[VIRALNUT]

Hi Mark,

I am creating a generic dataflow (No Projection) that loads a parquet file from blob to delta lake.
Also I need to partition the sink on a key column coming from the source file.

I have tried using a derived column to generate the key column from the drifted columns but that column is not flowing into the sink where I wanted to use it as the Partition Key.


Could you please help me/create a video on the same ?