Data validation between source and target table | PySpark Interview Question |

Hello Everyone,

source_data = [(1,'A'),(2,'B'),(3,'C'),(4,'D'),(5,'E')]
source_schema = ['id','name']

target_data = [(1,'A'),(2,'B'),(3,'X'),(4,'F'),(6,'G')]
target_schema = ['id','name']

This series is for beginners and intermediate level candidates who wants to crack PySpark interviews

#pyspark #interviewquestions #interview #pysparkinterview #dataengineer #aws #databricks #python

[beingnagur]

At 6.04 instead of copying the same statement you can use .otherwise("not matching")

[rishabhkesarwani-brrx]

I do below steps to compare source vs target table 
1) Count should be matching in source and target table
2) Schema should be matching in source and target table
3) Use the except and to check if any records are there which are present in source and not in target or vice versa.
4) Use the left anti join to find out the records which are not matching.
5) Trying to debug why there is record mismatch

[gudiatoka]

exceptAll can be usefull too or anti join

[jhonsen]

Main Problem i found in learning Pyspark is brackets every time it gives me some error.

[nishirajnikku]

I request you to please create a playlist for Pyspark Unit testing .

[shivamchandan]

plz make video on pyspark unit testing