data engineer interview questions

In this video I have talked about salting in spark

Project details for resume :-

.Successfully led a data engineering project in a retail environment using technologies such as Apache Spark, Python, SQL, and Amazon S3 to optimize data processing.

.Implemented structured data models, including dimension and fact tables, to provide valuable context for point-of-sale data analysis.

Designed and executed an incentive program based on sales performance, enhancing motivation among sales teams by rewarding top performers.

Managed extensive daily data volumes of approximately 100GB, demonstrating the ability to handle large-scale data pipelines.

Employed Spark optimization techniques like caching and broadcast joins to improve data processing speed and efficiency.

Utilized Azure CI/CD pipelines for code deployment, and orchestrated workflows using Airflow and CRON jobs.

Detailed writeup to explain more during interview:-

As a Data Engineer on a project for a prominent offline grocery and kitchen supplies retailer, I applied my expertise in data engineering to drive critical improvements in their data processing and analysis operations.

The project primarily focused on processing and analyzing point-of-sale data, which was structured into dimension and fact tables to provide meaningful context for sales analysis. To further enhance employee motivation and performance, we designed and implemented an incentive program that rewarded salespeople with the highest sales volumes in each store.

Handling a substantial daily data volume of approximately 100GB, we leveraged Apache Spark and applied optimization techniques like data caching and broadcast joins to significantly accelerate data processing. This not only improved the speed of our data pipelines but also increased the efficiency of our data analysis.

We seamlessly integrated the code deployment process into the Azure CI/CD pipeline. As part of workflow automation, we orchestrated task scheduling using Airflow and CRON jobs.

One of the project's major achievements was the implementation of a customer engagement strategy that identified infrequent buyers and provided incentives in the form of coupons. This initiative not only boosted customer retention but also had a positive impact on the overall business growth.

For more queries reach out to me on my below social media handle.

My Gear:-

My PC Components:-

[aprao]

bhai iss video se mein fan hogya aapka. "logon ke pass experiance nahi hain, aur company ko experiance chahiye" 🔥 🔥

[hritikapal]

What a gem content sir 🥺 thankyou so much for in-depth video!

[parameshwarbhange]

What you have face problem while doing your projects and how to resolve it . Please answer this question as experience person

[KhaderAliAfghan]

Sir got placed as azure data engineer, its all because of you really thank you for everything 🥹🥹 i would like to talk with you

[ramswaroop]

Hi bro.... Please do video in english also 🙏🙏 we really need these questions and answers for it

[DEwithDhairy]

DSA interview series for Data Engineer

[BigDataWithSky]

What Don't you talk in English 😢for non Hindi speaker😊

[GauravSubodh]

Wow... Manish bhai really loved this content. Please, I will encourage you to do more videos like this.

[rh]

How to do ONPREMISE to CLOUD migration.

[cretive]

Sir maths kitna required h data engineer profile me please reply

[shadabahmed]

Waiting for 2nd part eagerly, related to last project. Please next time usi ko upload krna.

[rameshjadhav]

Hey Manish, I'm extremely thankful to you and all of your playlists. Especially this video is super problem solver one! No one teaches in so much depth as you do.
Thanks for taking out time to teach us!!😇🤗

[rishav]

Thanks for amazing content . Spark playlist is amazing

[jayantakumarsahu]

I have followed your practical and theory series is just amazing.
can you make please make videos on 'Delta Lakehouse' ?

[WolfmaninKannada]

Sir your amazing. No one has created content till now on this.Wish to see more on this type of content .Being a fresher we need to have a clear idea about how the project works and we should know how to explain project to interviewer.

[vivekpuurkayastha]

Great video Manish .. What you have face problem while doing your projects and how to resolve it . Please answer this question as experience person.

[poojajoshi]

Hi Manish,
Got call wherein they are asking to hv exp into AWS glue n pyspark.
Please tell me how to incorporate glue with pyspark

[ritusrivastava]

Wholesome of Data engineer playlist. Great job Manish 🎉

[krishnaKumar-hjpu]

thank u very much for such a nice and valuable content

[CctnsHelpdesk]

Bhaiya, ,
ye spark submit config
cloud me toh kahin v nai mila databricks me cluster banate time
spark submit jo on prem me karte hai...unke liye hai ky?