AQE in spark | Lec-19

In this video I have talked about Adaptive Query Execution.

For more queries reach out to me on my below social media handle.

My Gear:-

My PC Components:-

[piyushjain]

Manish bhai, mai AQE pdh k gaya is video se, aaj interviewer ne puchha skew data ko kse optimize karoge, mene coleasce or split bataya pr usne puchha ki split kese hoga internally vo mujhe nahi samajh aya fir usne bataya SALTING krni hogi, bs vahi ek topic mene nhi padha tha qki mujhe laga important nhi hoga aapne b video me skip kr dia, pr vhi puch lia usne. Baki sb answer kr die mene thanks to your videos, but he rejected me, anyway I'm feeling more coinfident for upcoming interviews. Keep up the good work. You're helping guys like me a lot.

[vipul]

Best Explanation I found anywhere in youtube, Great Work!

[Fr]

Great .Very precise and detailed explanation on AQE. Do you also have the Github or the dbc file of the example.

[deepjyotimitra]

Har ek video apka master class hain. ❤

[greendaywithtrading]

GREAT EXPLANATION KINDLY PROVIDE DATA FOR PRACTICE

[RohitSingh-wemo]

@Manish Kumar -Aapne bataya tha ki coalesce me partition size reduce hota hai but yha pe add kyu kr rhe dynamically coalescing me.Iska explain kr denge

[dharmendersingh]

Tell me one thing Manish and you need to be clear this time


Assuming you have 2 executors and 2 partitions on each executor after shuffle


So keys will lies in different partitions in same executor?
Or

Keys like in same partition in a executor?

[shadabalam]

I have a small confusion, when the final plan is updated, and AQE chooses Broadcast join instead of Sort merge join as per DAG, then, we are avoiding shuffling infact, you told in the end of Dynamic switching of join strategy, that we cannot avoid shuffling, But we know, Broadcast join avoid shuffling, Can anyone clarify me pls

[Aryan-lujs]

Azure Synapse was launched in Nov 2019 but Indian Recruiters wants someone who has 5+ years of experience in Azure Synapse 😢

[stevedz]

Sir, plz make video on Orchestration & HDFS.

[prashantmehta]

Hello sir, at 11.40, is there any rule for coalesce when we do AQE?

[SouAzu-be]

Thank you, please explain the salting and how to implement it in next video.

[Amarjeet-fblk]

At 27.54
Apne note me dala he,

Skew join hone k baad,
1 partition me 8 gb data bhi aa sakta he, for example.

But,
1 partition he, wo ek core pe store hogi, uspe process hogi.
To,
Agar 8 gb ka ek partition ban gaya, join k baad,
But, core k pas memory hi, 4 gb ki hai,
To wo kaise process karega?

[mission_possible]

Please make video on how to read physical plan and DAG

[aryankhandelwal]

Why does it create duplicate partitions for the join? It simply takes Table 2 partitions and joins them with both Table 1 partitions.

[gauravkothe]

if we do split then how will join is perform on that split data please explain

[mohammedasif]

Ye AQE internally apne aap lag jata hai ya humein code me koi config set karna hota hai??

[SubhamKumar-orvc]

Why would the shuffling still happen in case we've got 8gb and 10mb tables after filtering?
The shuffling you talked about, is it before the tables are filtered or after the tables are filtered?

[ansarmobile]

Sir, how many more lectures are pending, pls?

[madanmohan]

Hi Manish, can you make detailed video on fact and dimension tables.