PySpark Interview Questions | Azure Data Engineer #azuredataengineer #databricks #pyspark

Q80. Find the top N most frequent words in a large text file using PySpark

"Need to find the most common words in a massive text file? 🔍

PySpark makes it a breeze! Learn how to extract the top N words using a simple and efficient approach. ⚡"

Don't forget to like, comment, and subscribe for more PySpark interview preparation content! 🔥💡

👉 If you found this video helpful, don’t forget to hit the like button and subscribe for more Spark tutorials!
📢 Have questions or tips of your own? Drop them in the comments below!

#PySpark #BigData #DataScience #CloudArchitectAbhiram

[CloudMaster_Abhiram]

Syntax for Find the top N most frequent words in a large text file

from pyspark import SparkContext
# create your spark context
sc = SparkContext("local", "WordCount")
# import a text file from a local path
lines =
# split and map the words
# then reduce by using the words as keys and add to the count
word_counts = lines.flatMap(lambda line: line.split(" ")) \
.map(lambda word: (word, 1)) \
.reduceByKey(lambda a, b: a + b)
# order the words and take only the top N frequent words
top_n_words = word_counts.takeOrdered(N, key=lambda x: -x[1])
print(top_n_words)