show distinct column values in pyspark dataframe

## Show Distinct Column Values in PySpark Dataframe: A Comprehensive Guide

This tutorial provides a comprehensive guide on how to retrieve and display distinct (unique) values from a specific column in a PySpark DataFrame. We'll cover different methods, discuss their performance implications, and provide code examples for each approach.

**Prerequisites:**

* **PySpark Installation:** Make sure you have PySpark installed. You can install it using pip: `pip install pyspark`
* **Spark Session:** You need to create a SparkSession to interact with Spark.

**1. Setting up the Spark Session and Sample Data:**

First, let's create a SparkSession and a sample DataFrame to work with. This will serve as the foundation for all our examples.

This code snippet does the following:

1. **Imports `SparkSession`:** Imports the necessary class to create a Spark session.
2. **Creates `SparkSession`:** Creates a SparkSession with the application name "DistinctValues". This is the entry point to Spark functionality. `getOrCreate()` ensures that if a SparkSession already exists, it will be reused; otherwise, a new one is created.
3. **Defines Sample Data:** Creates a list of tuples representing sample data with columns "name", "city", and "age". We have a duplicate row to showcase the 'distinct' functionality.
4. **Defines Schema:** Defines the schema (column names) for the DataFrame. This makes it easier to work with the data by providing names to the columns.

**2. Using `distinct()`:**

The `distinct()` method is the most straightforward way to get distinct rows from a DataFrame. When applied to a single column, it returns a new DataFrame containing only the unique v ...

#class12 #class12 #class12