Databricks Certified Associate Developer for Apache Spark practice questions

1. Question 1 of 25

   Which of the following describes the Spark driver?

   • AThe Spark driver is responsible for performing all execution in all execution modes – it is the entire Spark application.
   • BThe Spare driver is fault tolerant – if it fails, it will recover the entire Spark application.
   • CThe Spark driver is the coarsest level of the Spark execution hierarchy – it is synonymous with the Spark application.
   • DThe Spark driver is the program space in which the Spark application’s main method runs coordinating the Spark entire application.
   • EThe Spark driver is horizontally scaled to increase overall processing throughput of a Spark application.

   Show answer

   Correct answer: D

   How others answered

   • D100%

   Explanation

   The correct answer is D. The Spark driver is the program space in which the Spark application's main method runs, coordinating the entire Spark application. Explanation: The Spark driver is a program that runs the main method of a Spark application and coordinates the execution of the entire application. It is responsible for defining the SparkContext, which is the entry point for any Spark functionality. The driver program is responsible for dividing the Spark application into tasks, scheduling them on the cluster, and managing the overall execution. The driver communicates with the cluster manager to allocate resources and coordinate the distribution of tasks to the worker nodes. It also maintains the overall control and monitoring of the application. Horizontal scaling, fault tolerance, and execution modes are not directly related to the Spark driver.

   Adapted from community discussion by TmData · 2023-06-17 · 4 community upvotes. Third-party contribution; reviewed by our quality filter, not independently verified.

   View community discussion4

   Candidate context

   Question 1 discussion

   These are quality-filtered excerpts from third-party community discussion. They can be useful reasoning signals, but they are not official answers and have not been independently verified by SPARK-DEV or CertExamHQ.

   1. T

      TmData

      2023-06-17

      4 upvotes

      The correct answer is D. The Spark driver is the program space in which the Spark application's main method runs, coordinating the entire Spark application. Explanation: The Spark driver is a program that runs the main method of a Spark application and coordinates the execution of the entire application. It is responsible for defining the SparkContext, which is the entry point for any Spark functionality. The driver program is responsible for dividing the Spark application into tasks, scheduling them on the cluster, and managing the overall execution. The driver communicates with the cluster manager to allocate resources and coordinate the distribution of tasks to the worker nodes. It also maintains the overall control and monitoring of the application. Horizontal scaling, fault tolerance, and execution modes are not directly related to the Spark driver.

   2. 4

      4be8126

      2023-04-24

      Selected D3 upvotes

      D. The Spark driver is the program space in which the Spark application’s main method runs coordinating the Spark entire application. The Spark driver is responsible for coordinating the execution of a Spark application, and it runs in the program space where the Spark application's main method runs. It manages the scheduling, distribution, and monitoring of tasks across the cluster, and it communicates with the cluster manager to acquire resources and allocate them to the application. The driver also maintains the state of the application and collects results. It is not responsible for performing all execution in all execution modes, nor is it fault-tolerant or horizontally scalable.

   3. S

      singh100

      2023-07-31

      4 upvotes

      Answer: D Receives the user's code and breaks it into tasks for execution. Orchestrates the execution plan and optimizes the Spark job. Coordinates with cluster managers to allocate resources for tasks. Collects and aggregates results from distributed workers. Maintains the metadata and state of the Spark application during its execution.

   4. Z

      zic00

      2024-08-14

      Selected D1 upvote

      The Spark driver is responsible for orchestrating the execution of a Spark application, managing the SparkContext, and coordinating the execution of tasks across the Spark cluster. It does not perform the execution of tasks itself but rather schedules tasks on the worker nodes. The Spark driver is not fault-tolerant in the sense that if it fails, the entire Spark application usually fails. It also does not scale horizontally; only the executors (worker nodes) do that.

2. Question 2 of 25

   Which of the following DataFrame operations is classified as a wide transformation?

   • ADataFrame.filter()
   • BDataFrame.join()
   • CDataFrame.select()
   • DDataFrame.drop()
   • EDataFrame.union()

   Show answer

   Correct answer: B

   How others answered

   • B100%

   Explanation

   The DataFrame operation classified as a wide transformation is: B. DataFrame.join() Explanation: In Spark, transformations are operations on DataFrames that create a new DataFrame from an existing one. Wide transformations involve shuffling or redistributing data across partitions and typically require data movement across the network. Among the options provided, DataFrame.join() is a wide transformation because it involves combining two DataFrames based on a common key column, which often requires shuffling and redistributing the data across partitions.

   Adapted from community discussion by TmData · 2023-06-17 · 3 community upvotes. Third-party contribution; reviewed by our quality filter, not independently verified.

   View community discussion4

   Candidate context

   Question 2 discussion

   These are quality-filtered excerpts from third-party community discussion. They can be useful reasoning signals, but they are not official answers and have not been independently verified by SPARK-DEV or CertExamHQ.

   1. 4

      4be8126

      2023-05-03

      Selected B5 upvotes

      B. DataFrame.join() is classified as a wide transformation, as it shuffles the data across the network to perform the join operation.

   2. T

      TmData

      2023-06-17

      Selected B3 upvotes

      The DataFrame operation classified as a wide transformation is: B. DataFrame.join() Explanation: In Spark, transformations are operations on DataFrames that create a new DataFrame from an existing one. Wide transformations involve shuffling or redistributing data across partitions and typically require data movement across the network. Among the options provided, DataFrame.join() is a wide transformation because it involves combining two DataFrames based on a common key column, which often requires shuffling and redistributing the data across partitions.

   3. S

      SonicBoom10C9

      2023-05-15

      Selected B3 upvotes

      None of the other options are wide transformations, they are narrow (logically, they modify the length of a dataframe). Only a join can force shuffling of data between horizontally scaled partitions.

   4. B

      Becida

      2024-09-30

      Selected B1 upvote

      Join takes different partitions and combines into single dataframe.

3. Question 3 of 25

   Which of the following code blocks returns a DataFrame containing only the rows from DataFrame storesDF where the value in column sqft is less than or equal to 25,000 AND the value in column customerSatisfaction is greater than or equal to 30?

   • AstoresDF.filter(col("sqft") <= 25000 and col("customerSatisfaction") >= 30)
   • BstoresDF.filter(col("sqft") <= 25000 or col("customerSatisfaction") >= 30)
   • CstoresDF.filter(sqft) <= 25000 and customerSatisfaction >= 30)
   • DstoresDF.filter(col("sqft") <= 25000 & col("customerSatisfaction") >= 30)
   • EstoresDF.filter(sqft <= 25000) & customerSatisfaction >= 30)

   Show answer

   Correct answer: D

   How others answered

   • D67%
   • A33%

   Only 67% agreed on this answer — treat it as contested rather than settled.

   Explanation

   in pyspark, all wrong as the conditions inside the filter should be wrapped inside parentesis. should be: D. storesDF.filter((col("sqft") = 30))

   Adapted from community discussion by gaco · 2024-08-23 · 5 community upvotes. Third-party contribution; reviewed by our quality filter, not independently verified.

   View community discussion4

   Candidate context

   Question 3 discussion

   These are quality-filtered excerpts from third-party community discussion. They can be useful reasoning signals, but they are not official answers and have not been independently verified by SPARK-DEV or CertExamHQ.

   1. G

      gaco

      2024-08-23

      5 upvotes

      in pyspark, all wrong as the conditions inside the filter should be wrapped inside parentesis. should be: D. storesDF.filter((col("sqft") = 30))

   2. S

      Sowwy1

      2024-04-02

      2 upvotes

      It's D: https://sparkbyexamples.com/spark/spark-and-or-not-operators/ PySpark Logical operations use the bitwise operators: & for and | for or ~ for not

   3. J

      Jgo1986

      2024-09-03

      2 upvotes

      The most similar is D, And and OR are not valid statements for filtering in pySpark

   4. P

      PushpakKothekar

      2025-04-08

      Selected D1 upvote

      For dataframe we cannot AND, OR. This applicable for only spark.sql. hence correct answer is D.

4. Question 4 of 25

   Which of the following statements about Spark’s stability is incorrect?

   • ASpark is designed to support the loss of any set of worker nodes.
   • BSpark will rerun any failed tasks due to failed worker nodes.
   • CSpark will recompute data cached on failed worker nodes.
   • DSpark will spill data to disk if it does not fit in memory.
   • ESpark will reassign the driver to a worker node if the driver’s node fails.

   Show answer

   Correct answer: E

   How others answered

   • E100%

   Explanation

   Option E is incorrect because the driver program in Spark is not reassigned to another worker node if the driver's node fails. The driver program is responsible for the coordination and control of the Spark application and runs on a separate machine, typically the client machine or cluster manager. If the driver's node fails, the Spark application as a whole may fail or need to be restarted, but the driver is not automatically reassigned to another worker node.

   Adapted from community discussion by TmData · 2023-06-17 · 6 community upvotes. Third-party contribution; reviewed by our quality filter, not independently verified.

   View community discussion4

   Candidate context

   Question 4 discussion

   These are quality-filtered excerpts from third-party community discussion. They can be useful reasoning signals, but they are not official answers and have not been independently verified by SPARK-DEV or CertExamHQ.

   1. T

      TmData

      2023-06-17

      Selected E6 upvotes

      Option E is incorrect because the driver program in Spark is not reassigned to another worker node if the driver's node fails. The driver program is responsible for the coordination and control of the Spark application and runs on a separate machine, typically the client machine or cluster manager. If the driver's node fails, the Spark application as a whole may fail or need to be restarted, but the driver is not automatically reassigned to another worker node.

   2. 4

      4be8126

      2023-05-03

      Selected E2 upvotes

      All of the following statements about Spark's stability are correct except for: E. Spark will reassign the driver to a worker node if the driver’s node fails. The driver is a special process in Spark that is responsible for coordinating tasks and executing the main program. If the driver fails, the entire Spark application fails and cannot be restarted. Therefore, Spark does not reassign the driver to a worker node if the driver's node fails.

   3. G

      GuidoDC

      2023-03-25

      Selected E3 upvotes

      If the driver node fails your cluster will fail. If the worker node fails, Databricks will spawn a new worker node to replace the failed node and resumes the workload.

   4. Z

      zic00

      2024-08-21

      Selected E2 upvotes

      In Spark, the driver node is crucial for orchestrating the execution of the Spark application. If the driver node fails, the Spark application fails. Spark does not automatically reassign the driver to another node if the driver fails. This would require the application to be restarted manually or through external high-availability mechanisms.

5. Question 5 of 25

   Of the following situations, in which will it be most advantageous to store DataFrame df at the MEMORY_AND_DISK storage level rather than the MEMORY_ONLY storage level?

   • AWhen all of the computed data in DataFrame df can fit into memory.
   • BWhen the memory is full and it’s faster to recompute all the data in DataFrame df rather than read it from disk.
   • CWhen it’s faster to recompute all the data in DataFrame df that cannot fit into memory based on its logical plan rather than read it from disk.
   • DWhen it’s faster to read all the computed data in DataFrame df that cannot fit into memory from disk rather than recompute it based on its logical plan.
   • EThe storage level MENORY_ONLY will always be more advantageous because it’s faster to read data from memory than it is to read data from disk.

   Show answer

   Correct answer: D

   How others answered

   • D100%

   Explanation

   All other explanation is either wrong or misleading. To understand the question, you need to understand the difference between Memory_only and Memory_and_Disk 1. Memory_and_Disk, which is the default mode for cache ro persist. That means, if the data size is larger than the memory, it will store the extra data in disk. next time when we n eed to read data, we will read data firstly from memory, and then read from disk. 2. Memory_Only means, if the data size is larger than memory, it will not store the extra data. next time we read data, we will read from memory first and then recompute the extra data which cannot store in memory. PS. Mr. 4be8126 is wrong about raising error when out of memory. Therefore, the difference/balance between Memory_only and memory_and_disk lay in how they handle the extra data out of memory. which is option D, if it is faster to read data from disk is faster than recompute it, then memory_and_disk.

   Adapted from community discussion by ZSun · 2024-06-07 · 7 community upvotes. Third-party contribution; reviewed by our quality filter, not independently verified.

   View community discussion4

   Candidate context

   Question 5 discussion

   These are quality-filtered excerpts from third-party community discussion. They can be useful reasoning signals, but they are not official answers and have not been independently verified by SPARK-DEV or CertExamHQ.

   1. Z

      ZSun

      2024-06-07

      7 upvotes

      All other explanation is either wrong or misleading. To understand the question, you need to understand the difference between Memory_only and Memory_and_Disk 1. Memory_and_Disk, which is the default mode for cache ro persist. That means, if the data size is larger than the memory, it will store the extra data in disk. next time when we n eed to read data, we will read data firstly from memory, and then read from disk. 2. Memory_Only means, if the data size is larger than memory, it will not store the extra data. next time we read data, we will read from memory first and then recompute the extra data which cannot store in memory. PS. Mr. 4be8126 is wrong about raising error when out of memory. Therefore, the difference/balance between Memory_only and memory_and_disk lay in how they handle the extra data out of memory. which is option D, if it is faster to read data from disk is faster than recompute it, then memory_and_disk.

   2. S

      sousouka

      2024-03-29

      9 upvotes

      D. When it’s faster to read all the computed data in DataFrame df that cannot fit into memory from disk rather than recompute it based on its logical plan.

   3. 4

      4be8126

      2024-05-03

      Selected D1 upvote

      The most advantageous situation to store a DataFrame at the MEMORY_AND_DISK storage level instead of the MEMORY_ONLY storage level is option D - when it’s faster to read all the computed data in DataFrame df that cannot fit into memory from disk rather than recompute it based on its logical plan. This is because the MEMORY_ONLY storage level only stores data in memory, which can result in an out-of-memory error if the data exceeds the available memory. On the other hand, the MEMORY_AND_DISK storage level will spill data to disk if there is not enough memory available, allowing more data to be processed without errors. In situations where the computed data can fit entirely into memory, it is best to use the MEMORY_ONLY storage level as it will be faster than reading from disk. However, when there is not enough memory to store all the computed data, it may be necessary to use the MEMORY_AND_DISK storage level.

   4. S

      SonicBoom10C9

      2024-05-15

      Selected D1 upvote

      If it's faster to read from memory and can fit in, then there is no reason to use Memory_and_disk, Memory_only is sufficient. Also, if it's faster to compute than read from disk, that's what you would do. The only options is when it's too big to fit in memory and too expensive to recompute, so reading from disk (or rather caching from disk into memory on the fly) is faster.

6. Question 6 of 25

   Which of the following code blocks returns a DataFrame containing a column openDateString, a string representation of Java’s SimpleDateFormat? Note that column openDate is of type integer and represents a date in the UNIX epoch format — the number of seconds since midnight on January 1st, 1970. An example of Java's SimpleDateFormat is "Sunday, Dec 4, 2008 1:05 pm". A sample of storesDF is displayed below:

   

   • AstoresDF.withColumn("openDatestring", from unixtime(col("openDate“), “EEEE, MMM d, yyyy h:mm a"))
   • BstoresDF.withColumn("openDateString", from_unixtime(col("openDate“), "EEEE, MMM d, yyyy h:mm a", TimestampType()))
   • CstoresDF.withColumn("openDateString", date(col("openDate"), "EEEE, MMM d, yyyy h:mm a"))
   • DstoresDF.newColumn(col("openDateString"), from_unixtime("openDate", "EEEE, MMM d, yyyy h:mm a"))
   • EstoresDF.withColumn("openDateString", date(col("openDate“), "EEEE, MMM d, yyyy h:mm a", TimestampType))

   Show answer

   Correct answer: A

   How others answered

   • A60%
   • B40%

   Only 60% agreed on this answer — treat it as contested rather than settled.

   Explanation

   A is the right answer # Assuming you have a SparkSession created (replace with your setup if needed) from pyspark.sql import SparkSession from pyspark.sql.functions import from_unixtime # Sample data (replace with your actual DataFrame) data = [("store1", 1668518400), ("store2", 1668432000)] # Sample UNIX epoch timestamps df = spark.createDataFrame(data, ["storeName", "openDate"]) # Convert openDate to formatted string formatted_df = df.withColumn( "openDateString", from_unixtime(col("openDate"), "EEEE, MMM d, yyyy h:mm a") ) # Print the resulting DataFrame formatted_df.show(truncate=False) # Stop the SparkSession (optional) #spark.stop()

   Adapted from community discussion by c3d91ee · 2024-03-20 · 2 community upvotes. Third-party contribution; reviewed by our quality filter, not independently verified.

   View community discussion4

   Candidate context

   Question 6 discussion

   These are quality-filtered excerpts from third-party community discussion. They can be useful reasoning signals, but they are not official answers and have not been independently verified by SPARK-DEV or CertExamHQ.

   1. C

      c3d91ee

      2024-03-20

      2 upvotes

      A is the right answer # Assuming you have a SparkSession created (replace with your setup if needed) from pyspark.sql import SparkSession from pyspark.sql.functions import from_unixtime # Sample data (replace with your actual DataFrame) data = [("store1", 1668518400), ("store2", 1668432000)] # Sample UNIX epoch timestamps df = spark.createDataFrame(data, ["storeName", "openDate"]) # Convert openDate to formatted string formatted_df = df.withColumn( "openDateString", from_unixtime(col("openDate"), "EEEE, MMM d, yyyy h:mm a") ) # Print the resulting DataFrame formatted_df.show(truncate=False) # Stop the SparkSession (optional) #spark.stop()

   2. 6

      65bd33e

      2024-08-16

      Selected A1 upvote

      The correct code block that returns a DataFrame containing a column openDateString, which is a string representation of Java's SimpleDateFormat, is: A. storesDF.withColumn("openDatestring", from_unixtime(col("openDate"), "EEEE, MMM d, yyyy h:mm a")) This option correctly converts the openDate column from UNIX epoch format to a human-readable string format using the from_unixtime function in PySpark, which is equivalent to Java's SimpleDateFormat. The format "EEEE, MMM d, yyyy h:mm a" aligns with the example provided: "Sunday, Dec 4, 2008 1:05 pm".

   3. O

      oussa_ama

      2024-08-21

      Selected A1 upvote

      pyspark.sql.functions.from_unixtime(timestamp: ColumnOrName, format: str = 'yyyy-MM-dd HH:mm:ss') → pyspark.sql.column.Column[source] Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the given format.

   4. S

      Sowwy1

      2024-04-10

      1 upvote

      A. storesDF.withColumn("openDatestring", from unixtime(col("openDate“), “EEEE, MMM d, yyyy h:mm a"))

7. Question 7 of 25

   A Spark application has a 128 GB DataFrame A and a 1 GB DataFrame B. If a broadcast join were to be performed on these two DataFrames, which of the following describes which DataFrame should be broadcasted and why?

   • AEither DataFrame can be broadcasted. Their results will be identical in result and efficiency.
   • BDataFrame B should be broadcasted because it is smaller and will eliminate the need for the shuffling of itself.
   • CDataFrame A should be broadcasted because it is larger and will eliminate the need for the shuffling of DataFrame B.
   • DDataFrame B should be broadcasted because it is smaller and will eliminate the need for the shuffling of DataFrame A.
   • EDataFrame A should be broadcasted because it is smaller and will eliminate the need for the shuffling of itself.

   Show answer

   Correct answer: D

   How others answered

   • D57%
   • B43%

   Only 57% agreed on this answer — treat it as contested rather than settled.

   Explanation

   The correct answer is: B. DataFrame B should be broadcasted because it is smaller and will eliminate the need for the shuffling of itself. Explanation: In Spark, a broadcast join is a specific type of join where one DataFrame is sent to every node in the cluster to avoid the costly network shuffle that can occur with large datasets in regular joins. Generally, the smaller DataFrame should be broadcasted to optimize performance. This is because broadcasting a smaller DataFrame requires less network bandwidth and memory usage across the cluster. Broadcasting DataFrame B (the smaller DataFrame at 1 GB) means that each node will have a local copy of DataFrame B, allowing them to perform the join operation locally with their respective partitions of DataFrame A without needing to shuffle DataFrame B across the network. This approach significantly reduces the amount of data that needs to be shuffled (since only DataFrame A is partitioned across the nodes), thereby improving the performance of the join operation.

   Adapted from community discussion by atulrao · 2024-07-11 · 3 community upvotes. Third-party contribution; reviewed by our quality filter, not independently verified.

   View community discussion4

   Candidate context

   Question 7 discussion

   These are quality-filtered excerpts from third-party community discussion. They can be useful reasoning signals, but they are not official answers and have not been independently verified by SPARK-DEV or CertExamHQ.

   1. A

      atulrao

      2024-07-11

      3 upvotes

      The correct answer is: B. DataFrame B should be broadcasted because it is smaller and will eliminate the need for the shuffling of itself. Explanation: In Spark, a broadcast join is a specific type of join where one DataFrame is sent to every node in the cluster to avoid the costly network shuffle that can occur with large datasets in regular joins. Generally, the smaller DataFrame should be broadcasted to optimize performance. This is because broadcasting a smaller DataFrame requires less network bandwidth and memory usage across the cluster. Broadcasting DataFrame B (the smaller DataFrame at 1 GB) means that each node will have a local copy of DataFrame B, allowing them to perform the join operation locally with their respective partitions of DataFrame A without needing to shuffle DataFrame B across the network. This approach significantly reduces the amount of data that needs to be shuffled (since only DataFrame A is partitioned across the nodes), thereby improving the performance of the join operation.

   2. T

      TmData

      2023-06-17

      Selected D3 upvotes

      Option A is incorrect because not both DataFrames can be broadcasted. Only one of the DataFrames should be broadcasted to minimize shuffling. Option B is correct because DataFrame B is smaller and broadcasting it will eliminate the shuffling of DataFrame B, improving the join operation's efficiency. Option C is incorrect because DataFrame A is larger and shuffling DataFrame B is not a concern in this scenario. Option E is incorrect because DataFrame A is larger, and broadcasting it would not eliminate the shuffling of itself. The larger DataFrame typically undergoes shuffling in a broadcast join. Therefore, the correct option is D.

   3. E

      eendee

      2023-08-11

      Selected D3 upvotes

      https://sparkbyexamples.com/spark/broadcast-join-in-spark/ Spark Broadcast Join is an important part of the Spark SQL execution engine, With broadcast join, Spark broadcast the smaller DataFrame to all executors and the executor keeps this DataFrame in memory and the larger DataFrame is split and distributed across all executors so that Spark can perform a join without shuffling any data from the larger DataFrame as the data required for join colocated on every executor.

   4. 4

      4be8126

      2023-05-03

      Selected D1 upvote

      The correct answer is D. DataFrame B should be broadcasted because it is smaller and will eliminate the need for the shuffling of DataFrame A. A broadcast join is a technique where the smaller DataFrame is broadcast to all the worker nodes in the cluster, so that it can be joined with the larger DataFrame without requiring any shuffling of the larger DataFrame. This is generally more efficient than a shuffle join, which requires data to be shuffled across the network. In this scenario, DataFrame B is much smaller than DataFrame A, so it is more efficient to broadcast DataFrame B to all worker nodes in the cluster. This will eliminate the need for shuffling of DataFrame A, making the join more efficient.

8. Question 8 of 25

   The code block shown below contains an error. The code block is intended to return a DataFrame containing all columns from DataFrame storesDF except for column sqft and column customerSatisfaction. Identify the error. Code block: storesDF.drop(sqft, customerSatisfaction)

   • AThe drop() operation only works if one column name is called at a time – there should be two calls in succession like storesDF.drop("sqft").drop("customerSatisfaction").
   • BThe drop() operation only works if column names are wrapped inside the col() function like storesDF.drop(col(sqft), col(customerSatisfaction)).
   • CThere is no drop() operation for storesDF.
   • DThe sqft and customerSatisfaction column names should be quoted like "sqft" and "customerSatisfaction".
   • EThe sqft and customerSatisfaction column names should be subset from the DataFrame storesDF like storesDF."sqft" and storesDF."customerSatisfaction".

   Show answer

   Correct answer: D

   How others answered

   • D100%

   Explanation

   The error in the code block is that the column names sqft and customerSatisfaction should be quoted, like "sqft" and "customerSatisfaction", since they are strings. The correct code block should be: storesDF.drop("sqft", "customerSatisfaction") Option D correctly identifies this error.

   Adapted from community discussion by 4be8126 · 2023-10-26 · 5 community upvotes. Third-party contribution; reviewed by our quality filter, not independently verified.

   View community discussion4

   Candidate context

   Question 8 discussion

   These are quality-filtered excerpts from third-party community discussion. They can be useful reasoning signals, but they are not official answers and have not been independently verified by SPARK-DEV or CertExamHQ.

   1. 4

      4be8126

      2023-10-26

      Selected D5 upvotes

      The error in the code block is that the column names sqft and customerSatisfaction should be quoted, like "sqft" and "customerSatisfaction", since they are strings. The correct code block should be: storesDF.drop("sqft", "customerSatisfaction") Option D correctly identifies this error.

   2. T

      TmData

      2023-12-17

      Selected D1 upvote

      When using the drop() operation in Spark DataFrame, the column names should be specified as strings and enclosed in quotes. In the given code block, the column names "sqft" and "customerSatisfaction" are not quoted, which results in a syntax error.

   3. A

      azurearch

      2024-09-06

      1 upvote

      https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.drop.html option A is correct, drop expects only one argument, if its more than one, you would have to use as listofcols=['col1','col2'] and drop(*listofcols)

   4. Z

      ZSun

      2023-12-07

      1 upvote

      The correct one is B: storesDF.drop("sqft").drop("customerSatisfaction") For D, it should be list of column name: storesDF.drop(["sqft", "customerSatisfaction"])

9. Question 9 of 25

   Which of the following describes the relationship between nodes and executors?

   • AExecutors and nodes are not related.
   • BAnode is a processing engine running on an executor.
   • CAn executor is a processing engine running on a node.
   • DThere are always the same number of executors and nodes.
   • EThere are always more nodes than executors.

   Show answer

   Correct answer: C

   How others answered

   • C100%

   Explanation

   In a Spark cluster, each node typically has multiple executors, which are responsible for executing tasks on that node. An executor is a separate process that runs on a node and is responsible for executing tasks assigned to it by the driver. Therefore, a node can have multiple executors running on it. The number of executors on a node depends on the resources available on that node and the configuration settings of the Spark application. So, option C is the correct answer.

   Adapted from community discussion by TC007 · 2023-10-03 · 7 community upvotes. Third-party contribution; reviewed by our quality filter, not independently verified.

   View community discussion4

   Candidate context

   Question 9 discussion

   These are quality-filtered excerpts from third-party community discussion. They can be useful reasoning signals, but they are not official answers and have not been independently verified by SPARK-DEV or CertExamHQ.

   1. T

      TC007

      2023-10-03

      Selected C7 upvotes

      In a Spark cluster, each node typically has multiple executors, which are responsible for executing tasks on that node. An executor is a separate process that runs on a node and is responsible for executing tasks assigned to it by the driver. Therefore, a node can have multiple executors running on it. The number of executors on a node depends on the resources available on that node and the configuration settings of the Spark application. So, option C is the correct answer.

   2. T

      TmData

      2023-12-17

      Selected C1 upvote

      The correct answer is C. An executor is a processing engine running on a node. Explanation: In Apache Spark, a node refers to a physical or virtual machine in a cluster that is part of the Spark cluster. Each node can have one or more executors running on it. An executor is a Spark component responsible for executing tasks and storing data in memory or on disk. It is a worker process that runs on a node and performs the actual computation and data processing tasks assigned to it by the driver program. Executors are created and managed by the cluster manager, and they are responsible for executing the tasks and managing the data partitions assigned to them.

   3. O

      oda_rasheed_basha

      2024-09-12

      3 upvotes

      A) is C. nodes are the physical machines in the Spark cluster, while executors are the worker processes running on these nodes, responsible for executing tasks and processing data as part of a Spark application.

   4. S

      Saurabh_221b

      2025-04-04

      Selected C1 upvote

      Executors are the processing engines that run on nodes to execute tasks in a Spark application.

10. Question 10 of 25

    Which of the following code blocks returns a DataFrame containing only the rows from DataFrame storesDF where the value in column sqft is less than or equal to 25,000 OR the value in column customerSatisfaction is greater than or equal to 30?

    • AstoresDF.filter(col("sqft") <= 25000 | col("customerSatisfaction") >= 30)
    • BstoresDF.filter(col("sqft") <= 25000 or col("customerSatisfaction") >= 30)
    • CstoresDF.filter(sqft <= 25000 or customerSatisfaction >= 30)
    • DstoresDF.filter(col(sqft) <= 25000 | col(customerSatisfaction) >= 30)
    • EstoresDF.filter((col("sqft") <= 25000) | (col("customerSatisfaction") >= 30))

    Show answer

    Correct answer: E

    How others answered

    • E89%

    Explanation

    The correct code block to return a DataFrame containing only the rows from DataFrame storesDF where the value in column sqft is less than or equal to 25,000 OR the value in column customerSatisfaction is greater than or equal to 30 is: storesDF.filter((col("sqft") = 30)) Option A uses a single pipe (|) instead of the correct syntax of two vertical bars (||) to represent "OR" logic, and also uses the wrong syntax for column referencing. Option B uses the correct or operator, but also uses the wrong syntax for column referencing. Option C uses the correct operator and syntax for column referencing, but does not use the col() function to reference column names. Option D uses the col() function, but also uses the wrong syntax for column referencing. Option E uses the correct syntax for both column referencing and logical operator, and correctly specifies the parentheses to ensure the proper order of operations. Therefore, the correct answer is E. storesDF.filter((col("sqft") = 30))

    Adapted from community discussion by 4be8126 · 2023-04-26 · 2 community upvotes. Third-party contribution; reviewed by our quality filter, not independently verified.

    View community discussion4

    Candidate context

    Question 10 discussion

    These are quality-filtered excerpts from third-party community discussion. They can be useful reasoning signals, but they are not official answers and have not been independently verified by SPARK-DEV or CertExamHQ.

    1. 4

       4be8126

       2023-04-26

       2 upvotes

       The correct code block to return a DataFrame containing only the rows from DataFrame storesDF where the value in column sqft is less than or equal to 25,000 OR the value in column customerSatisfaction is greater than or equal to 30 is: storesDF.filter((col("sqft") = 30)) Option A uses a single pipe (|) instead of the correct syntax of two vertical bars (||) to represent "OR" logic, and also uses the wrong syntax for column referencing. Option B uses the correct or operator, but also uses the wrong syntax for column referencing. Option C uses the correct operator and syntax for column referencing, but does not use the col() function to reference column names. Option D uses the col() function, but also uses the wrong syntax for column referencing. Option E uses the correct syntax for both column referencing and logical operator, and correctly specifies the parentheses to ensure the proper order of operations. Therefore, the correct answer is E. storesDF.filter((col("sqft") = 30))

    2. T

       TmData

       2023-06-17

       Selected E4 upvotes

       Option E, storesDF.filter((col("sqft") = 30)), is the correct option. It uses the filter() operation with the conditions (col("sqft") = 30) to filter the rows where the value in column sqft is less than or equal to 25,000 OR the value in column customerSatisfaction is greater than or equal to 30.

    3. E

       evertonllins

       2023-08-26

       5 upvotes

       Congrats man, not everyone goes back to tell they were wrong and corrects them selves. We need more people like this on this platform

    4. J

       jds0

       2024-07-23

       Selected E2 upvotes

       Answer: E I tried it with the code below, all other options raised an error: # register UDF with udf function from pyspark.sql import SparkSession from pyspark.sql.functions import col, udf from pyspark.sql.types import IntegerType spark = SparkSession.builder.appName("MyApp").getOrCreate() data = [ (0, 3, 20000, "A"), (1, 1, 50000, "A"), (2, 2, 70000, "A"), (3, 5, 10000, "B"), (4, 4, 100000, "B"), ] storesDF = spark.createDataFrame(data, ["storeID", "customerSatisfaction", "sqft", "division"]) try: storesDF.filter(col("sqft") = 30).show() except Exception as e: print(e) try: storesDF.filter((col("sqft") = 30)).show() except Exception as e: print(e)

11. Question 11 of 25

    Which of the following code blocks returns a new DataFrame from DataFrame storesDF where column storeId is of the type string?

    • AstoresDF.withColumn("storeId, cast(col("storeId"), StringType()))
    • BstoresDF.withColumn("storeId, col("storeId").cast(StringType()))
    • CstoresDF.withColumn("storeId, cast(storeId).as(StringType)
    • DstoresDF.withColumn("storeId, col(storeId).cast(StringType)
    • EstoresDF.withColumn("storeId, cast("storeId").as(StringType()))

    Show answer

    Correct answer: B

    How others answered

    • B100%

    Explanation

    The correct code block to return a new DataFrame from DataFrame storesDF where column storeId is of the type string is: storesDF.withColumn("storeId", col("storeId").cast(StringType())) Option A has an extra quotation mark after "storeId" and is missing a closing parenthesis for the cast() function. Option B correctly uses the cast() function to change the data type, but has a typo where "storeId" is repeated inside the string argument for the withColumn() function. Option C is missing the col() function to reference the storeId column, and also has a typo with the closing parentheses for the cast() function. Option D correctly references the storeId column using col(), but has a typo with the quotation marks and parentheses. Option E has a syntax error where the cast() function is inside the quotation marks, and is also missing the col() function to reference the storeId column. Therefore, the correct answer is B. storesDF.withColumn("storeId", col("storeId").cast(StringType()))

    Adapted from community discussion by 4be8126 · 2023-04-26 · 2 community upvotes. Third-party contribution; reviewed by our quality filter, not independently verified.

    View community discussion4

    Candidate context

    Question 11 discussion

    These are quality-filtered excerpts from third-party community discussion. They can be useful reasoning signals, but they are not official answers and have not been independently verified by SPARK-DEV or CertExamHQ.

    1. 4

       4be8126

       2023-04-26

       Selected B2 upvotes

       The correct code block to return a new DataFrame from DataFrame storesDF where column storeId is of the type string is: storesDF.withColumn("storeId", col("storeId").cast(StringType())) Option A has an extra quotation mark after "storeId" and is missing a closing parenthesis for the cast() function. Option B correctly uses the cast() function to change the data type, but has a typo where "storeId" is repeated inside the string argument for the withColumn() function. Option C is missing the col() function to reference the storeId column, and also has a typo with the closing parentheses for the cast() function. Option D correctly references the storeId column using col(), but has a typo with the quotation marks and parentheses. Option E has a syntax error where the cast() function is inside the quotation marks, and is also missing the col() function to reference the storeId column. Therefore, the correct answer is B. storesDF.withColumn("storeId", col("storeId").cast(StringType()))

    2. J

       jds0

       2024-07-23

       Selected B1 upvote

       Answer is B but with a typo: See code below (Spark 3.5.1): from pyspark.sql import SparkSession from pyspark.sql.functions import col, cast from pyspark.sql.types import StringType spark = SparkSession.builder.appName("MyApp").getOrCreate() data = [ (0, 3, 20000, "A"), (1, 1, 50000, "A"), (2, 2, 70000, "A"), ] storesDF = spark.createDataFrame(data, ["storeID", "customerSatisfaction", "sqft", "division"]) storesDF.withColumn("storeId", col("storeId").cast(StringType())).printSchema() # root # |-- storeId: string (nullable = true) # |-- customerSatisfaction: long (nullable = true) # |-- sqft: long (nullable = true) # |-- division: string (nullable = true)

    3. D

       dduque10

       2023-05-15

       Selected B3 upvotes

       All answers are wrong because the first argument does not have the closing quotes :D, apart from that, it is B

    4. Z

       ZSun

       2023-06-06

       1 upvote

       cast is a method belongs to class pyspark.sql.column therefore, A C E are wrong. it should be dataframe.column.cast() or col('col_name').cast() B is correct, with small typo

12. Question 12 of 25

    Which of the following code blocks returns a new DataFrame with a new column employeesPerSqft that is the quotient of column numberOfEmployees and column sqft, both of which are from DataFrame storesDF? Note that column employeesPerSqft is not in the original DataFrame storesDF.

    • AstoresDF.withColumn("employeesPerSqft", col("numberOfEmployees") / col("sqft"))
    • BstoresDF.withColumn("employeesPerSqft", "numberOfEmployees" / "sqft")
    • CstoresDF.select("employeesPerSqft", "numberOfEmployees" / "sqft")
    • DstoresDF.select("employeesPerSqft", col("numberOfEmployees") / col("sqft"))
    • EstoresDF.withColumn(col("employeesPerSqft"), col("numberOfEmployees") / col("sqft"))

    Show answer

    Correct answer: A

    How others answered

    • A100%

    Explanation

    Answer: A All other options do not work Code: from pyspark.sql import SparkSession from pyspark.sql.functions import col spark = SparkSession.builder.appName("MyApp").getOrCreate() data = [ (0, 3, 20, "A"), (1, 1, 50, "A"), (2, 2, 70, "A"), ] storesDF = spark.createDataFrame(data, ["storeID", "numberOfEmployees", "sqft", "division"]) storesDF.withColumn("employeesPerSqft", col("numberOfEmployees") / col("sqft")) # A.

    Adapted from community discussion by jds0 · 2024-07-23 · 2 community upvotes. Third-party contribution; reviewed by our quality filter, not independently verified.

    View community discussion4

    Candidate context

    Question 12 discussion

    These are quality-filtered excerpts from third-party community discussion. They can be useful reasoning signals, but they are not official answers and have not been independently verified by SPARK-DEV or CertExamHQ.

    1. 4

       4be8126

       2023-04-26

       1 upvote

       The correct code block to return a new DataFrame with a new column employeesPerSqft that is the quotient of column numberOfEmployees and column sqft from DataFrame storesDF is: storesDF.withColumn("employeesPerSqft", col("numberOfEmployees") / col("sqft")) Option A correctly uses the withColumn() function to create a new column employeesPerSqft by dividing column numberOfEmployees by column sqft. Option B has a syntax error because it uses quotation marks to reference column names instead of the col() function. Option C also has a syntax error because it uses quotation marks to reference column names instead of the col() function, and also uses the select() function instead of withColumn() to create a new column. Option D correctly references column names using col() and uses the select() function to return a DataFrame with only the two selected columns. Option E has a syntax error where col() is used as a first argument instead of a second argument for the withColumn() function. Therefore, the correct answer is A. storesDF.withColumn("employeesPerSqft", col("numberOfEmployees") / col("sqft"))

    2. N

       newusername

       2023-09-11

       Selected A1 upvote

       Test: from pyspark.sql import SparkSession from pyspark.sql.functions import col # Initializing Spark session (if not already initialized) spark = SparkSession.builder.appName("databricks_example").getOrCreate() # Creating some synthetic data for storesDF data = [ {"storeId": 1, "numberOfEmployees": 10, "sqft": 500}, {"storeId": 2, "numberOfEmployees": 15, "sqft": 750}, {"storeId": 3, "numberOfEmployees": 8, "sqft": 400} ] storesDF = spark.createDataFrame(data) # Option A: try: df_a = storesDF.withColumn("employeesPerSqft", col("numberOfEmployees") / col("sqft")) df_a.show() print("Option A works") except Exception as e: print("Option A doesn't work:", str(e))

    3. J

       jds0

       2024-07-23

       Selected A2 upvotes

       Answer: A All other options do not work Code: from pyspark.sql import SparkSession from pyspark.sql.functions import col spark = SparkSession.builder.appName("MyApp").getOrCreate() data = [ (0, 3, 20, "A"), (1, 1, 50, "A"), (2, 2, 70, "A"), ] storesDF = spark.createDataFrame(data, ["storeID", "numberOfEmployees", "sqft", "division"]) storesDF.withColumn("employeesPerSqft", col("numberOfEmployees") / col("sqft")) # A.

    4. 4

       4be8126

       2023-04-26

       1 upvote

       storesDF.select("employeesPerSqft", col("numberOfEmployees") / col("sqft")) This code block selects the columns "employeesPerSqft" and the quotient of "numberOfEmployees" and "sqft" from the DataFrame storesDF. However, since "employeesPerSqft" is not a column in the original storesDF, this code block would throw an error. To create a new column "employeesPerSqft" in the resulting DataFrame, we need to use the withColumn() method instead of select(). Here's the corrected code block: storesDF.withColumn("employeesPerSqft", col("numberOfEmployees") / col("sqft")) This code block adds a new column "employeesPerSqft" to the storesDF DataFrame. The new column is created by dividing the values in column "numberOfEmployees" by the values in column "sqft".

13. Question 13 of 25

    Which of the following code blocks returns a DataFrame where column storeCategory from DataFrame storesDF is split at the underscore character into column storeValueCategory and column storeSizeCategory? A sample of DataFrame storesDF is displayed below:

    

    • A(storesDF.withColumn("storeValueCategory", split(col("storeCategory"), "_")[1]) .withColumn("storeSizeCategory", split(col("storeCategory"), "_")[2]))
    • B(storesDF.withColumn("storeValueCategory", col("storeCategory").split("_")[0]) .withColumn("storeSizeCategory", col("storeCategory").split("_")[1]))
    • C(storesDF.withColumn("storeValueCategory", split(col("storeCategory"), "_")[0]) .withColumn("storeSizeCategory", split(col("storeCategory"), "_")[1]))
    • D(storesDF.withColumn("storeValueCategory", split("storeCategory", "_")[0]) .withColumn("storeSizeCategory", split("storeCategory", "_")[1]))
    • E(storesDF.withColumn("storeValueCategory", col("storeCategory").split("_")[1]) .withColumn("storeSizeCategory", col("storeCategory").split("_")[2]))

    Show answer

    Correct answer: C

    How others answered

    • C100%

    Explanation

    Option C returns a DataFrame where column storeCategory from DataFrame storesDF is split at the underscore character into column storeValueCategory and column storeSizeCategory. The correct code is: (storesDF.withColumn("storeValueCategory", split(col("storeCategory"), "_")[0]) .withColumn("storeSizeCategory", split(col("storeCategory"), "_")[1])) Explanation: split(col("storeCategory"), "_") splits the values in column storeCategory by the "_" character and returns an array of strings. [0] gets the first element of the resulting array and assigns it to the new column storeValueCategory. [1] gets the second element of the resulting array and assigns it to the new column storeSizeCategory. withColumn is used to create the new columns and returns a new DataFrame.

    Adapted from community discussion by 4be8126 · 2023-04-26 · 2 community upvotes. Third-party contribution; reviewed by our quality filter, not independently verified.

    View community discussion4

    Candidate context

    Question 13 discussion

    These are quality-filtered excerpts from third-party community discussion. They can be useful reasoning signals, but they are not official answers and have not been independently verified by SPARK-DEV or CertExamHQ.

    1. R

       ronfun

       2023-04-09

       8 upvotes

       Both C or D are correct. Function split accepts both col and str. https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.split.html?highlight=split#pyspark.sql.functions.split

    2. 4

       4be8126

       2023-04-26

       Selected C2 upvotes

       Option C returns a DataFrame where column storeCategory from DataFrame storesDF is split at the underscore character into column storeValueCategory and column storeSizeCategory. The correct code is: (storesDF.withColumn("storeValueCategory", split(col("storeCategory"), "_")[0]) .withColumn("storeSizeCategory", split(col("storeCategory"), "_")[1])) Explanation: split(col("storeCategory"), "_") splits the values in column storeCategory by the "_" character and returns an array of strings. [0] gets the first element of the resulting array and assigns it to the new column storeValueCategory. [1] gets the second element of the resulting array and assigns it to the new column storeSizeCategory. withColumn is used to create the new columns and returns a new DataFrame.

    3. N

       newusername

       2023-09-11

       Selected C2 upvotes

       C you can check, by running the code below: from pyspark.sql import SparkSession # Initialize Spark session spark = SparkSession.builder.appName("split_test").getOrCreate() # Create synthetic data data = [ {"storeCategory": "value1_size1"}, {"storeCategory": "value2_size2"}, {"storeCategory": "value3_size3"}, ] storesDF = spark.createDataFrame(data) storesDF.show() from pyspark.sql.functions import split, col # Option C newDF = (storesDF.withColumn("storeValueCategory", split(col("storeCategory"), "_")[0]) .withColumn("storeSizeCategory", split(col("storeCategory"), "_")[1])) newDF.show()

    4. J

       jds0

       2024-07-23

       Selected C1 upvote

       Both C or D work in Spark 3.5.1, but C is probably better for backward compatibility. See code example below: from pyspark.sql import SparkSession from pyspark.sql.functions import split, col spark = SparkSession.builder.appName("MyApp").getOrCreate() data = [ (0, True, 10020, "VALUE_MEDIUM"), (1, True, 10050, "MAINSTREAM_SMALL"), (2, False, 10070, "PREMIUM_LARGE"), ] storesDF = spark.createDataFrame(data, ["storeID", "open", "openDate", "storeCategory"]) (storesDF.withColumn("storeValueCategory", split(col("storeCategory"), "_")[0]).withColumn("storeSizeCategory", split(col("storeCategory"), "_")[1])).show() (storesDF.withColumn("storeValueCategory", split("storeCategory", "_")[0]).withColumn("storeSizeCategory", split("storeCategory", "_")[1])).show()

14. Question 14 of 25

    Which of the following code blocks returns a new DataFrame where column productCategories only has one word per row, resulting in a DataFrame with many more rows than DataFrame storesDF? A sample of storesDF is displayed below:

    

    • AstoresDF.withColumn("productCategories", explode(col("productCategories")))
    • BstoresDF.withColumn("productCategories", split(col("productCategories")))
    • CstoresDF.withColumn("productCategories", col("productCategories").explode())
    • DstoresDF.withColumn("productCategories", col("productCategories").split())
    • EstoresDF.withColumn("productCategories", explode("productCategories"))

    Show answer

    Correct answer: A

    How others answered

    • A88%

    Explanation

    A is correct, use below code to test: from pyspark.sql import SparkSession # Initializing Spark session spark = SparkSession.builder.appName("test").getOrCreate() # 1. Creating DataFrame with an array column data_array = [ (1, ["electronics", "clothes", "toys"]), (2, ["groceries", "electronics"]), (3, ["books", "clothes"]), ] storesDF = spark.createDataFrame(data_array, ["ID", "productCategories"]) storesDF.show() df_array = storesDF.withColumn("productCategories", explode(col("productCategories"))) df_array.show()

    Adapted from community discussion by newusername · 2023-09-11 · 3 community upvotes. Third-party contribution; reviewed by our quality filter, not independently verified.

    View community discussion4

    Candidate context

    Question 14 discussion

    These are quality-filtered excerpts from third-party community discussion. They can be useful reasoning signals, but they are not official answers and have not been independently verified by SPARK-DEV or CertExamHQ.

    1. N

       newusername

       2023-09-11

       Selected A3 upvotes

       A is correct, use below code to test: from pyspark.sql import SparkSession # Initializing Spark session spark = SparkSession.builder.appName("test").getOrCreate() # 1. Creating DataFrame with an array column data_array = [ (1, ["electronics", "clothes", "toys"]), (2, ["groceries", "electronics"]), (3, ["books", "clothes"]), ] storesDF = spark.createDataFrame(data_array, ["ID", "productCategories"]) storesDF.show() df_array = storesDF.withColumn("productCategories", explode(col("productCategories"))) df_array.show()

    2. N

       newusername

       2023-09-11

       2 upvotes

       But E works as well, sadly. What has to be chosen then? from pyspark.sql import SparkSession # Initializing Spark session spark = SparkSession.builder.appName("test").getOrCreate() # 1. Creating DataFrame with an array column data_array = [ (1, ["electronics", "clothes", "toys"]), (2, ["groceries", "electronics"]), (3, ["books", "clothes"]), ] storesDF = spark.createDataFrame(data_array, ["ID", "productCategories"]) storesDF.show() #df_array = storesDF.withColumn("productCategories", explode(col("productCategories"))) #df_array.show() #check E df_array = storesDF.withColumn("productCategories", explode("productCategories")) df_array.show()

    3. J

       jds0

       2024-07-24

       Selected A1 upvote

       Both option A and E work with spark 3.5.1. But A is better for backward compatibility. See code example below: from pyspark.sql import SparkSession from pyspark.sql.functions import col, explode spark = SparkSession.builder.appName("MyApp").getOrCreate() data = [ (0, ["value 1", "value 2", "value 3"]), (1, ["value 1", "value 2", "value 3"]), (2, ["value 1", "value 2", "value 3"]), ] storesDF = spark.createDataFrame(data, ["storeID", "productCategories"]) storesDF.withColumn("productCategories", explode(col("productCategories"))).show() # A. storesDF.withColumn("productCategories", explode("productCategories")).show() # E.

    4. M

       mhaskins

       2023-05-22

       Selected A2 upvotes

       While the Explode function allows for a str or Column input, this requires the col() wrapper because it is used in a withColumn() call, where the 2nd parameter requires the column object. https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.withColumn.html?highlight=withcolumn#pyspark.sql.DataFrame.withColumn

15. Question 15 of 25

    Which of the following code blocks returns a new DataFrame with column storeDescription where the pattern "Description: " has been removed from the beginning of column storeDescription in DataFrame storesDF? A sample of DataFrame storesDF is below:

    

    • AstoresDF.withColumn("storeDescription", regexp_replace(col("storeDescription"), "^Description: "))
    • BstoresDF.withColumn("storeDescription", col("storeDescription").regexp_replace("^Description: ", ""))
    • CstoresDF.withColumn("storeDescription", regexp_extract(col("storeDescription"), "^Description: ", ""))
    • DstoresDF.withColumn("storeDescription", regexp_replace("storeDescription", "^Description: ", ""))
    • EstoresDF.withColumn("storeDescription", regexp_replace(col("storeDescription"), "^Description: ", ""))

    Show answer

    Correct answer: E

    How others answered

    • E56%
    • D31%

    Only 56% agreed on this answer — treat it as contested rather than settled.

    Explanation

    The correct answer is option E: storesDF.withColumn("storeDescription", regexp_replace(col("storeDescription"), "^Description: ", "")). This code block uses the withColumn() function to create a new column called storeDescription. It uses the regexp_replace() function to replace the pattern "^Description: " at the beginning of the string in the storeDescription column with an empty string. This effectively removes the pattern from the beginning of the string in each row of the column.

    Adapted from community discussion by 4be8126 · 2023-04-26 · 4 community upvotes. Third-party contribution; reviewed by our quality filter, not independently verified.

    View community discussion4

    Candidate context

    Question 15 discussion

    These are quality-filtered excerpts from third-party community discussion. They can be useful reasoning signals, but they are not official answers and have not been independently verified by SPARK-DEV or CertExamHQ.

    1. Z

       ZSun

       2023-06-06

       7 upvotes

       this is completely wrong explanation. Both D and E has replacement expression, the only difference is how they call the replaced column. Both D and E are correct, but D works for Pyspark 2.0. D and E both work Pyspark 3.0+. Period!

    2. 4

       4be8126

       2023-04-26

       Selected E4 upvotes

       The correct answer is option E: storesDF.withColumn("storeDescription", regexp_replace(col("storeDescription"), "^Description: ", "")). This code block uses the withColumn() function to create a new column called storeDescription. It uses the regexp_replace() function to replace the pattern "^Description: " at the beginning of the string in the storeDescription column with an empty string. This effectively removes the pattern from the beginning of the string in each row of the column.

    3. N

       newusername

       2023-09-11

       3 upvotes

       Both work: from pyspark.sql import SparkSession from pyspark.sql.functions import regexp_replace,regexp_extract, col spark = SparkSession.builder.appName("test").getOrCreate() data = [ (1, "Description: This is a tech store. Description: This"), (2, "Description: This is a grocery store."), (3, "Description: This is a book store."), ] storesDF = spark.createDataFrame(data, ["storeID", "storeDescription"]) storesDF.show(truncate=False) #Case D print ("Case D") storesDF = storesDF.withColumn("storeDescription", regexp_replace("storeDescription", "^Description: ", "")) storesDF.show(truncate=False) #Case E print ("Case E") storesDF = storesDF.withColumn("storeDescription", regexp_replace(col("storeDescription"), "^Description: ", "")) storesDF.show(truncate=False)

    4. 4

       4be8126

       2023-04-26

       1 upvote

       The correct code block that returns a new DataFrame with column storeDescription where the pattern "Description: " has been removed from the beginning of column storeDescription in DataFrame storesDF is: A. storesDF.withColumn("storeDescription", regexp_replace(col("storeDescription"), "^Description: ")) This code uses the regexp_replace function to replace the pattern "^Description: " (which matches the string "Description: " at the beginning of the string) with an empty string in the column storeDescription. The resulting DataFrame will have the modified storeDescription column. Option B has a syntax error because the regexp_replace function should be called on the column using the dot notation instead of passing it as the second argument. Option C uses the regexp_extract function, which extracts a substring matching a regular expression pattern. It doesn't remove the pattern from the string. Option D has a syntax error because the column name is not wrapped in the col function. Option E is the same as option A, except that it uses the col function unnecessarily.

16. Question 16 of 25

    The code block shown contains an error. The code block is intended to return a new DataFrame where column sqft from DataFrame storesDF has had its missing values replaced with the value 30,000. Identify the error. A sample of DataFrame storesDF is displayed below:

    

    Code block: storesDF.na.fill(30000, col("sqft"))

    • AThe argument to the subset parameter of fill() should be a string column name or a list of string column names rather than a Column object.
    • BThe na.fill() operation does not work and should be replaced by the dropna() operation.
    • Che argument to the subset parameter of fill() should be a the numerical position of the column rather than a Column object.
    • DThe na.fill() operation does not work and should be replaced by the nafill() operation.
    • EThe na.fill() operation does not work and should be replaced by the fillna() operation.

    Show answer

    Correct answer: A

    How others answered

    • A83%
    • E17%

    Explanation

    The most correct answer seems to be A: Code below with Spark 3.5.1. ```python from pyspark.sql import SparkSession from pyspark.sql.functions import col from pyspark.errors import PySparkTypeError spark = SparkSession.builder.appName("MyApp").getOrCreate() data = [ (0, 43161), (1, 51200), (2, None), (3, 78367), (4, None), ] storesDF = spark.createDataFrame(data, ["storeID", "sqft"]) # storesDF.show() try: storesDF.na.fill(30000, col("sqft")) except PySparkTypeError as e: print(e) storesDF.na.fill(30000, "sqft").show() storesDF.na.fill(30000, ["sqft"]).show() storesDF.fillna(30000, ["sqft"]).show() storesDF.fillna(30000, "sqft").show() ```

    Adapted from community discussion by jds0 · 2024-07-23 · 2 community upvotes. Third-party contribution; reviewed by our quality filter, not independently verified.

    View community discussion4

    Candidate context

    Question 16 discussion

    These are quality-filtered excerpts from third-party community discussion. They can be useful reasoning signals, but they are not official answers and have not been independently verified by SPARK-DEV or CertExamHQ.

    1. Z

       ZSun

       2023-06-06

       6 upvotes

       Correct anwser is A. even for most updated version, spark 3.4. na.fill() still functioning, it is an alias of fillna() Mr. 4be8126 , 你可真是张嘴就来啊

    2. J

       jds0

       2024-07-23

       Selected A2 upvotes

       The most correct answer seems to be A: Code below with Spark 3.5.1. ```python from pyspark.sql import SparkSession from pyspark.sql.functions import col from pyspark.errors import PySparkTypeError spark = SparkSession.builder.appName("MyApp").getOrCreate() data = [ (0, 43161), (1, 51200), (2, None), (3, 78367), (4, None), ] storesDF = spark.createDataFrame(data, ["storeID", "sqft"]) # storesDF.show() try: storesDF.na.fill(30000, col("sqft")) except PySparkTypeError as e: print(e) storesDF.na.fill(30000, "sqft").show() storesDF.na.fill(30000, ["sqft"]).show() storesDF.fillna(30000, ["sqft"]).show() storesDF.fillna(30000, "sqft").show() ```

    3. 4

       4be8126

       2023-04-26

       2 upvotes

       The correct answer is either A or E, depending on the version of Spark being used. In Spark 2.x, the correct method to replace missing values is na.fill(). Option A is correct in Spark 2.x, as it correctly specifies the column to apply the fill operation to using a Column object. However, in Spark 3.x, the method has been renamed to fillna(). Therefore, in Spark 3.x, the correct answer is E, as it uses the correct method name. Both A and E accomplish the same task of replacing missing values in the sqft column with 30,000, so either answer can be considered correct depending on the version of Spark being used.

    4. P

       peekaboo15

       2023-04-13

       Selected A2 upvotes

       the answer should be A. See this link for reference https://sparkbyexamples.com/pyspark/pyspark-fillna-fill-replace-null-values/

17. Question 17 of 25

    Which of the following will occur if there are more slots than there are tasks?

    • AThe Spark job will likely not run as efficiently as possible.
    • BThe Spark application will fail – there must be at least as many tasks as there are slots.
    • CSome executors will shut down and allocate all slots on larger executors first.
    • DMore tasks will be automatically generated to ensure all slots are being used.
    • EThe Spark job will use just one single slot to perform all tasks.

    Show answer

    Correct answer: A

    How others answered

    • A88%

    Explanation

    Slots are the basic unit of parallelism in Spark, and represent a unit of resource allocation on a single executor. If there are more slots than there are tasks, it means that some of the slots will be idle and not being used to execute any tasks, leading to inefficient resource utilization. In this scenario, the Spark job will likely not run as efficiently as possible. However, it is still possible for the Spark job to complete successfully. Therefore, option A is the correct answer.

    Adapted from community discussion by TC007 · 2023-04-03 · 6 community upvotes. Third-party contribution; reviewed by our quality filter, not independently verified.

    View community discussion4

    Candidate context

    Question 17 discussion

    These are quality-filtered excerpts from third-party community discussion. They can be useful reasoning signals, but they are not official answers and have not been independently verified by SPARK-DEV or CertExamHQ.

    1. T

       TC007

       2023-04-03

       Selected A6 upvotes

       Slots are the basic unit of parallelism in Spark, and represent a unit of resource allocation on a single executor. If there are more slots than there are tasks, it means that some of the slots will be idle and not being used to execute any tasks, leading to inefficient resource utilization. In this scenario, the Spark job will likely not run as efficiently as possible. However, it is still possible for the Spark job to complete successfully. Therefore, option A is the correct answer.

    2. 4

       4be8126

       2023-04-24

       Selected A4 upvotes

       A. The Spark job will likely not run as efficiently as possible. In Spark, a slot represents a unit of processing capacity that an executor can offer to run a task. If there are more slots than there are tasks, some of the slots will remain unused, and the Spark job will likely not run as efficiently as possible. Spark automatically assigns tasks to slots, and if there are more slots than necessary, some of them may remain idle, resulting in wasted resources and slower job execution. However, the job will not fail as long as there are enough resources to execute the tasks, and Spark will not generate more tasks than needed. Also, executors will not shut down because there are unused slots. They will remain active until the end of the job or until explicitly terminated.

    3. Z

       zozoshanky

       2023-12-30

       2 upvotes

       C. Some executors will shut down and allocate all slots on larger executors first. explanation :If there are more slots than there are tasks in Apache Spark, some executors may shut down, and the available slots will be allocated to larger executors first. This process is part of the dynamic resource allocation mechanism in Spark, where resources are adjusted based on the workload. It helps in efficient resource utilization by shutting down unnecessary executors and allocating resources to larger executors to perform tasks more efficiently.

    4. S

       singh100

       2023-07-31

       1 upvote

       A. IF there are more slots than there are tasks, the extra slots will not be utilized, and they will remain idle, resulting in some resource waste. To maximize resource usage efficiency, it is essential to configure the cluster properly and adjust the number of tasks and slots based on the workload demands. Dynamic resource allocation features in cluster managers can also help improve resource utilization by adjusting the cluster size dynamically based on the task requirements.

18. Question 18 of 25

    Which of the following operations fails to return a DataFrame with no duplicate rows?

    • ADataFrame.dropDuplicates()
    • BDataFrame.distinct()
    • CDataFrame.drop_duplicates()
    • DDataFrame.drop_duplicates(subset = None)
    • EDataFrame.drop_duplicates(subset = "all")

    Show answer

    Correct answer: E

    How others answered

    • E96%

    Explanation

    A. DataFrame.dropDuplicates(): This method returns a new DataFrame with distinct rows based on all columns. It should return a DataFrame with no duplicate rows. B. DataFrame.distinct(): This method returns a new DataFrame with distinct rows based on all columns. It should also return a DataFrame with no duplicate rows. C. DataFrame.drop_duplicates(): This is an alias for DataFrame.dropDuplicates(). It should also return a DataFrame with no duplicate rows. D. DataFrame.drop_duplicates(subset=None): This method returns a new DataFrame with distinct rows based on all columns. It should return a DataFrame with no duplicate rows. E. DataFrame.drop_duplicates(subset="all"): This method attempts to drop duplicates based on all columns but returns an error, because "all" is not a valid argument for the subset parameter. So this operation fails to return a DataFrame with no duplicate rows. Therefore, the correct answer is E.

    Adapted from community discussion by 4be8126 · 2023-04-26 · 10 community upvotes. Third-party contribution; reviewed by our quality filter, not independently verified.

    View community discussion4

    Candidate context

    Question 18 discussion

    These are quality-filtered excerpts from third-party community discussion. They can be useful reasoning signals, but they are not official answers and have not been independently verified by SPARK-DEV or CertExamHQ.

    1. 4

       4be8126

       2023-04-26

       Selected E10 upvotes

       A. DataFrame.dropDuplicates(): This method returns a new DataFrame with distinct rows based on all columns. It should return a DataFrame with no duplicate rows. B. DataFrame.distinct(): This method returns a new DataFrame with distinct rows based on all columns. It should also return a DataFrame with no duplicate rows. C. DataFrame.drop_duplicates(): This is an alias for DataFrame.dropDuplicates(). It should also return a DataFrame with no duplicate rows. D. DataFrame.drop_duplicates(subset=None): This method returns a new DataFrame with distinct rows based on all columns. It should return a DataFrame with no duplicate rows. E. DataFrame.drop_duplicates(subset="all"): This method attempts to drop duplicates based on all columns but returns an error, because "all" is not a valid argument for the subset parameter. So this operation fails to return a DataFrame with no duplicate rows. Therefore, the correct answer is E.

    2. T

       TC007

       2023-03-26

       Selected E10 upvotes

       Option E is incorrect as "all" is not a valid value for the subset parameter in the drop_duplicates() method. The correct value should be a column name or a list of column names to be used as the subset to identify duplicate rows. All other options (A, B, C, and D) can be used to return a DataFrame with no duplicate rows. The dropDuplicates(), distinct(), and drop_duplicates() methods are all equivalent and return a new DataFrame with distinct rows. The drop_duplicates() method also accepts a subset parameter to specify the columns to use for identifying duplicates, and when the subset parameter is not specified, all columns are used. Therefore, both option A and C are valid, and option D is also valid as it is equivalent to drop_duplicates() with no subset parameter.

    3. J

       jds0

       2024-07-23

       Selected E2 upvotes

       It's E, see code below: # Drop duplicates in a DataFrame from pyspark.sql import SparkSession from pyspark.sql.functions import col from pyspark.errors import PySparkTypeError spark = SparkSession.builder.appName("MyApp").getOrCreate() data = [ (0, 43161), (0, 43161), (1, 51200), (2, None), (2, None), (3, 78367), (4, None), ] storesDF = spark.createDataFrame(data, ["storeID", "sqft"]) try: storesDF.dropDuplicates().show() except PySparkTypeError as e: print(e) try: storesDF.distinct().show() except PySparkTypeError as e: print(e) try: storesDF.drop_duplicates().show() except PySparkTypeError as e: print(e) try: storesDF.drop_duplicates(subset=None).show() except PySparkTypeError as e: print(e) try: storesDF.drop_duplicates(subset="all").show() except PySparkTypeError as e: print(e)

    4. A

       azurearch

       2024-03-07

       2 upvotes

       Option E . df.drop_duplicates(subset = "all") returns error SparkTypeError: [NOT_LIST_OR_TUPLE] Argument `subset` should be a list or tuple, got str.

19. Question 19 of 25

    Which of the following code blocks will most quickly return an approximation for the number of distinct values in column division in DataFrame storesDF?

    • AstoresDF.agg(approx_count_distinct(col("division")).alias("divisionDistinct"))
    • BstoresDF.agg(approx_count_distinct(col("division"), 0.01).alias("divisionDistinct"))
    • CstoresDF.agg(approx_count_distinct(col("division"), 0.15).alias("divisionDistinct"))
    • DstoresDF.agg(approx_count_distinct(col("division"), 0.0).alias("divisionDistinct"))
    • EstoresDF.agg(approx_count_distinct(col("division"), 0.05).alias("divisionDistinct"))

    Show answer

    Correct answer: C

    How others answered

    • C69%
    • B31%

    Only 69% agreed on this answer — treat it as contested rather than settled.

    Explanation

    I see you reply in a lot of question, barely correct. bro, you need to stop comment wrong information here. This question only ask for efficiency, no need to balance between accuracy and efficiency. Stop posting ChatGPT answer here

    Adapted from community discussion by ZSun · 2023-06-06 · 19 community upvotes. Third-party contribution; reviewed by our quality filter, not independently verified.

    View community discussion4

    Candidate context

    Question 19 discussion

    These are quality-filtered excerpts from third-party community discussion. They can be useful reasoning signals, but they are not official answers and have not been independently verified by SPARK-DEV or CertExamHQ.

    1. Z

       ZSun

       2023-06-06

       19 upvotes

       I see you reply in a lot of question, barely correct. bro, you need to stop comment wrong information here. This question only ask for efficiency, no need to balance between accuracy and efficiency. Stop posting ChatGPT answer here

    2. T

       thanab

       2023-09-06

       4 upvotes

       C The code block that will most quickly return an approximation for the number of distinct values in column `division` in DataFrame `storesDF` is **C**, `storesDF.agg(approx_count_distinct(col("division"), 0.15).alias("divisionDistinct"))`. The `approx_count_distinct` function can be used to quickly estimate the number of distinct values in a column by using a probabilistic data structure. The second parameter of the `approx_count_distinct` function specifies the maximum estimation error allowed, with a smaller value resulting in a more accurate but slower estimation. In this case, an error of 0.15 is specified, which will result in a faster but less accurate estimation than the other options.

    3. S

       SonicBoom10C9

       2023-05-15

       Selected C3 upvotes

       While not an option I would use, the question says most quickly (relatively), and this will be the fastest. Note that a 15% error is too high.

    4. O

       oussa_ama

       2024-08-19

       Selected C3 upvotes

       C is correct because it will provide the fastest approximate count with a standard deviation of 0.15

20. Question 20 of 25

    The code block shown below contains an error. The code block is intended to return a new DataFrame with the mean of column sqft from DataFrame storesDF in column sqftMean. Identify the error. Code block: storesDF.agg(mean("sqft").alias("sqftMean"))

    • AThe argument to the mean() operation should be a Column abject rather than a string column name.
    • BThe argument to the mean() operation should not be quoted.
    • CThe mean() operation is not a standalone function – it’s a method of the Column object.
    • DThe agg() operation is not appropriate here – the withColumn() operation should be used instead.
    • EThe only way to compute a mean of a column is with the mean() method from a DataFrame.

    Show answer

    Correct answer: A

    How others answered

    • A47%
    • E37%
    • D16%

    Only 47% agreed on this answer — treat it as contested rather than settled.

    Explanation

    Correct answer is A: from pyspark.sql.functions import col, mean students =[ {'rollno':'001','name':'sravan','sqft':23, 'height':5.79,'weight':67,'address':'guntur'}, {'rollno':'002','name':'ojaswi','sqft':16, 'height':3.79,'weight':34,'address':'hyd'}] storesDF = spark.createDataFrame( students) storesDF.agg(mean(col('sqft')).alias('sqftMean')).show()

    Adapted from community discussion by juliom6 · 2023-11-01 · 3 community upvotes. Third-party contribution; reviewed by our quality filter, not independently verified.

    View community discussion4

    Candidate context

    Question 20 discussion

    These are quality-filtered excerpts from third-party community discussion. They can be useful reasoning signals, but they are not official answers and have not been independently verified by SPARK-DEV or CertExamHQ.

    1. J

       juliom6

       2023-11-01

       Selected A3 upvotes

       Correct answer is A: from pyspark.sql.functions import col, mean students =[ {'rollno':'001','name':'sravan','sqft':23, 'height':5.79,'weight':67,'address':'guntur'}, {'rollno':'002','name':'ojaswi','sqft':16, 'height':3.79,'weight':34,'address':'hyd'}] storesDF = spark.createDataFrame( students) storesDF.agg(mean(col('sqft')).alias('sqftMean')).show()

    2. C

       cookiemonster42

       2023-08-10

       Selected A3 upvotes

       There's a similar question in the official Databricks samples and the right answer there is: Code block: storesDF.__1__(__2__(__3__).alias("sqftMean")) A. 1. agg 2. mean 3. col("sqft") If we stick to this logic, the answer is A.

    3. A

       azurearch

       2024-03-07

       3 upvotes

       from pyspark.sql.functions import col, mean students =[ {'rollno':'001','name':'sravan','sqft':23, 'height':5.79,'weight':67,'address':'guntur'}, {'rollno':'002','name':'ojaswi','sqft':16, 'height':3.79,'weight':34,'address':'hyd'}] storesDF = spark.createDataFrame( students) storesDF.agg(mean('sqft').alias('sqftMean')).show() this works as well! not sure which one is wrong then

    4. O

       outwalker

       2023-11-08

       1 upvote

       it appears that there might be some flexibility in how the mean function can be used with either a string column name or a col() function. However, the most accurate and recommended approach is to use the col() function to create a Column object explicitly. With this in mind, the best choice is: A. The argument to the mean() operation should be a Column object rather than a string column name. The mean function takes a Column object as an argument, not a string column name. To fix the error, the code block should be rewritten as storesDF.agg(mean(col("sqft")).alias("sqftMean")), where the col function is used to create a Column object from the string column name "sqft". While there might be situations where using a string column name works, following the standard practice of creating a Column object with col() ensures compatibility and clarity in code.

21. Question 21 of 25

    Which of the following code blocks returns a collection of summary statistics for all columns in DataFrame storesDF?

    • AstoresDF.summary("mean")
    • BstoresDF.describe(all = True)
    • CstoresDF.describe("all")
    • DstoresDF.summary("all")
    • EstoresDF.describe()

    Show answer

    Correct answer: E

    How others answered

    • E83%
    • B17%

    Explanation

    E is the right option. See code below with Spark 3.5.1 # Summary statistics of a DataFrame from pyspark.sql import SparkSession from pyspark.sql.functions import col from pyspark.errors import PySparkTypeError spark = SparkSession.builder.appName("MyApp").getOrCreate() data = [ (0, 43161), (1, 51200), (2, None), (3, 78367), (4, None), ] storesDF = spark.createDataFrame(data, ["storeID", "sqft"]) try: storesDF.summary("mean").show() except Exception as e: print(e) try: storesDF.describe(all = True).show() except Exception as e: print(e) try: storesDF.describe("all").show() except Exception as e: print(e) try: storesDF.summary("all").show() except Exception as e: print(e) try: storesDF.describe().show() except Exception as e: print(e)

    Adapted from community discussion by jds0 · 2024-07-23 · 2 community upvotes. Third-party contribution; reviewed by our quality filter, not independently verified.

    View community discussion4

    Candidate context

    Question 21 discussion

    These are quality-filtered excerpts from third-party community discussion. They can be useful reasoning signals, but they are not official answers and have not been independently verified by SPARK-DEV or CertExamHQ.

    1. Z

       ZSun

       2023-06-06

       6 upvotes

       Did you really try this in pyspark, or look up the document? TypeError: describe() got an unexpected keyword argument 'all'

    2. J

       jds0

       2024-07-23

       Selected E2 upvotes

       E is the right option. See code below with Spark 3.5.1 # Summary statistics of a DataFrame from pyspark.sql import SparkSession from pyspark.sql.functions import col from pyspark.errors import PySparkTypeError spark = SparkSession.builder.appName("MyApp").getOrCreate() data = [ (0, 43161), (1, 51200), (2, None), (3, 78367), (4, None), ] storesDF = spark.createDataFrame(data, ["storeID", "sqft"]) try: storesDF.summary("mean").show() except Exception as e: print(e) try: storesDF.describe(all = True).show() except Exception as e: print(e) try: storesDF.describe("all").show() except Exception as e: print(e) try: storesDF.summary("all").show() except Exception as e: print(e) try: storesDF.describe().show() except Exception as e: print(e)

    3. C

       cookiemonster42

       2023-08-03

       Selected E1 upvote

       check the documentation, mates. both methods receive names of columns as arguments, so E is correct!

    4. J

       juadaves

       2023-10-19

       1 upvote

       TypeError Traceback (most recent call last) in () ----> 1 storesDF.describe(all = True) TypeError: DataFrame.describe() got an unexpected keyword argument 'all'

22. Question 22 of 25

    The code block shown below contains an error. The code block is intended to create a Python UDF assessPerformanceUDF() using the integer-returning Python function assessPerformance() and apply it to column customerSatisfaction in DataFrame storesDF. Identify the error. Code block: assessPerformanceUDF – udf(assessPerformance) storesDF.withColumn("result", assessPerformanceUDF(col("customerSatisfaction")))

    • AThe assessPerformance() operation is not properly registered as a UDF.
    • BThe withColumn() operation is not appropriate here – UDFs should be applied by iterating over rows instead.
    • CUDFs can only be applied vie SQL and not through the DataFrame API.
    • DThe return type of the assessPerformanceUDF() is not specified in the udf() operation.
    • EThe assessPerformance() operation should be used on column customerSatisfaction rather than the assessPerformanceUDF() operation.

    Show answer

    Correct answer: D

    How others answered

    • D70%
    • A30%

    Only 70% agreed on this answer — treat it as contested rather than settled.

    Explanation

    The right answer is D. pyspark.sql.functions.udf(f=None, returnType=StringType) The default return type is string, but this question requires integer returning. so it should be D. "The return type of the assessPerformanceUDF() is not specified in the udf() operation."

    Adapted from community discussion by ZSun · 2023-06-06 · 11 community upvotes. Third-party contribution; reviewed by our quality filter, not independently verified.

    View community discussion4

    Candidate context

    Question 22 discussion

    These are quality-filtered excerpts from third-party community discussion. They can be useful reasoning signals, but they are not official answers and have not been independently verified by SPARK-DEV or CertExamHQ.

    1. Z

       ZSun

       2023-06-06

       11 upvotes

       The right answer is D. pyspark.sql.functions.udf(f=None, returnType=StringType) The default return type is string, but this question requires integer returning. so it should be D. "The return type of the assessPerformanceUDF() is not specified in the udf() operation."

    2. J

       jds0

       2024-07-24

       Selected D3 upvotes

       D is the right answer as otherwise the return type is the default StringType(). Test code below: from pyspark.sql import SparkSession from pyspark.sql.functions import col, udf from pyspark.sql.types import IntegerType spark = SparkSession.builder.appName("MyApp").getOrCreate() data = [ (0, 3, "A"), (1, 1, "A"), (2, 2, "A"), ] storesDF = spark.createDataFrame(data, ["storeID", "customerSatisfaction", "division"]) def assessPerformance(x): return 1 if x > 3 else 0 print("IntegerType()") assessPerformanceUDF = udf(assessPerformance, IntegerType()) df = storesDF.withColumn("result", assessPerformanceUDF(col("customerSatisfaction"))) df.printSchema() print("Default") assessPerformanceUDF = udf(assessPerformance) df = storesDF.withColumn("result", assessPerformanceUDF(col("customerSatisfaction"))) df.printSchema()

    3. T

       thanab

       2023-09-16

       Selected D2 upvotes

       The error in the code block is that the return type of the assessPerformanceUDF() is not specified in the udf() operation. In PySpark, when you register a Python function as a UDF, you should also specify the return type. This is important because Spark SQL needs to understand the return type to properly handle the UDF. Therefore, the correct answer is:

    4. Z

       ZSun

       2023-06-06

       3 upvotes

       what is the difference between your code and question itsefl? assessPerformanceUDF – udf(assessPerformance) assessPerformanceUDF = udf(assessPerformance) changing "-" to "="?

23. Question 23 of 25

    The code block shown below should create a single-column DataFrame from Python list years which is made up of integers. Choose the response that correctly fills in the numbered blanks within the code block to complete this task. Code block: _1_._2_(_3_, _4_)

    • A1. spark 2. createDataFrame 3. years 4. IntegerType
    • B1. DataFrame 2. create 3. [years] 4. IntegerType
    • C1. spark 2. createDataFrame 3. [years] 4. IntegertType
    • D1. spark 2. createDataFrame 3. [years] 4. IntegertType()
    • E1. spark 2. createDataFrame 3. years 4. IntegertType()

    Show answer

    Correct answer: E

    How others answered

    • E100%

    Explanation

    E is the right answer See code below: # Create a DataFrame from a list of integers from pyspark.sql import SparkSession from pyspark.sql.types import IntegerType spark = SparkSession.builder.appName("MyApp").getOrCreate() years = [2017, 2018, 2019] df = spark.createDataFrame(years, IntegerType()) df.show() df.printSchema()

    Adapted from community discussion by jds0 · 2024-07-24 · 4 community upvotes. Third-party contribution; reviewed by our quality filter, not independently verified.

    View community discussion4

    Candidate context

    Question 23 discussion

    These are quality-filtered excerpts from third-party community discussion. They can be useful reasoning signals, but they are not official answers and have not been independently verified by SPARK-DEV or CertExamHQ.

    1. P

       peekaboo15

       2023-04-13

       10 upvotes

       The answer should be E because Year is already a python list.

    2. J

       jds0

       2024-07-24

       Selected E4 upvotes

       E is the right answer See code below: # Create a DataFrame from a list of integers from pyspark.sql import SparkSession from pyspark.sql.types import IntegerType spark = SparkSession.builder.appName("MyApp").getOrCreate() years = [2017, 2018, 2019] df = spark.createDataFrame(years, IntegerType()) df.show() df.printSchema()

    3. Z

       zozoshanky

       2023-07-30

       1 upvote

       D throws a big error. /usr/local/spark/python/pyspark/sql/types.py in verify_acceptable_types(obj) 1291 # subclass of them can not be fromInternal in JVM 1292 if type(obj) not in _acceptable_types[_type]: -> 1293 raise TypeError(new_msg("%s can not accept object %r in type %s" 1294 % (dataType, obj, type(obj)))) 1295 TypeError: field value: IntegerType can not accept object [1, 2, 3, 4, 5] in type E is correct answer from pyspark.sql.types import IntegerType a = [1,2,3,4,5] spark.createDataFrame(a, IntegerType()).show()

    4. C

       cookiemonster42

       2023-08-03

       Selected E1 upvote

       if years is variable, it works, just tested it: years = [1, 3, 4, 5 , 9] df7 = spark.createDataFrame(years, IntegerType()) df7.show() this works as well: df7 = spark.createDataFrame([1, 3, 4, 5 , 9], IntegerType()) df7.show() this won't work: df7 = spark.createDataFrame([years], IntegerType()) df7.show() so, the answer is E

24. Question 24 of 25

    The code block shown below contains an error. The code block is intended to cache DataFrame storesDF only in Spark’s memory and then return the number of rows in the cached DataFrame. Identify the error. Code block: storesDF.cache().count()

    • AThe cache() operation caches DataFrames at the MEMORY_AND_DISK level by default – the storage level must be specified to MEMORY_ONLY as an argument to cache().
    • BThe cache() operation caches DataFrames at the MEMORY_AND_DISK level by default – the storage level must be set via storesDF.storageLevel prior to calling cache().
    • CThe storesDF DataFrame has not been checkpointed – it must have a checkpoint in order to be cached.
    • DDataFrames themselves cannot be cached – DataFrame storesDF must be cached as a table.
    • EThe cache() operation can only cache DataFrames at the MEMORY_AND_DISK level (the default) – persist() should be used instead.

    Show answer

    Correct answer: E

    How others answered

    • E67%
    • B33%

    Only 67% agreed on this answer — treat it as contested rather than settled.

    Explanation

    E is correct. You cannot set StorageLevel Memory_only with cache(), if memory available then it keeps everything into memory else it will spill to disk. To keep everything into Memory you need to use Persist() with Storage Level Memory only.

    Adapted from community discussion by singh100 · 2023-08-01 · 5 community upvotes. Third-party contribution; reviewed by our quality filter, not independently verified.

    View community discussion4

    Candidate context

    Question 24 discussion

    These are quality-filtered excerpts from third-party community discussion. They can be useful reasoning signals, but they are not official answers and have not been independently verified by SPARK-DEV or CertExamHQ.

    1. S

       singh100

       2023-08-01

       5 upvotes

       E is correct. You cannot set StorageLevel Memory_only with cache(), if memory available then it keeps everything into memory else it will spill to disk. To keep everything into Memory you need to use Persist() with Storage Level Memory only.

    2. J

       Jtic

       2023-05-27

       Selected E1 upvote

       E B. The cache() operation caches DataFrames at the MEMORY_AND_DISK level by default – the storage level must be set via storesDF.storageLevel prior to calling cache(). This option is incorrect. The storage level does not need to be set via storesDF.storageLevel prior to calling cache(). The cache() operation can be used directly on the DataFrame without explicitly setting the storage level. E. The cache() operation can only cache DataFrames at the MEMORY_AND_DISK level (the default) – persist() should be used instead. This option is the correct answer. The error in the code block is that the cache() operation is used instead of persist(). While cache() caches DataFrames at the default MEMORY_AND_DISK level, persist() provides more flexibility by allowing different storage levels to be specified, such as MEMORY_ONLY for caching only in memory. Therefore, persist() should be used instead of cache() to achieve the desired caching behavior.

    3. Z

       ZSun

       2023-06-06

       3 upvotes

       Wrong explanation. you can call cache() or persist() without set storage level, it will use default Memoery_and_disk. You clearly misunderstand the question itself. storesDF.cache().count() is a workable code, but fail the requirement. This is the issue. The question asked "only in memory", that means, if the data size is out of the memory, i do not want to store it in disk, but rather recompute. Therefore, you need to specifically set the storage level as "MEMORY ONLY". A is correct

    4. J

       juliom6

       2023-11-02

       Selected E1 upvote

       E is correct! from pyspark.sql.types import IntegerType from pyspark import StorageLevel storesDF = spark.createDataFrame([2023, 2024], IntegerType()) print(storesDF.persist(StorageLevel.MEMORY_ONLY).storageLevel)

25. Question 25 of 25

    Which of the following operations can be used to return a new DataFrame from DataFrame storesDF without inducing a shuffle?

    • AstoresDF.intersect()
    • BstoresDF.repartition(1)
    • CstoresDF.union()
    • DstoresDF.coalesce(1)
    • EstoresDF.rdd.getNumPartitions()

    Show answer

    Correct answer: D

    How others answered

    • D67%
    • C33%

    Only 67% agreed on this answer — treat it as contested rather than settled.

    Explanation

    The correct answer is D. coalesce() can be used to return a new DataFrame with a reduced number of partitions, without inducing a shuffle. A shuffle is an expensive operation that involves the redistribution of data across a cluster, so it's important to minimize its use whenever possible. In this case, repartition() and union() both involve shuffles, while intersect() returns only the common rows between two DataFrames, and rdd.getNumPartitions() returns the number of partitions in the RDD underlying the DataFrame.

    Adapted from community discussion by 4be8126 · 2023-11-03 · 3 community upvotes. Third-party contribution; reviewed by our quality filter, not independently verified.

    View community discussion4

    Candidate context

    Question 25 discussion

    These are quality-filtered excerpts from third-party community discussion. They can be useful reasoning signals, but they are not official answers and have not been independently verified by SPARK-DEV or CertExamHQ.

    1. 4

       4be8126

       2023-11-03

       Selected D3 upvotes

       The correct answer is D. coalesce() can be used to return a new DataFrame with a reduced number of partitions, without inducing a shuffle. A shuffle is an expensive operation that involves the redistribution of data across a cluster, so it's important to minimize its use whenever possible. In this case, repartition() and union() both involve shuffles, while intersect() returns only the common rows between two DataFrames, and rdd.getNumPartitions() returns the number of partitions in the RDD underlying the DataFrame.

    2. Z

       ZSun

       2023-12-08

       3 upvotes

       I think this question contains error, it should not be which one without shuffle, it should be which one cause shuffle. union is a narrow transformation, not causing shuffle. coalesce simply combine partitions together into one, not shuffle them. rdd.getNumPartitions just evaluate the number of partition of a dataframe, no shuffle. even for repartition(1), since there is only one partition in the end, it also not causing shuffle, it simply combine all partition together. Therefore, it should be A, this is the only one inducing a shuffle. or, B C D E without inducing a shuffle

    3. A

       azurearch

       2024-09-07

       3 upvotes

       Though Union does not cause a shuffle, you need another dataframe to do union. in this question its limited to storesDF. coalesce(1) is the correct answer, as it does not cause shuffle rather combines multiple partitions into 1, i.e. reducing partitions = no shuffle. execute storedDF.coalesce(1) and check DAG

    4. Z

       ZSun

       2023-12-07

       2 upvotes

       The Answer is C. Union rather than coalesce. Union is a narrow transformation. unlike wide transformationl, narrow transformation does not require shuffle. Coalesce is wide transformation, combine multiple partition to smaller number of partition. Don't this process require shuffling partition together? if you ask ChatGPT, it will tell you what 4be8126 comment.

Keyboard:J Kmove between questionsA–Dselect an answerRshow or hide the answer← →change page

SPARK-DEV practice, question 1 of 25