Databricks Databricks-Certified-Associate-Developer-for-Apache-Spark-3.0 Exam Dumps - Actual Questions Answers

Explanation

transactionsDf.groupBy("productId").count()

Correct. This code block first groups DataFrame transactionsDf by column productId and then counts the rows in each group.

transactionsDf.groupBy("productId").select(count("value"))

Incorrect. You cannot call select on a GroupedData object (the output of a groupBy) statement.

transactionsDf.count("productId")

No. DataFrame.count() does not take any arguments.

transactionsDf.count("productId").distinct()

Wrong. Since DataFrame.count() does not take any arguments, this option cannot be right.

transactionsDf.groupBy("productId").agg(col("value").count())

False. A Column object, as returned by col("value"), does not have a count() method. You can see all available methods for Column object linked in the Spark documentation below.

More info: pyspark.sql.DataFrame.count — PySpark 3.1.2 documentation, pyspark.sql.Column — PySpark 3.1.2 documentation

Static notebook | Dynamic notebook: See test 3, QUESTION NO: 41 (Databricks import instructions)

Explanation

Correct code block:

from pyspark.sql.functions import udf

from pyspark.sql import types as T

transactionsDf.createOrReplaceTempView('transactions')

def pow_5(x):

if x:

return x**5

return x

spark.udf.register('power_5_udf', pow_5, T.LongType())

spark.sql('SELECT power_5_udf(value) AS result FROM transactions')

Here it is important to understand how the pow_5 method handles empty values. In the wrong code block above, the pow_5 method is unable to handle empty values and will throw an error, since

Python's ** operator cannot deal with any null value Spark passes into method pow_5.

The order of arguments for registering the UDF function with Spark via spark.udf.register matters. In the code snippet in the question, the arguments for the SQL method name and the actual Python

function are switched. You can read more about the arguments of spark.udf.register and see some examples of its usage in the documentation (link below).

Finally, you should recognize that in the original code block, an expression to rename column created through the UDF function is missing. The renaming is done by SQL's AS result argument.

Omitting that argument, you end up with the column name power_5_udf(value) and not result.

More info: pyspark.sql.functions.udf — PySpark 3.1.1 documentation

Explanation

Correct code block:

transactionsDf.filter(col('predError').isin([3, 6])).count()

The isin method is the correct one to use here – the in method does not exist for the Column object.

More info: pyspark.sql.Column.isin — PySpark 3.1.2 documentation