Question #1
Verified Q&A
An upstream system has been configured to pass the date for a given batch of data to the Databricks Jobs API as a parameter. The notebook to be scheduled will use this parameter to load data with the following code:<br />
df = spark.read.format("parquet").load(f"/mnt/source/(date)")<br />
<br />
Which code block should be used to create the date Python variable used in the above code block?
df = spark.read.format("parquet").load(f"/mnt/source/(date)")<br />
<br />
Which code block should be used to create the date Python variable used in the above code block?
A
dbutils.widgets.text("date", "null")date = dbutils.widgets.get("date")
B
input_dict = input()date= input_dict["date"]
C
date = spark.conf.get("date")
D
import sysdate = sys.argv[1]
E
date = dbutils.notebooks.getParam("date")
Correct Answer:
dbutils.widgets.text("date", "null")date = dbutils.widgets.get("date")
Explanation:
The code block that should be used to create the date Python variable used in the above code block is: dbutils.widgets.text(“date”, “null”) date = dbutils.widgets.get(“date”) This code block uses the dbutils.widgets API to create and get a text widget named “date” that can accept a string value as a parameter1. The default value of the widget is “null”, which means that if no parameter is passed, the date variable will be “null”. However, if a parameter is passed through the Databricks Jobs API, the date variable will be assigned the value of the parameter. For example, if the parameter is “2021-11-01”, the date variable will be “2021-11-01”. This way, the notebook can use the date variable to load data from the specified path.<br />
<br />
The other options are not correct, because:<br />
<br />
Option A is incorrect because spark.conf.get(“date”) is not a valid way to get a parameter passed through the Databricks Jobs API. The spark.conf API is used to get or set Spark configuration properties, not notebook parameters2.<br />
<br />
Option B is incorrect because input() is not a valid way to get a parameter passed through the Databricks Jobs API. The input() function is used to get user input from the standard input stream, not from the API request3.<br />
<br />
Option C is incorrect because sys.argv1 is not a valid way to get a parameter passed through the Databricks Jobs API. The sys.argv list is used to get the command-line arguments passed to a Python script, not to a notebook4.<br />
<br />
Option D is incorrect because dbutils.notebooks.getParam(“date”) is not a valid way to get a parameter passed through the Databricks Jobs API. The dbutils.notebooks API is used to get or set notebook parameters when running a notebook as a job or as a subnotebook, not when passing parameters through the API5.<br />
<br />
Reference: Widgets, Spark Configuration, input(), sys.argv, Notebooks
<br />
The other options are not correct, because:<br />
<br />
Option A is incorrect because spark.conf.get(“date”) is not a valid way to get a parameter passed through the Databricks Jobs API. The spark.conf API is used to get or set Spark configuration properties, not notebook parameters2.<br />
<br />
Option B is incorrect because input() is not a valid way to get a parameter passed through the Databricks Jobs API. The input() function is used to get user input from the standard input stream, not from the API request3.<br />
<br />
Option C is incorrect because sys.argv1 is not a valid way to get a parameter passed through the Databricks Jobs API. The sys.argv list is used to get the command-line arguments passed to a Python script, not to a notebook4.<br />
<br />
Option D is incorrect because dbutils.notebooks.getParam(“date”) is not a valid way to get a parameter passed through the Databricks Jobs API. The dbutils.notebooks API is used to get or set notebook parameters when running a notebook as a job or as a subnotebook, not when passing parameters through the API5.<br />
<br />
Reference: Widgets, Spark Configuration, input(), sys.argv, Notebooks
