Master Databricks and Apache Spark Step by Step: Lesson 24 - Creating PySpark Dataframe Scalar UDFs
In this video, you learn how to create PySpark dataframe User Defined Functions (UDF) to perform distributed transformations on each row. You will learn about using Apache Arrow to get optimal performance and how to use these functions from Spark SQL and dataframes.
Video demo notebook at:
https://github.com/bcafferky/shared/b...
For information on how to upload files to Databricks and create tables see:
• Video
Blog on creating Apache Spark Scalar UDFs
https://docs.databricks.com/spark/lat...
