functions
Built-in functions for working with Columns, mirroring org.apache.spark.sql.functions.
import org.apache.spark.sql.functions._
df.select(col("id"), upper(col("name")), (col("x") + 1).as("x1"))
df.groupBy("dept").agg(avg("salary"), count(lit(1)))
This object exposes a comprehensive subset of Spark's function library. Any Spark function not listed here can still be invoked by name via callUDF / expr.
Following Spark's convention, a String argument denotes a column name for most functions (e.g. sum("salary") aggregates the salary column), while functions whose parameters are genuinely literal (regex patterns, date formats, JSON paths, ...) treat their String arguments as literal values.
Attributes
- Graph
-
- Supertypes
-
class Objecttrait Matchableclass Any
- Self type
-
functions.type
Members list
Value members
Concrete methods
Applies a binary operator to an initial state and all array elements, then a finish step.
Applies a binary operator to an initial state and all array elements, then a finish step.
Attributes
Marks a DataFrame as small enough for a broadcast join.
Marks a DataFrame as small enough for a broadcast join.
Attributes
Calls a Spark function by name with the given columns as arguments.
Calls a Spark function by name with the given columns as arguments.
Attributes
True if the predicate holds for any element of the array.
True if the predicate holds for any element of the array.
Attributes
Parses the expression string into the column it represents.
Parses the expression string into the column it represents.
Attributes
Filters an array keeping elements for which the predicate holds.
Filters an array keeping elements for which the predicate holds.
Attributes
Filters an array using the (element, index) predicate.
Filters an array using the (element, index) predicate.
Attributes
True if the predicate holds for every element of the array.
True if the predicate holds for every element of the array.
Attributes
Filters a map keeping entries for which the (key, value) predicate holds.
Filters a map keeping entries for which the (key, value) predicate holds.
Attributes
Merges two maps by key using the (key, value1, value2) function.
Merges two maps by key using the (key, value1, value2) function.
Attributes
Transforms elements of an array using the given function.
Transforms elements of an array using the given function.
Attributes
Transforms elements of an array using the (element, index) function.
Transforms elements of an array using the (element, index) function.
Attributes
Applies a function to every (key, value) entry of a map and returns transformed keys.
Applies a function to every (key, value) entry of a map and returns transformed keys.
Attributes
Applies a function to every (key, value) entry of a map and returns transformed values.
Applies a function to every (key, value) entry of a map and returns transformed values.
Attributes
Evaluates a list of conditions and returns one of multiple possible result expressions.
Evaluates a list of conditions and returns one of multiple possible result expressions.