A column in a Dataset: a lazily-evaluated reference to a column or a computation over columns. Columns are immutable; operators and methods return new Columns.
Build columns with functions.col, functions.lit, by indexing a DataFrame (df("id")), or by combining other columns with operators.
import org.apache.spark.sql.functions._
(col("age") + 1).as("next_age")
col("name").like("a%") && (col("age") >= 18)
The operators mirror org.apache.spark.sql.Column and build the same Spark Connect expressions as the Apache Spark Scala client.
Attributes
- Companion
- object
- Graph
-
- Supertypes
-
class Objecttrait Matchableclass Any
- Known subtypes
-
class ColumnName
Members list
Value members
Constructors
Wraps an existing protobuf expression.
Wraps an existing protobuf expression.
Attributes
A column reference by (possibly dotted) name; "*" expands to all columns.
A column reference by (possibly dotted) name; "*" expands to all columns.
Attributes
Concrete methods
Extract an array element by index, a map value by key, or a struct field.
Extract an array element by index, a map value by key, or a struct field.
Attributes
Gives this column an Encoder, producing a TypedColumn usable with the typed Dataset.select overloads.
Gives this column an Encoder, producing a TypedColumn usable with the typed Dataset.select overloads.
Attributes
True if lowerBound <= this <= upperBound.
True if lowerBound <= this <= upperBound.
Attributes
Drops fields of a struct column by name; a no-op for fields that do not exist.
Drops fields of a struct column by name; a no-op for fields that do not exist.
Attributes
Compares the receiver object (this) with the argument object (that) for equivalence.
Compares the receiver object (this) with the argument object (that) for equivalence.
Any implementation of this method should be an equivalence relation:
- It is reflexive: for any instance
xof typeAny,x.equals(x)should returntrue. - It is symmetric: for any instances
xandyof typeAny,x.equals(y)should returntrueif and only ify.equals(x)returnstrue. - It is transitive: for any instances
x,y, andzof typeAnyifx.equals(y)returnstrueandy.equals(z)returnstrue, thenx.equals(z)should returntrue.
If you override this method, you should verify that your implementation remains an equivalence relation. Additionally, when overriding this method it is usually necessary to override hashCode to ensure that objects which are "equal" (o1.equals(o2) returns true) hash to the same scala.Int. (o1.hashCode.equals(o2.hashCode)).
Value parameters
- that
-
the object to compare against this object for equality.
Attributes
- Returns
-
trueif the receiver object is equivalent to the argument;falseotherwise. - Definition Classes
-
Any
Prints the expression to the console: the SQL-like form, or the full protobuf if extended.
Prints the expression to the console: the SQL-like form, or the full protobuf if extended.
Attributes
Calculates a hash code value for the object.
Calculates a hash code value for the object.
The default hashing algorithm is platform dependent.
Note that it is allowed for two objects to have identical hash codes (o1.hashCode.equals(o2.hashCode)) yet not be equal (o1.equals(o2) returns false). A degenerate implementation could always return 0. However, it is required that if two objects are equal (o1.equals(o2) returns true) that they have identical hash codes (o1.hashCode.equals(o2.hashCode)). Therefore, when overriding this method, be sure to verify that the behavior is consistent with the equals method.
Attributes
- Returns
-
the hash code value for this object.
- Definition Classes
-
Any
True if the column's value is in the given collection.
True if the column's value is in the given collection.
Attributes
(Java-specific) True if the column's value is in the given collection.
(Java-specific) True if the column's value is in the given collection.
Attributes
True if the column's value is in the given list.
True if the column's value is in the given list.
Attributes
True if the column's value is in the result of the single-column (or, for a struct(...) column, multi-column) subquery ds. Builds an IN (subquery) expression, which needs a Spark 4.1+ server.
True if the column's value is in the result of the single-column (or, for a struct(...) column, multi-column) subquery ds. Builds an IN (subquery) expression, which needs a Spark 4.1+ server.
Attributes
Provides the default (ELSE) value for a CASE expression.
Provides the default (ELSE) value for a CASE expression.
Attributes
Marks this column as an outer reference for a correlated subquery or a lateral join's correlated right side. Spark Connect resolves outer references on the server, so (as in the Apache Spark Connect client) this does not change the expression sent to the server.
Marks this column as an outer reference for a correlated subquery or a lateral join's correlated right side. Spark Connect resolves outer references on the server, so (as in the Apache Spark Connect client) this does not change the expression sent to the server.
Attributes
A SQL-like rendering of this column, e.g. (a + 1) AS b.
A SQL-like rendering of this column, e.g. (a + 1) AS b.
Attributes
- Definition Classes
-
Any
Applies f to this column, for chaining custom transformations.
Applies f to this column, for chaining custom transformations.
Attributes
Casts to to, returning null instead of failing when the cast is not possible.
Casts to to, returning null instead of failing when the cast is not possible.
Attributes
Casts to the type given as a SQL string, returning null when not possible.
Casts to the type given as a SQL string, returning null when not possible.
Attributes
Adds a branch to a CASE expression started by functions.when.
Adds or replaces a field in a struct column by name.
Adds or replaces a field in a struct column by name.