Spark Connect for Scala 3
API
Generated with
Spark Connect for Scala 3/org.apache.spark/org.apache.spark.sql/Column

Column

org.apache.spark.sql.Column
See theColumn companion object
class Column(val expr: Expression)

A column in a Dataset: a lazily-evaluated reference to a column or a computation over columns. Columns are immutable; operators and methods return new Columns.

Build columns with functions.col, functions.lit, by indexing a DataFrame (df("id")), or by combining other columns with operators.

 import org.apache.spark.sql.functions._
 (col("age") + 1).as("next_age")
 col("name").like("a%") && (col("age") >= 18)

Attributes

Companion
object
Graph
Supertypes
class Object
trait Matchable
class Any

Members list

Value members

Concrete methods

def %(other: Any): Column
def &&(other: Column): Column
def *(other: Any): Column
def +(other: Any): Column
def -(other: Any): Column
def /(other: Any): Column
def <(other: Any): Column
def <=(other: Any): Column
def <=>(other: Any): Column
def =!=(other: Any): Column
def ===(other: Any): Column
def >(other: Any): Column
def >=(other: Any): Column
def alias(alias: String): Column
def and(other: Column): Column
def apply(key: Any): Column

Extract an array element by index, a map value by key, or a struct field.

Extract an array element by index, a map value by key, or a struct field.

Attributes

def as(alias: String): Column
def as(aliases: Seq[String]): Column
def asc: Column
def between(lowerBound: Any, upperBound: Any): Column

True if lowerBound <= this <= upperBound.

True if lowerBound <= this <= upperBound.

Attributes

def bitwiseAND(other: Any): Column
def bitwiseOR(other: Any): Column
def bitwiseXOR(other: Any): Column
def cast(to: DataType): Column
def cast(to: String): Column
def contains(other: Any): Column
def desc: Column
def endsWith(other: Any): Column
def eqNullSafe(other: Any): Column
override def equals(obj: Any): Boolean

Compares the receiver object (this) with the argument object (that) for equivalence.

Compares the receiver object (this) with the argument object (that) for equivalence.

Any implementation of this method should be an equivalence relation:

  • It is reflexive: for any instance x of type Any, x.equals(x) should return true.
  • It is symmetric: for any instances x and y of type Any, x.equals(y) should return true if and only if y.equals(x) returns true.
  • It is transitive: for any instances x, y, and z of type Any if x.equals(y) returns true and y.equals(z) returns true, then x.equals(z) should return true.

If you override this method, you should verify that your implementation remains an equivalence relation. Additionally, when overriding this method it is usually necessary to override hashCode to ensure that objects which are "equal" (o1.equals(o2) returns true) hash to the same scala.Int. (o1.hashCode.equals(o2.hashCode)).

Value parameters

that

the object to compare against this object for equality.

Attributes

Returns

true if the receiver object is equivalent to the argument; false otherwise.

Definition Classes
Any
def getField(fieldName: String): Column
def getItem(key: Any): Column
override def hashCode(): Int

Calculates a hash code value for the object.

Calculates a hash code value for the object.

The default hashing algorithm is platform dependent.

Note that it is allowed for two objects to have identical hash codes (o1.hashCode.equals(o2.hashCode)) yet not be equal (o1.equals(o2) returns false). A degenerate implementation could always return 0. However, it is required that if two objects are equal (o1.equals(o2) returns true) that they have identical hash codes (o1.hashCode.equals(o2.hashCode)). Therefore, when overriding this method, be sure to verify that the behavior is consistent with the equals method.

Attributes

Returns

the hash code value for this object.

Definition Classes
Any
def ilike(literal: String): Column
def isNaN: Column
def isNull: Column
def isin(values: Any*): Column

True if the column's value is in the given list.

True if the column's value is in the given list.

Attributes

def like(literal: String): Column
def name(alias: String): Column
def notEqual(other: Any): Column
def or(other: Column): Column
def otherwise(value: Any): Column

Provides the default (ELSE) value for a CASE expression.

Provides the default (ELSE) value for a CASE expression.

Attributes

def over(window: WindowSpec): Column
def over(): Column
def rlike(literal: String): Column
def startsWith(other: Any): Column
def substr(startPos: Int, len: Int): Column
override def toString: String

Returns a string representation of the object.

Returns a string representation of the object.

The default representation is platform dependent.

Attributes

Returns

a string representation of the object.

Definition Classes
Any
def unary_-: Column
def when(condition: Column, value: Any): Column

Adds a branch to a CASE expression started by functions.when.

Adds a branch to a CASE expression started by functions.when.

Attributes

def ||(other: Column): Column
Generated with