Spark Connect for Scala 3
API
Generated with
Spark Connect for Scala 3/org.apache.spark/org.apache.spark.sql/Dataset

Dataset

org.apache.spark.sql.Dataset
class Dataset[T]

A distributed collection of rows. A Dataset is lazy: transformations build up a protobuf logical plan, and nothing executes until an action (e.g. collect, show, count) is called.

Mirrors the public surface of org.apache.spark.sql.Dataset over the Spark Connect protocol: operations that keep the row type (filtering, sorting, set operations, repartitioning, ...) return a Dataset[T], while operations that change the schema return a DataFrame.

Attributes

Graph
Supertypes
class Object
trait Matchable
class Any

Members list

Value members

Concrete methods

def agg(expr: Column, exprs: Column*): DataFrame
def agg(aggExpr: (String, String), aggExprs: (String, String)*): DataFrame
def agg(exprs: Map[String, String]): DataFrame
def agg(exprs: Map[String, String]): DataFrame
def alias(alias: String): Dataset[T]
def alias(alias: Symbol): Dataset[T]
def apply(colName: String): Column

Selects a column by name. Alias for col.

Selects a column by name. Alias for col.

Attributes

def as[U](using enc: Encoder[U]): Dataset[U]

Returns a new Dataset where each record is mapped to type U via its Encoder. This is a purely client-side reinterpretation (no server-side closure), so it works over Spark Connect.

Returns a new Dataset where each record is mapped to type U via its Encoder. This is a purely client-side reinterpretation (no server-side closure), so it works over Spark Connect.

Attributes

def as(alias: String): Dataset[T]
def as(alias: Symbol): Dataset[T]
def cache(): this.type

Persists this Dataset with the default storage level.

Persists this Dataset with the default storage level.

Attributes

def checkpoint(): Dataset[T]

Eagerly checkpoints this Dataset to reliable storage and returns the checkpointed copy.

Eagerly checkpoints this Dataset to reliable storage and returns the checkpointed copy.

Attributes

def checkpoint(eager: Boolean): Dataset[T]

Checkpoints this Dataset to reliable storage.

Checkpoints this Dataset to reliable storage.

Attributes

def coalesce(numPartitions: Int): Dataset[T]
def col(colName: String): Column

Selects a column by name, qualified by this Dataset's plan id so that it resolves unambiguously even in self-joins.

Selects a column by name, qualified by this Dataset's plan id so that it resolves unambiguously even in self-joins.

Attributes

def colRegex(colName: String): Column

Selects columns based on a column name regular expression.

Selects columns based on a column name regular expression.

Attributes

def collect(): Array[T]
def collectAsList(): List[T]
def columns: Array[String]
def count(): Long
def createGlobalTempView(viewName: String): Unit
def createOrReplaceGlobalTempView(viewName: String): Unit
def createOrReplaceTempView(viewName: String): Unit
def createTempView(viewName: String): Unit
def crossJoin(right: Dataset[_]): DataFrame
def cube(col1: String, cols: String*): RelationalGroupedDataset
def describe(cols: String*): DataFrame

Computes basic statistics (count, mean, stddev, min, max) for numeric and string columns.

Computes basic statistics (count, mean, stddev, min, max) for numeric and string columns.

Attributes

def distinct(): Dataset[T]
def drop(colName: String): DataFrame
def drop(colNames: String*): DataFrame
def drop(col: Column): DataFrame
def drop(col: Column, cols: Column*): DataFrame
def dropDuplicates(colNames: Seq[String]): Dataset[T]
def dropDuplicates(colNames: Array[String]): Dataset[T]
def dropDuplicates(col1: String, cols: String*): Dataset[T]
def dropDuplicatesWithinWatermark(): Dataset[T]

Drops duplicates within the event-time watermark, keeping state bounded for streaming.

Drops duplicates within the event-time watermark, keeping state bounded for streaming.

Attributes

def dropDuplicatesWithinWatermark(colNames: Seq[String]): Dataset[T]
def dropDuplicatesWithinWatermark(colNames: Array[String]): Dataset[T]
def dropDuplicatesWithinWatermark(col1: String, cols: String*): Dataset[T]
def dtypes: Array[(String, String)]
def except(other: Dataset[T]): Dataset[T]
def exceptAll(other: Dataset[T]): Dataset[T]
def exists(): Column

Returns a Column that is true if this Dataset has any rows (an EXISTS subquery).

Returns a Column that is true if this Dataset has any rows (an EXISTS subquery).

Attributes

def explain(): Unit
def explain(extended: Boolean): Unit
def explain(mode: String): Unit
def filter(condition: Column): Dataset[T]
def filter(conditionExpr: String): Dataset[T]
def first(): T
def groupBy(col1: String, cols: String*): RelationalGroupedDataset
def groupingSets(groupingSets: Seq[Seq[Column]], cols: Column*): RelationalGroupedDataset

Creates a multi-dimensional aggregation over the given grouping sets, e.g. ds.groupingSets(Seq(Seq($"dept", $"gender"), Seq()), $"dept", $"gender").agg(...).

Creates a multi-dimensional aggregation over the given grouping sets, e.g. ds.groupingSets(Seq(Seq($"dept", $"gender"), Seq()), $"dept", $"gender").agg(...).

Attributes

def head(n: Int): Array[T]
def head(): T
def hint(name: String, parameters: Any*): Dataset[T]
def inputFiles: Array[String]
def intersect(other: Dataset[T]): Dataset[T]
def intersectAll(other: Dataset[T]): Dataset[T]
def isEmpty: Boolean
def isLocal: Boolean
def isStreaming: Boolean
def join(right: Dataset[_]): DataFrame
def join(right: Dataset[_], usingColumn: String): DataFrame
def join(right: Dataset[_], usingColumns: Array[String]): DataFrame
def join(right: Dataset[_], usingColumns: Seq[String]): DataFrame
def join(right: Dataset[_], usingColumn: String, joinType: String): DataFrame
def join(right: Dataset[_], usingColumns: Array[String], joinType: String): DataFrame
def join(right: Dataset[_], usingColumns: Seq[String], joinType: String): DataFrame
def join(right: Dataset[_], joinExprs: Column): DataFrame
def join(right: Dataset[_], joinExprs: Column, joinType: String): DataFrame
def joinWith[U](other: Dataset[U], condition: Column): Dataset[(T, U)]

Joins with other (an inner join), returning each matching pair of records as a tuple. Like Apache Spark, a struct-typed (case class) side becomes a nested struct column.

Joins with other (an inner join), returning each matching pair of records as a tuple. Like Apache Spark, a struct-typed (case class) side becomes a nested struct column.

Attributes

def joinWith[U](other: Dataset[U], condition: Column, joinType: String): Dataset[(T, U)]

Joins with other using the given join type (inner, cross, outer/full, left or right variants), returning each matching pair of records as a tuple.

Joins with other using the given join type (inner, cross, outer/full, left or right variants), returning each matching pair of records as a tuple.

Attributes

def lateralJoin(right: Dataset[_]): DataFrame

Lateral join with a correlated right relation.

Lateral join with a correlated right relation.

Attributes

def lateralJoin(right: Dataset[_], joinExprs: Column): DataFrame
def lateralJoin(right: Dataset[_], joinType: String): DataFrame
def lateralJoin(right: Dataset[_], joinExprs: Column, joinType: String): DataFrame
def limit(n: Int): Dataset[T]
def localCheckpoint(): Dataset[T]

Eagerly locally checkpoints this Dataset.

Eagerly locally checkpoints this Dataset.

Attributes

def localCheckpoint(eager: Boolean): Dataset[T]

Locally checkpoints this Dataset.

Locally checkpoints this Dataset.

Attributes

def localCheckpoint(eager: Boolean, storageLevel: StorageLevel): Dataset[T]

Locally checkpoints this Dataset, caching it at the given storage level.

Locally checkpoints this Dataset, caching it at the given storage level.

Attributes

def melt(ids: Array[Column], values: Array[Column], variableColumnName: String, valueColumnName: String): DataFrame

Alias for unpivot.

Alias for unpivot.

Attributes

def melt(ids: Array[Column], variableColumnName: String, valueColumnName: String): DataFrame

Alias for unpivot.

Alias for unpivot.

Attributes

def mergeInto(table: String, condition: Column): MergeIntoWriter[T]

Merges this Dataset (the source) into the table (the target) using condition to match rows. Returns a MergeIntoWriter to configure the WHEN clauses; call merge() to run it.

Merges this Dataset (the source) into the table (the target) using condition to match rows. Returns a MergeIntoWriter to configure the WHEN clauses; call merge() to run it.

Attributes

def metadataColumn(colName: String): Column

Selects a metadata column (such as _metadata of file sources) by name, qualified by this Dataset's plan id.

Selects a metadata column (such as _metadata of file sources) by name, qualified by this Dataset's plan id.

Attributes

def na: DataFrameNaFunctions

Returns a DataFrameNaFunctions for working with missing data.

Returns a DataFrameNaFunctions for working with missing data.

Attributes

def nearestByJoin(right: Dataset[_], rankingExpression: Column, numResults: Int, mode: String, direction: String): DataFrame

Nearest-by top-K ranking join with the default inner join type: for each row of this Dataset, returns up to numResults rows of right ranked by rankingExpression. mode is approx or exact; direction is distance (smallest first) or similarity (largest first). Requires Spark 4.2+.

Nearest-by top-K ranking join with the default inner join type: for each row of this Dataset, returns up to numResults rows of right ranked by rankingExpression. mode is approx or exact; direction is distance (smallest first) or similarity (largest first). Requires Spark 4.2+.

Attributes

def nearestByJoin(right: Dataset[_], rankingExpression: Column, numResults: Int, mode: String, direction: String, joinType: String): DataFrame

Nearest-by top-K ranking join; joinType is inner or leftouter. The arguments are validated client-side with the same NEAREST_BY_JOIN.* errors as the server. Requires Spark 4.2+.

Nearest-by top-K ranking join; joinType is inner or leftouter. The arguments are validated client-side with the same NEAREST_BY_JOIN.* errors as the server. Requires Spark 4.2+.

Attributes

def observe(name: String, expr: Column, exprs: Column*): Dataset[T]

Defines named observed metrics computed while this Dataset is processed. The metrics are reported with the query result; use the Observation overload to read them back.

Defines named observed metrics computed while this Dataset is processed. The metrics are reported with the query result; use the Observation overload to read them back.

Attributes

def observe(observation: Observation, expr: Column, exprs: Column*): Dataset[T]

Defines observed metrics whose values can be read back through observation.get.

Defines observed metrics whose values can be read back through observation.get.

Attributes

def offset(n: Int): Dataset[T]
def orderBy(sortCol: String, sortCols: String*): Dataset[T]
def orderBy(sortExprs: Column*): Dataset[T]
def persist(): this.type

Persists this Dataset with the default storage level (MEMORY_AND_DISK).

Persists this Dataset with the default storage level (MEMORY_AND_DISK).

Attributes

def persist(newLevel: StorageLevel): this.type

Persists this Dataset with the given storage level.

Persists this Dataset with the given storage level.

Attributes

def plan: Plan

The full protobuf proto.Plan (a root plan wrapping relation) that would be sent to the server to execute this Dataset. Exposed for plugin authors and tooling; ordinary code never needs it.

The full protobuf proto.Plan (a root plan wrapping relation) that would be sent to the server to execute this Dataset. Exposed for plugin authors and tooling; ordinary code never needs it.

Attributes

def printSchema(): Unit
def printSchema(level: Int): Unit
def randomSplit(weights: Array[Double], seed: Long): Array[Dataset[T]]

Randomly splits this Dataset with the given weights and a fixed seed.

Randomly splits this Dataset with the given weights and a fixed seed.

Attributes

def randomSplit(weights: Array[Double]): Array[Dataset[T]]

Randomly splits this Dataset with the given weights.

Randomly splits this Dataset with the given weights.

Attributes

def randomSplitAsList(weights: Array[Double], seed: Long): List[Dataset[T]]

(Java-specific) Randomly splits this Dataset with the given weights and seed.

(Java-specific) Randomly splits this Dataset with the given weights and seed.

Attributes

def repartition(numPartitions: Int): Dataset[T]
def repartition(numPartitions: Int, partitionExprs: Column*): Dataset[T]
def repartition(partitionExprs: Column*): Dataset[T]
def repartitionById(numPartitions: Int, partitionIdExpr: Column): Dataset[T]

Repartitions into numPartitions partitions, placing each row in the partition whose id is given by partitionIdExpr (evaluated per row), without hashing.

Repartitions into numPartitions partitions, placing each row in the partition whose id is given by partitionIdExpr (evaluated per row), without hashing.

Attributes

def repartitionByRange(numPartitions: Int, partitionExprs: Column*): Dataset[T]

Range-partitions by the given expressions into numPartitions.

Range-partitions by the given expressions into numPartitions.

Attributes

def repartitionByRange(partitionExprs: Column*): Dataset[T]

Range-partitions by the given expressions.

Range-partitions by the given expressions.

Attributes

def rollup(col1: String, cols: String*): RelationalGroupedDataset
def sameSemantics(other: Dataset[T]): Boolean
def sample(fraction: Double): Dataset[T]
def sample(fraction: Double, seed: Long): Dataset[T]
def sample(withReplacement: Boolean, fraction: Double): Dataset[T]
def sample(withReplacement: Boolean, fraction: Double, seed: Long): Dataset[T]
def scalar(): Column

Returns a Column for this single-column, single-row Dataset as a scalar subquery, e.g. df.where(col("v") > spark.table("t").select(max("v")).scalar()).

Returns a Column for this single-column, single-row Dataset as a scalar subquery, e.g. df.where(col("v") > spark.table("t").select(max("v")).scalar()).

Attributes

def schema: StructType

The schema of this Dataset.

The schema of this Dataset.

Attributes

def select(cols: Column*): DataFrame
def select(col: String, cols: String*): DataFrame
def select[U1](c1: TypedColumn[T, U1]): Dataset[U1]

Selects one typed column, returning a Dataset of its values.

Selects one typed column, returning a Dataset of its values.

Attributes

def select[U1, U2](c1: TypedColumn[T, U1], c2: TypedColumn[T, U2]): Dataset[(U1, U2)]

Selects two typed columns, returning a Dataset of tuples.

Selects two typed columns, returning a Dataset of tuples.

Attributes

def select[U1, U2, U3](c1: TypedColumn[T, U1], c2: TypedColumn[T, U2], c3: TypedColumn[T, U3]): Dataset[(U1, U2, U3)]

Selects three typed columns, returning a Dataset of tuples.

Selects three typed columns, returning a Dataset of tuples.

Attributes

def select[U1, U2, U3, U4](c1: TypedColumn[T, U1], c2: TypedColumn[T, U2], c3: TypedColumn[T, U3], c4: TypedColumn[T, U4]): Dataset[(U1, U2, U3, U4)]

Selects four typed columns, returning a Dataset of tuples.

Selects four typed columns, returning a Dataset of tuples.

Attributes

def select[U1, U2, U3, U4, U5](c1: TypedColumn[T, U1], c2: TypedColumn[T, U2], c3: TypedColumn[T, U3], c4: TypedColumn[T, U4], c5: TypedColumn[T, U5]): Dataset[(U1, U2, U3, U4, U5)]

Selects five typed columns, returning a Dataset of tuples.

Selects five typed columns, returning a Dataset of tuples.

Attributes

def selectExpr(exprs: String*): DataFrame
def semanticHash(): Int
def show(): Unit
def show(numRows: Int): Unit
def show(truncate: Boolean): Unit
def show(numRows: Int, truncate: Boolean): Unit
def show(numRows: Int, truncate: Int): Unit
def show(numRows: Int, truncate: Int, vertical: Boolean): Unit
def sort(sortCol: String, sortCols: String*): Dataset[T]
def sort(sortExprs: Column*): Dataset[T]
def sortWithinPartitions(sortExprs: Column*): Dataset[T]
def sortWithinPartitions(sortCol: String, sortCols: String*): Dataset[T]
def stat: DataFrameStatFunctions

Returns a DataFrameStatFunctions for statistic functions.

Returns a DataFrameStatFunctions for statistic functions.

Attributes

def storageLevel: StorageLevel

Returns the current storage level of this Dataset.

Returns the current storage level of this Dataset.

Attributes

def summary(statistics: String*): DataFrame

Computes the requested summary statistics; defaults match Spark's summary().

Computes the requested summary statistics; defaults match Spark's summary().

Attributes

def tail(n: Int): Array[T]

Returns the last n rows. Like collect(), this moves the data into the client.

Returns the last n rows. Like collect(), this moves the data into the client.

Attributes

def take(n: Int): Array[T]
def takeAsList(n: Int): List[T]
def to(schema: StructType): DataFrame

Returns a new Dataset where each row is reconciled to match the specified schema (by column name, reordering and casting as needed).

Returns a new Dataset where each row is reconciled to match the specified schema (by column name, reordering and casting as needed).

Attributes

def toDF(): DataFrame
def toDF(colNames: String*): DataFrame
def toJSON: Dataset[String]

Returns the content as a Dataset[String] of JSON documents in a value column.

Returns the content as a Dataset[String] of JSON documents in a value column.

Attributes

def toLocalIterator(): Iterator[T]
def transform[U](t: (Dataset[T]) => Dataset[U]): Dataset[U]

Concisely applies a transformation to this Dataset.

Concisely applies a transformation to this Dataset.

Attributes

def transpose(): DataFrame

Transposes the DataFrame, turning the first column into the new column names.

Transposes the DataFrame, turning the first column into the new column names.

Attributes

def transpose(indexColumn: Column): DataFrame

Transposes the DataFrame using indexColumn for the new column names.

Transposes the DataFrame using indexColumn for the new column names.

Attributes

def union(other: Dataset[T]): Dataset[T]
def unionAll(other: Dataset[T]): Dataset[T]
def unionByName(other: Dataset[T]): Dataset[T]
def unionByName(other: Dataset[T], allowMissingColumns: Boolean): Dataset[T]
def unpersist(blocking: Boolean): this.type

Marks this Dataset as non-persistent.

Marks this Dataset as non-persistent.

Attributes

def unpersist(): this.type

Marks this Dataset as non-persistent.

Marks this Dataset as non-persistent.

Attributes

def unpivot(ids: Array[Column], values: Array[Column], variableColumnName: String, valueColumnName: String): DataFrame

Unpivots (melts) a DataFrame from wide to long format.

Unpivots (melts) a DataFrame from wide to long format.

Attributes

def unpivot(ids: Array[Column], variableColumnName: String, valueColumnName: String): DataFrame

Unpivots, inferring the value columns from those not in ids.

Unpivots, inferring the value columns from those not in ids.

Attributes

def where(condition: Column): Dataset[T]
def where(conditionExpr: String): Dataset[T]
def withColumn(colName: String, col: Column): DataFrame
def withColumnRenamed(existingName: String, newName: String): DataFrame
def withColumns(colsMap: Map[String, Column]): DataFrame

Adds or replaces several columns at once.

Adds or replaces several columns at once.

Attributes

def withColumns(colsMap: Map[String, Column]): DataFrame

(Java-specific) Adds or replaces several columns at once.

(Java-specific) Adds or replaces several columns at once.

Attributes

def withColumnsRenamed(colsMap: Map[String, String]): DataFrame
def withColumnsRenamed(colsMap: Map[String, String]): DataFrame

(Java-specific) Renames several columns at once.

(Java-specific) Renames several columns at once.

Attributes

def withMetadata(columnName: String, metadata: Metadata): DataFrame

Returns a new Dataset where the metadata of column columnName is replaced by metadata.

Returns a new Dataset where the metadata of column columnName is replaced by metadata.

Attributes

def withWatermark(eventTime: String, delayThreshold: String): Dataset[T]

Defines an event-time watermark for this streaming Dataset.

Defines an event-time watermark for this streaming Dataset.

Attributes

def write: DataFrameWriter

Interface for saving the content of this Dataset to external storage.

Interface for saving the content of this Dataset to external storage.

Attributes

def writeStream: DataStreamWriter

Interface for saving the content of a streaming Dataset to external storage.

Interface for saving the content of a streaming Dataset to external storage.

Attributes

def writeTo(table: String): DataFrameWriterV2

Creates a v2 (catalog) write configuration builder.

Creates a v2 (catalog) write configuration builder.

Attributes

def zip(other: Dataset[_]): DataFrame

Combines the columns of this DataFrame with other side by side, preserving row alignment. Both must derive from the same source through projection-only operations (select, withColumn, ...); otherwise the server raises an AnalysisException. Requires Spark 4.3+.

Combines the columns of this DataFrame with other side by side, preserving row alignment. Both must derive from the same source through projection-only operations (select, withColumn, ...); otherwise the server raises an AnalysisException. Requires Spark 4.3+.

Attributes

def zipWithIndex(): DataFrame

Appends a column of consecutive 0-based Long indices named index, similar to RDD.zipWithIndex(). The index column is the last column of the result.

Appends a column of consecutive 0-based Long indices named index, similar to RDD.zipWithIndex(). The index column is the last column of the result.

Attributes

def zipWithIndex(indexColName: String): DataFrame

Appends a column of consecutive 0-based Long indices named indexColName, similar to RDD.zipWithIndex(). If a column with that name already exists, the result has duplicate column names.

Appends a column of consecutive 0-based Long indices named indexColName, similar to RDD.zipWithIndex(). If a column with that name already exists, the result has duplicate column names.

Attributes

Deprecated methods

def registerTempTable(tableName: String): Unit

Registers this Dataset as a temporary view. Use createOrReplaceTempView instead.

Registers this Dataset as a temporary view. Use createOrReplaceTempView instead.

Attributes

Deprecated
true

Concrete fields

val relation: Relation
Generated with