Spark Connect for Scala 3
API
Generated with
Spark Connect for Scala 3/org.apache.spark/org.apache.spark.sql/DataFrameWriterV2

DataFrameWriterV2

org.apache.spark.sql.DataFrameWriterV2
final class DataFrameWriterV2

Interface used to write a Dataset to external storage using the v2 (catalog) API. Use Dataset.writeTo to access this.

 df.writeTo("catalog.db.table").using("parquet").create()
 df.writeTo("catalog.db.table").append()

Attributes

Graph
Supertypes
class Object
trait Matchable
class Any

Members list

Value members

Concrete methods

def append(): Unit

Append the contents of the DataFrame to the output table.

Append the contents of the DataFrame to the output table.

Attributes

def clusterBy(colName: String, colNames: String*): this.type

Clusters the output by the given columns.

Clusters the output by the given columns.

Attributes

def create(): Unit

Create a new table from the contents of the DataFrame.

Create a new table from the contents of the DataFrame.

Attributes

def createOrReplace(): Unit

Create a new table, or replace an existing one, with the contents of the DataFrame.

Create a new table, or replace an existing one, with the contents of the DataFrame.

Attributes

def option(key: String, value: String): this.type
def options(opts: Map[String, String]): this.type
def overwrite(condition: Column): Unit

Overwrite rows matching the given filter.

Overwrite rows matching the given filter.

Attributes

def overwritePartitions(): Unit

Overwrite all partitions touched by the contents of the DataFrame.

Overwrite all partitions touched by the contents of the DataFrame.

Attributes

def partitionedBy(column: Column, columns: Column*): this.type

Partitions the output by the given columns.

Partitions the output by the given columns.

Attributes

def replace(): Unit

Replace an existing table with the contents of the DataFrame.

Replace an existing table with the contents of the DataFrame.

Attributes

def tableProperty(property: String, value: String): this.type
def using(provider: String): this.type

Specifies a provider for the underlying output data source.

Specifies a provider for the underlying output data source.

Attributes

Generated with