Spark Connect for Scala 3
API
Generated with
Spark Connect for Scala 3/org.apache.spark/org.apache.spark.sql/DataFrameWriter

DataFrameWriter

org.apache.spark.sql.DataFrameWriter
class DataFrameWriter

Saves the contents of a DataFrame to external storage systems (e.g. file systems, key-value stores, tables). Use Dataset.write to access this.

Mirrors the public surface of org.apache.spark.sql.DataFrameWriter over the Spark Connect protocol.

Attributes

Example 
 df.write.format("parquet").mode("overwrite").save("out.parquet")
 df.write.mode("append").saveAsTable("my_table")
Graph
Supertypes
class Object
trait Matchable
class Any

Members list

Value members

Concrete methods

def bucketBy(numBuckets: Int, colName: String, colNames: String*): DataFrameWriter

Buckets the output by the given columns into numBuckets buckets. If specified, the output is laid out on the file system similar to Hive's bucketing scheme.

Buckets the output by the given columns into numBuckets buckets. If specified, the output is laid out on the file system similar to Hive's bucketing scheme.

Attributes

Returns

this writer, for chaining.

def clusterBy(colName: String, colNames: String*): DataFrameWriter

Clusters the output by the given columns, for table formats that support clustering (e.g. liquid clustering). Cannot be combined with partitionBy or bucketBy.

Clusters the output by the given columns, for table formats that support clustering (e.g. liquid clustering). Cannot be combined with partitionBy or bucketBy.

Attributes

Returns

this writer, for chaining.

def csv(path: String): Unit

Saves the content as CSV at the given path.

Saves the content as CSV at the given path.

Attributes

def format(source: String): DataFrameWriter

Specifies the output data source format (e.g. "csv", "json", "parquet", "orc").

Specifies the output data source format (e.g. "csv", "json", "parquet", "orc").

Attributes

Returns

this writer, for chaining.

def insertInto(tableName: String): Unit

Inserts the DataFrame into the given existing table (by position).

Inserts the DataFrame into the given existing table (by position).

Attributes

def jdbc(url: String, table: String, connectionProperties: Properties): Unit

Saves the content to the external database table table over JDBC, using url and the connection properties. Partitioning, bucketing, and clustering are not supported here.

Saves the content to the external database table table over JDBC, using url and the connection properties. Partitioning, bucketing, and clustering are not supported here.

Attributes

def json(path: String): Unit

Saves the content as JSON at the given path.

Saves the content as JSON at the given path.

Attributes

def mode(saveMode: SaveMode): DataFrameWriter

Specifies the behavior when data or table already exists, using a SaveMode.

Specifies the behavior when data or table already exists, using a SaveMode.

Attributes

def mode(saveMode: String): DataFrameWriter
def option(key: String, value: String): DataFrameWriter

Adds an output option for the underlying data source.

Adds an output option for the underlying data source.

Attributes

Returns

this writer, for chaining.

def option(key: String, value: Boolean): DataFrameWriter

Adds a boolean output option. @return this writer, for chaining.

Adds a boolean output option. @return this writer, for chaining.

Attributes

def option(key: String, value: Long): DataFrameWriter

Adds a long output option. @return this writer, for chaining.

Adds a long output option. @return this writer, for chaining.

Attributes

def option(key: String, value: Double): DataFrameWriter

Adds a double output option. @return this writer, for chaining.

Adds a double output option. @return this writer, for chaining.

Attributes

def options(options: Map[String, String]): DataFrameWriter

Adds multiple output options.

Adds multiple output options.

Attributes

Returns

this writer, for chaining.

def options(options: Map[String, String]): DataFrameWriter

(Java-specific) Adds multiple output options. @return this writer, for chaining.

(Java-specific) Adds multiple output options. @return this writer, for chaining.

Attributes

def orc(path: String): Unit

Saves the content as ORC at the given path.

Saves the content as ORC at the given path.

Attributes

def parquet(path: String): Unit

Saves the content as Parquet at the given path.

Saves the content as Parquet at the given path.

Attributes

def partitionBy(colNames: String*): DataFrameWriter

Partitions the output by the given columns on the file system.

Partitions the output by the given columns on the file system.

Attributes

Returns

this writer, for chaining.

def save(): Unit

Saves the DataFrame as output, for data sources that do not require a path (e.g. external key-value stores).

Saves the DataFrame as output, for data sources that do not require a path (e.g. external key-value stores).

Attributes

def save(path: String): Unit

Saves the DataFrame at the given path.

Saves the DataFrame at the given path.

Attributes

def saveAsTable(tableName: String): Unit

Saves the DataFrame as the given managed/registered table.

Saves the DataFrame as the given managed/registered table.

Attributes

def sortBy(colName: String, colNames: String*): DataFrameWriter

Sorts the output in each bucket by the given columns.

Sorts the output in each bucket by the given columns.

Attributes

Returns

this writer, for chaining.

def text(path: String): Unit

Saves the content as text at the given path.

Saves the content as text at the given path.

Attributes

def withSchemaEvolution(): DataFrameWriter

Enables automatic schema evolution for this write (Spark 4.2+). The target table must declare the AUTOMATIC_SCHEMA_EVOLUTION capability.

Enables automatic schema evolution for this write (Spark 4.2+). The target table must declare the AUTOMATIC_SCHEMA_EVOLUTION capability.

Attributes

Returns

this writer, for chaining.

def xml(path: String): Unit

Saves the content as XML at the given path.

Saves the content as XML at the given path.

Attributes

Generated with