Spark Connect for Scala 3
API
Generated with
Spark Connect for Scala 3/org.apache.spark/org.apache.spark.sql/DataFrameReader

DataFrameReader

org.apache.spark.sql.DataFrameReader
class DataFrameReader

Loads data from external storage systems (e.g. file systems, key-value stores, JDBC) into a DataFrame. Use SparkSession.read to access this.

Mirrors the public surface of org.apache.spark.sql.DataFrameReader over the Spark Connect protocol.

Attributes

Example 
 spark.read.format("csv").option("header", true).load("data.csv")
 spark.read.json("events.json")
 spark.read.table("my_table")
Graph
Supertypes
class Object
trait Matchable
class Any

Members list

Value members

Concrete methods

def changes(tableName: String): DataFrame

Returns the row-level changes (Change Data Capture) of the table as a DataFrame (Spark 4.2+). Only Data Source V2 tables whose catalog supports change logs implement this. Use options such as startingVersion, endingVersion, startingTimestamp, endingTimestamp, deduplicationMode, and computeUpdates to choose the range and post-processing:

Returns the row-level changes (Change Data Capture) of the table as a DataFrame (Spark 4.2+). Only Data Source V2 tables whose catalog supports change logs implement this. Use options such as startingVersion, endingVersion, startingTimestamp, endingTimestamp, deduplicationMode, and computeUpdates to choose the range and post-processing:

 spark.read.option("startingVersion", "10").changes("my_table")

Attributes

def csv(path: String): DataFrame

Loads CSV file(s) and returns the result as a DataFrame.

Loads CSV file(s) and returns the result as a DataFrame.

Attributes

def csv(paths: String*): DataFrame

Loads CSV file(s) and returns the result as a DataFrame.

Loads CSV file(s) and returns the result as a DataFrame.

Attributes

def csv(csvDataset: Dataset[String]): DataFrame

Parses each row of a Dataset[String] as a CSV record, returning a DataFrame.

Parses each row of a Dataset[String] as a CSV record, returning a DataFrame.

Attributes

def format(source: String): DataFrameReader

Specifies the input data source format (e.g. "csv", "json", "parquet", "orc").

Specifies the input data source format (e.g. "csv", "json", "parquet", "orc").

Attributes

Returns

this reader, for chaining.

def jdbc(url: String, table: String, properties: Map[String, String]): DataFrame

Constructs a DataFrame representing the database table accessible via JDBC.

Constructs a DataFrame representing the database table accessible via JDBC.

Value parameters

properties

connection properties (e.g. "user", "password"); these are merged into the read options.

table

the name of the table in the external database (or a subquery).

url

the JDBC URL of the form jdbc:subprotocol:subname.

Attributes

def jdbc(url: String, table: String, properties: Properties): DataFrame

Constructs a DataFrame over a JDBC table, with connection properties.

Constructs a DataFrame over a JDBC table, with connection properties.

Attributes

def jdbc(url: String, table: String, columnName: String, lowerBound: Long, upperBound: Long, numPartitions: Int, connectionProperties: Properties): DataFrame

Constructs a DataFrame over a JDBC table, partitioned on columnName into numPartitions ranges between lowerBound and upperBound.

Constructs a DataFrame over a JDBC table, partitioned on columnName into numPartitions ranges between lowerBound and upperBound.

Attributes

def jdbc(url: String, table: String, predicates: Array[String], connectionProperties: Properties): DataFrame

Constructs a DataFrame over a JDBC table with one partition per WHERE-clause predicate in predicates.

Constructs a DataFrame over a JDBC table with one partition per WHERE-clause predicate in predicates.

Attributes

def json(path: String): DataFrame

Loads JSON file(s) and returns the result as a DataFrame.

Loads JSON file(s) and returns the result as a DataFrame.

Attributes

def json(paths: String*): DataFrame

Loads JSON file(s) and returns the result as a DataFrame.

Loads JSON file(s) and returns the result as a DataFrame.

Attributes

def json(jsonDataset: Dataset[String]): DataFrame

Parses each row of a Dataset[String] as a JSON object, returning a DataFrame.

Parses each row of a Dataset[String] as a JSON object, returning a DataFrame.

Attributes

def load(): DataFrame

Loads input as a DataFrame, for data sources that do not require a path (e.g. external key-value stores).

Loads input as a DataFrame, for data sources that do not require a path (e.g. external key-value stores).

Attributes

def load(path: String): DataFrame

Loads input from the given path as a DataFrame.

Loads input from the given path as a DataFrame.

Attributes

def load(paths: String*): DataFrame

Loads input from the given paths as a DataFrame, for data sources that support reading multiple paths.

Loads input from the given paths as a DataFrame, for data sources that support reading multiple paths.

Attributes

def option(key: String, value: String): DataFrameReader

Adds an input option for the underlying data source.

Adds an input option for the underlying data source.

Attributes

Returns

this reader, for chaining.

def option(key: String, value: Boolean): DataFrameReader

Adds a boolean input option. @return this reader, for chaining.

Adds a boolean input option. @return this reader, for chaining.

Attributes

def option(key: String, value: Long): DataFrameReader

Adds a long input option. @return this reader, for chaining.

Adds a long input option. @return this reader, for chaining.

Attributes

def option(key: String, value: Double): DataFrameReader

Adds a double input option. @return this reader, for chaining.

Adds a double input option. @return this reader, for chaining.

Attributes

def options(options: Map[String, String]): DataFrameReader

Adds multiple input options.

Adds multiple input options.

Attributes

Returns

this reader, for chaining.

def options(options: Map[String, String]): DataFrameReader

(Java-specific) Adds multiple input options. @return this reader, for chaining.

(Java-specific) Adds multiple input options. @return this reader, for chaining.

Attributes

def orc(path: String): DataFrame

Loads ORC file(s) and returns the result as a DataFrame.

Loads ORC file(s) and returns the result as a DataFrame.

Attributes

def orc(paths: String*): DataFrame

Loads ORC file(s) and returns the result as a DataFrame.

Loads ORC file(s) and returns the result as a DataFrame.

Attributes

def parquet(path: String): DataFrame

Loads Parquet file(s) and returns the result as a DataFrame.

Loads Parquet file(s) and returns the result as a DataFrame.

Attributes

def parquet(paths: String*): DataFrame

Loads Parquet file(s) and returns the result as a DataFrame.

Loads Parquet file(s) and returns the result as a DataFrame.

Attributes

def schema(schemaString: String): DataFrameReader

Specifies the input schema using a DDL-formatted string (e.g. "a INT, b STRING").

Specifies the input schema using a DDL-formatted string (e.g. "a INT, b STRING").

Attributes

Returns

this reader, for chaining.

def schema(schema: StructType): DataFrameReader

Specifies the input schema using a StructType.

Specifies the input schema using a StructType.

Attributes

Returns

this reader, for chaining.

def table(tableName: String): DataFrame

Returns the specified table/view as a DataFrame.

Returns the specified table/view as a DataFrame.

Attributes

def text(path: String): DataFrame

Loads text file(s) and returns the result as a DataFrame with a single value column.

Loads text file(s) and returns the result as a DataFrame with a single value column.

Attributes

def text(paths: String*): DataFrame

Loads text file(s) and returns the result as a DataFrame with a single value column.

Loads text file(s) and returns the result as a DataFrame with a single value column.

Attributes

def textFile(path: String): Dataset[String]

Loads text file(s) and returns a Dataset[String] of their lines.

Loads text file(s) and returns a Dataset[String] of their lines.

Attributes

def textFile(paths: String*): Dataset[String]

Loads text file(s) and returns a Dataset[String] of their lines.

Loads text file(s) and returns a Dataset[String] of their lines.

Attributes

def xml(path: String): DataFrame

Loads XML file(s) and returns the result as a DataFrame.

Loads XML file(s) and returns the result as a DataFrame.

Attributes

def xml(paths: String*): DataFrame

Loads XML file(s) and returns the result as a DataFrame.

Loads XML file(s) and returns the result as a DataFrame.

Attributes

def xml(xmlDataset: Dataset[String]): DataFrame

Parses each row of a Dataset[String] as an XML record (Spark 4.2+ servers).

Parses each row of a Dataset[String] as an XML record (Spark 4.2+ servers).

Attributes

Generated with