Spark Connect for Scala 3
API
Generated with
Spark Connect for Scala 3/org.apache.spark/org.apache.spark.sql/SparkSession

SparkSession

org.apache.spark.sql.SparkSession
See theSparkSession companion object
class SparkSession extends AutoCloseable

The entry point to programming Spark with the DataFrame API over Spark Connect.

 val spark = SparkSession.builder
   .remote("sc://localhost:15002")
   .appName("my-app")
   .getOrCreate()

 spark.range(10).filter(col("id") % 2 === 0).show()
 spark.stop()

A session owns the SparkConnectClient, a monotonic plan-id allocator (so every relation is uniquely identifiable to the server), an Arrow allocator for decoding results, and the RuntimeConfig facade.

Attributes

Companion
object
Graph
Supertypes
trait AutoCloseable
class Object
trait Matchable
class Any

Members list

Type members

Classlikes

object implicits extends SQLImplicits

Implicit conversions ($"col", Seq(...).toDF(...)); use import spark.implicits.*.

Implicit conversions ($"col", Seq(...).toDF(...)); use import spark.implicits.*.

Attributes

Supertypes
class SQLImplicits
class Object
trait Matchable
class Any
Self type
implicits.type

Value members

Concrete methods

override def close(): Unit

Attributes

Definition Classes
AutoCloseable
def createDataFrame(rows: Seq[Row], schema: StructType): DataFrame

Creates a DataFrame from a local sequence of Rows using the given schema.

Creates a DataFrame from a local sequence of Rows using the given schema.

Attributes

def createDataFrame(rows: List[Row], schema: StructType): DataFrame

Creates a DataFrame from a Java list of Rows using the given schema.

Creates a DataFrame from a Java list of Rows using the given schema.

Attributes

def createDataset[T](data: Seq[T])(using enc: Encoder[T]): Dataset[T]

Creates a typed Dataset from a local sequence of T using its Encoder. The data is serialized via the encoder and shipped as a local relation; no server-side closure is involved.

Creates a typed Dataset from a local sequence of T using its Encoder. The data is serialized via the encoder and shipped as a local relation; no server-side closure is involved.

Attributes

def createDataset[T](data: List[T])(using enc: Encoder[T]): Dataset[T]

Creates a typed Dataset from a Java list of T using its Encoder.

Creates a typed Dataset from a Java list of T using its Encoder.

Attributes

def emptyDataFrame: DataFrame

Returns a DataFrame with no rows or columns.

Returns a DataFrame with no rows or columns.

Attributes

def newDataFrame(relType: RelType): DataFrame

Creates a DataFrame from an arbitrary RelType, tagging it with a unique plan id.

Creates a DataFrame from an arbitrary RelType, tagging it with a unique plan id.

This is the primary extension point for Spark Connect plugins: build any relation the server understands -- including a custom extension relation -- and turn it into a DataFrame. For the common case of a packed plugin message, prefer the extension overload.

Attributes

def newDataFrame(extension: Any): DataFrame

Creates a DataFrame backed by an extension relation carrying an already packed protobuf message. This is the Scala counterpart of the PySpark client's

Creates a DataFrame backed by an extension relation carrying an already packed protobuf message. This is the Scala counterpart of the PySpark client's

plan.extension.Pack(message); DataFrame(plan, session)

and targets server-side org.apache.spark.sql.connect.plugin.RelationPlugin implementations.

 import com.google.protobuf.any.Any as ProtoAny

 // `myPluginMessage` is a ScalaPB message generated from the plugin's own .proto.
 // Existing DataFrames can be embedded via their `relation` (see Dataset.relation):
 //   val payload = MyPlugin(input = Some(existingDf.relation))
 val payload: ProtoAny = ProtoAny.pack(myPluginMessage)
 val df = spark.newDataFrame(payload)

Attributes

def newRelation(relType: RelType): Relation

Wraps a RelType into a fresh proto.Relation tagged with a unique plan id.

Wraps a RelType into a fresh proto.Relation tagged with a unique plan id.

This is a low-level building block intended for Spark Connect plugin authors who need to construct custom relations (for example, a relation handled by a server-side org.apache.spark.sql.connect.plugin.RelationPlugin). It is a stable extension point but not part of the day-to-day DataFrame API; ordinary code should use range, sql, createDataFrame, and friends instead.

Attributes

def newSession(): SparkSession

Starts a new independent session against the same endpoint (fresh server-side session).

Starts a new independent session against the same endpoint (fresh server-side session).

Attributes

def pipeline(defaultCatalog: Option[String], defaultDatabase: Option[String], sqlConf: Map[String, String]): Pipeline

Creates a new Spark Declarative Pipeline (a dataflow graph) in this session. Available on Spark 4.1 and later servers.

Creates a new Spark Declarative Pipeline (a dataflow graph) in this session. Available on Spark 4.1 and later servers.

Attributes

def range(end: Long): DataFrame

Creates a DataFrame with a single id column of [0, end).

Creates a DataFrame with a single id column of [0, end).

Attributes

def range(start: Long, end: Long): DataFrame

Creates a DataFrame with a single id column of [start, end).

Creates a DataFrame with a single id column of [start, end).

Attributes

def range(start: Long, end: Long, step: Long): DataFrame
def range(start: Long, end: Long, step: Long, numPartitions: Int): DataFrame
def read: DataFrameReader

Returns a DataFrameReader that can be used to read non-streaming data as a DataFrame.

Returns a DataFrameReader that can be used to read non-streaming data as a DataFrame.

Attributes

def readStream: DataStreamReader

Returns a org.apache.spark.sql.streaming.DataStreamReader for reading streaming data.

Returns a org.apache.spark.sql.streaming.DataStreamReader for reading streaming data.

Attributes

def sessionId: String

The client session id (a UUID).

The client session id (a UUID).

Attributes

def setActive(): SparkSession

Make this the active session for the current thread.

Make this the active session for the current thread.

Attributes

def sql(query: String): DataFrame

Executes a SQL query and returns a lazy DataFrame over its result.

Executes a SQL query and returns a lazy DataFrame over its result.

Attributes

def sql(query: String, args: Array[Any]): DataFrame

Executes a SQL query with positional parameters bound into the query.

Executes a SQL query with positional parameters bound into the query.

Attributes

def sql(query: String, args: Map[String, Any]): DataFrame

Executes a SQL query with named parameters bound into the query.

Executes a SQL query with named parameters bound into the query.

Attributes

def stop(): Unit

Releases the server-side session resources and closes the channel.

Releases the server-side session resources and closes the channel.

Attributes

def table(tableName: String): DataFrame

Returns the named table or view as a DataFrame.

Returns the named table or view as a DataFrame.

Attributes

def version: String

The version of Spark on which the connected server is running.

The version of Spark on which the connected server is running.

Attributes

Concrete fields

lazy val catalog: Catalog

Returns the org.apache.spark.sql.catalog.Catalog interface for this session.

Returns the org.apache.spark.sql.catalog.Catalog interface for this session.

Attributes

lazy val conf: RuntimeConfig

Runtime configuration for Spark.

Runtime configuration for Spark.

Attributes

Generated with