SparkSession

Implicit conversions ($"col", Seq(...).toDF(...)); use import spark.implicits.*.

Adds a tag to every operation started by the current thread (and threads it creates later), so they can be interrupted together with interruptTag. A tag must be non-empty and must not contain ,.

Removes all of the current thread's operation tags.

Clones this session on the server (Spark 4.1+): the new session starts with a copy of this session's SQL configurations, temporary views, registered functions, and catalog state, and is independent afterwards.

Like cloneSession, with the given session id (a UUID) for the clone.

Creates a DataFrame from a local sequence of Rows using the given schema.

Creates a DataFrame from a Java list of Rows using the given schema.

Creates a DataFrame from a local sequence of case classes or tuples.

Creates a typed Dataset from a local sequence of T using its Encoder. The data is serialized via the encoder and shipped as a local relation; no server-side closure is involved.

Creates a typed Dataset from a Java list of T using its Encoder.

Returns a DataFrame with no rows or columns.

Returns an empty Dataset of type T.

Runs command in an external execution engine (runner, e.g. a JDBC data source) rather than Spark, returning the command's output, if any, as a DataFrame.

The tags the current thread attaches to the operations it starts.

Interrupts all operations of this session; returns the ids of interrupted operations.

Interrupts the operation with the given id; returns the ids of interrupted operations.

Interrupts the operations with the given tag; returns the ids of interrupted operations.

Creates a DataFrame from an arbitrary RelType, tagging it with a unique plan id.

This is the primary extension point for Spark Connect plugins: build any relation the server understands -- including a custom extension relation -- and turn it into a DataFrame. For the common case of a packed plugin message, prefer the extension overload.

Creates a DataFrame backed by an extension relation carrying an already packed protobuf message. This is the Scala counterpart of the PySpark client's

plan.extension.Pack(message); DataFrame(plan, session)

and targets server-side org.apache.spark.sql.connect.plugin.RelationPlugin implementations.

 import com.google.protobuf.any.Any as ProtoAny

 // `myPluginMessage` is a ScalaPB message generated from the plugin's own .proto.
 // Existing DataFrames can be embedded via their `relation` (see Dataset.relation):
 //   val payload = MyPlugin(input = Some(existingDf.relation))
 val payload: ProtoAny = ProtoAny.pack(myPluginMessage)
 val df = spark.newDataFrame(payload)

Wraps a RelType into a fresh proto.Relation tagged with a unique plan id.

This is a low-level building block intended for Spark Connect plugin authors who need to construct custom relations (for example, a relation handled by a server-side org.apache.spark.sql.connect.plugin.RelationPlugin). It is a stable extension point but not part of the day-to-day DataFrame API; ordinary code should use range, sql, createDataFrame, and friends instead.

Starts a new independent session against the same endpoint: a fresh server-side session with its own configuration and temporary views.

Creates a new Spark Declarative Pipeline (a dataflow graph) in this session. Available on Spark 4.1 and later servers.

Creates a Dataset with a single id column of [0, end).

Creates a Dataset with a single id column of [start, end).

Creates a Dataset with a single id column of [start, end) stepping by step.

Creates a Dataset with a single id column of [start, end) in numPartitions.

Returns a DataFrameReader that can be used to read non-streaming data as a DataFrame.

Returns a org.apache.spark.sql.streaming.DataStreamReader for reading streaming data.

Removes a tag previously added by addTag on the current thread.

The client session id (a UUID).

Make this the active session for the current thread.

Executes a SQL statement, returning its result as a DataFrame. Like Apache Spark, the statement is sent to the server right away: commands (DDL / DML such as CREATE TABLE or INSERT) run immediately, while queries stay lazy until an action.

Executes a SQL statement with positional parameters bound into the query.

Executes a SQL statement with named parameters bound into the query.

(Java-specific) Executes a SQL statement with named parameters bound into the query.

Releases the server-side session resources and closes the channel.

Returns the named table or view as a DataFrame.

Runs f, printing the time it took to the console, and returns its result.

Returns the table-valued functions, e.g. spark.tvf.explode(...).

The version of Spark on which the connected server is running.

Runs block with this session as the active session, restoring the previous one after.

Returns the org.apache.spark.sql.catalog.Catalog interface for this session.

Runtime configuration for Spark.

Returns a org.apache.spark.sql.streaming.StreamingQueryManager for this session.