Spark Connect for Scala 3
API
Generated with
Spark Connect for Scala 3/org.apache.spark/org.apache.spark.sql/org.apache.spark.sql.connect/org.apache.spark.sql.connect.client

org.apache.spark.sql.connect.client

package org.apache.spark.sql.connect.client

Members list

Type members

Classlikes

object ArrowSerializer

Encodes local rows into an Apache Arrow IPC stream so that the client can ship local data to the server as a LocalRelation (used by SparkSession.createDataFrame).

Encodes local rows into an Apache Arrow IPC stream so that the client can ship local data to the server as a LocalRelation (used by SparkSession.createDataFrame).

This is the symmetric ENCODE counterpart to SparkResult (which DECODES the Arrow batches returned by the server). The Spark-type-to-Arrow-type mapping and the per-cell value conversions here mirror the decode logic in SparkResult.getValue so that a value round-trips faithfully.

Modelled on the same-author Ruby reference (spark_connect/arrow.rb, from_rows / build_arrow_schema / arrow_field_type).

Attributes

Supertypes
class Object
trait Matchable
class Any
Self type
case class RetryPolicy(maxRetries: Option[Int], initialBackoff: FiniteDuration, maxBackoff: Option[FiniteDuration], backoffMultiplier: Double, jitter: FiniteDuration, minJitterThreshold: FiniteDuration, canRetry: Throwable => Boolean, name: String, recognizeServerRetryDelay: Boolean, maxServerRetryDelay: Option[FiniteDuration])

Configures the retry mechanism used by GrpcRetryHandler. Ported from Apache Spark's org.apache.spark.sql.connect.client.RetryPolicy.

Configures the retry mechanism used by GrpcRetryHandler. Ported from Apache Spark's org.apache.spark.sql.connect.client.RetryPolicy.

Value parameters

backoffMultiplier

multiplicative base of the exponential backoff.

canRetry

predicate deciding whether an error is retryable.

initialBackoff

start of the exponential backoff.

jitter

random value in [0, jitter] added to the backoff.

maxBackoff

cap on the exponential backoff.

maxRetries

maximum number of retries (None = unbounded).

maxServerRetryDelay

cap on the server-provided retry delay.

minJitterThreshold

minimum backoff at which jitter is added.

name

policy name (for diagnostics).

recognizeServerRetryDelay

honor a server-provided RetryInfo.retry_delay.

Attributes

Companion
object
Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
object RetryPolicy

Attributes

Companion
class
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
case class RpcDeadlines(reattachableExecutePlan: Option[FiniteDuration], reattachExecute: Option[FiniteDuration], analyzePlan: Option[FiniteDuration], addArtifacts: Option[FiniteDuration], config: Option[FiniteDuration], interrupt: Option[FiniteDuration], releaseSession: Option[FiniteDuration], artifactStatus: Option[FiniteDuration], cloneSession: Option[FiniteDuration], getStatus: Option[FiniteDuration], fetchErrorDetails: Option[FiniteDuration])

Per-RPC gRPC deadlines (timeouts). Each field bounds one RPC type; None disables the deadline for that RPC. The defaults match the Apache Spark 4.3 Connect clients.

Per-RPC gRPC deadlines (timeouts). Each field bounds one RPC type; None disables the deadline for that RPC. The defaults match the Apache Spark 4.3 Connect clients.

reattachableExecutePlan and reattachExecute bound each individual result-stream segment, not the query: when one fires, the server-side operation keeps running and the client resumes with a fresh ReattachExecute stream. Non-reattachable ExecutePlan has no deadline because a timeout there would kill the execution with no way to recover.

 SparkConnectClient.builder().rpcDeadlines(RpcDeadlines(analyzePlan = Some(10.minutes)))
 SparkConnectClient.builder().rpcDeadlines(RpcDeadlines.disabled)

Attributes

Companion
object
Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
object RpcDeadlines

Attributes

Companion
class
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
object ShowString

Renders rows as the familiar df.show() table, client-side, matching the layout of Dataset.showString in Apache Spark (horizontal and vertical). Rendering locally from collected rows keeps show() deterministic and exactly consistent with collect().

Renders rows as the familiar df.show() table, client-side, matching the layout of Dataset.showString in Apache Spark (horizontal and vertical). Rendering locally from collected rows keeps show() deterministic and exactly consistent with collect().

The caller passes up to numRows + 1 rows; the extra row (if present) only triggers the "only showing top N rows" footer and is not displayed.

Attributes

Supertypes
class Object
trait Matchable
class Any
Self type
ShowString.type
class SparkConnectClient

The low-level client that talks to a Spark Connect server over gRPC.

The low-level client that talks to a Spark Connect server over gRPC.

It owns the gRPC channel, the (stable) session id, the user context and the user agent, and exposes the RPCs used by the public API: ExecutePlan, AnalyzePlan, Config, Interrupt, ReleaseSession, and GetStatus.

Attributes

Companion
object
Supertypes
class Object
trait Matchable
class Any
object SparkConnectClient

Attributes

Companion
class
Supertypes
class Object
trait Matchable
class Any
Self type
Generated with