Module: SparkConnect::PlanBuilder

Defined in:
lib/spark_connect/plan.rb

Overview

Low-level helpers for assembling the protobuf logical plan that the client sends to the server. DataFrame and SparkSession build relations through these helpers so that every relation carries a unique plan_id (used by the server to resolve columns to a specific subtree, e.g. for self-joins).

Constant Summary collapse

Proto = 
SparkConnect::Proto

Class Method Summary collapse

Class Method Details

.command_plan(command) ⇒ Spark::Connect::Plan

Wrap a command as an executable Spark::Connect::Plan.

Parameters:

  • command (Spark::Connect::Command)

Returns:

  • (Spark::Connect::Plan) 


35
36
37
 
# File 'lib/spark_connect/plan.rb', line 35

def command_plan(command)
  Proto::Plan.new(command: command)
end

.relation(id_source, **rel) ⇒ Spark::Connect::Relation

Wrap a rel_type oneof keyword into a Spark::Connect::Relation, attaching a fresh plan_id from id_source.

Parameters:

  • id_source (#next_plan_id) —

    usually a SparkSession.

  • rel (Hash) —

    exactly one rel_type keyword, e.g. project:.

Returns:

  • (Spark::Connect::Relation) 


19
20
21
 
# File 'lib/spark_connect/plan.rb', line 19

def relation(id_source, **rel)
  Proto::Relation.new(common: Proto::RelationCommon.new(plan_id: id_source.next_plan_id), **rel)
end

.root_plan(relation) ⇒ Spark::Connect::Plan

Wrap a relation as the root of an executable Spark::Connect::Plan.

Parameters:

  • relation (Spark::Connect::Relation)

Returns:

  • (Spark::Connect::Plan) 


27
28
29
 
# File 'lib/spark_connect/plan.rb', line 27

def root_plan(relation)
  Proto::Plan.new(root: relation)
end