Pipeline

org.apache.spark.sql.pipelines.Pipeline
See thePipeline companion class
object Pipeline

Attributes

Companion
class
Graph
Supertypes
class Object
trait Matchable
class Any
Self type
Pipeline.type

Members list

Value members

Concrete methods

def create(session: SparkSession, defaultCatalog: Option[String], defaultDatabase: Option[String], sqlConf: Map[String, String]): Pipeline

Create a new dataflow graph on the server and return a Pipeline bound to it.

Create a new dataflow graph on the server and return a Pipeline bound to it.

Value parameters

defaultCatalog

the default catalog for datasets in this graph.

defaultDatabase

the default database for datasets in this graph.

session

the session to run the pipeline against.

sqlConf

SQL configurations for all flows in this graph.

Attributes

Returns

a Pipeline bound to the newly created graph.