Spark Connect for Scala 3
API
Generated with
Spark Connect for Scala 3/org.apache.spark/org.apache.spark.sql/org.apache.spark.sql.catalog/Catalog

Catalog

org.apache.spark.sql.catalog.Catalog
class Catalog

The catalog interface for inspecting and managing databases, tables, functions, and the query cache.

Methods that return rows (listDatabases, listTables, ...) return a DataFrame; predicate methods return a Boolean; command-like methods return Unit. Mirrors PySpark's Catalog.

 spark.catalog.listTables().show()
 spark.catalog.tableExists("my_table")  // => true

Attributes

Graph
Supertypes
class Object
trait Matchable
class Any

Members list

Value members

Concrete methods

def cacheTable(tableName: String): Unit

Caches the given table in memory.

Caches the given table in memory.

Attributes

def clearCache(): Unit

Clears all cached tables.

Clears all cached tables.

Attributes

def createExternalTable(tableName: String, path: String): DataFrame

Creates a table from the given path and returns the corresponding DataFrame.

Creates a table from the given path and returns the corresponding DataFrame.

Attributes

def createExternalTable(tableName: String, source: String, options: Map[String, String]): DataFrame

Creates a table from the given source and options, returning the corresponding DataFrame.

Creates a table from the given source and options, returning the corresponding DataFrame.

Attributes

def createTable(tableName: String, path: String): DataFrame

Creates a table from the given path (using the default data source) and returns the corresponding DataFrame.

Creates a table from the given path (using the default data source) and returns the corresponding DataFrame.

Attributes

def createTable(tableName: String, source: String, options: Map[String, String]): DataFrame

Creates a table from the given source and options, returning the corresponding DataFrame.

Creates a table from the given source and options, returning the corresponding DataFrame.

Attributes

def createTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame

Creates a table from the given source, schema and options, returning the corresponding DataFrame.

Creates a table from the given source, schema and options, returning the corresponding DataFrame.

Attributes

def currentCatalog: String

Returns the current default catalog.

Returns the current default catalog.

Attributes

def currentDatabase: String

Returns the current default database.

Returns the current default database.

Attributes

def databaseExists(dbName: String): Boolean

Returns whether a database with the given name exists.

Returns whether a database with the given name exists.

Attributes

def dropGlobalTempView(viewName: String): Boolean

Drops a global temporary view. Returns whether the view was dropped.

Drops a global temporary view. Returns whether the view was dropped.

Attributes

def dropTempView(viewName: String): Boolean

Drops a session-local temporary view. Returns whether the view was dropped.

Drops a session-local temporary view. Returns whether the view was dropped.

Attributes

def functionExists(functionName: String): Boolean

Returns whether a function with the given name exists.

Returns whether a function with the given name exists.

Attributes

def getDatabase(dbName: String): Database

Gets the database with the given name as a Database.

Gets the database with the given name as a Database.

Attributes

def getFunction(functionName: String): Function

Gets the function with the given name as a Function.

Gets the function with the given name as a Function.

Attributes

def getFunction(dbName: String, functionName: String): Function

Gets the function of the given name in the given database as a Function.

Gets the function of the given name in the given database as a Function.

Attributes

def getTable(tableName: String): Table

Gets the table or view with the given name as a Table.

Gets the table or view with the given name as a Table.

Attributes

def getTable(dbName: String, tableName: String): Table

Gets the table or view of the given name in the given database as a Table.

Gets the table or view of the given name in the given database as a Table.

Attributes

def isCached(tableName: String): Boolean

Returns whether the given table is cached.

Returns whether the given table is cached.

Attributes

def listCatalogs(): DataFrame

Returns a DataFrame of all catalogs.

Returns a DataFrame of all catalogs.

Attributes

def listColumns(tableName: String): DataFrame

Returns a DataFrame of the columns of the given table.

Returns a DataFrame of the columns of the given table.

Attributes

def listColumns(dbName: String, tableName: String): DataFrame

Returns a DataFrame of the columns of the given table in the given database.

Returns a DataFrame of the columns of the given table in the given database.

Attributes

def listDatabases(): DataFrame

Returns a DataFrame of all databases.

Returns a DataFrame of all databases.

Attributes

def listFunctions(): DataFrame

Returns a DataFrame of functions registered in the catalog.

Returns a DataFrame of functions registered in the catalog.

Attributes

def listTables(): DataFrame

Returns a DataFrame of tables (and views) in the current database.

Returns a DataFrame of tables (and views) in the current database.

Attributes

def listTables(dbName: String): DataFrame

Returns a DataFrame of tables (and views) in the given database.

Returns a DataFrame of tables (and views) in the given database.

Attributes

def recoverPartitions(tableName: String): Unit

Recovers all partitions of the given table.

Recovers all partitions of the given table.

Attributes

def refreshByPath(path: String): Unit

Invalidates and refreshes cached data (and metadata) at the given path.

Invalidates and refreshes cached data (and metadata) at the given path.

Attributes

def refreshTable(tableName: String): Unit

Invalidates and refreshes cached metadata for the given table.

Invalidates and refreshes cached metadata for the given table.

Attributes

def setCurrentCatalog(catalogName: String): Unit

Sets the current default catalog.

Sets the current default catalog.

Attributes

def setCurrentDatabase(dbName: String): Unit

Sets the current default database.

Sets the current default database.

Attributes

def tableExists(tableName: String): Boolean

Returns whether a table or view with the given name exists.

Returns whether a table or view with the given name exists.

Attributes

def tableExists(dbName: String, tableName: String): Boolean

Returns whether a table or view with the given name in the given database exists.

Returns whether a table or view with the given name in the given database exists.

Attributes

def uncacheTable(tableName: String): Unit

Removes the given table from the cache.

Removes the given table from the cache.

Attributes

Generated with