Spark Connect for Scala 3
API
Generated with
Spark Connect for Scala 3/org.apache.spark/org.apache.spark.sql/org.apache.spark.sql.catalog/Catalog

Catalog

org.apache.spark.sql.catalog.Catalog
class Catalog

The catalog interface for inspecting and managing catalogs, databases, tables, views, functions, and the query cache. Access it with spark.catalog. Mirrors org.apache.spark.sql.catalog.

Listing methods return typed Datasets (Dataset[Table], Dataset[Database], ...); predicate methods return a Boolean; command-like methods return Unit and run immediately.

 spark.catalog.listTables().show()
 spark.catalog.listTables().collect().map(_.name)
 spark.catalog.tableExists("my_table")  // => true

Attributes

Graph
Supertypes
class Object
trait Matchable
class Any

Members list

Value members

Concrete methods

def analyzeTable(tableName: String, noScan: Boolean): Unit

Computes table statistics like ANALYZE TABLE ... COMPUTE STATISTICS (Spark 4.2+).

Computes table statistics like ANALYZE TABLE ... COMPUTE STATISTICS (Spark 4.2+).

Attributes

def cacheTable(tableName: String): Unit

Caches the table in memory.

Caches the table in memory.

Attributes

def cacheTable(tableName: String, storageLevel: StorageLevel): Unit

Caches the table with the given storage level.

Caches the table with the given storage level.

Attributes

def clearCache(): Unit

Removes all cached tables from the in-memory cache.

Removes all cached tables from the in-memory cache.

Attributes

def createDatabase(dbName: String, ifNotExists: Boolean): Unit

Creates a database (namespace); dbName may be multi-part (Spark 4.2+).

Creates a database (namespace); dbName may be multi-part (Spark 4.2+).

Attributes

def createDatabase(dbName: String, ifNotExists: Boolean, properties: Map[String, String]): Unit

Creates a database with the given properties (e.g. comment, location), like CREATE NAMESPACE ... WITH DBPROPERTIES (Spark 4.2+).

Creates a database with the given properties (e.g. comment, location), like CREATE NAMESPACE ... WITH DBPROPERTIES (Spark 4.2+).

Attributes

def createDatabase(dbName: String, ifNotExists: Boolean, properties: Map[String, String]): Unit

(Java-specific) Creates a database with the given properties (Spark 4.2+).

(Java-specific) Creates a database with the given properties (Spark 4.2+).

Attributes

def createTable(tableName: String, path: String): DataFrame

Creates a table from the files at path, using the default data source (spark.sql.sources.default), and returns it as a DataFrame.

Creates a table from the files at path, using the default data source (spark.sql.sources.default), and returns it as a DataFrame.

Attributes

def createTable(tableName: String, path: String, source: String): DataFrame

Creates a table from the files at path using the data source source.

Creates a table from the files at path using the data source source.

Attributes

def createTable(tableName: String, source: String, options: Map[String, String]): DataFrame

Creates a table from a data source and options.

Creates a table from a data source and options.

Attributes

def createTable(tableName: String, source: String, options: Map[String, String]): DataFrame

(Java-specific) Creates a table from a data source and options.

(Java-specific) Creates a table from a data source and options.

Attributes

def createTable(tableName: String, source: String, description: String, options: Map[String, String]): DataFrame

Creates a table from a data source, a description, and options.

Creates a table from a data source, a description, and options.

Attributes

def createTable(tableName: String, source: String, description: String, options: Map[String, String]): DataFrame

(Java-specific) Creates a table from a data source, a description, and options.

(Java-specific) Creates a table from a data source, a description, and options.

Attributes

def createTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame

Creates a table from a data source, a schema, and options.

Creates a table from a data source, a schema, and options.

Attributes

def createTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame

(Java-specific) Creates a table from a data source, a schema, and options.

(Java-specific) Creates a table from a data source, a schema, and options.

Attributes

def createTable(tableName: String, source: String, schema: StructType, description: String, options: Map[String, String]): DataFrame

Creates a table from a data source, a schema, a description, and options.

Creates a table from a data source, a schema, a description, and options.

Attributes

def createTable(tableName: String, source: String, schema: StructType, description: String, options: Map[String, String]): DataFrame

(Java-specific) Creates a table from a data source, schema, description, and options.

(Java-specific) Creates a table from a data source, schema, description, and options.

Attributes

def currentCatalog(): String

Returns the current default catalog.

Returns the current default catalog.

Attributes

def currentDatabase: String

Returns the current default database.

Returns the current default database.

Attributes

def databaseExists(dbName: String): Boolean

Returns true if the database exists.

Returns true if the database exists.

Attributes

def dropDatabase(dbName: String, ifExists: Boolean, cascade: Boolean): Unit

Drops a database; with cascade, also drops the objects it contains (Spark 4.2+).

Drops a database; with cascade, also drops the objects it contains (Spark 4.2+).

Attributes

def dropGlobalTempView(viewName: String): Boolean

Drops a global temporary view; returns true if it existed.

Drops a global temporary view; returns true if it existed.

Attributes

def dropTable(tableName: String, ifExists: Boolean, purge: Boolean): Unit

Drops a persistent table; temporary views are dropped with dropTempView (Spark 4.2+).

Drops a persistent table; temporary views are dropped with dropTempView (Spark 4.2+).

Attributes

def dropTempView(viewName: String): Boolean

Drops a local temporary view; returns true if it existed.

Drops a local temporary view; returns true if it existed.

Attributes

def dropView(viewName: String, ifExists: Boolean): Unit

Drops a persistent view; temporary views are dropped with dropTempView (Spark 4.2+).

Drops a persistent view; temporary views are dropped with dropTempView (Spark 4.2+).

Attributes

def functionExists(functionName: String): Boolean

Returns true if the function exists.

Returns true if the function exists.

Attributes

def functionExists(dbName: String, functionName: String): Boolean

Returns true if the function exists in the given database.

Returns true if the function exists in the given database.

Attributes

def getCreateTableString(tableName: String, asSerde: Boolean): String

Returns the SHOW CREATE TABLE statement of a table or view; with asSerde, the Hive SerDe form when applicable (Spark 4.2+).

Returns the SHOW CREATE TABLE statement of a table or view; with asSerde, the Hive SerDe form when applicable (Spark 4.2+).

Attributes

def getDatabase(dbName: String): Database

Returns the database with the given name; throws if it does not exist.

Returns the database with the given name; throws if it does not exist.

Attributes

def getFunction(functionName: String): Function

Returns the function with the given name; throws if it does not exist.

Returns the function with the given name; throws if it does not exist.

Attributes

def getFunction(dbName: String, functionName: String): Function

Returns the function with the given name in the given database.

Returns the function with the given name in the given database.

Attributes

def getTable(tableName: String): Table

Returns the table or view with the given name; throws if it does not exist.

Returns the table or view with the given name; throws if it does not exist.

Attributes

def getTable(dbName: String, tableName: String): Table

Returns the table or view with the given name in the given database.

Returns the table or view with the given name in the given database.

Attributes

def getTableProperties(tableName: String): Map[String, String]

Returns all properties of a table or view, like SHOW TBLPROPERTIES (Spark 4.2+).

Returns all properties of a table or view, like SHOW TBLPROPERTIES (Spark 4.2+).

Attributes

def isCached(tableName: String): Boolean

Returns true if the table is cached in memory.

Returns true if the table is cached in memory.

Attributes

def listCatalogs(): Dataset[CatalogMetadata]

Returns a list of the catalogs available in this session.

Returns a list of the catalogs available in this session.

Attributes

def listCatalogs(pattern: String): Dataset[CatalogMetadata]

Returns the catalogs whose name matches the SQL LIKE pattern.

Returns the catalogs whose name matches the SQL LIKE pattern.

Attributes

def listColumns(tableName: String): Dataset[Column]

Returns the columns of the given table or view.

Returns the columns of the given table or view.

Attributes

def listColumns(dbName: String, tableName: String): Dataset[Column]

Returns the columns of the given table or view in the given database.

Returns the columns of the given table or view in the given database.

Attributes

def listDatabases(): Dataset[Database]

Returns a list of the databases (namespaces) in the current catalog.

Returns a list of the databases (namespaces) in the current catalog.

Attributes

def listDatabases(pattern: String): Dataset[Database]

Returns the databases whose name matches the SQL LIKE pattern.

Returns the databases whose name matches the SQL LIKE pattern.

Attributes

def listFunctions(): Dataset[Function]

Returns the functions registered in the current database, including temporary ones.

Returns the functions registered in the current database, including temporary ones.

Attributes

def listFunctions(dbName: String): Dataset[Function]

Returns the functions registered in the given database, including temporary ones.

Returns the functions registered in the given database, including temporary ones.

Attributes

def listFunctions(dbName: String, pattern: String): Dataset[Function]

Returns the functions of dbName whose name matches the SQL LIKE pattern.

Returns the functions of dbName whose name matches the SQL LIKE pattern.

Attributes

def listPartitions(tableName: String): Dataset[TablePartition]

Returns the partitions of a table, like SHOW PARTITIONS (Spark 4.2+).

Returns the partitions of a table, like SHOW PARTITIONS (Spark 4.2+).

Attributes

def listTables(): Dataset[Table]

Returns the tables and views in the current database, including temporary views.

Returns the tables and views in the current database, including temporary views.

Attributes

def listTables(dbName: String): Dataset[Table]

Returns the tables and views in the given database, including temporary views.

Returns the tables and views in the given database, including temporary views.

Attributes

def listTables(dbName: String, pattern: String): Dataset[Table]

Returns the tables and views of dbName whose name matches the SQL LIKE pattern.

Returns the tables and views of dbName whose name matches the SQL LIKE pattern.

Attributes

def listViews(): Dataset[Table]

Returns the views in the current database (Spark 4.2+).

Returns the views in the current database (Spark 4.2+).

Attributes

def listViews(dbName: String): Dataset[Table]

Returns the views in the given (possibly catalog-qualified) database (Spark 4.2+).

Returns the views in the given (possibly catalog-qualified) database (Spark 4.2+).

Attributes

def listViews(dbName: String, pattern: String): Dataset[Table]

Returns the views of dbName whose name matches the SQL LIKE pattern (Spark 4.2+).

Returns the views of dbName whose name matches the SQL LIKE pattern (Spark 4.2+).

Attributes

def recoverPartitions(tableName: String): Unit

Recovers all the partitions of the table and updates the catalog.

Recovers all the partitions of the table and updates the catalog.

Attributes

def refreshByPath(path: String): Unit

Invalidates and refreshes the cached data of any Dataset that reads from path.

Invalidates and refreshes the cached data of any Dataset that reads from path.

Attributes

def refreshTable(tableName: String): Unit

Invalidates and refreshes the cached data and metadata of the table.

Invalidates and refreshes the cached data and metadata of the table.

Attributes

def setCurrentCatalog(catalogName: String): Unit

Sets the current default catalog.

Sets the current default catalog.

Attributes

def setCurrentDatabase(dbName: String): Unit

Sets the current default database.

Sets the current default database.

Attributes

def tableExists(tableName: String): Boolean

Returns true if the table or view exists.

Returns true if the table or view exists.

Attributes

def tableExists(dbName: String, tableName: String): Boolean

Returns true if the table or view exists in the given database.

Returns true if the table or view exists in the given database.

Attributes

def truncateTable(tableName: String): Unit

Removes all rows from a table; not supported for views (Spark 4.2+).

Removes all rows from a table; not supported for views (Spark 4.2+).

Attributes

def uncacheTable(tableName: String): Unit

Removes the table from the in-memory cache.

Removes the table from the in-memory cache.

Attributes

Deprecated methods

def createExternalTable(tableName: String, path: String): DataFrame

Attributes

Deprecated
true
def createExternalTable(tableName: String, path: String, source: String): DataFrame

Attributes

Deprecated
true
def createExternalTable(tableName: String, source: String, options: Map[String, String]): DataFrame

Attributes

Deprecated
true
def createExternalTable(tableName: String, source: String, options: Map[String, String]): DataFrame

Attributes

Deprecated
true
def createExternalTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame

Attributes

Deprecated
true
def createExternalTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame

Attributes

Deprecated
true
Generated with