org.apache.spark.sql.catalog

The catalog interface for inspecting and managing catalogs, databases, tables, views, functions, and the query cache. Access it with spark.catalog. Mirrors org.apache.spark.sql.catalog.

Listing methods return typed Datasets (Dataset[Table], Dataset[Database], ...); predicate methods return a Boolean; command-like methods return Unit and run immediately.

 spark.catalog.listTables().show()
 spark.catalog.listTables().collect().map(_.name)
 spark.catalog.tableExists("my_table")  // => true

Attributes

Supertypes: class Object

trait Matchable

class Any

A catalog in Spark, as returned by Catalog.listCatalogs.

Value parameters

description: description of the catalog (may be null).
name: name of the catalog.

Attributes

Supertypes: trait Serializable

trait Product

trait Equals

class Object

trait Matchable

class Any
Show all

A table property key and value, as returned by Catalog.getTableProperties.

Attributes

Supertypes: trait Serializable

trait Product

trait Equals

class Object

trait Matchable

class Any
Show all

A column of a table, as returned by Catalog.listColumns.

Value parameters

dataType: the column's data type as a string.
description: description of the column (may be null).
isBucket: whether the column is a bucket column.
isCluster: whether the column is a clustering column (always false on Spark 3.5 servers).
isPartition: whether the column is a partition column.
name: name of the column.
nullable: whether the column is nullable.

Attributes

Supertypes: trait Serializable

trait Product

trait Equals

class Object

trait Matchable

class Any
Show all

A database in the catalog, as returned by Catalog.getDatabase / Catalog.listDatabases.

Value parameters

catalog: name of the catalog the database belongs to (may be null on older servers).
description: description of the database.
locationUri: path (in the form of a uri) to data files.
name: name of the database.

Attributes

Supertypes: trait Serializable

trait Product

trait Equals

class Object

trait Matchable

class Any
Show all

A function in the catalog, as returned by Catalog.getFunction / Catalog.listFunctions.

Value parameters

catalog: name of the catalog the function belongs to (may be null on older servers).
className: the fully qualified class name of the function.
description: description of the function.
isTemporary: whether the function is a temporary function.
name: name of the function.
namespace: the namespace the function belongs to; null/empty for temporary functions.

Attributes

Supertypes: trait Serializable

trait Product

trait Equals

class Object

trait Matchable

class Any
Show all

A table or view in the catalog, as returned by Catalog.getTable / Catalog.listTables.

Value parameters

catalog: name of the catalog the table belongs to (may be null on older servers).
description: description of the table.
isTemporary: whether the table is a temporary view.
name: name of the table.
namespace: the namespace (database) the table belongs to; empty for temporary views.
tableType: type of the table (e.g. MANAGED, EXTERNAL, VIEW, TEMPORARY).

Attributes

Supertypes: trait Serializable

trait Product

trait Equals

class Object

trait Matchable

class Any
Show all

A partition of a table, as returned by Catalog.listPartitions (the same form as SHOW PARTITIONS, e.g. year=2024/month=1).

Value parameters

partition: the partition description string.

Attributes

Supertypes: trait Serializable

trait Product

trait Equals

class Object

trait Matchable

class Any
Show all

org.apache.spark.sql.catalog

Members list

Type members

Classlikes

Attributes

Value parameters

Attributes

Attributes

Value parameters

Attributes

Value parameters

Attributes

Value parameters

Attributes

Value parameters

Attributes

Value parameters

Attributes