org.apache.spark.sql.types

Members list

Type members

Classlikes

case class ArrayType(elementType: DataType, containsNull: Boolean) extends DataType

An array type containing elements of elementType.

An array type containing elements of elementType.

Attributes

Companion
object
Supertypes
trait Product
trait Equals
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all
object ArrayType

Attributes

Companion
class
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
ArrayType.type
abstract class AtomicType extends DataType

A type for which the values can be ordered and compared.

A type for which the values can be ordered and compared.

Attributes

Supertypes
class DataType
trait Serializable
class Object
trait Matchable
class Any
Known subtypes
case object BinaryType extends AtomicType

Attributes

Supertypes
trait Singleton
trait Product
trait Mirror
trait Product
trait Equals
class AtomicType
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all
Self type
BinaryType.type
case object BooleanType extends AtomicType

Attributes

Supertypes
trait Singleton
trait Product
trait Mirror
trait Product
trait Equals
class AtomicType
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all
Self type
case object ByteType extends NumericType

Attributes

Supertypes
trait Singleton
trait Product
trait Mirror
trait Product
trait Equals
class NumericType
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all
Self type
ByteType.type
case object CalendarIntervalType extends DataType

Attributes

Supertypes
trait Singleton
trait Product
trait Mirror
trait Product
trait Equals
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all
Self type
case class CharType(length: Int) extends AtomicType

A char(n) type.

A char(n) type.

Attributes

Supertypes
trait Product
trait Equals
class AtomicType
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all
abstract class DataType extends Serializable

The base type of all Spark SQL data types.

The base type of all Spark SQL data types.

This mirrors org.apache.spark.sql.types.DataType from Apache Spark closely enough that schemas returned by a Spark Connect server can be represented faithfully on a pure Scala 3 client without depending on spark-catalyst.

Attributes

Companion
object
Supertypes
trait Serializable
class Object
trait Matchable
class Any
Known subtypes
object DataType

Attributes

Companion
class
Supertypes
class Object
trait Matchable
class Any
Self type
DataType.type
case object DateType extends AtomicType

Attributes

Supertypes
trait Singleton
trait Product
trait Mirror
trait Product
trait Equals
class AtomicType
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all
Self type
DateType.type
case class DayTimeIntervalType(startField: Byte, endField: Byte) extends AtomicType

A day-time interval type covering a contiguous range of fields.

A day-time interval type covering a contiguous range of fields.

Attributes

Companion
object
Supertypes
trait Product
trait Equals
class AtomicType
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all

Attributes

Companion
class
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
case class DecimalType(precision: Int, scale: Int) extends NumericType

A decimal type with the given precision and scale.

A decimal type with the given precision and scale.

Attributes

Companion
object
Supertypes
trait Product
trait Equals
class NumericType
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all
object DecimalType

Attributes

Companion
class
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
case object DoubleType extends NumericType

Attributes

Supertypes
trait Singleton
trait Product
trait Mirror
trait Product
trait Equals
class NumericType
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all
Self type
DoubleType.type
case object FloatType extends NumericType

Attributes

Supertypes
trait Singleton
trait Product
trait Mirror
trait Product
trait Equals
class NumericType
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all
Self type
FloatType.type
final class Geography extends Serializable

A GEOGRAPHY value: its Well-Known Binary (WKB) bytes plus a spatial reference identifier. Returned for GeographyType columns. Mirrors org.apache.spark.sql.types.Geography.

A GEOGRAPHY value: its Well-Known Binary (WKB) bytes plus a spatial reference identifier. Returned for GeographyType columns. Mirrors org.apache.spark.sql.types.Geography.

Attributes

Companion
object
Supertypes
trait Serializable
class Object
trait Matchable
class Any
object Geography

Attributes

Companion
class
Supertypes
class Object
trait Matchable
class Any
Self type
Geography.type
case class GeographyType(srid: Int) extends AtomicType

The GEOGRAPHY type with spatial reference identifier srid (-1, rendered as any, means a mixed-SRID column). Values are represented as Geography. Mirrors org.apache.spark.sql.types.GeographyType (Spark 4.1+).

The GEOGRAPHY type with spatial reference identifier srid (-1, rendered as any, means a mixed-SRID column). Values are represented as Geography. Mirrors org.apache.spark.sql.types.GeographyType (Spark 4.1+).

Attributes

Companion
object
Supertypes
trait Product
trait Equals
class AtomicType
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all
object GeographyType

Attributes

Companion
class
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
final class Geometry extends Serializable

A GEOMETRY value: its Well-Known Binary (WKB) bytes plus a spatial reference identifier. Returned for GeometryType columns. Mirrors org.apache.spark.sql.types.Geometry.

A GEOMETRY value: its Well-Known Binary (WKB) bytes plus a spatial reference identifier. Returned for GeometryType columns. Mirrors org.apache.spark.sql.types.Geometry.

Attributes

Companion
object
Supertypes
trait Serializable
class Object
trait Matchable
class Any
object Geometry

Attributes

Companion
class
Supertypes
class Object
trait Matchable
class Any
Self type
Geometry.type
case class GeometryType(srid: Int) extends AtomicType

The GEOMETRY type with spatial reference identifier srid (-1, rendered as any, means a mixed-SRID column). Values are represented as Geometry. Mirrors org.apache.spark.sql.types.GeometryType (Spark 4.1+).

The GEOMETRY type with spatial reference identifier srid (-1, rendered as any, means a mixed-SRID column). Values are represented as Geometry. Mirrors org.apache.spark.sql.types.GeometryType (Spark 4.1+).

Attributes

Companion
object
Supertypes
trait Product
trait Equals
class AtomicType
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all
object GeometryType

Attributes

Companion
class
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
case object IntegerType extends NumericType

Attributes

Supertypes
trait Singleton
trait Product
trait Mirror
trait Product
trait Equals
class NumericType
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all
Self type
case object LongType extends NumericType

Attributes

Supertypes
trait Singleton
trait Product
trait Mirror
trait Product
trait Equals
class NumericType
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all
Self type
LongType.type
case class MapType(keyType: DataType, valueType: DataType, valueContainsNull: Boolean) extends DataType

A map type with the given key and value types.

A map type with the given key and value types.

Attributes

Companion
object
Supertypes
trait Product
trait Equals
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all
object MapType

Attributes

Companion
class
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
MapType.type
final class Metadata extends Serializable

Metadata is a wrapper over a map that can be used to associate extra information with a StructField. The supported value types mirror Spark's Metadata: Long, Double, Boolean, String, and arrays / nested metadata thereof.

Metadata is a wrapper over a map that can be used to associate extra information with a StructField. The supported value types mirror Spark's Metadata: Long, Double, Boolean, String, and arrays / nested metadata thereof.

Attributes

Companion
object
Supertypes
trait Serializable
class Object
trait Matchable
class Any
object Metadata

Attributes

Companion
class
Supertypes
class Object
trait Matchable
class Any
Self type
Metadata.type
final class MetadataBuilder

A builder for Metadata.

A builder for Metadata.

Attributes

Supertypes
class Object
trait Matchable
class Any
case object NullType extends DataType

Attributes

Supertypes
trait Singleton
trait Product
trait Mirror
trait Product
trait Equals
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all
Self type
NullType.type
abstract class NumericType extends DataType

Numeric data types.

Numeric data types.

Attributes

Supertypes
class DataType
trait Serializable
class Object
trait Matchable
class Any
Known subtypes
object ByteType
class DecimalType
object DoubleType
object FloatType
object IntegerType
object LongType
object ShortType
Show all
case object ShortType extends NumericType

Attributes

Supertypes
trait Singleton
trait Product
trait Mirror
trait Product
trait Equals
class NumericType
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all
Self type
ShortType.type
case object StringType extends AtomicType

Attributes

Supertypes
trait Singleton
trait Product
trait Mirror
trait Product
trait Equals
class AtomicType
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all
Self type
StringType.type
case class StructField(name: String, dataType: DataType, nullable: Boolean, metadata: Metadata)

A field inside a StructType.

A field inside a StructType.

Attributes

Companion
object
Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
object StructField

Attributes

Companion
class
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
case class StructType(fields: Array[StructField]) extends DataType, Iterable[StructField]

A struct type, i.e. the schema of a row.

A struct type, i.e. the schema of a row.

Attributes

Companion
object
Supertypes
trait Product
trait Equals
trait Iterable[StructField]
trait IterableFactoryDefaults[StructField, Iterable]
trait IterableOps[StructField, Iterable, Iterable[StructField]]
trait IterableOnceOps[StructField, Iterable, Iterable[StructField]]
trait IterableOnce[StructField]
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all
object StructType

Attributes

Companion
class
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
StructType.type
case class TimeType(precision: Int) extends AtomicType

The TIME type: a time of day without a time zone, with precision digits of fractional seconds (0 to 9). Values are represented as java.time.LocalTime. Mirrors org.apache.spark.sql.types.TimeType (Spark 4.1+; the server gates it behind spark.sql.timeType.enabled).

The TIME type: a time of day without a time zone, with precision digits of fractional seconds (0 to 9). Values are represented as java.time.LocalTime. Mirrors org.apache.spark.sql.types.TimeType (Spark 4.1+; the server gates it behind spark.sql.timeType.enabled).

Attributes

Companion
object
Supertypes
trait Product
trait Equals
class AtomicType
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all
object TimeType

Attributes

Companion
class
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
TimeType.type
case class TimestampLTZNanosType(precision: Int) extends AtomicType

A timestamp with local time zone with nanosecond-capable precision (7 to 9 fractional-second digits). Values are represented as java.time.Instant. Mirrors org.apache.spark.sql.types.TimestampLTZNanosType (Spark 4.3+, a preview feature the server gates behind spark.sql.timestampNanosTypes.enabled).

A timestamp with local time zone with nanosecond-capable precision (7 to 9 fractional-second digits). Values are represented as java.time.Instant. Mirrors org.apache.spark.sql.types.TimestampLTZNanosType (Spark 4.3+, a preview feature the server gates behind spark.sql.timestampNanosTypes.enabled).

Attributes

Companion
object
Supertypes
trait Product
trait Equals
class AtomicType
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all

Attributes

Companion
class
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
case class TimestampNTZNanosType(precision: Int) extends AtomicType

A timestamp without time zone with nanosecond-capable precision (7 to 9 fractional-second digits). Values are represented as java.time.LocalDateTime. Mirrors org.apache.spark.sql.types.TimestampNTZNanosType (Spark 4.3+, a preview feature the server gates behind spark.sql.timestampNanosTypes.enabled).

A timestamp without time zone with nanosecond-capable precision (7 to 9 fractional-second digits). Values are represented as java.time.LocalDateTime. Mirrors org.apache.spark.sql.types.TimestampNTZNanosType (Spark 4.3+, a preview feature the server gates behind spark.sql.timestampNanosTypes.enabled).

Attributes

Companion
object
Supertypes
trait Product
trait Equals
class AtomicType
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all

Attributes

Companion
class
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
case object TimestampNTZType extends AtomicType

Attributes

Supertypes
trait Singleton
trait Product
trait Mirror
trait Product
trait Equals
class AtomicType
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all
Self type
case object TimestampType extends AtomicType

Attributes

Supertypes
trait Singleton
trait Product
trait Mirror
trait Product
trait Equals
class AtomicType
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all
Self type
case class VarcharType(length: Int) extends AtomicType

A varchar(n) type.

A varchar(n) type.

Attributes

Supertypes
trait Product
trait Equals
class AtomicType
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all
case object VariantType extends AtomicType

Attributes

Supertypes
trait Singleton
trait Product
trait Mirror
trait Product
trait Equals
class AtomicType
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all
Self type
case class YearMonthIntervalType(startField: Byte, endField: Byte) extends AtomicType

A year-month interval type covering a contiguous range of fields.

A year-month interval type covering a contiguous range of fields.

Attributes

Companion
object
Supertypes
trait Product
trait Equals
class AtomicType
class DataType
trait Serializable
class Object
trait Matchable
class Any
Show all

Attributes

Companion
class
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type