Spark Connect for Scala 3
API
Generated with
Spark Connect for Scala 3/org.apache.spark/org.apache.spark.sql/functions

functions

org.apache.spark.sql.functions
object functions

Built-in functions for working with Columns, mirroring org.apache.spark.sql.functions.

 import org.apache.spark.sql.functions._
 df.select(col("id"), upper(col("name")), (col("x") + 1).as("x1"))
 df.groupBy("dept").agg(avg("salary"), count(lit(1)))

This object exposes a comprehensive subset of Spark's function library. Any Spark function not listed here can still be invoked by name via callUDF / expr.

Following Spark's convention, a String argument denotes a column name for most functions (e.g. sum("salary") aggregates the salary column), while functions whose parameters are genuinely literal (regex patterns, date formats, JSON paths, ...) treat their String arguments as literal values.

Attributes

Graph
Supertypes
class Object
trait Matchable
class Any
Self type
functions.type

Members list

Type members

Classlikes

object partitioning

Partition transform functions for DataFrameWriterV2.partitionedBy, e.g. df.writeTo("t").partitionedBy(partitioning.years(col("ts"))). The deprecated top-level aliases below forward here.

Partition transform functions for DataFrameWriterV2.partitionedBy, e.g. df.writeTo("t").partitionedBy(partitioning.years(col("ts"))). The deprecated top-level aliases below forward here.

Attributes

Supertypes
class Object
trait Matchable
class Any
Self type

Value members

Concrete methods

def abs(e: Column): Column
def acos(e: Column): Column
def acos(columnName: String): Column
def acosh(e: Column): Column
def acosh(columnName: String): Column
def add_months(start: Column, numMonths: Int): Column
def add_months(start: Column, numMonths: Column): Column
def aes_decrypt(input: Column, key: Column, mode: Column, padding: Column, aad: Column): Column

Returns a decrypted value of input using AES in mode with padding.

Returns a decrypted value of input using AES in mode with padding.

Attributes

def aes_decrypt(input: Column, key: Column, mode: Column, padding: Column): Column

Returns a decrypted value of input.

Returns a decrypted value of input.

Attributes

def aes_decrypt(input: Column, key: Column, mode: Column): Column

Returns a decrypted value of input.

Returns a decrypted value of input.

Attributes

def aes_decrypt(input: Column, key: Column): Column

Returns a decrypted value of input.

Returns a decrypted value of input.

Attributes

def aes_encrypt(input: Column, key: Column, mode: Column, padding: Column, iv: Column, aad: Column): Column

Returns an encrypted value of input using AES in given mode with the specified padding.

Returns an encrypted value of input using AES in given mode with the specified padding.

Attributes

def aes_encrypt(input: Column, key: Column, mode: Column, padding: Column, iv: Column): Column

Returns an encrypted value of input.

Returns an encrypted value of input.

Attributes

def aes_encrypt(input: Column, key: Column, mode: Column, padding: Column): Column

Returns an encrypted value of input.

Returns an encrypted value of input.

Attributes

def aes_encrypt(input: Column, key: Column, mode: Column): Column

Returns an encrypted value of input.

Returns an encrypted value of input.

Attributes

def aes_encrypt(input: Column, key: Column): Column

Returns an encrypted value of input.

Returns an encrypted value of input.

Attributes

def aggregate(expr: Column, initialValue: Column, merge: (Column, Column) => Column, finish: Column => Column): Column

Applies a binary operator to an initial state and all array elements, then a finish step.

Applies a binary operator to an initial state and all array elements, then a finish step.

Attributes

def aggregate(expr: Column, initialValue: Column, merge: (Column, Column) => Column): Column
def any(e: Column): Column
def any_value(e: Column, ignoreNulls: Column): Column
def approx_count_distinct(columnName: String): Column
def approx_count_distinct(e: Column, rsd: Double): Column
def approx_count_distinct(columnName: String, rsd: Double): Column
def approx_percentile(e: Column, percentage: Column, accuracy: Column): Column

Aggregate function: returns the approximate percentile of the numeric column col which is the smallest value in the ordered col values (sorted from least to greatest) such that no more than percentage of col values is less than the value or equal to that value.

Aggregate function: returns the approximate percentile of the numeric column col which is the smallest value in the ordered col values (sorted from least to greatest) such that no more than percentage of col values is less than the value or equal to that value.

Attributes

def array(cols: Column*): Column
def array(colName: String, colNames: String*): Column
def array_agg(e: Column): Column

Aggregate function: returns a list of objects with duplicates.

Aggregate function: returns a list of objects with duplicates.

Attributes

def array_append(column: Column, element: Any): Column
def array_compact(column: Column): Column
def array_contains(column: Column, value: Any): Column
def array_except(col1: Column, col2: Column): Column
def array_insert(arr: Column, pos: Column, value: Column): Column
def array_intersect(col1: Column, col2: Column): Column
def array_join(column: Column, delimiter: String): Column
def array_join(column: Column, delimiter: String, nullReplacement: String): Column
def array_position(column: Column, value: Any): Column
def array_prepend(column: Column, element: Any): Column
def array_remove(column: Column, element: Any): Column
def array_repeat(e: Column, count: Int): Column
def array_repeat(e: Column, count: Column): Column
def array_size(e: Column): Column

Returns the total number of elements in the array.

Returns the total number of elements in the array.

Attributes

def array_sort(e: Column, comparator: (Column, Column) => Column): Column

Sorts the input array based on the given comparator function.

Sorts the input array based on the given comparator function.

Attributes

def array_union(col1: Column, col2: Column): Column
def asc(columnName: String): Column
def asc_nulls_first(columnName: String): Column
def asc_nulls_last(columnName: String): Column
def ascii(e: Column): Column
def asin(e: Column): Column
def asin(columnName: String): Column
def asinh(e: Column): Column
def asinh(columnName: String): Column
def assert_true(c: Column): Column

Returns null if the condition is true, and throws an exception otherwise.

Returns null if the condition is true, and throws an exception otherwise.

Attributes

def assert_true(c: Column, e: Column): Column

Returns null if the condition is true; throws an exception with the error message otherwise.

Returns null if the condition is true; throws an exception with the error message otherwise.

Attributes

def atan(e: Column): Column
def atan(columnName: String): Column
def atan2(y: Column, x: Column): Column
def atan2(y: Column, xName: String): Column
def atan2(yName: String, x: Column): Column
def atan2(yName: String, xName: String): Column
def atan2(y: Column, xValue: Double): Column
def atan2(yValue: Double, x: Column): Column
def atan2(yName: String, xValue: Double): Column
def atan2(yValue: Double, xName: String): Column
def atanh(e: Column): Column
def atanh(columnName: String): Column
def avg(e: Column): Column
def avg(columnName: String): Column
def base64(e: Column): Column
def bin(e: Column): Column
def bin(columnName: String): Column
def bit_and(e: Column): Column
def bit_get(e: Column, pos: Column): Column

Returns the value of the bit (0 or 1) at the specified position.

Returns the value of the bit (0 or 1) at the specified position.

Attributes

def bit_or(e: Column): Column
def bit_xor(e: Column): Column
def bitmap_and_agg(col: Column): Column

Returns a bitmap that is the bitwise AND of all of the bitmaps from the input column.

Returns a bitmap that is the bitwise AND of all of the bitmaps from the input column.

Attributes

def bitmap_bit_position(col: Column): Column

Returns the bit position for the given input column.

Returns the bit position for the given input column.

Attributes

def bitmap_bucket_number(col: Column): Column

Returns the bucket number for the given input column.

Returns the bucket number for the given input column.

Attributes

def bitmap_construct_agg(col: Column): Column

Returns a bitmap with the positions of the bits set from all the values from the input column.

Returns a bitmap with the positions of the bits set from all the values from the input column.

Attributes

def bitmap_count(col: Column): Column

Returns the number of set bits in the input bitmap.

Returns the number of set bits in the input bitmap.

Attributes

def bitmap_or_agg(col: Column): Column

Returns a bitmap that is the bitwise OR of all of the bitmaps from the input column.

Returns a bitmap that is the bitwise OR of all of the bitmaps from the input column.

Attributes

def bool_or(e: Column): Column
def broadcast[U](df: Dataset[U]): Dataset[U]
def bround(e: Column): Column
def bround(e: Column, scale: Int): Column
def bround(e: Column, scale: Column): Column

Round the value of e to scale decimal places with HALF_EVEN round mode if scale is greater than or equal to 0 or at integral part when scale is less than 0.

Round the value of e to scale decimal places with HALF_EVEN round mode if scale is greater than or equal to 0 or at integral part when scale is less than 0.

Attributes

def btrim(str: Column): Column
def btrim(str: Column, trim: Column): Column
def bucket(numBuckets: Column, e: Column): Column

(Java-specific) A transform for any type that partitions by a hash of the input column.

(Java-specific) A transform for any type that partitions by a hash of the input column.

Attributes

def bucket(numBuckets: Int, e: Column): Column

(Java-specific) A transform for any type that partitions by a hash of the input column.

(Java-specific) A transform for any type that partitions by a hash of the input column.

Attributes

def call_function(funcName: String, cols: Column*): Column

Calls a SQL function by name, which may be a built-in function, a temporary function, or a persistent (possibly qualified, e.g. my_db.my_fn) function.

Calls a SQL function by name, which may be a built-in function, a temporary function, or a persistent (possibly qualified, e.g. my_db.my_fn) function.

Attributes

def call_udf(udfName: String, cols: Column*): Column

Call an user-defined function.

Call an user-defined function.

Attributes

def cbrt(e: Column): Column
def cbrt(columnName: String): Column
def ceil(e: Column): Column
def ceil(columnName: String): Column
def ceil(e: Column, scale: Column): Column
def ceiling(e: Column): Column
def ceiling(e: Column, scale: Column): Column

Computes the ceiling of the given value of e to scale decimal places.

Computes the ceiling of the given value of e to scale decimal places.

Attributes

def char(n: Column): Column

Returns the ASCII character having the binary equivalent to n.

Returns the ASCII character having the binary equivalent to n.

Attributes

def chr(n: Column): Column

Returns the ASCII character having the binary equivalent to n.

Returns the ASCII character having the binary equivalent to n.

Attributes

def coalesce(e: Column*): Column
def col(colName: String): Column

Returns a Column based on the given column name.

Returns a Column based on the given column name.

Attributes

def collate(e: Column, collation: String): Column

Marks a given column with specified collation.

Marks a given column with specified collation.

Attributes

def collation(e: Column): Column

Returns the collation name of a given column.

Returns the collation name of a given column.

Attributes

def collect_list(columnName: String): Column
def collect_set(columnName: String): Column
def collect_union(e: Column): Column

Aggregate function: returns the distinct union of the elements of an array-typed column across rows.

Aggregate function: returns the distinct union of the elements of an array-typed column across rows.

Attributes

def collect_union(columnName: String): Column

Aggregate function: returns the distinct union of the elements of an array-typed column across rows.

Aggregate function: returns the distinct union of the elements of an array-typed column across rows.

Attributes

def column(colName: String): Column

Returns a Column based on the given column name. Alias of col.

Returns a Column based on the given column name. Alias of col.

Attributes

def concat(exprs: Column*): Column
def concat_ws(sep: String, exprs: Column*): Column
def contains(left: Column, right: Column): Column
def conv(num: Column, fromBase: Int, toBase: Int): Column
def convert_timezone(sourceTz: Column, targetTz: Column, sourceTs: Column): Column

Converts the timestamp without time zone sourceTs from the sourceTz time zone to targetTz.

Converts the timestamp without time zone sourceTs from the sourceTz time zone to targetTz.

Attributes

def convert_timezone(targetTz: Column, sourceTs: Column): Column

Converts the timestamp without time zone sourceTs from the current time zone to targetTz.

Converts the timestamp without time zone sourceTs from the current time zone to targetTz.

Attributes

def corr(column1: Column, column2: Column): Column
def corr(columnName1: String, columnName2: String): Column
def cos(e: Column): Column
def cos(columnName: String): Column
def cosh(e: Column): Column
def cosh(columnName: String): Column
def cot(e: Column): Column
def count(e: Column): Column
def count(columnName: String): Column
def countDistinct(expr: Column, exprs: Column*): Column
def countDistinct(columnName: String, columnNames: String*): Column
def count_distinct(expr: Column, exprs: Column*): Column
def count_min_sketch(e: Column, eps: Column, confidence: Column, seed: Column): Column

Returns a count-min sketch of a column with the given esp, confidence and seed.

Returns a count-min sketch of a column with the given esp, confidence and seed.

Attributes

def count_min_sketch(e: Column, eps: Column, confidence: Column): Column

Returns a count-min sketch of a column with the given esp, confidence and seed.

Returns a count-min sketch of a column with the given esp, confidence and seed.

Attributes

def counter_diff(value: Column): Column

Window function: computes the differences between consecutive cumulative counter values in a time series, thereby converting the counter from the cumulative to the delta format.

Window function: computes the differences between consecutive cumulative counter values in a time series, thereby converting the counter from the cumulative to the delta format.

Attributes

def counter_diff(value: Column, startTime: Column): Column

Window function: computes the differences between consecutive cumulative counter values in a time series, thereby converting the counter from the cumulative to the delta format.

Window function: computes the differences between consecutive cumulative counter values in a time series, thereby converting the counter from the cumulative to the delta format.

Attributes

def covar_pop(column1: Column, column2: Column): Column
def covar_pop(columnName1: String, columnName2: String): Column
def covar_samp(column1: Column, column2: Column): Column
def covar_samp(columnName1: String, columnName2: String): Column
def crc32(e: Column): Column
def csc(e: Column): Column
def cume_dist(): Column
def curdate(): Column

Returns the current date at the start of query evaluation as a date column.

Returns the current date at the start of query evaluation as a date column.

Attributes

def current_path(): Column

Returns the current SQL path as a comma-separated list of qualified schema names.

Returns the current SQL path as a comma-separated list of qualified schema names.

Attributes

def current_time(): Column

Returns the current time at the start of query evaluation.

Returns the current time at the start of query evaluation.

Attributes

def current_time(precision: Int): Column

Returns the current time at the start of query evaluation.

Returns the current time at the start of query evaluation.

Attributes

def date_add(start: Column, days: Int): Column
def date_add(start: Column, days: Column): Column
def date_diff(end: Column, start: Column): Column
def date_format(dateExpr: Column, format: String): Column
def date_part(field: Column, source: Column): Column
def date_sub(start: Column, days: Int): Column
def date_sub(start: Column, days: Column): Column
def date_trunc(format: String, timestamp: Column): Column
def dateadd(start: Column, days: Column): Column

Returns the date that is days days after start

Returns the date that is days days after start

Attributes

def datediff(end: Column, start: Column): Column
def datepart(field: Column, source: Column): Column
def day(e: Column): Column
def dayname(timeExp: Column): Column

Extracts the three-letter abbreviated day name from a given date/timestamp/string.

Extracts the three-letter abbreviated day name from a given date/timestamp/string.

Attributes

def days(e: Column): Column

(Java-specific) A transform for timestamps and dates to partition data into days.

(Java-specific) A transform for timestamps and dates to partition data into days.

Attributes

def decode(value: Column, charset: String): Column
def degrees(e: Column): Column
def degrees(columnName: String): Column
def desc(columnName: String): Column
def desc_nulls_first(columnName: String): Column
def desc_nulls_last(columnName: String): Column
def e(): Column

Returns Euler's number.

Returns Euler's number.

Attributes

def element_at(column: Column, value: Any): Column
def elt(inputs: Column*): Column

Returns the n-th input, e.g., returns input2 when n is 2.

Returns the n-th input, e.g., returns input2 when n is 2.

Attributes

def encode(value: Column, charset: String): Column
def endswith(str: Column, suffix: Column): Column
def equal_null(col1: Column, col2: Column): Column

Returns same result as the EQUAL(=) operator for non-null operands, but returns true if both are null, false if one of the them is null.

Returns same result as the EQUAL(=) operator for non-null operands, but returns true if both are null, false if one of the them is null.

Attributes

def every(e: Column): Column
def exists(column: Column, f: Column => Column): Column

True if the predicate holds for any element of the array.

True if the predicate holds for any element of the array.

Attributes

def exp(e: Column): Column
def exp(columnName: String): Column
def explode(e: Column): Column
def expm1(e: Column): Column
def expm1(columnName: String): Column
def expr(expr: String): Column

Parses the expression string into the column it represents.

Parses the expression string into the column it represents.

Attributes

def extract(field: Column, source: Column): Column
def filter(column: Column, f: Column => Column): Column

Filters an array keeping elements for which the predicate holds.

Filters an array keeping elements for which the predicate holds.

Attributes

def filter(column: Column, f: (Column, Column) => Column): Column

Filters an array using the (element, index) predicate.

Filters an array using the (element, index) predicate.

Attributes

def find_in_set(str: Column, strArray: Column): Column

Returns the index (1-based) of the given string (str) in the comma-delimited list (strArray).

Returns the index (1-based) of the given string (str) in the comma-delimited list (strArray).

Attributes

def first(e: Column): Column
def first(columnName: String): Column
def first(e: Column, ignoreNulls: Boolean): Column
def first(columnName: String, ignoreNulls: Boolean): Column

Aggregate function: returns the first value of a column in a group.

Aggregate function: returns the first value of a column in a group.

Attributes

def first_value(e: Column, ignoreNulls: Column): Column
def flatten(e: Column): Column
def floor(e: Column): Column
def floor(columnName: String): Column
def floor(e: Column, scale: Column): Column
def forall(column: Column, f: Column => Column): Column

True if the predicate holds for every element of the array.

True if the predicate holds for every element of the array.

Attributes

def format_number(x: Column, d: Int): Column
def format_string(format: String, arguments: Column*): Column
def from_base32(e: Column): Column

Decodes a BASE32 (RFC 4648) encoded string column and returns it as a binary column.

Decodes a BASE32 (RFC 4648) encoded string column and returns it as a binary column.

Attributes

def from_csv(e: Column, schema: Column): Column
def from_csv(e: Column, schema: StructType, options: Map[String, String]): Column

Parses a column containing a CSV string into a StructType with the specified schema.

Parses a column containing a CSV string into a StructType with the specified schema.

Attributes

def from_csv(e: Column, schema: Column, options: Map[String, String]): Column

(Java-specific) Parses a column containing a CSV string into a StructType with the specified schema.

(Java-specific) Parses a column containing a CSV string into a StructType with the specified schema.

Attributes

def from_json(e: Column, schema: String): Column
def from_json(e: Column, schema: Column): Column
def from_json(e: Column, schema: StructType, options: Map[String, String]): Column

(Scala-specific) Parses a column containing a JSON string into a StructType with the specified schema.

(Scala-specific) Parses a column containing a JSON string into a StructType with the specified schema.

Attributes

def from_json(e: Column, schema: DataType, options: Map[String, String]): Column

(Scala-specific) Parses a column containing a JSON string into a MapType with StringType as keys type, StructType or ArrayType with the specified schema.

(Scala-specific) Parses a column containing a JSON string into a MapType with StringType as keys type, StructType or ArrayType with the specified schema.

Attributes

def from_json(e: Column, schema: StructType, options: Map[String, String]): Column

(Java-specific) Parses a column containing a JSON string into a StructType with the specified schema.

(Java-specific) Parses a column containing a JSON string into a StructType with the specified schema.

Attributes

def from_json(e: Column, schema: DataType, options: Map[String, String]): Column

(Java-specific) Parses a column containing a JSON string into a MapType with StringType as keys type, StructType or ArrayType with the specified schema.

(Java-specific) Parses a column containing a JSON string into a MapType with StringType as keys type, StructType or ArrayType with the specified schema.

Attributes

def from_json(e: Column, schema: StructType): Column

Parses a column containing a JSON string into a StructType with the specified schema.

Parses a column containing a JSON string into a StructType with the specified schema.

Attributes

def from_json(e: Column, schema: DataType): Column

Parses a column containing a JSON string into a MapType with StringType as keys type, StructType or ArrayType with the specified schema.

Parses a column containing a JSON string into a MapType with StringType as keys type, StructType or ArrayType with the specified schema.

Attributes

def from_json(e: Column, schema: String, options: Map[String, String]): Column

(Java-specific) Parses a column containing a JSON string into a MapType with StringType as keys type, StructType or ArrayType with the specified schema.

(Java-specific) Parses a column containing a JSON string into a MapType with StringType as keys type, StructType or ArrayType with the specified schema.

Attributes

def from_json(e: Column, schema: String, options: Map[String, String]): Column

(Scala-specific) Parses a column containing a JSON string into a MapType with StringType as keys type, StructType or ArrayType with the specified schema.

(Scala-specific) Parses a column containing a JSON string into a MapType with StringType as keys type, StructType or ArrayType with the specified schema.

Attributes

def from_json(e: Column, schema: Column, options: Map[String, String]): Column

(Java-specific) Parses a column containing a JSON string into a MapType with StringType as keys type, StructType or ArrayType of StructTypes with the specified schema.

(Java-specific) Parses a column containing a JSON string into a MapType with StringType as keys type, StructType or ArrayType of StructTypes with the specified schema.

Attributes

def from_unixtime(ut: Column, f: String): Column
def from_utc_timestamp(ts: Column, tz: String): Column
def from_xml(e: Column, schema: StructType, options: Map[String, String]): Column

Parses a column containing a XML string into the data type corresponding to the specified schema.

Parses a column containing a XML string into the data type corresponding to the specified schema.

Attributes

def from_xml(e: Column, schema: String, options: Map[String, String]): Column

(Java-specific) Parses a column containing a XML string into a StructType with the specified schema.

(Java-specific) Parses a column containing a XML string into a StructType with the specified schema.

Attributes

def from_xml(e: Column, schema: Column): Column

(Java-specific) Parses a column containing a XML string into a StructType with the specified schema.

(Java-specific) Parses a column containing a XML string into a StructType with the specified schema.

Attributes

def from_xml(e: Column, schema: Column, options: Map[String, String]): Column

(Java-specific) Parses a column containing a XML string into a StructType with the specified schema.

(Java-specific) Parses a column containing a XML string into a StructType with the specified schema.

Attributes

def from_xml(e: Column, schema: StructType): Column

Parses a column containing a XML string into the data type corresponding to the specified schema.

Parses a column containing a XML string into the data type corresponding to the specified schema.

Attributes

def get(column: Column, index: Column): Column
def get_json_object(e: Column, path: String): Column
def getbit(e: Column, pos: Column): Column

Returns the value of the bit (0 or 1) at the specified position.

Returns the value of the bit (0 or 1) at the specified position.

Attributes

def greatest(exprs: Column*): Column
def greatest(columnName: String, columnNames: String*): Column

Returns the greatest value of the list of column names, skipping null values.

Returns the greatest value of the list of column names, skipping null values.

Attributes

def grouping(columnName: String): Column
def grouping_id(cols: Column*): Column
def grouping_id(colName: String, colNames: String*): Column
def hash(cols: Column*): Column
def hex(column: Column): Column
def histogram_numeric(e: Column, nBins: Column): Column

Aggregate function: computes a histogram on numeric 'expr' using nb bins.

Aggregate function: computes a histogram on numeric 'expr' using nb bins.

Attributes

def hll_sketch_agg(e: Column, lgConfigK: Column): Column

Aggregate function: returns the updatable binary representation of the Datasketches HllSketch configured with lgConfigK arg.

Aggregate function: returns the updatable binary representation of the Datasketches HllSketch configured with lgConfigK arg.

Attributes

def hll_sketch_agg(e: Column, lgConfigK: Int): Column

Aggregate function: returns the updatable binary representation of the Datasketches HllSketch configured with lgConfigK arg.

Aggregate function: returns the updatable binary representation of the Datasketches HllSketch configured with lgConfigK arg.

Attributes

def hll_sketch_agg(columnName: String, lgConfigK: Int): Column

Aggregate function: returns the updatable binary representation of the Datasketches HllSketch configured with lgConfigK arg.

Aggregate function: returns the updatable binary representation of the Datasketches HllSketch configured with lgConfigK arg.

Attributes

def hll_sketch_agg(e: Column): Column

Aggregate function: returns the updatable binary representation of the Datasketches HllSketch configured with default lgConfigK value.

Aggregate function: returns the updatable binary representation of the Datasketches HllSketch configured with default lgConfigK value.

Attributes

def hll_sketch_agg(columnName: String): Column

Aggregate function: returns the updatable binary representation of the Datasketches HllSketch configured with default lgConfigK value.

Aggregate function: returns the updatable binary representation of the Datasketches HllSketch configured with default lgConfigK value.

Attributes

def hll_sketch_estimate(c: Column): Column

Returns the estimated number of unique values given the binary representation of a Datasketches HllSketch.

Returns the estimated number of unique values given the binary representation of a Datasketches HllSketch.

Attributes

def hll_sketch_estimate(columnName: String): Column

Returns the estimated number of unique values given the binary representation of a Datasketches HllSketch.

Returns the estimated number of unique values given the binary representation of a Datasketches HllSketch.

Attributes

def hll_union(c1: Column, c2: Column): Column

Merges two binary representations of Datasketches HllSketch objects, using a Datasketches Union object.

Merges two binary representations of Datasketches HllSketch objects, using a Datasketches Union object.

Attributes

def hll_union(columnName1: String, columnName2: String): Column

Merges two binary representations of Datasketches HllSketch objects, using a Datasketches Union object.

Merges two binary representations of Datasketches HllSketch objects, using a Datasketches Union object.

Attributes

def hll_union(c1: Column, c2: Column, allowDifferentLgConfigK: Boolean): Column

Merges two binary representations of Datasketches HllSketch objects, using a Datasketches Union object.

Merges two binary representations of Datasketches HllSketch objects, using a Datasketches Union object.

Attributes

def hll_union(columnName1: String, columnName2: String, allowDifferentLgConfigK: Boolean): Column

Merges two binary representations of Datasketches HllSketch objects, using a Datasketches Union object.

Merges two binary representations of Datasketches HllSketch objects, using a Datasketches Union object.

Attributes

def hll_union_agg(e: Column, allowDifferentLgConfigK: Column): Column

Aggregate function: returns the updatable binary representation of the Datasketches HllSketch, generated by merging previously created Datasketches HllSketch instances via a Datasketches Union instance.

Aggregate function: returns the updatable binary representation of the Datasketches HllSketch, generated by merging previously created Datasketches HllSketch instances via a Datasketches Union instance.

Attributes

def hll_union_agg(e: Column, allowDifferentLgConfigK: Boolean): Column

Aggregate function: returns the updatable binary representation of the Datasketches HllSketch, generated by merging previously created Datasketches HllSketch instances via a Datasketches Union instance.

Aggregate function: returns the updatable binary representation of the Datasketches HllSketch, generated by merging previously created Datasketches HllSketch instances via a Datasketches Union instance.

Attributes

def hll_union_agg(columnName: String, allowDifferentLgConfigK: Boolean): Column

Aggregate function: returns the updatable binary representation of the Datasketches HllSketch, generated by merging previously created Datasketches HllSketch instances via a Datasketches Union instance.

Aggregate function: returns the updatable binary representation of the Datasketches HllSketch, generated by merging previously created Datasketches HllSketch instances via a Datasketches Union instance.

Attributes

def hll_union_agg(e: Column): Column

Aggregate function: returns the updatable binary representation of the Datasketches HllSketch, generated by merging previously created Datasketches HllSketch instances via a Datasketches Union instance.

Aggregate function: returns the updatable binary representation of the Datasketches HllSketch, generated by merging previously created Datasketches HllSketch instances via a Datasketches Union instance.

Attributes

def hll_union_agg(columnName: String): Column

Aggregate function: returns the updatable binary representation of the Datasketches HllSketch, generated by merging previously created Datasketches HllSketch instances via a Datasketches Union instance.

Aggregate function: returns the updatable binary representation of the Datasketches HllSketch, generated by merging previously created Datasketches HllSketch instances via a Datasketches Union instance.

Attributes

def hmac(key: Column, message: Column, algorithm: Column): Column

Returns the keyed-hash message authentication code (HMAC) of message using key and the given hash algorithm.

Returns the keyed-hash message authentication code (HMAC) of message using key and the given hash algorithm.

Attributes

def hmac(key: Column, message: Column): Column

Returns the keyed-hash message authentication code (HMAC) of message using key and SHA-256.

Returns the keyed-hash message authentication code (HMAC) of message using key and SHA-256.

Attributes

def hour(e: Column): Column
def hours(e: Column): Column

(Java-specific) A transform for timestamps to partition data into hours.

(Java-specific) A transform for timestamps to partition data into hours.

Attributes

def hypot(l: Column, r: Column): Column
def hypot(l: Column, rightName: String): Column
def hypot(leftName: String, r: Column): Column
def hypot(leftName: String, rightName: String): Column
def hypot(l: Column, r: Double): Column
def hypot(l: Double, r: Column): Column
def hypot(leftName: String, r: Double): Column

Computes sqrt(a^2^ + b^2^) without intermediate overflow or underflow.

Computes sqrt(a^2^ + b^2^) without intermediate overflow or underflow.

Attributes

def hypot(l: Double, rightName: String): Column

Computes sqrt(a^2^ + b^2^) without intermediate overflow or underflow.

Computes sqrt(a^2^ + b^2^) without intermediate overflow or underflow.

Attributes

def ifnull(col1: Column, col2: Column): Column
def ilike(str: Column, pattern: Column, escapeChar: Column): Column

Returns true if str matches pattern with escapeChar case-insensitively, null if any arguments are null, false otherwise.

Returns true if str matches pattern with escapeChar case-insensitively, null if any arguments are null, false otherwise.

Attributes

def ilike(str: Column, pattern: Column): Column

Returns true if str matches pattern with escapeChar('') case-insensitively, null if any arguments are null, false otherwise.

Returns true if str matches pattern with escapeChar('') case-insensitively, null if any arguments are null, false otherwise.

Attributes

def initcap(e: Column): Column
def inline(e: Column): Column
def instr(str: Column, substring: String): Column
def instr(str: Column, substring: Column): Column

Locate the position of the first occurrence of substr column in the given string.

Locate the position of the first occurrence of substr column in the given string.

Attributes

def instr(str: Column, substring: Column, start: Int): Column

Locate the position of the first occurrence of substring in str, starting the search from position start.

Locate the position of the first occurrence of substring in str, starting the search from position start.

Attributes

def instr(str: Column, substring: Column, start: Column): Column

Locate the position of the first occurrence of substring in str, starting the search from position start.

Locate the position of the first occurrence of substring in str, starting the search from position start.

Attributes

def instr(str: Column, substring: Column, start: Int, occurrence: Int): Column

Locate the position of the occurrence-th occurrence of substring in str, starting the search from position start.

Locate the position of the occurrence-th occurrence of substring in str, starting the search from position start.

Attributes

def instr(str: Column, substring: Column, start: Column, occurrence: Column): Column

Locate the position of the occurrence-th occurrence of substring in str, starting the search from position start.

Locate the position of the occurrence-th occurrence of substring in str, starting the search from position start.

Attributes

def is_valid_utf8(str: Column): Column

Returns true if the input is a valid UTF-8 string, otherwise returns false.

Returns true if the input is a valid UTF-8 string, otherwise returns false.

Attributes

def is_valid_variant(v: Column): Column

Check if a variant value is valid.

Check if a variant value is valid.

Attributes

def is_variant_null(v: Column): Column

Check if a variant value is a variant null.

Check if a variant value is a variant null.

Attributes

def isnan(e: Column): Column
def isnotnull(col: Column): Column

Returns true if col is not null, or false otherwise.

Returns true if col is not null, or false otherwise.

Attributes

def isnull(e: Column): Column
def jaro_winkler_similarity(l: Column, r: Column): Column

Computes the Jaro-Winkler similarity between the two given string columns.

Computes the Jaro-Winkler similarity between the two given string columns.

Attributes

def java_method(cols: Column*): Column

Calls a method with reflection.

Calls a method with reflection.

Attributes

def json_array_length(e: Column): Column

Returns the number of elements in the outermost JSON array.

Returns the number of elements in the outermost JSON array.

Attributes

def json_object_keys(e: Column): Column

Returns all the keys of the outermost JSON object as an array.

Returns all the keys of the outermost JSON object as an array.

Attributes

def json_tuple(json: Column, fields: String*): Column
def kll_merge_agg_bigint(e: Column, k: Column): Column

Aggregate function: merges binary KllLongsSketch representations and returns the merged sketch.

Aggregate function: merges binary KllLongsSketch representations and returns the merged sketch.

Attributes

def kll_merge_agg_bigint(e: Column, k: Int): Column

Aggregate function: merges binary KllLongsSketch representations and returns the merged sketch.

Aggregate function: merges binary KllLongsSketch representations and returns the merged sketch.

Attributes

def kll_merge_agg_bigint(columnName: String, k: Int): Column

Aggregate function: merges binary KllLongsSketch representations and returns the merged sketch.

Aggregate function: merges binary KllLongsSketch representations and returns the merged sketch.

Attributes

def kll_merge_agg_bigint(e: Column): Column

Aggregate function: merges binary KllLongsSketch representations and returns the merged sketch.

Aggregate function: merges binary KllLongsSketch representations and returns the merged sketch.

Attributes

def kll_merge_agg_bigint(columnName: String): Column

Aggregate function: merges binary KllLongsSketch representations and returns the merged sketch.

Aggregate function: merges binary KllLongsSketch representations and returns the merged sketch.

Attributes

def kll_merge_agg_double(e: Column, k: Column): Column

Aggregate function: merges binary KllDoublesSketch representations and returns merged sketch.

Aggregate function: merges binary KllDoublesSketch representations and returns merged sketch.

Attributes

def kll_merge_agg_double(e: Column, k: Int): Column

Aggregate function: merges binary KllDoublesSketch representations and returns merged sketch.

Aggregate function: merges binary KllDoublesSketch representations and returns merged sketch.

Attributes

def kll_merge_agg_double(columnName: String, k: Int): Column

Aggregate function: merges binary KllDoublesSketch representations and returns merged sketch.

Aggregate function: merges binary KllDoublesSketch representations and returns merged sketch.

Attributes

def kll_merge_agg_double(e: Column): Column

Aggregate function: merges binary KllDoublesSketch representations and returns merged sketch.

Aggregate function: merges binary KllDoublesSketch representations and returns merged sketch.

Attributes

def kll_merge_agg_double(columnName: String): Column

Aggregate function: merges binary KllDoublesSketch representations and returns merged sketch.

Aggregate function: merges binary KllDoublesSketch representations and returns merged sketch.

Attributes

def kll_merge_agg_float(e: Column, k: Column): Column

Aggregate function: merges binary KllFloatsSketch representations and returns merged sketch.

Aggregate function: merges binary KllFloatsSketch representations and returns merged sketch.

Attributes

def kll_merge_agg_float(e: Column, k: Int): Column

Aggregate function: merges binary KllFloatsSketch representations and returns merged sketch.

Aggregate function: merges binary KllFloatsSketch representations and returns merged sketch.

Attributes

def kll_merge_agg_float(columnName: String, k: Int): Column

Aggregate function: merges binary KllFloatsSketch representations and returns merged sketch.

Aggregate function: merges binary KllFloatsSketch representations and returns merged sketch.

Attributes

def kll_merge_agg_float(e: Column): Column

Aggregate function: merges binary KllFloatsSketch representations and returns merged sketch.

Aggregate function: merges binary KllFloatsSketch representations and returns merged sketch.

Attributes

def kll_merge_agg_float(columnName: String): Column

Aggregate function: merges binary KllFloatsSketch representations and returns merged sketch.

Aggregate function: merges binary KllFloatsSketch representations and returns merged sketch.

Attributes

def kll_sketch_agg_bigint(e: Column, k: Column): Column

Aggregate function: returns the compact binary representation of the Datasketches KllLongsSketch built with the values in the input column.

Aggregate function: returns the compact binary representation of the Datasketches KllLongsSketch built with the values in the input column.

Attributes

def kll_sketch_agg_bigint(e: Column, k: Int): Column

Aggregate function: returns the compact binary representation of the Datasketches KllLongsSketch built with the values in the input column.

Aggregate function: returns the compact binary representation of the Datasketches KllLongsSketch built with the values in the input column.

Attributes

def kll_sketch_agg_bigint(columnName: String, k: Int): Column

Aggregate function: returns the compact binary representation of the Datasketches KllLongsSketch built with the values in the input column.

Aggregate function: returns the compact binary representation of the Datasketches KllLongsSketch built with the values in the input column.

Attributes

def kll_sketch_agg_bigint(e: Column): Column

Aggregate function: returns the compact binary representation of the Datasketches KllLongsSketch built with the values in the input column with default k value of 200.

Aggregate function: returns the compact binary representation of the Datasketches KllLongsSketch built with the values in the input column with default k value of 200.

Attributes

def kll_sketch_agg_bigint(columnName: String): Column

Aggregate function: returns the compact binary representation of the Datasketches KllLongsSketch built with the values in the input column with default k value of 200.

Aggregate function: returns the compact binary representation of the Datasketches KllLongsSketch built with the values in the input column with default k value of 200.

Attributes

def kll_sketch_agg_double(e: Column, k: Column): Column

Aggregate function: returns the compact binary representation of the Datasketches KllDoublesSketch built with the values in the input column.

Aggregate function: returns the compact binary representation of the Datasketches KllDoublesSketch built with the values in the input column.

Attributes

def kll_sketch_agg_double(e: Column, k: Int): Column

Aggregate function: returns the compact binary representation of the Datasketches KllDoublesSketch built with the values in the input column.

Aggregate function: returns the compact binary representation of the Datasketches KllDoublesSketch built with the values in the input column.

Attributes

def kll_sketch_agg_double(columnName: String, k: Int): Column

Aggregate function: returns the compact binary representation of the Datasketches KllDoublesSketch built with the values in the input column.

Aggregate function: returns the compact binary representation of the Datasketches KllDoublesSketch built with the values in the input column.

Attributes

def kll_sketch_agg_double(e: Column): Column

Aggregate function: returns the compact binary representation of the Datasketches KllDoublesSketch built with the values in the input column with default k value of 200.

Aggregate function: returns the compact binary representation of the Datasketches KllDoublesSketch built with the values in the input column with default k value of 200.

Attributes

def kll_sketch_agg_double(columnName: String): Column

Aggregate function: returns the compact binary representation of the Datasketches KllDoublesSketch built with the values in the input column with default k value of 200.

Aggregate function: returns the compact binary representation of the Datasketches KllDoublesSketch built with the values in the input column with default k value of 200.

Attributes

def kll_sketch_agg_float(e: Column, k: Column): Column

Aggregate function: returns the compact binary representation of the Datasketches KllFloatsSketch built with the values in the input column.

Aggregate function: returns the compact binary representation of the Datasketches KllFloatsSketch built with the values in the input column.

Attributes

def kll_sketch_agg_float(e: Column, k: Int): Column

Aggregate function: returns the compact binary representation of the Datasketches KllFloatsSketch built with the values in the input column.

Aggregate function: returns the compact binary representation of the Datasketches KllFloatsSketch built with the values in the input column.

Attributes

def kll_sketch_agg_float(columnName: String, k: Int): Column

Aggregate function: returns the compact binary representation of the Datasketches KllFloatsSketch built with the values in the input column.

Aggregate function: returns the compact binary representation of the Datasketches KllFloatsSketch built with the values in the input column.

Attributes

def kll_sketch_agg_float(e: Column): Column

Aggregate function: returns the compact binary representation of the Datasketches KllFloatsSketch built with the values in the input column with default k value of 200.

Aggregate function: returns the compact binary representation of the Datasketches KllFloatsSketch built with the values in the input column with default k value of 200.

Attributes

def kll_sketch_agg_float(columnName: String): Column

Aggregate function: returns the compact binary representation of the Datasketches KllFloatsSketch built with the values in the input column with default k value of 200.

Aggregate function: returns the compact binary representation of the Datasketches KllFloatsSketch built with the values in the input column with default k value of 200.

Attributes

def kll_sketch_get_n_bigint(e: Column): Column

Returns the number of items collected in the KLL bigint sketch.

Returns the number of items collected in the KLL bigint sketch.

Attributes

def kll_sketch_get_n_double(e: Column): Column

Returns the number of items collected in the KLL double sketch.

Returns the number of items collected in the KLL double sketch.

Attributes

def kll_sketch_get_n_float(e: Column): Column

Returns the number of items collected in the KLL float sketch.

Returns the number of items collected in the KLL float sketch.

Attributes

def kll_sketch_get_quantile_bigint(sketch: Column, rank: Column): Column

Extracts a quantile value from a KLL bigint sketch given an input rank value.

Extracts a quantile value from a KLL bigint sketch given an input rank value.

Attributes

def kll_sketch_get_quantile_double(sketch: Column, rank: Column): Column

Extracts a quantile value from a KLL double sketch given an input rank value.

Extracts a quantile value from a KLL double sketch given an input rank value.

Attributes

def kll_sketch_get_quantile_float(sketch: Column, rank: Column): Column

Extracts a quantile value from a KLL float sketch given an input rank value.

Extracts a quantile value from a KLL float sketch given an input rank value.

Attributes

def kll_sketch_get_rank_bigint(sketch: Column, quantile: Column): Column

Extracts a rank value from a KLL bigint sketch given an input quantile value.

Extracts a rank value from a KLL bigint sketch given an input quantile value.

Attributes

def kll_sketch_get_rank_double(sketch: Column, quantile: Column): Column

Extracts a rank value from a KLL double sketch given an input quantile value.

Extracts a rank value from a KLL double sketch given an input quantile value.

Attributes

def kll_sketch_get_rank_float(sketch: Column, quantile: Column): Column

Extracts a rank value from a KLL float sketch given an input quantile value.

Extracts a rank value from a KLL float sketch given an input quantile value.

Attributes

def kll_sketch_merge_bigint(left: Column, right: Column): Column

Merges two KLL bigint sketch buffers together into one.

Merges two KLL bigint sketch buffers together into one.

Attributes

def kll_sketch_merge_double(left: Column, right: Column): Column

Merges two KLL double sketch buffers together into one.

Merges two KLL double sketch buffers together into one.

Attributes

def kll_sketch_merge_float(left: Column, right: Column): Column

Merges two KLL float sketch buffers together into one.

Merges two KLL float sketch buffers together into one.

Attributes

def kll_sketch_to_string_bigint(e: Column): Column

Returns a string with human readable summary information about the KLL bigint sketch.

Returns a string with human readable summary information about the KLL bigint sketch.

Attributes

def kll_sketch_to_string_double(e: Column): Column

Returns a string with human readable summary information about the KLL double sketch.

Returns a string with human readable summary information about the KLL double sketch.

Attributes

def kll_sketch_to_string_float(e: Column): Column

Returns a string with human readable summary information about the KLL float sketch.

Returns a string with human readable summary information about the KLL float sketch.

Attributes

def kurtosis(columnName: String): Column
def lag(e: Column, offset: Int): Column
def lag(columnName: String, offset: Int): Column
def lag(e: Column, offset: Int, defaultValue: Any): Column
def lag(columnName: String, offset: Int, defaultValue: Any): Column

Window function: returns the value that is offset rows before the current row, and defaultValue if there is less than offset rows before the current row.

Window function: returns the value that is offset rows before the current row, and defaultValue if there is less than offset rows before the current row.

Attributes

def lag(e: Column, offset: Int, defaultValue: Any, ignoreNulls: Boolean): Column

Window function: returns the value that is offset rows before the current row, and defaultValue if there is less than offset rows before the current row.

Window function: returns the value that is offset rows before the current row, and defaultValue if there is less than offset rows before the current row.

Attributes

def last(e: Column): Column
def last(columnName: String): Column
def last(e: Column, ignoreNulls: Boolean): Column
def last(columnName: String, ignoreNulls: Boolean): Column

Aggregate function: returns the last value of the column in a group.

Aggregate function: returns the last value of the column in a group.

Attributes

def last_value(e: Column, ignoreNulls: Column): Column
def lcase(e: Column): Column
def lead(e: Column, offset: Int): Column
def lead(columnName: String, offset: Int): Column
def lead(e: Column, offset: Int, defaultValue: Any): Column
def lead(columnName: String, offset: Int, defaultValue: Any): Column

Window function: returns the value that is offset rows after the current row, and defaultValue if there is less than offset rows after the current row.

Window function: returns the value that is offset rows after the current row, and defaultValue if there is less than offset rows after the current row.

Attributes

def lead(e: Column, offset: Int, defaultValue: Any, ignoreNulls: Boolean): Column

Window function: returns the value that is offset rows after the current row, and defaultValue if there is less than offset rows after the current row.

Window function: returns the value that is offset rows after the current row, and defaultValue if there is less than offset rows after the current row.

Attributes

def least(exprs: Column*): Column
def least(columnName: String, columnNames: String*): Column

Returns the least value of the list of column names, skipping null values.

Returns the least value of the list of column names, skipping null values.

Attributes

def left(str: Column, len: Column): Column

Returns the leftmost len(len can be string type) characters from the string str, if len is less or equal than 0 the result is an empty string.

Returns the leftmost len(len can be string type) characters from the string str, if len is less or equal than 0 the result is an empty string.

Attributes

def len(e: Column): Column

Computes the character length of a given string or number of bytes of a binary string.

Computes the character length of a given string or number of bytes of a binary string.

Attributes

def length(e: Column): Column
def levenshtein(l: Column, r: Column, threshold: Int): Column
def like(str: Column, pattern: Column, escapeChar: Column): Column

Returns true if str matches pattern with escapeChar, null if any arguments are null, false otherwise.

Returns true if str matches pattern with escapeChar, null if any arguments are null, false otherwise.

Attributes

def like(str: Column, pattern: Column): Column

Returns true if str matches pattern with escapeChar(''), null if any arguments are null, false otherwise.

Returns true if str matches pattern with escapeChar(''), null if any arguments are null, false otherwise.

Attributes

def listagg(e: Column): Column

Aggregate function: returns the concatenation of non-null input values.

Aggregate function: returns the concatenation of non-null input values.

Attributes

def listagg(e: Column, delimiter: Column): Column

Aggregate function: returns the concatenation of non-null input values, separated by the delimiter.

Aggregate function: returns the concatenation of non-null input values, separated by the delimiter.

Attributes

def listagg_distinct(e: Column): Column

Aggregate function: returns the concatenation of distinct non-null input values.

Aggregate function: returns the concatenation of distinct non-null input values.

Attributes

def listagg_distinct(e: Column, delimiter: Column): Column

Aggregate function: returns the concatenation of distinct non-null input values, separated by the delimiter.

Aggregate function: returns the concatenation of distinct non-null input values, separated by the delimiter.

Attributes

def lit(literal: Any): Column

Creates a Column of literal value.

Creates a Column of literal value.

Attributes

def ln(e: Column): Column
def locate(substr: String, str: Column): Column
def locate(substr: String, str: Column, pos: Int): Column
def log(e: Column): Column
def log(columnName: String): Column
def log(base: Double, e: Column): Column
def log(base: Double, columnName: String): Column
def log10(e: Column): Column
def log10(columnName: String): Column
def log1p(e: Column): Column
def log1p(columnName: String): Column
def log2(e: Column): Column
def log2(columnName: String): Column
def lower(e: Column): Column
def lpad(str: Column, len: Int, pad: String): Column
def lpad(str: Column, len: Int, pad: Array[Byte]): Column

Left-pad the binary column with pad to a byte length of len.

Left-pad the binary column with pad to a byte length of len.

Attributes

def lpad(str: Column, len: Column, pad: Column): Column

Left-pad the string column with pad to a length of len.

Left-pad the string column with pad to a length of len.

Attributes

def ltrim(e: Column): Column
def ltrim(e: Column, trimString: String): Column
def ltrim(e: Column, trim: Column): Column

Trim the specified character string from left end for the specified string column.

Trim the specified character string from left end for the specified string column.

Attributes

def make_date(year: Column, month: Column, day: Column): Column
def make_dt_interval(days: Column, hours: Column, mins: Column, secs: Column): Column

Make DayTimeIntervalType duration from days, hours, mins and secs.

Make DayTimeIntervalType duration from days, hours, mins and secs.

Attributes

def make_dt_interval(days: Column, hours: Column, mins: Column): Column

Make DayTimeIntervalType duration from days, hours and mins.

Make DayTimeIntervalType duration from days, hours and mins.

Attributes

def make_dt_interval(days: Column, hours: Column): Column

Make DayTimeIntervalType duration from days and hours.

Make DayTimeIntervalType duration from days and hours.

Attributes

def make_dt_interval(days: Column): Column

Make DayTimeIntervalType duration from days.

Make DayTimeIntervalType duration from days.

Attributes

def make_dt_interval(): Column

Make DayTimeIntervalType duration.

Make DayTimeIntervalType duration.

Attributes

def make_interval(years: Column, months: Column, weeks: Column, days: Column, hours: Column, mins: Column, secs: Column): Column

Make interval from years, months, weeks, days, hours, mins and secs.

Make interval from years, months, weeks, days, hours, mins and secs.

Attributes

def make_interval(years: Column, months: Column, weeks: Column, days: Column, hours: Column, mins: Column): Column

Make interval from years, months, weeks, days, hours and mins.

Make interval from years, months, weeks, days, hours and mins.

Attributes

def make_interval(years: Column, months: Column, weeks: Column, days: Column, hours: Column): Column

Make interval from years, months, weeks, days and hours.

Make interval from years, months, weeks, days and hours.

Attributes

def make_interval(years: Column, months: Column, weeks: Column, days: Column): Column

Make interval from years, months, weeks and days.

Make interval from years, months, weeks and days.

Attributes

def make_interval(years: Column, months: Column, weeks: Column): Column

Make interval from years, months and weeks.

Make interval from years, months and weeks.

Attributes

def make_interval(years: Column, months: Column): Column

Make interval from years and months.

Make interval from years and months.

Attributes

def make_interval(years: Column): Column

Make interval from years.

Make interval from years.

Attributes

def make_interval(): Column

Make interval.

Make interval.

Attributes

def make_time(hour: Column, minute: Column, second: Column): Column

Create time from hour, minute and second fields.

Create time from hour, minute and second fields.

Attributes

def make_timestamp(years: Column, months: Column, days: Column, hours: Column, mins: Column, secs: Column): Column
def make_timestamp(years: Column, months: Column, days: Column, hours: Column, mins: Column, secs: Column, timezone: Column): Column

Create timestamp from years, months, days, hours, mins, secs and timezone fields.

Create timestamp from years, months, days, hours, mins, secs and timezone fields.

Attributes

def make_timestamp(date: Column, time: Column, timezone: Column): Column

Create a local date-time from date, time, and timezone fields.

Create a local date-time from date, time, and timezone fields.

Attributes

def make_timestamp(date: Column, time: Column): Column

Create a local date-time from date and time fields.

Create a local date-time from date and time fields.

Attributes

def make_timestamp_ltz(years: Column, months: Column, days: Column, hours: Column, mins: Column, secs: Column, timezone: Column): Column

Create the current timestamp with local time zone from years, months, days, hours, mins, secs and timezone fields.

Create the current timestamp with local time zone from years, months, days, hours, mins, secs and timezone fields.

Attributes

def make_timestamp_ltz(years: Column, months: Column, days: Column, hours: Column, mins: Column, secs: Column): Column

Create the current timestamp with local time zone from years, months, days, hours, mins and secs fields.

Create the current timestamp with local time zone from years, months, days, hours, mins and secs fields.

Attributes

def make_timestamp_ntz(years: Column, months: Column, days: Column, hours: Column, mins: Column, secs: Column): Column

Create local date-time from years, months, days, hours, mins, secs fields.

Create local date-time from years, months, days, hours, mins, secs fields.

Attributes

def make_timestamp_ntz(date: Column, time: Column): Column

Create a local date-time from date and time fields.

Create a local date-time from date and time fields.

Attributes

def make_valid_utf8(str: Column): Column

Returns a new string in which all invalid UTF-8 byte sequences, if any, are replaced by the Unicode replacement character (U+FFFD).

Returns a new string in which all invalid UTF-8 byte sequences, if any, are replaced by the Unicode replacement character (U+FFFD).

Attributes

def make_ym_interval(years: Column, months: Column): Column

Make year-month interval from years, months.

Make year-month interval from years, months.

Attributes

def make_ym_interval(years: Column): Column

Make year-month interval from years.

Make year-month interval from years.

Attributes

def make_ym_interval(): Column

Make year-month interval.

Make year-month interval.

Attributes

def map(cols: Column*): Column
def map_concat(cols: Column*): Column
def map_contains_key(column: Column, key: Any): Column
def map_filter(expr: Column, f: (Column, Column) => Column): Column

Filters a map keeping entries for which the (key, value) predicate holds.

Filters a map keeping entries for which the (key, value) predicate holds.

Attributes

def map_from_arrays(keys: Column, values: Column): Column
def map_zip_with(left: Column, right: Column, f: (Column, Column, Column) => Column): Column

Merges two maps by key using the (key, value1, value2) function.

Merges two maps by key using the (key, value1, value2) function.

Attributes

def mask(input: Column): Column

Masks the given string value.

Masks the given string value.

Attributes

def mask(input: Column, upperChar: Column): Column

Masks the given string value.

Masks the given string value.

Attributes

def mask(input: Column, upperChar: Column, lowerChar: Column): Column

Masks the given string value.

Masks the given string value.

Attributes

def mask(input: Column, upperChar: Column, lowerChar: Column, digitChar: Column): Column

Masks the given string value.

Masks the given string value.

Attributes

def mask(input: Column, upperChar: Column, lowerChar: Column, digitChar: Column, otherChar: Column): Column

Masks the given string value.

Masks the given string value.

Attributes

def max(e: Column): Column
def max(columnName: String): Column
def max_by(e: Column, ord: Column): Column
def max_by(e: Column, ord: Column, k: Int): Column

Aggregate function: returns an array of values associated with the top k values of ord.

Aggregate function: returns an array of values associated with the top k values of ord.

Attributes

def max_by(e: Column, ord: Column, k: Column): Column

Aggregate function: returns an array of values associated with the top k values of ord.

Aggregate function: returns an array of values associated with the top k values of ord.

Attributes

def md5(e: Column): Column
def mean(e: Column): Column
def mean(columnName: String): Column
def median(e: Column): Column
def min(e: Column): Column
def min(columnName: String): Column
def min_by(e: Column, ord: Column): Column
def min_by(e: Column, ord: Column, k: Int): Column

Aggregate function: returns an array of values associated with the bottom k values of ord.

Aggregate function: returns an array of values associated with the bottom k values of ord.

Attributes

def min_by(e: Column, ord: Column, k: Column): Column

Aggregate function: returns an array of values associated with the bottom k values of ord.

Aggregate function: returns an array of values associated with the bottom k values of ord.

Attributes

def minute(e: Column): Column
def mode(e: Column): Column
def mode(e: Column, deterministic: Boolean): Column

Aggregate function: returns the most frequent value in a group.

Aggregate function: returns the most frequent value in a group.

Attributes

def month(e: Column): Column
def monthname(timeExp: Column): Column

Extracts the three-letter abbreviated month name from a given date/timestamp/string.

Extracts the three-letter abbreviated month name from a given date/timestamp/string.

Attributes

def months(e: Column): Column

(Java-specific) A transform for timestamps and dates to partition data into months.

(Java-specific) A transform for timestamps and dates to partition data into months.

Attributes

def months_between(end: Column, start: Column): Column
def months_between(end: Column, start: Column, roundOff: Boolean): Column
def named_struct(cols: Column*): Column
def nanvl(col1: Column, col2: Column): Column
def negate(e: Column): Column
def next_day(date: Column, dayOfWeek: String): Column
def next_day(date: Column, dayOfWeek: Column): Column
def not(e: Column): Column

Inversion of boolean expression, i.e.

Inversion of boolean expression, i.e.

Attributes

def now(): Column
def nth_value(e: Column, offset: Int): Column
def nth_value(e: Column, offset: Int, ignoreNulls: Boolean): Column
def ntile(n: Int): Column
def nullif(col1: Column, col2: Column): Column
def nullifzero(col: Column): Column

Returns null if col is equal to zero, or col otherwise.

Returns null if col is equal to zero, or col otherwise.

Attributes

def nvl(col1: Column, col2: Column): Column
def nvl2(col1: Column, col2: Column, col3: Column): Column
def overlay(src: Column, replace: Column, pos: Column): Column
def overlay(src: Column, replace: Column, pos: Column, len: Column): Column
def parse_json(json: Column): Column

Parses a JSON string and constructs a Variant value.

Parses a JSON string and constructs a Variant value.

Attributes

def parse_url(url: Column, partToExtract: Column, key: Column): Column

Extracts a part from a URL.

Extracts a part from a URL.

Attributes

def parse_url(url: Column, partToExtract: Column): Column

Extracts a part from a URL.

Extracts a part from a URL.

Attributes

def percentile(e: Column, percentage: Column): Column
def percentile(e: Column, percentage: Column, frequency: Column): Column
def percentile_approx(e: Column, percentage: Column, accuracy: Column): Column
def pi(): Column

Returns Pi.

Returns Pi.

Attributes

def pmod(dividend: Column, divisor: Column): Column
def position(substr: Column, str: Column, start: Column): Column

Returns the position of the first occurrence of substr in str after position start.

Returns the position of the first occurrence of substr in str after position start.

Attributes

def position(substr: Column, str: Column): Column

Returns the position of the first occurrence of substr in str after position 1.

Returns the position of the first occurrence of substr in str after position 1.

Attributes

def pow(l: Column, r: Column): Column
def pow(l: Column, r: Double): Column
def pow(l: Double, r: Column): Column
def pow(l: Column, rightName: String): Column
def pow(leftName: String, r: Column): Column
def pow(leftName: String, rightName: String): Column
def pow(leftName: String, r: Double): Column

Returns the value of the first argument raised to the power of the second argument.

Returns the value of the first argument raised to the power of the second argument.

Attributes

def pow(l: Double, rightName: String): Column

Returns the value of the first argument raised to the power of the second argument.

Returns the value of the first argument raised to the power of the second argument.

Attributes

def power(l: Column, r: Column): Column
def printf(format: Column, arguments: Column*): Column

Formats the arguments in printf-style and returns the result as a string column.

Formats the arguments in printf-style and returns the result as a string column.

Attributes

def product(e: Column): Column
def quarter(e: Column): Column
def quote(str: Column): Column

Returns str enclosed by single quotes and each instance of single quote in it is preceded by a backslash.

Returns str enclosed by single quotes and each instance of single quote in it is preceded by a backslash.

Attributes

def radians(e: Column): Column
def radians(columnName: String): Column
def raise_error(c: Column): Column

Throws an exception with the provided error message.

Throws an exception with the provided error message.

Attributes

def rand(): Column
def rand(seed: Long): Column
def randn(): Column
def randn(seed: Long): Column
def random(seed: Column): Column

Returns a random value with independent and identically distributed (i.i.d.) uniformly distributed values in [0, 1).

Returns a random value with independent and identically distributed (i.i.d.) uniformly distributed values in [0, 1).

Attributes

def random(): Column

Returns a random value with independent and identically distributed (i.i.d.) uniformly distributed values in [0, 1).

Returns a random value with independent and identically distributed (i.i.d.) uniformly distributed values in [0, 1).

Attributes

def randstr(length: Column): Column

Returns a string of the specified length whose characters are chosen uniformly at random from the following pool of characters: 0-9, a-z, A-Z.

Returns a string of the specified length whose characters are chosen uniformly at random from the following pool of characters: 0-9, a-z, A-Z.

Attributes

def randstr(length: Column, seed: Column): Column

Returns a string of the specified length whose characters are chosen uniformly at random from the following pool of characters: 0-9, a-z, A-Z, with the chosen random seed.

Returns a string of the specified length whose characters are chosen uniformly at random from the following pool of characters: 0-9, a-z, A-Z, with the chosen random seed.

Attributes

def rank(): Column
def reduce(expr: Column, initialValue: Column, merge: (Column, Column) => Column, finish: Column => Column): Column

Applies a binary operator to an initial state and all elements in the array, and reduces this to a single state.

Applies a binary operator to an initial state and all elements in the array, and reduces this to a single state.

Attributes

def reduce(expr: Column, initialValue: Column, merge: (Column, Column) => Column): Column

Applies a binary operator to an initial state and all elements in the array, and reduces this to a single state.

Applies a binary operator to an initial state and all elements in the array, and reduces this to a single state.

Attributes

def reflect(cols: Column*): Column

Calls a method with reflection.

Calls a method with reflection.

Attributes

def regexp(str: Column, regexp: Column): Column

Returns true if str matches regexp, or false otherwise.

Returns true if str matches regexp, or false otherwise.

Attributes

def regexp_count(e: Column, pattern: Column): Column
def regexp_extract(e: Column, exp: String, groupIdx: Int): Column
def regexp_extract_all(e: Column, exp: Column, groupIdx: Column): Column
def regexp_instr(e: Column, pattern: Column): Column
def regexp_instr(str: Column, regexp: Column, idx: Column): Column

Searches a string for a regular expression and returns an integer that indicates the beginning position of the matched substring.

Searches a string for a regular expression and returns an integer that indicates the beginning position of the matched substring.

Attributes

def regexp_like(e: Column, pattern: Column): Column
def regexp_replace(e: Column, pattern: String, replacement: String): Column
def regexp_replace(e: Column, pattern: Column, replacement: Column): Column
def regexp_replace(e: Column, pattern: String, replacement: String, pos: Int): Column

Replace all substrings of the specified string value that match regexp with rep, starting at the specified position pos.

Replace all substrings of the specified string value that match regexp with rep, starting at the specified position pos.

Attributes

def regexp_replace(e: Column, pattern: Column, replacement: Column, pos: Column): Column

Replace all substrings of the specified string value that match regexp with rep, starting at the specified position pos.

Replace all substrings of the specified string value that match regexp with rep, starting at the specified position pos.

Attributes

def regexp_substr(e: Column, pattern: Column): Column
def regr_avgx(y: Column, x: Column): Column

Aggregate function: returns the average of the independent variable for non-null pairs in a group, where y is the dependent variable and x is the independent variable.

Aggregate function: returns the average of the independent variable for non-null pairs in a group, where y is the dependent variable and x is the independent variable.

Attributes

def regr_avgy(y: Column, x: Column): Column

Aggregate function: returns the average of the dependent variable for non-null pairs in a group, where y is the dependent variable and x is the independent variable.

Aggregate function: returns the average of the dependent variable for non-null pairs in a group, where y is the dependent variable and x is the independent variable.

Attributes

def regr_count(y: Column, x: Column): Column

Aggregate function: returns the number of non-null number pairs in a group, where y is the dependent variable and x is the independent variable.

Aggregate function: returns the number of non-null number pairs in a group, where y is the dependent variable and x is the independent variable.

Attributes

def regr_intercept(y: Column, x: Column): Column

Aggregate function: returns the intercept of the univariate linear regression line for non-null pairs in a group, where y is the dependent variable and x is the independent variable.

Aggregate function: returns the intercept of the univariate linear regression line for non-null pairs in a group, where y is the dependent variable and x is the independent variable.

Attributes

def regr_r2(y: Column, x: Column): Column

Aggregate function: returns the coefficient of determination for non-null pairs in a group, where y is the dependent variable and x is the independent variable.

Aggregate function: returns the coefficient of determination for non-null pairs in a group, where y is the dependent variable and x is the independent variable.

Attributes

def regr_slope(y: Column, x: Column): Column

Aggregate function: returns the slope of the linear regression line for non-null pairs in a group, where y is the dependent variable and x is the independent variable.

Aggregate function: returns the slope of the linear regression line for non-null pairs in a group, where y is the dependent variable and x is the independent variable.

Attributes

def regr_sxx(y: Column, x: Column): Column

Aggregate function: returns REGR_COUNT(y, x) * VAR_POP(x) for non-null pairs in a group, where y is the dependent variable and x is the independent variable.

Aggregate function: returns REGR_COUNT(y, x) * VAR_POP(x) for non-null pairs in a group, where y is the dependent variable and x is the independent variable.

Attributes

def regr_sxy(y: Column, x: Column): Column

Aggregate function: returns REGR_COUNT(y, x) * COVAR_POP(y, x) for non-null pairs in a group, where y is the dependent variable and x is the independent variable.

Aggregate function: returns REGR_COUNT(y, x) * COVAR_POP(y, x) for non-null pairs in a group, where y is the dependent variable and x is the independent variable.

Attributes

def regr_syy(y: Column, x: Column): Column

Aggregate function: returns REGR_COUNT(y, x) * VAR_POP(y) for non-null pairs in a group, where y is the dependent variable and x is the independent variable.

Aggregate function: returns REGR_COUNT(y, x) * VAR_POP(y) for non-null pairs in a group, where y is the dependent variable and x is the independent variable.

Attributes

def repeat(str: Column, n: Int): Column
def repeat(str: Column, n: Column): Column

Repeats a string column n times, and returns it as a new string column.

Repeats a string column n times, and returns it as a new string column.

Attributes

def replace(src: Column, search: Column): Column
def replace(src: Column, search: Column, replace: Column): Column
def reverse(e: Column): Column
def right(str: Column, len: Column): Column

Returns the rightmost len(len can be string type) characters from the string str, if len is less or equal than 0 the result is an empty string.

Returns the rightmost len(len can be string type) characters from the string str, if len is less or equal than 0 the result is an empty string.

Attributes

def rint(e: Column): Column
def rint(columnName: String): Column
def rlike(str: Column, regexp: Column): Column

Returns true if str matches regexp, or false otherwise.

Returns true if str matches regexp, or false otherwise.

Attributes

def round(e: Column): Column
def round(e: Column, scale: Int): Column
def round(e: Column, scale: Column): Column

Round the value of e to scale decimal places with HALF_UP round mode if scale is greater than or equal to 0 or at integral part when scale is less than 0.

Round the value of e to scale decimal places with HALF_UP round mode if scale is greater than or equal to 0 or at integral part when scale is less than 0.

Attributes

def rpad(str: Column, len: Int, pad: String): Column
def rpad(str: Column, len: Int, pad: Array[Byte]): Column

Right-pad the binary column with pad to a byte length of len.

Right-pad the binary column with pad to a byte length of len.

Attributes

def rpad(str: Column, len: Column, pad: Column): Column

Right-pad the string column with pad to a length of len.

Right-pad the string column with pad to a length of len.

Attributes

def rtrim(e: Column): Column
def rtrim(e: Column, trimString: String): Column
def rtrim(e: Column, trim: Column): Column

Trim the specified character string from right end for the specified string column.

Trim the specified character string from right end for the specified string column.

Attributes

def schema_of_csv(csv: String): Column
def schema_of_csv(csv: Column, options: Map[String, String]): Column

Parses a CSV string and infers its schema in DDL format using options.

Parses a CSV string and infers its schema in DDL format using options.

Attributes

def schema_of_json(json: String): Column
def schema_of_json(json: Column, options: Map[String, String]): Column

Parses a JSON string and infers its schema in DDL format using options.

Parses a JSON string and infers its schema in DDL format using options.

Attributes

def schema_of_variant(v: Column): Column

Returns schema in the SQL format of a variant.

Returns schema in the SQL format of a variant.

Attributes

def schema_of_variant_agg(v: Column): Column

Returns the merged schema in the SQL format of a variant column.

Returns the merged schema in the SQL format of a variant column.

Attributes

def schema_of_xml(xml: String): Column

Parses a XML string and infers its schema in DDL format.

Parses a XML string and infers its schema in DDL format.

Attributes

def schema_of_xml(xml: Column): Column

Parses a XML string and infers its schema in DDL format.

Parses a XML string and infers its schema in DDL format.

Attributes

def schema_of_xml(xml: Column, options: Map[String, String]): Column

Parses a XML string and infers its schema in DDL format using options.

Parses a XML string and infers its schema in DDL format using options.

Attributes

def sec(e: Column): Column
def second(e: Column): Column
def sentences(string: Column): Column
def sentences(string: Column, language: Column, country: Column): Column
def sentences(string: Column, language: Column): Column

Splits a string into arrays of sentences, where each sentence is an array of words.

Splits a string into arrays of sentences, where each sentence is an array of words.

Attributes

def sequence(start: Column, stop: Column): Column
def sequence(start: Column, stop: Column, step: Column): Column
def session_user(): Column

Returns the user name of current execution context.

Returns the user name of current execution context.

Attributes

def session_window(timeColumn: Column, gapDuration: String): Column
def session_window(timeColumn: Column, gapDuration: Column): Column

Generates session window given a timestamp specifying column.

Generates session window given a timestamp specifying column.

Attributes

def sha(col: Column): Column

Returns a sha1 hash value as a hex string of the col.

Returns a sha1 hash value as a hex string of the col.

Attributes

def sha1(e: Column): Column
def sha2(e: Column, numBits: Int): Column
def shiftleft(e: Column, numBits: Int): Column
def shiftright(e: Column, numBits: Int): Column
def shiftrightunsigned(e: Column, numBits: Int): Column
def shuffle(e: Column): Column
def shuffle(e: Column, seed: Column): Column

Returns a random permutation of the given array.

Returns a random permutation of the given array.

Attributes

def sign(e: Column): Column

Computes the signum of the given value.

Computes the signum of the given value.

Attributes

def signum(e: Column): Column
def signum(columnName: String): Column
def sin(e: Column): Column
def sin(columnName: String): Column
def sinh(e: Column): Column
def sinh(columnName: String): Column
def size(e: Column): Column
def skewness(columnName: String): Column
def slice(x: Column, start: Int, length: Int): Column
def slice(x: Column, start: Column, length: Column): Column
def some(e: Column): Column
def sort_array(e: Column, asc: Boolean): Column
def soundex(e: Column): Column
def split(str: Column, pattern: String): Column
def split(str: Column, pattern: String, limit: Int): Column
def split(str: Column, pattern: Column): Column

Splits str around matches of the given pattern.

Splits str around matches of the given pattern.

Attributes

def split(str: Column, pattern: Column, limit: Column): Column

Splits str around matches of the given pattern.

Splits str around matches of the given pattern.

Attributes

def split_part(str: Column, delimiter: Column, partNum: Column): Column
def sqrt(e: Column): Column
def sqrt(colName: String): Column
def st_asbinary(geo: Column): Column

Returns the input GEOGRAPHY or GEOMETRY value in WKB format.

Returns the input GEOGRAPHY or GEOMETRY value in WKB format.

Attributes

def st_asbinary(geo: Column, endianness: Column): Column

Returns the input GEOGRAPHY or GEOMETRY value in WKB format using the specified endianness.

Returns the input GEOGRAPHY or GEOMETRY value in WKB format using the specified endianness.

Attributes

def st_asbinary(geo: Column, endianness: String): Column

Returns the input GEOGRAPHY or GEOMETRY value in WKB format using the specified endianness.

Returns the input GEOGRAPHY or GEOMETRY value in WKB format using the specified endianness.

Attributes

def st_geogfromwkb(wkb: Column): Column

Parses the WKB description of a geography and returns the corresponding GEOGRAPHY value.

Parses the WKB description of a geography and returns the corresponding GEOGRAPHY value.

Attributes

def st_geomfromwkb(wkb: Column): Column

Parses the WKB description of a geometry and returns the corresponding GEOMETRY value.

Parses the WKB description of a geometry and returns the corresponding GEOMETRY value.

Attributes

def st_geomfromwkb(wkb: Column, srid: Column): Column

Parses the WKB description of a geometry and returns the corresponding GEOMETRY value.

Parses the WKB description of a geometry and returns the corresponding GEOMETRY value.

Attributes

def st_geomfromwkb(wkb: Column, srid: Int): Column

Parses the WKB description of a geometry and returns the corresponding GEOMETRY value.

Parses the WKB description of a geometry and returns the corresponding GEOMETRY value.

Attributes

def st_setsrid(geo: Column, srid: Column): Column

Returns a new GEOGRAPHY or GEOMETRY value whose SRID is the specified SRID value.

Returns a new GEOGRAPHY or GEOMETRY value whose SRID is the specified SRID value.

Attributes

def st_setsrid(geo: Column, srid: Int): Column

Returns a new GEOGRAPHY or GEOMETRY value whose SRID is the specified SRID value.

Returns a new GEOGRAPHY or GEOMETRY value whose SRID is the specified SRID value.

Attributes

def st_srid(geo: Column): Column

Returns the SRID of the input GEOGRAPHY or GEOMETRY value.

Returns the SRID of the input GEOGRAPHY or GEOMETRY value.

Attributes

def stack(cols: Column*): Column

Separates col1, ..., colk into n rows.

Separates col1, ..., colk into n rows.

Attributes

def startswith(str: Column, prefix: Column): Column
def std(e: Column): Column

Aggregate function: alias for stddev_samp.

Aggregate function: alias for stddev_samp.

Attributes

def stddev(e: Column): Column
def stddev(columnName: String): Column
def stddev_pop(columnName: String): Column
def stddev_samp(columnName: String): Column
def str_to_map(text: Column, pairDelim: Column, keyValueDelim: Column): Column

Creates a map after splitting the text into key/value pairs using delimiters.

Creates a map after splitting the text into key/value pairs using delimiters.

Attributes

def str_to_map(text: Column, pairDelim: Column): Column

Creates a map after splitting the text into key/value pairs using delimiters.

Creates a map after splitting the text into key/value pairs using delimiters.

Attributes

def str_to_map(text: Column): Column

Creates a map after splitting the text into key/value pairs using delimiters.

Creates a map after splitting the text into key/value pairs using delimiters.

Attributes

def string_agg(e: Column): Column

Aggregate function: returns the concatenation of non-null input values.

Aggregate function: returns the concatenation of non-null input values.

Attributes

def string_agg(e: Column, delimiter: Column): Column

Aggregate function: returns the concatenation of non-null input values, separated by the delimiter.

Aggregate function: returns the concatenation of non-null input values, separated by the delimiter.

Attributes

def string_agg_distinct(e: Column): Column

Aggregate function: returns the concatenation of distinct non-null input values.

Aggregate function: returns the concatenation of distinct non-null input values.

Attributes

def string_agg_distinct(e: Column, delimiter: Column): Column

Aggregate function: returns the concatenation of distinct non-null input values, separated by the delimiter.

Aggregate function: returns the concatenation of distinct non-null input values, separated by the delimiter.

Attributes

def struct(cols: Column*): Column
def struct(colName: String, colNames: String*): Column
def substr(str: Column, pos: Column, len: Column): Column

Returns the substring of str that starts at pos and is of length len, or the slice of byte array that starts at pos and is of length len.

Returns the substring of str that starts at pos and is of length len, or the slice of byte array that starts at pos and is of length len.

Attributes

def substr(str: Column, pos: Column): Column

Returns the substring of str that starts at pos, or the slice of byte array that starts at pos.

Returns the substring of str that starts at pos, or the slice of byte array that starts at pos.

Attributes

def substring(str: Column, pos: Int, len: Int): Column
def substring(str: Column, pos: Column, len: Column): Column
def substring_index(str: Column, delim: String, count: Int): Column
def sum(e: Column): Column
def sum(columnName: String): Column
def sumDistinct(columnName: String): Column
def tan(e: Column): Column
def tan(columnName: String): Column
def tanh(e: Column): Column
def tanh(columnName: String): Column
def theta_difference(c1: Column, c2: Column): Column

Subtracts two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches AnotB object

Subtracts two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches AnotB object

Attributes

def theta_difference(columnName1: String, columnName2: String): Column

Subtracts two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches AnotB object

Subtracts two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches AnotB object

Attributes

def theta_intersection(c1: Column, c2: Column): Column

Intersects two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Intersection object

Intersects two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Intersection object

Attributes

def theta_intersection(columnName1: String, columnName2: String): Column

Intersects two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Intersection object

Intersects two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Intersection object

Attributes

def theta_intersection_agg(e: Column): Column

Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by intersecting the Datasketches ThetaSketch instances in the input column via a Datasketches Intersection instance.

Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by intersecting the Datasketches ThetaSketch instances in the input column via a Datasketches Intersection instance.

Attributes

def theta_intersection_agg(columnName: String): Column

Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by intersecting the Datasketches ThetaSketch instances in the input volumn via a Datasketches Intersection instance.

Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by intersecting the Datasketches ThetaSketch instances in the input volumn via a Datasketches Intersection instance.

Attributes

def theta_sketch_agg(e: Column, lgNomEntries: Column): Column

Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch built with the values in the input column and configured with the lgNomEntries nominal entries.

Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch built with the values in the input column and configured with the lgNomEntries nominal entries.

Attributes

def theta_sketch_agg(e: Column, lgNomEntries: Int): Column

Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch built with the values in the input column and configured with the lgNomEntries nominal entries.

Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch built with the values in the input column and configured with the lgNomEntries nominal entries.

Attributes

def theta_sketch_agg(columnName: String, lgNomEntries: Int): Column

Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch built with the values in the input column and configured with the lgNomEntries nominal entries.

Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch built with the values in the input column and configured with the lgNomEntries nominal entries.

Attributes

def theta_sketch_agg(e: Column): Column

Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch built with the values in the input column and configured with the default value of 12 for lgNomEntries.

Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch built with the values in the input column and configured with the default value of 12 for lgNomEntries.

Attributes

def theta_sketch_agg(columnName: String): Column

Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch built with the values in the input column and configured with the default value of 12 for lgNomEntries.

Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch built with the values in the input column and configured with the default value of 12 for lgNomEntries.

Attributes

def theta_sketch_estimate(c: Column): Column

Returns the estimated number of unique values given the binary representation of a Datasketches ThetaSketch.

Returns the estimated number of unique values given the binary representation of a Datasketches ThetaSketch.

Attributes

def theta_sketch_estimate(columnName: String): Column

Returns the estimated number of unique values given the binary representation of a Datasketches ThetaSketch.

Returns the estimated number of unique values given the binary representation of a Datasketches ThetaSketch.

Attributes

def theta_union(c1: Column, c2: Column): Column

Unions two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Union object.

Unions two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Union object.

Attributes

def theta_union(columnName1: String, columnName2: String): Column

Unions two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Union object.

Unions two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Union object.

Attributes

def theta_union(c1: Column, c2: Column, lgNomEntries: Int): Column

Unions two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Union object.

Unions two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Union object.

Attributes

def theta_union(columnName1: String, columnName2: String, lgNomEntries: Int): Column

Unions two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Union object.

Unions two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Union object.

Attributes

def theta_union(c1: Column, c2: Column, lgNomEntries: Column): Column

Unions two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Union object.

Unions two binary representations of Datasketches ThetaSketch objects in the input columns using a Datasketches Union object.

Attributes

def theta_union_agg(e: Column, lgNomEntries: Column): Column

Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by the union of Datasketches ThetaSketch instances in the input column via a Datasketches Union instance.

Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by the union of Datasketches ThetaSketch instances in the input column via a Datasketches Union instance.

Attributes

def theta_union_agg(e: Column, lgNomEntries: Int): Column

Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by the union of Datasketches ThetaSketch instances in the input column via a Datasketches Union instance.

Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by the union of Datasketches ThetaSketch instances in the input column via a Datasketches Union instance.

Attributes

def theta_union_agg(columnName: String, lgNomEntries: Int): Column

Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by the union of Datasketches ThetaSketch instances in the input column via a Datasketches Union instance.

Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by the union of Datasketches ThetaSketch instances in the input column via a Datasketches Union instance.

Attributes

def theta_union_agg(e: Column): Column

Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by the union of Datasketches ThetaSketch instances in the input column via a Datasketches Union instance.

Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by the union of Datasketches ThetaSketch instances in the input column via a Datasketches Union instance.

Attributes

def theta_union_agg(columnName: String): Column

Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by the union of Datasketches ThetaSketch instances in the input column via a Datasketches Union instance.

Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch, generated by the union of Datasketches ThetaSketch instances in the input column via a Datasketches Union instance.

Attributes

def time_bucket(bucketSize: Column, ts: Column): Column

Returns the start of the fixed-size bucket of bucketSize that contains ts, with buckets aligned to the default origin (1970-01-01 00:00:00).

Returns the start of the fixed-size bucket of bucketSize that contains ts, with buckets aligned to the default origin (1970-01-01 00:00:00).

Attributes

def time_bucket(bucketSize: Column, ts: Column, origin: Column): Column

Returns the start of the fixed-size bucket of bucketSize that contains ts, with buckets aligned to origin.

Returns the start of the fixed-size bucket of bucketSize that contains ts, with buckets aligned to origin.

Attributes

def time_diff(unit: Column, start: Column, end: Column): Column

Returns the difference between two times, measured in specified units.

Returns the difference between two times, measured in specified units.

Attributes

def time_from_micros(e: Column): Column

Creates a TIME from the number of microseconds since midnight.

Creates a TIME from the number of microseconds since midnight.

Attributes

def time_from_millis(e: Column): Column

Creates a TIME from the number of milliseconds since midnight.

Creates a TIME from the number of milliseconds since midnight.

Attributes

def time_from_seconds(e: Column): Column

Creates a TIME from the number of seconds since midnight.

Creates a TIME from the number of seconds since midnight.

Attributes

def time_to_micros(e: Column): Column

Extracts the number of microseconds since midnight from a TIME value.

Extracts the number of microseconds since midnight from a TIME value.

Attributes

def time_to_millis(e: Column): Column

Extracts the number of milliseconds since midnight from a TIME value.

Extracts the number of milliseconds since midnight from a TIME value.

Attributes

def time_to_seconds(e: Column): Column

Extracts the number of seconds (including fractional seconds) from a TIME value.

Extracts the number of seconds (including fractional seconds) from a TIME value.

Attributes

def time_trunc(unit: Column, time: Column): Column

Returns time truncated to the unit.

Returns time truncated to the unit.

Attributes

def timestamp_add(unit: String, quantity: Column, ts: Column): Column

Adds the specified number of units to the given timestamp.

Adds the specified number of units to the given timestamp.

Attributes

def timestamp_diff(unit: String, start: Column, end: Column): Column

Gets the difference between the timestamps in the specified units by truncating the fraction part.

Gets the difference between the timestamps in the specified units by truncating the fraction part.

Attributes

def timestamp_nanos(e: Column): Column

Creates a timestamp with the local time zone and nanosecond precision (TIMESTAMP_LTZ(9)) from the number of nanoseconds since UTC epoch.

Creates a timestamp with the local time zone and nanosecond precision (TIMESTAMP_LTZ(9)) from the number of nanoseconds since UTC epoch.

Attributes

def to_base32(e: Column): Column

Computes the BASE32 (RFC 4648) encoding of a binary column and returns it as a string column.

Computes the BASE32 (RFC 4648) encoding of a binary column and returns it as a string column.

Attributes

def to_binary(e: Column, f: Column): Column

Converts the input e to a binary value based on the supplied format.

Converts the input e to a binary value based on the supplied format.

Attributes

def to_binary(e: Column): Column

Converts the input e to a binary value based on the default format "hex".

Converts the input e to a binary value based on the default format "hex".

Attributes

def to_char(e: Column, format: Column): Column
def to_csv(e: Column): Column
def to_csv(e: Column, options: Map[String, String]): Column

(Java-specific) Converts a column containing a StructType into a CSV string with the specified schema.

(Java-specific) Converts a column containing a StructType into a CSV string with the specified schema.

Attributes

def to_date(e: Column): Column
def to_date(e: Column, fmt: String): Column
def to_json(e: Column): Column
def to_json(e: Column, options: Map[String, String]): Column

(Scala-specific) Converts a column containing a StructType, ArrayType or a MapType into a JSON string with the specified schema.

(Scala-specific) Converts a column containing a StructType, ArrayType or a MapType into a JSON string with the specified schema.

Attributes

def to_json(e: Column, options: Map[String, String]): Column

(Java-specific) Converts a column containing a StructType, ArrayType or a MapType into a JSON string with the specified schema.

(Java-specific) Converts a column containing a StructType, ArrayType or a MapType into a JSON string with the specified schema.

Attributes

def to_number(e: Column, format: Column): Column
def to_time(str: Column): Column

Parses a string value to a time value.

Parses a string value to a time value.

Attributes

def to_time(str: Column, format: Column): Column

Parses a string value to a time value.

Parses a string value to a time value.

Attributes

def to_timestamp(e: Column, fmt: String): Column
def to_timestamp_ltz(timestamp: Column, format: Column): Column

Parses the timestamp expression with the format expression to a timestamp with local time zone.

Parses the timestamp expression with the format expression to a timestamp with local time zone.

Attributes

def to_timestamp_ltz(timestamp: Column): Column

Parses the timestamp expression with the default format to a timestamp with local time zone.

Parses the timestamp expression with the default format to a timestamp with local time zone.

Attributes

def to_timestamp_ntz(timestamp: Column, format: Column): Column

Parses the timestamp_str expression with the format expression to a timestamp without time zone.

Parses the timestamp_str expression with the format expression to a timestamp without time zone.

Attributes

def to_timestamp_ntz(timestamp: Column): Column

Parses the timestamp expression with the default format to a timestamp without time zone.

Parses the timestamp expression with the default format to a timestamp without time zone.

Attributes

def to_unix_timestamp(timeExp: Column, format: Column): Column

Returns the UNIX timestamp of the given time.

Returns the UNIX timestamp of the given time.

Attributes

def to_unix_timestamp(timeExp: Column): Column

Returns the UNIX timestamp of the given time.

Returns the UNIX timestamp of the given time.

Attributes

def to_utc_timestamp(ts: Column, tz: String): Column
def to_varchar(e: Column, format: Column): Column
def to_variant_object(col: Column): Column

Converts a column containing nested inputs (array/map/struct) into a variants where maps and structs are converted to variant objects which are unordered unlike SQL structs.

Converts a column containing nested inputs (array/map/struct) into a variants where maps and structs are converted to variant objects which are unordered unlike SQL structs.

Attributes

def to_xml(e: Column, options: Map[String, String]): Column

(Java-specific) Converts a column containing a StructType into a XML string with the specified schema.

(Java-specific) Converts a column containing a StructType into a XML string with the specified schema.

Attributes

def to_xml(e: Column): Column

Converts a column containing a StructType into a XML string with the specified schema.

Converts a column containing a StructType into a XML string with the specified schema.

Attributes

def transform(column: Column, f: Column => Column): Column

Transforms elements of an array using the given function.

Transforms elements of an array using the given function.

Attributes

def transform(column: Column, f: (Column, Column) => Column): Column

Transforms elements of an array using the (element, index) function.

Transforms elements of an array using the (element, index) function.

Attributes

def transform_keys(expr: Column, f: (Column, Column) => Column): Column

Applies a function to every (key, value) entry of a map and returns transformed keys.

Applies a function to every (key, value) entry of a map and returns transformed keys.

Attributes

def transform_values(expr: Column, f: (Column, Column) => Column): Column

Applies a function to every (key, value) entry of a map and returns transformed values.

Applies a function to every (key, value) entry of a map and returns transformed values.

Attributes

def translate(src: Column, matchingString: String, replaceString: String): Column
def trim(e: Column): Column
def trim(e: Column, trimString: String): Column
def trim(e: Column, trim: Column): Column

Trim the specified character from both ends for the specified string column.

Trim the specified character from both ends for the specified string column.

Attributes

def trunc(date: Column, format: String): Column
def try_add(left: Column, right: Column): Column

Returns the sum of left and right and the result is null on overflow.

Returns the sum of left and right and the result is null on overflow.

Attributes

def try_aes_decrypt(input: Column, key: Column, mode: Column, padding: Column, aad: Column): Column

This is a special version of aes_decrypt that performs the same operation, but returns a NULL value instead of raising an error if the decryption cannot be performed.

This is a special version of aes_decrypt that performs the same operation, but returns a NULL value instead of raising an error if the decryption cannot be performed.

Attributes

def try_aes_decrypt(input: Column, key: Column, mode: Column, padding: Column): Column

Returns a decrypted value of input.

Returns a decrypted value of input.

Attributes

def try_aes_decrypt(input: Column, key: Column, mode: Column): Column

Returns a decrypted value of input.

Returns a decrypted value of input.

Attributes

def try_aes_decrypt(input: Column, key: Column): Column

Returns a decrypted value of input.

Returns a decrypted value of input.

Attributes

def try_avg(e: Column): Column

Returns the mean calculated from values of a group and the result is null on overflow.

Returns the mean calculated from values of a group and the result is null on overflow.

Attributes

def try_divide(left: Column, right: Column): Column

Returns dividend``/``divisor.

Returns dividend``/``divisor.

Attributes

def try_element_at(column: Column, value: Column): Column

(array, index) - Returns element of array at given (1-based) index.

(array, index) - Returns element of array at given (1-based) index.

Attributes

def try_make_interval(years: Column, months: Column, weeks: Column, days: Column, hours: Column, mins: Column, secs: Column): Column

This is a special version of make_interval that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.

This is a special version of make_interval that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.

Attributes

def try_make_interval(years: Column, months: Column, weeks: Column, days: Column, hours: Column, mins: Column): Column

This is a special version of make_interval that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.

This is a special version of make_interval that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.

Attributes

def try_make_interval(years: Column, months: Column, weeks: Column, days: Column, hours: Column): Column

This is a special version of make_interval that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.

This is a special version of make_interval that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.

Attributes

def try_make_interval(years: Column, months: Column, weeks: Column, days: Column): Column

This is a special version of make_interval that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.

This is a special version of make_interval that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.

Attributes

def try_make_interval(years: Column, months: Column, weeks: Column): Column

This is a special version of make_interval that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.

This is a special version of make_interval that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.

Attributes

def try_make_interval(years: Column, months: Column): Column

This is a special version of make_interval that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.

This is a special version of make_interval that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.

Attributes

def try_make_interval(years: Column): Column

This is a special version of make_interval that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.

This is a special version of make_interval that performs the same operation, but returns a NULL value instead of raising an error if interval cannot be created.

Attributes

def try_make_timestamp(years: Column, months: Column, days: Column, hours: Column, mins: Column, secs: Column, timezone: Column): Column

Try to create a timestamp from years, months, days, hours, mins, secs and timezone fields.

Try to create a timestamp from years, months, days, hours, mins, secs and timezone fields.

Attributes

def try_make_timestamp(years: Column, months: Column, days: Column, hours: Column, mins: Column, secs: Column): Column

Try to create a timestamp from years, months, days, hours, mins, and secs fields.

Try to create a timestamp from years, months, days, hours, mins, and secs fields.

Attributes

def try_make_timestamp(date: Column, time: Column, timezone: Column): Column

Try to create a local date-time from date, time, and timezone fields.

Try to create a local date-time from date, time, and timezone fields.

Attributes

def try_make_timestamp(date: Column, time: Column): Column

Try to create a local date-time from date and time fields.

Try to create a local date-time from date and time fields.

Attributes

def try_make_timestamp_ltz(years: Column, months: Column, days: Column, hours: Column, mins: Column, secs: Column, timezone: Column): Column

Try to create the current timestamp with local time zone from years, months, days, hours, mins, secs and timezone fields.

Try to create the current timestamp with local time zone from years, months, days, hours, mins, secs and timezone fields.

Attributes

def try_make_timestamp_ltz(years: Column, months: Column, days: Column, hours: Column, mins: Column, secs: Column): Column

Try to create the current timestamp with local time zone from years, months, days, hours, mins and secs fields.

Try to create the current timestamp with local time zone from years, months, days, hours, mins and secs fields.

Attributes

def try_make_timestamp_ntz(years: Column, months: Column, days: Column, hours: Column, mins: Column, secs: Column): Column

Try to create a local date-time from years, months, days, hours, mins, secs fields.

Try to create a local date-time from years, months, days, hours, mins, secs fields.

Attributes

def try_make_timestamp_ntz(date: Column, time: Column): Column

Try to create a local date-time from date and time fields.

Try to create a local date-time from date and time fields.

Attributes

def try_mod(left: Column, right: Column): Column

Returns the remainder of dividend``/``divisor.

Returns the remainder of dividend``/``divisor.

Attributes

def try_multiply(left: Column, right: Column): Column

Returns left``*``right and the result is null on overflow.

Returns left``*``right and the result is null on overflow.

Attributes

def try_parse_json(json: Column): Column

Parses a JSON string and constructs a Variant value.

Parses a JSON string and constructs a Variant value.

Attributes

def try_parse_url(url: Column, partToExtract: Column, key: Column): Column

Extracts a part from a URL.

Extracts a part from a URL.

Attributes

def try_parse_url(url: Column, partToExtract: Column): Column

Extracts a part from a URL.

Extracts a part from a URL.

Attributes

def try_reflect(cols: Column*): Column

This is a special version of reflect that performs the same operation, but returns a NULL value instead of raising an error if the invoke method thrown exception.

This is a special version of reflect that performs the same operation, but returns a NULL value instead of raising an error if the invoke method thrown exception.

Attributes

def try_subtract(left: Column, right: Column): Column

Returns left``-``right and the result is null on overflow.

Returns left``-``right and the result is null on overflow.

Attributes

def try_sum(e: Column): Column

Returns the sum calculated from values of a group and the result is null on overflow.

Returns the sum calculated from values of a group and the result is null on overflow.

Attributes

def try_to_binary(e: Column, f: Column): Column

This is a special version of to_binary that performs the same operation, but returns a NULL value instead of raising an error if the conversion cannot be performed.

This is a special version of to_binary that performs the same operation, but returns a NULL value instead of raising an error if the conversion cannot be performed.

Attributes

def try_to_binary(e: Column): Column

This is a special version of to_binary that performs the same operation, but returns a NULL value instead of raising an error if the conversion cannot be performed.

This is a special version of to_binary that performs the same operation, but returns a NULL value instead of raising an error if the conversion cannot be performed.

Attributes

def try_to_date(e: Column): Column

This is a special version of to_date that performs the same operation, but returns a NULL value instead of raising an error if date cannot be created.

This is a special version of to_date that performs the same operation, but returns a NULL value instead of raising an error if date cannot be created.

Attributes

def try_to_date(e: Column, fmt: String): Column

This is a special version of to_date that performs the same operation, but returns a NULL value instead of raising an error if date cannot be created.

This is a special version of to_date that performs the same operation, but returns a NULL value instead of raising an error if date cannot be created.

Attributes

def try_to_number(e: Column, format: Column): Column

Convert string e to a number based on the string format format.

Convert string e to a number based on the string format format.

Attributes

def try_to_time(str: Column): Column

Parses a string value to a time value.

Parses a string value to a time value.

Attributes

def try_to_time(str: Column, format: Column): Column

Parses a string value to a time value.

Parses a string value to a time value.

Attributes

def try_to_timestamp(s: Column, format: Column): Column

Parses the s with the format to a timestamp.

Parses the s with the format to a timestamp.

Attributes

def try_to_timestamp(s: Column): Column

Parses the s to a timestamp.

Parses the s to a timestamp.

Attributes

def try_url_decode(str: Column): Column

This is a special version of url_decode that performs the same operation, but returns a NULL value instead of raising an error if the decoding cannot be performed.

This is a special version of url_decode that performs the same operation, but returns a NULL value instead of raising an error if the decoding cannot be performed.

Attributes

def try_validate_utf8(str: Column): Column

Returns the input value if it corresponds to a valid UTF-8 string, or NULL otherwise.

Returns the input value if it corresponds to a valid UTF-8 string, or NULL otherwise.

Attributes

def try_variant_array_append(v: Column, path: Column, value: Column): Column

Appends a value to the array in a variant at the given JSONPath location.

Appends a value to the array in a variant at the given JSONPath location.

Attributes

def try_variant_array_append(v: Column, path: String, value: Column): Column

Appends a value to the array in a variant at the given JSONPath location.

Appends a value to the array in a variant at the given JSONPath location.

Attributes

def try_variant_get(v: Column, path: String, targetType: String): Column

Extracts a sub-variant from v according to path string, and then cast the sub-variant to targetType.

Extracts a sub-variant from v according to path string, and then cast the sub-variant to targetType.

Attributes

def try_variant_get(v: Column, path: Column, targetType: String): Column

Extracts a sub-variant from v according to path column, and then cast the sub-variant to targetType.

Extracts a sub-variant from v according to path column, and then cast the sub-variant to targetType.

Attributes

def try_variant_insert(v: Column, path: Column, value: Column): Column

Inserts a value into a variant at the given JSONPath location.

Inserts a value into a variant at the given JSONPath location.

Attributes

def try_variant_insert(v: Column, path: String, value: Column): Column

Inserts a value into a variant at the given JSONPath location.

Inserts a value into a variant at the given JSONPath location.

Attributes

def try_variant_set(v: Column, path: Column, value: Column): Column

Sets or upserts a value in a variant at the given JSONPath location.

Sets or upserts a value in a variant at the given JSONPath location.

Attributes

def try_variant_set(v: Column, path: String, value: Column): Column

Sets or upserts a value in a variant at the given JSONPath location.

Sets or upserts a value in a variant at the given JSONPath location.

Attributes

def try_variant_set(v: Column, path: Column, value: Column, createIfMissing: Boolean): Column

Sets or upserts a value in a variant at the given JSONPath location.

Sets or upserts a value in a variant at the given JSONPath location.

Attributes

def try_variant_set(v: Column, path: String, value: Column, createIfMissing: Boolean): Column

Sets or upserts a value in a variant at the given JSONPath location.

Sets or upserts a value in a variant at the given JSONPath location.

Attributes

def tuple_difference_double(c1: Column, c2: Column): Column

Subtracts two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches AnotB object.

Subtracts two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches AnotB object.

Attributes

def tuple_difference_double(columnName1: String, columnName2: String): Column

Subtracts two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches AnotB object.

Subtracts two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches AnotB object.

Attributes

def tuple_difference_integer(c1: Column, c2: Column): Column

Subtracts two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches AnotB object.

Subtracts two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches AnotB object.

Attributes

def tuple_difference_integer(columnName1: String, columnName2: String): Column

Subtracts two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches AnotB object.

Subtracts two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches AnotB object.

Attributes

def tuple_difference_theta_double(c1: Column, c2: Column): Column

Subtracts the binary representation of a Datasketches ThetaSketch from a TupleSketch with double summary data type in the input columns using a Datasketches AnotB object.

Subtracts the binary representation of a Datasketches ThetaSketch from a TupleSketch with double summary data type in the input columns using a Datasketches AnotB object.

Attributes

def tuple_difference_theta_double(columnName1: String, columnName2: String): Column

Subtracts the binary representation of a Datasketches ThetaSketch from a TupleSketch with double summary data type in the input columns using a Datasketches AnotB object.

Subtracts the binary representation of a Datasketches ThetaSketch from a TupleSketch with double summary data type in the input columns using a Datasketches AnotB object.

Attributes

def tuple_difference_theta_integer(c1: Column, c2: Column): Column

Subtracts the binary representation of a Datasketches ThetaSketch from a TupleSketch with integer summary data type in the input columns using a Datasketches AnotB object.

Subtracts the binary representation of a Datasketches ThetaSketch from a TupleSketch with integer summary data type in the input columns using a Datasketches AnotB object.

Attributes

def tuple_difference_theta_integer(columnName1: String, columnName2: String): Column

Subtracts the binary representation of a Datasketches ThetaSketch from a TupleSketch with integer summary data type in the input columns using a Datasketches AnotB object.

Subtracts the binary representation of a Datasketches ThetaSketch from a TupleSketch with integer summary data type in the input columns using a Datasketches AnotB object.

Attributes

def tuple_intersection_agg_double(e: Column, mode: Column): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.

Attributes

def tuple_intersection_agg_double(e: Column, mode: String): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.

Attributes

def tuple_intersection_agg_double(columnName: String, mode: String): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.

Attributes

def tuple_intersection_agg_double(e: Column): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.

Attributes

def tuple_intersection_agg_double(columnName: String): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.

Attributes

def tuple_intersection_agg_integer(e: Column, mode: Column): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.

Attributes

def tuple_intersection_agg_integer(e: Column, mode: String): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.

Attributes

def tuple_intersection_agg_integer(columnName: String, mode: String): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.

Attributes

def tuple_intersection_agg_integer(e: Column): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.

Attributes

def tuple_intersection_agg_integer(columnName: String): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by intersecting the Datasketches TupleSketch instances in the input column via a Datasketches Intersection instance.

Attributes

def tuple_intersection_double(c1: Column, c2: Column): Column

Intersects two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Intersection object.

Intersects two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Intersection object.

Attributes

def tuple_intersection_double(columnName1: String, columnName2: String): Column

Intersects two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Intersection object.

Intersects two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Intersection object.

Attributes

def tuple_intersection_double(c1: Column, c2: Column, mode: String): Column

Intersects two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Intersection object.

Intersects two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Intersection object.

Attributes

def tuple_intersection_double(columnName1: String, columnName2: String, mode: String): Column

Intersects two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Intersection object.

Intersects two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Intersection object.

Attributes

def tuple_intersection_double(c1: Column, c2: Column, mode: Column): Column

Intersects two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Intersection object.

Intersects two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Intersection object.

Attributes

def tuple_intersection_integer(c1: Column, c2: Column): Column

Intersects two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Intersection object.

Intersects two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Intersection object.

Attributes

def tuple_intersection_integer(columnName1: String, columnName2: String): Column

Intersects two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Intersection object.

Intersects two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Intersection object.

Attributes

def tuple_intersection_integer(c1: Column, c2: Column, mode: String): Column

Intersects two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Intersection object.

Intersects two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Intersection object.

Attributes

def tuple_intersection_integer(columnName1: String, columnName2: String, mode: String): Column

Intersects two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Intersection object.

Intersects two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Intersection object.

Attributes

def tuple_intersection_integer(c1: Column, c2: Column, mode: Column): Column

Intersects two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Intersection object.

Intersects two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Intersection object.

Attributes

def tuple_intersection_theta_double(c1: Column, c2: Column): Column

Intersects the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.

Intersects the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.

Attributes

def tuple_intersection_theta_double(columnName1: String, columnName2: String): Column

Intersects the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.

Intersects the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.

Attributes

def tuple_intersection_theta_double(c1: Column, c2: Column, mode: String): Column

Intersects the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.

Intersects the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.

Attributes

def tuple_intersection_theta_double(columnName1: String, columnName2: String, mode: String): Column

Intersects the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.

Intersects the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.

Attributes

def tuple_intersection_theta_double(c1: Column, c2: Column, mode: Column): Column

Intersects the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.

Intersects the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.

Attributes

def tuple_intersection_theta_integer(c1: Column, c2: Column): Column

Intersects the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.

Intersects the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.

Attributes

def tuple_intersection_theta_integer(columnName1: String, columnName2: String): Column

Intersects the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.

Intersects the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.

Attributes

def tuple_intersection_theta_integer(c1: Column, c2: Column, mode: String): Column

Intersects the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.

Intersects the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.

Attributes

def tuple_intersection_theta_integer(columnName1: String, columnName2: String, mode: String): Column

Intersects the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.

Intersects the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.

Attributes

def tuple_intersection_theta_integer(c1: Column, c2: Column, mode: Column): Column

Intersects the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.

Intersects the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Intersection object.

Attributes

def tuple_sketch_agg_double(key: Column, summary: Column, lgNomEntries: Column, mode: Column): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns and configured with the lgNomEntries nominal entries and aggregation mode.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns and configured with the lgNomEntries nominal entries and aggregation mode.

Attributes

def tuple_sketch_agg_double(key: Column, summary: Column, lgNomEntries: Int, mode: String): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns and configured with the lgNomEntries nominal entries and aggregation mode.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns and configured with the lgNomEntries nominal entries and aggregation mode.

Attributes

def tuple_sketch_agg_double(keyColumnName: String, summaryColumnName: String, lgNomEntries: Int, mode: String): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns and configured with the lgNomEntries nominal entries and aggregation mode.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns and configured with the lgNomEntries nominal entries and aggregation mode.

Attributes

def tuple_sketch_agg_double(key: Column, summary: Column, lgNomEntries: Int): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns and configured with the lgNomEntries nominal entries.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns and configured with the lgNomEntries nominal entries.

Attributes

def tuple_sketch_agg_double(keyColumnName: String, summaryColumnName: String, lgNomEntries: Int): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns and configured with the lgNomEntries nominal entries.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns and configured with the lgNomEntries nominal entries.

Attributes

def tuple_sketch_agg_double(key: Column, summary: Column): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns.

Attributes

def tuple_sketch_agg_double(keyColumnName: String, summaryColumnName: String): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary built with the key and summary values in the input columns.

Attributes

def tuple_sketch_agg_integer(key: Column, summary: Column, lgNomEntries: Column, mode: Column): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns and configured with the lgNomEntries nominal entries and aggregation mode.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns and configured with the lgNomEntries nominal entries and aggregation mode.

Attributes

def tuple_sketch_agg_integer(key: Column, summary: Column, lgNomEntries: Int, mode: String): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns and configured with the lgNomEntries nominal entries and aggregation mode.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns and configured with the lgNomEntries nominal entries and aggregation mode.

Attributes

def tuple_sketch_agg_integer(keyColumnName: String, summaryColumnName: String, lgNomEntries: Int, mode: String): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns and configured with the lgNomEntries nominal entries and aggregation mode.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns and configured with the lgNomEntries nominal entries and aggregation mode.

Attributes

def tuple_sketch_agg_integer(key: Column, summary: Column, lgNomEntries: Int): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns and configured with the lgNomEntries nominal entries.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns and configured with the lgNomEntries nominal entries.

Attributes

def tuple_sketch_agg_integer(keyColumnName: String, summaryColumnName: String, lgNomEntries: Int): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns and configured with the lgNomEntries nominal entries.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns and configured with the lgNomEntries nominal entries.

Attributes

def tuple_sketch_agg_integer(key: Column, summary: Column): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns.

Attributes

def tuple_sketch_agg_integer(keyColumnName: String, summaryColumnName: String): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary built with the key and summary values in the input columns.

Attributes

def tuple_sketch_estimate_double(c: Column): Column

Returns the estimated number of unique values given the binary representation of a Datasketches TupleSketch with double summary data type.

Returns the estimated number of unique values given the binary representation of a Datasketches TupleSketch with double summary data type.

Attributes

def tuple_sketch_estimate_double(columnName: String): Column

Returns the estimated number of unique values given the binary representation of a Datasketches TupleSketch with double summary data type.

Returns the estimated number of unique values given the binary representation of a Datasketches TupleSketch with double summary data type.

Attributes

def tuple_sketch_estimate_integer(c: Column): Column

Returns the estimated number of unique values given the binary representation of a Datasketches TupleSketch with integer summary data type.

Returns the estimated number of unique values given the binary representation of a Datasketches TupleSketch with integer summary data type.

Attributes

def tuple_sketch_estimate_integer(columnName: String): Column

Returns the estimated number of unique values given the binary representation of a Datasketches TupleSketch with integer summary data type.

Returns the estimated number of unique values given the binary representation of a Datasketches TupleSketch with integer summary data type.

Attributes

def tuple_sketch_summary_double(c: Column): Column

Aggregates the summary values from a Datasketches TupleSketch with double summary data type.

Aggregates the summary values from a Datasketches TupleSketch with double summary data type.

Attributes

def tuple_sketch_summary_double(columnName: String): Column

Aggregates the summary values from a Datasketches TupleSketch with double summary data type.

Aggregates the summary values from a Datasketches TupleSketch with double summary data type.

Attributes

def tuple_sketch_summary_double(c: Column, mode: String): Column

Aggregates the summary values from a Datasketches TupleSketch with double summary data type.

Aggregates the summary values from a Datasketches TupleSketch with double summary data type.

Attributes

def tuple_sketch_summary_double(columnName: String, mode: String): Column

Aggregates the summary values from a Datasketches TupleSketch with double summary data type.

Aggregates the summary values from a Datasketches TupleSketch with double summary data type.

Attributes

def tuple_sketch_summary_double(c: Column, mode: Column): Column

Aggregates the summary values from a Datasketches TupleSketch with double summary data type.

Aggregates the summary values from a Datasketches TupleSketch with double summary data type.

Attributes

def tuple_sketch_summary_integer(c: Column): Column

Aggregates the summary values from a Datasketches TupleSketch with integer summary data type.

Aggregates the summary values from a Datasketches TupleSketch with integer summary data type.

Attributes

def tuple_sketch_summary_integer(columnName: String): Column

Aggregates the summary values from a Datasketches TupleSketch with integer summary data type.

Aggregates the summary values from a Datasketches TupleSketch with integer summary data type.

Attributes

def tuple_sketch_summary_integer(c: Column, mode: String): Column

Aggregates the summary values from a Datasketches TupleSketch with integer summary data type.

Aggregates the summary values from a Datasketches TupleSketch with integer summary data type.

Attributes

def tuple_sketch_summary_integer(columnName: String, mode: String): Column

Aggregates the summary values from a Datasketches TupleSketch with integer summary data type.

Aggregates the summary values from a Datasketches TupleSketch with integer summary data type.

Attributes

def tuple_sketch_summary_integer(c: Column, mode: Column): Column

Aggregates the summary values from a Datasketches TupleSketch with integer summary data type.

Aggregates the summary values from a Datasketches TupleSketch with integer summary data type.

Attributes

def tuple_sketch_theta_double(c: Column): Column

Returns the theta value (sampling rate) from a Datasketches TupleSketch with double summary data type.

Returns the theta value (sampling rate) from a Datasketches TupleSketch with double summary data type.

Attributes

def tuple_sketch_theta_double(columnName: String): Column

Returns the theta value (sampling rate) from a Datasketches TupleSketch with double summary data type.

Returns the theta value (sampling rate) from a Datasketches TupleSketch with double summary data type.

Attributes

def tuple_sketch_theta_integer(c: Column): Column

Returns the theta value (sampling rate) from a Datasketches TupleSketch with integer summary data type.

Returns the theta value (sampling rate) from a Datasketches TupleSketch with integer summary data type.

Attributes

def tuple_sketch_theta_integer(columnName: String): Column

Returns the theta value (sampling rate) from a Datasketches TupleSketch with integer summary data type.

Returns the theta value (sampling rate) from a Datasketches TupleSketch with integer summary data type.

Attributes

def tuple_union_agg_double(e: Column, lgNomEntries: Column, mode: Column): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Attributes

def tuple_union_agg_double(e: Column, lgNomEntries: Int, mode: String): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Attributes

def tuple_union_agg_double(columnName: String, lgNomEntries: Int, mode: String): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Attributes

def tuple_union_agg_double(e: Column, lgNomEntries: Int): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Attributes

def tuple_union_agg_double(columnName: String, lgNomEntries: Int): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Attributes

def tuple_union_agg_double(e: Column): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Attributes

def tuple_union_agg_double(columnName: String): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with a double type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Attributes

def tuple_union_agg_integer(e: Column, lgNomEntries: Column, mode: Column): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Attributes

def tuple_union_agg_integer(e: Column, lgNomEntries: Int, mode: String): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Attributes

def tuple_union_agg_integer(columnName: String, lgNomEntries: Int, mode: String): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Attributes

def tuple_union_agg_integer(e: Column, lgNomEntries: Int): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Attributes

def tuple_union_agg_integer(columnName: String, lgNomEntries: Int): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Attributes

def tuple_union_agg_integer(e: Column): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Attributes

def tuple_union_agg_integer(columnName: String): Column

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with an integer type summary, generated by the union of Datasketches TupleSketch instances in the input column via a Datasketches Union instance.

Attributes

def tuple_union_double(c1: Column, c2: Column): Column

Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object.

Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object.

Attributes

def tuple_union_double(columnName1: String, columnName2: String): Column

Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object.

Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object.

Attributes

def tuple_union_double(c1: Column, c2: Column, lgNomEntries: Int): Column

Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object.

Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object.

Attributes

def tuple_union_double(columnName1: String, columnName2: String, lgNomEntries: Int): Column

Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object.

Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object.

Attributes

def tuple_union_double(c1: Column, c2: Column, lgNomEntries: Int, mode: String): Column

Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object.

Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object.

Attributes

def tuple_union_double(columnName1: String, columnName2: String, lgNomEntries: Int, mode: String): Column

Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object.

Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object.

Attributes

def tuple_union_double(c1: Column, c2: Column, lgNomEntries: Column, mode: Column): Column

Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object.

Unions two binary representations of Datasketches TupleSketch objects with double summary data type in the input columns using a Datasketches Union object.

Attributes

def tuple_union_integer(c1: Column, c2: Column): Column

Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object.

Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object.

Attributes

def tuple_union_integer(columnName1: String, columnName2: String): Column

Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object.

Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object.

Attributes

def tuple_union_integer(c1: Column, c2: Column, lgNomEntries: Int): Column

Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object.

Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object.

Attributes

def tuple_union_integer(columnName1: String, columnName2: String, lgNomEntries: Int): Column

Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object.

Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object.

Attributes

def tuple_union_integer(c1: Column, c2: Column, lgNomEntries: Int, mode: String): Column

Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object.

Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object.

Attributes

def tuple_union_integer(columnName1: String, columnName2: String, lgNomEntries: Int, mode: String): Column

Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object.

Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object.

Attributes

def tuple_union_integer(c1: Column, c2: Column, lgNomEntries: Column, mode: Column): Column

Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object.

Unions two binary representations of Datasketches TupleSketch objects with integer summary data type in the input columns using a Datasketches Union object.

Attributes

def tuple_union_theta_double(c1: Column, c2: Column): Column

Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Attributes

def tuple_union_theta_double(columnName1: String, columnName2: String): Column

Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Attributes

def tuple_union_theta_double(c1: Column, c2: Column, lgNomEntries: Int): Column

Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Attributes

def tuple_union_theta_double(columnName1: String, columnName2: String, lgNomEntries: Int): Column

Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Attributes

def tuple_union_theta_double(c1: Column, c2: Column, lgNomEntries: Int, mode: String): Column

Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Attributes

def tuple_union_theta_double(columnName1: String, columnName2: String, lgNomEntries: Int, mode: String): Column

Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Attributes

def tuple_union_theta_double(c1: Column, c2: Column, lgNomEntries: Column, mode: Column): Column

Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Unions the binary representation of a Datasketches TupleSketch with double summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Attributes

def tuple_union_theta_integer(c1: Column, c2: Column): Column

Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Attributes

def tuple_union_theta_integer(columnName1: String, columnName2: String): Column

Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Attributes

def tuple_union_theta_integer(c1: Column, c2: Column, lgNomEntries: Int): Column

Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Attributes

def tuple_union_theta_integer(columnName1: String, columnName2: String, lgNomEntries: Int): Column

Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Attributes

def tuple_union_theta_integer(c1: Column, c2: Column, lgNomEntries: Int, mode: String): Column

Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Attributes

def tuple_union_theta_integer(columnName1: String, columnName2: String, lgNomEntries: Int, mode: String): Column

Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Attributes

def tuple_union_theta_integer(c1: Column, c2: Column, lgNomEntries: Column, mode: Column): Column

Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Unions the binary representation of a Datasketches TupleSketch with integer summary data type with a Datasketches ThetaSketch in the input columns using a Datasketches Union object.

Attributes

def typedLit[T](literal: T)(using enc: Encoder[T]): Column

Creates a Column of literal value whose Spark type comes from the value's Encoder, so that parameterized types (Seq, Map, Option, case classes and tuples) keep their full type, including for empty collections and None.

Creates a Column of literal value whose Spark type comes from the value's Encoder, so that parameterized types (Seq, Map, Option, case classes and tuples) keep their full type, including for empty collections and None.

Attributes

def typedlit[T](literal: T)(using enc: Encoder[T]): Column

Creates a Column of literal value whose Spark type comes from the value's Encoder. An alias of typedLit.

Creates a Column of literal value whose Spark type comes from the value's Encoder. An alias of typedLit.

Attributes

def typeof(e: Column): Column
def ucase(e: Column): Column
def unhex(column: Column): Column
def uniform(min: Column, max: Column): Column

Returns a random value with independent and identically distributed (i.i.d.) values with the specified range of numbers.

Returns a random value with independent and identically distributed (i.i.d.) values with the specified range of numbers.

Attributes

def uniform(min: Column, max: Column, seed: Column): Column

Returns a random value with independent and identically distributed (i.i.d.) values with the specified range of numbers, with the chosen random seed.

Returns a random value with independent and identically distributed (i.i.d.) values with the specified range of numbers, with the chosen random seed.

Attributes

def unix_nanos(e: Column): Column

Returns the number of nanoseconds since 1970-01-01 00:00:00 UTC for a nanosecond-precision timestamp (TIMESTAMP_LTZ(p) / TIMESTAMP_NTZ(p), p in [7, 9]).

Returns the number of nanoseconds since 1970-01-01 00:00:00 UTC for a nanosecond-precision timestamp (TIMESTAMP_LTZ(p) / TIMESTAMP_NTZ(p), p in [7, 9]).

Attributes

def unix_timestamp(s: Column, p: String): Column
def unwrap_udt(column: Column): Column

Unwrap UDT data type column into its underlying type.

Unwrap UDT data type column into its underlying type.

Attributes

def upper(e: Column): Column
def url_decode(str: Column): Column

Decodes a str in 'application/x-www-form-urlencoded' format using a specific encoding scheme.

Decodes a str in 'application/x-www-form-urlencoded' format using a specific encoding scheme.

Attributes

def url_encode(str: Column): Column

Translates a string into 'application/x-www-form-urlencoded' format using a specific encoding scheme.

Translates a string into 'application/x-www-form-urlencoded' format using a specific encoding scheme.

Attributes

def user(): Column

Returns the user name of current execution context.

Returns the user name of current execution context.

Attributes

def uuid(): Column
def uuid(seed: Column): Column

Returns an universally unique identifier (UUID) string.

Returns an universally unique identifier (UUID) string.

Attributes

def validate_utf8(str: Column): Column

Returns the input value if it corresponds to a valid UTF-8 string, or emits a SparkIllegalArgumentException exception otherwise.

Returns the input value if it corresponds to a valid UTF-8 string, or emits a SparkIllegalArgumentException exception otherwise.

Attributes

def var_pop(e: Column): Column
def var_pop(columnName: String): Column
def var_samp(columnName: String): Column
def variance(columnName: String): Column
def variant_array_append(v: Column, path: Column, value: Column): Column

Appends a value to the array in a variant at the given JSONPath location.

Appends a value to the array in a variant at the given JSONPath location.

Attributes

def variant_array_append(v: Column, path: String, value: Column): Column

Appends a value to the array in a variant at the given JSONPath location.

Appends a value to the array in a variant at the given JSONPath location.

Attributes

def variant_delete(v: Column, path: Column, paths: Column*): Column

Removes fields or array elements from a variant at the given JSONPath locations.

Removes fields or array elements from a variant at the given JSONPath locations.

Attributes

def variant_delete(v: Column, path: String, paths: String*): Column

Removes fields or array elements from a variant at the given JSONPath locations.

Removes fields or array elements from a variant at the given JSONPath locations.

Attributes

def variant_get(v: Column, path: String, targetType: String): Column

Extracts a sub-variant from v according to path string, and then cast the sub-variant to targetType.

Extracts a sub-variant from v according to path string, and then cast the sub-variant to targetType.

Attributes

def variant_get(v: Column, path: Column, targetType: String): Column

Extracts a sub-variant from v according to path column, and then cast the sub-variant to targetType.

Extracts a sub-variant from v according to path column, and then cast the sub-variant to targetType.

Attributes

def variant_insert(v: Column, path: Column, value: Column): Column

Inserts a value into a variant at the given JSONPath location.

Inserts a value into a variant at the given JSONPath location.

Attributes

def variant_insert(v: Column, path: String, value: Column): Column

Inserts a value into a variant at the given JSONPath location.

Inserts a value into a variant at the given JSONPath location.

Attributes

def variant_set(v: Column, path: Column, value: Column): Column

Sets or upserts a value in a variant at the given JSONPath location.

Sets or upserts a value in a variant at the given JSONPath location.

Attributes

def variant_set(v: Column, path: String, value: Column): Column

Sets or upserts a value in a variant at the given JSONPath location.

Sets or upserts a value in a variant at the given JSONPath location.

Attributes

def variant_set(v: Column, path: Column, value: Column, createIfMissing: Boolean): Column

Sets or upserts a value in a variant at the given JSONPath location.

Sets or upserts a value in a variant at the given JSONPath location.

Attributes

def variant_set(v: Column, path: String, value: Column, createIfMissing: Boolean): Column

Sets or upserts a value in a variant at the given JSONPath location.

Sets or upserts a value in a variant at the given JSONPath location.

Attributes

def variant_strip_nulls(v: Column): Column

Recursively removes object fields and array elements whose value is a variant null.

Recursively removes object fields and array elements whose value is a variant null.

Attributes

def variant_strip_nulls(v: Column, includeArrays: Boolean): Column

Recursively removes object fields and array elements whose value is a variant null, unless includeArrays is false, in which case null array elements are kept.

Recursively removes object fields and array elements whose value is a variant null, unless includeArrays is false, in which case null array elements are kept.

Attributes

def vector_avg(col: Column): Column

Aggregate function: returns the element-wise mean of float vectors in a group.

Aggregate function: returns the element-wise mean of float vectors in a group.

Attributes

def vector_cosine_similarity(left: Column, right: Column): Column

Returns the cosine similarity between two float vectors.

Returns the cosine similarity between two float vectors.

Attributes

def vector_inner_product(left: Column, right: Column): Column

Returns the inner product (dot product) between two float vectors.

Returns the inner product (dot product) between two float vectors.

Attributes

def vector_l2_distance(left: Column, right: Column): Column

Returns the Euclidean (L2) distance between two float vectors.

Returns the Euclidean (L2) distance between two float vectors.

Attributes

def vector_norm(vector: Column, degree: Column): Column

Returns the Lp norm of a float vector.

Returns the Lp norm of a float vector.

Attributes

def vector_norm(vector: Column): Column

Returns the Lp norm of a float vector using degree 2.0 (Euclidean norm).

Returns the Lp norm of a float vector using degree 2.0 (Euclidean norm).

Attributes

def vector_normalize(vector: Column, degree: Column): Column

Normalizes a float vector to unit length.

Normalizes a float vector to unit length.

Attributes

def vector_normalize(vector: Column): Column

Normalizes a float vector to unit length using degree 2.0 (Euclidean norm).

Normalizes a float vector to unit length using degree 2.0 (Euclidean norm).

Attributes

def vector_sum(col: Column): Column

Aggregate function: returns the element-wise sum of float vectors in a group.

Aggregate function: returns the element-wise sum of float vectors in a group.

Attributes

def version(): Column
def weekday(e: Column): Column
def when(condition: Column, value: Any): Column

Evaluates a list of conditions and returns one of multiple possible result expressions.

Evaluates a list of conditions and returns one of multiple possible result expressions.

Attributes

def width_bucket(v: Column, min: Column, max: Column, numBucket: Column): Column

Returns the bucket number into which the value of this expression would fall after being evaluated.

Returns the bucket number into which the value of this expression would fall after being evaluated.

Attributes

def window(timeColumn: Column, windowDuration: String): Column
def window(timeColumn: Column, windowDuration: String, slideDuration: String): Column
def window(timeColumn: Column, windowDuration: String, slideDuration: String, startTime: String): Column
def window_time(windowColumn: Column): Column

Extracts the event time from the window column.

Extracts the event time from the window column.

Attributes

def xpath(xml: Column, path: Column): Column

Returns a string array of values within the nodes of xml that match the XPath expression.

Returns a string array of values within the nodes of xml that match the XPath expression.

Attributes

def xpath_boolean(xml: Column, path: Column): Column

Returns true if the XPath expression evaluates to true, or if a matching node is found.

Returns true if the XPath expression evaluates to true, or if a matching node is found.

Attributes

def xpath_double(xml: Column, path: Column): Column

Returns a double value, the value zero if no match is found, or NaN if a match is found but the value is non-numeric.

Returns a double value, the value zero if no match is found, or NaN if a match is found but the value is non-numeric.

Attributes

def xpath_float(xml: Column, path: Column): Column

Returns a float value, the value zero if no match is found, or NaN if a match is found but the value is non-numeric.

Returns a float value, the value zero if no match is found, or NaN if a match is found but the value is non-numeric.

Attributes

def xpath_int(xml: Column, path: Column): Column

Returns an integer value, or the value zero if no match is found, or a match is found but the value is non-numeric.

Returns an integer value, or the value zero if no match is found, or a match is found but the value is non-numeric.

Attributes

def xpath_long(xml: Column, path: Column): Column

Returns a long integer value, or the value zero if no match is found, or a match is found but the value is non-numeric.

Returns a long integer value, or the value zero if no match is found, or a match is found but the value is non-numeric.

Attributes

def xpath_number(xml: Column, path: Column): Column

Returns a double value, the value zero if no match is found, or NaN if a match is found but the value is non-numeric.

Returns a double value, the value zero if no match is found, or NaN if a match is found but the value is non-numeric.

Attributes

def xpath_short(xml: Column, path: Column): Column

Returns a short integer value, or the value zero if no match is found, or a match is found but the value is non-numeric.

Returns a short integer value, or the value zero if no match is found, or a match is found but the value is non-numeric.

Attributes

def xpath_string(xml: Column, path: Column): Column

Returns the text contents of the first xml node that matches the XPath expression.

Returns the text contents of the first xml node that matches the XPath expression.

Attributes

def xxhash64(cols: Column*): Column
def year(e: Column): Column
def years(e: Column): Column

(Java-specific) A transform for timestamps and dates to partition data into years.

(Java-specific) A transform for timestamps and dates to partition data into years.

Attributes

def zeroifnull(col: Column): Column

Returns zero if col is null, or col otherwise.

Returns zero if col is null, or col otherwise.

Attributes

def zip_with(left: Column, right: Column, f: (Column, Column) => Column): Column

Merges two arrays element-wise using the given function.

Merges two arrays element-wise using the given function.

Attributes

Deprecated methods

def approxCountDistinct(e: Column): Column

Attributes

Deprecated
true
def approxCountDistinct(columnName: String): Column

Attributes

Deprecated
true
def approxCountDistinct(e: Column, rsd: Double): Column

Attributes

Deprecated
true
def approxCountDistinct(columnName: String, rsd: Double): Column

Attributes

Deprecated
true
def bitwiseNOT(e: Column): Column

Attributes

Deprecated
true
def callUDF(funcName: String, cols: Column*): Column

Calls a Spark function by name with the given columns as arguments.

Calls a Spark function by name with the given columns as arguments.

Attributes

Deprecated
true
def shiftLeft(e: Column, numBits: Int): Column

Shift the given value numBits left.

Shift the given value numBits left.

Attributes

Deprecated
true
def shiftRight(e: Column, numBits: Int): Column

(Signed) shift the given value numBits right.

(Signed) shift the given value numBits right.

Attributes

Deprecated
true
def shiftRightUnsigned(e: Column, numBits: Int): Column

Unsigned shift the given value numBits right.

Unsigned shift the given value numBits right.

Attributes

Deprecated
true
def toDegrees(columnName: String): Column

Attributes

Deprecated
true
def toRadians(columnName: String): Column

Attributes

Deprecated
true
Generated with