Module: SparkConnect::Functions

Extended by:
Functions
Included in:
Functions
Defined in:
lib/spark_connect/functions.rb

Overview

The standard Spark SQL function library, mirroring PySpark's pyspark.sql.functions. Every function returns a Column.

Available both as SparkConnect::Functions and the shorthand SparkConnect::F. All methods are module functions.

Following PySpark's convention, a String argument denotes a column name for most functions (e.g. F.sum("salary") aggregates the salary column), while functions whose parameters are genuinely literal (regex patterns, date formats, JSON paths, ...) treat their String arguments as literal values.

Examples:

F = SparkConnect::F
F.col("a") + F.lit(1)
F.when(F.col("x") > 0, "pos").otherwise("non-pos")
F.sum("amount").alias("total")

Constant Summary collapse

Proto = 
SparkConnect::Proto
UNIFORM =

The following functions are generated programmatically below (UNIFORM and NO_ARG). The @!method directives document them so they appear in the API reference; each returns a Column.

---- Generated uniform functions -------------------------------------- Functions whose arguments are all ColumnOrName (a String denotes a column name). Defined programmatically to keep the surface complete and compact.

%w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze
NO_ARG =

No-argument functions.

%w[
  current_date current_timestamp now current_timezone current_user current_catalog
  current_database current_schema monotonically_increasing_id spark_partition_id
  input_file_name input_file_block_start input_file_block_length version uuid
  row_number rank dense_rank percent_rank cume_dist
].freeze

Class Attribute Summary collapse

Instance Method Summary collapse

Class Attribute Details

.lambda_counter ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future. 



880
881
882
 
# File 'lib/spark_connect/functions.rb', line 880

def lambda_counter
  @lambda_counter
end

Instance Method Details

#_col(value) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

ColumnOrName coercion: String/Symbol -> column reference, Column -> itself, everything else -> literal.



863
864
865
866
867
868
869
 
# File 'lib/spark_connect/functions.rb', line 863

def _col(value)
  case value
  when Column then value
  when String, Symbol then col(value.to_s)
  else lit(value)
  end
end

#_lambda(block) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Build a Column wrapping a LambdaFunction from a Ruby block. The block is called with one or more lambda-variable columns and must return a Column.



886
887
888
889
890
891
892
893
894
895
896
897
898
 
# File 'lib/spark_connect/functions.rb', line 886

def _lambda(block)
  arity = block.arity.negative? ? 1 : [block.arity, 1].max
  Functions.lambda_counter += 1
  names = (0...arity).map { |i| "x_#{Functions.lambda_counter}_#{i}" }
  vars = names.map do |n|
    Proto::Expression::UnresolvedNamedLambdaVariable.new(name_parts: [n])
  end
  cols = vars.map { |v| Column.new(Proto::Expression.new(unresolved_named_lambda_variable: v)) }
  body = block.call(*cols)
  Column.new(Proto::Expression.new(
               lambda_function: Proto::Expression::LambdaFunction.new(function: body.to_expr, arguments: vars)
             ))
end

#_lit_or_col(value) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future. 



872
873
874
 
# File 'lib/spark_connect/functions.rb', line 872

def _lit_or_col(value)
  value.is_a?(Column) ? value : lit(value)
end

#abs(*cols) ⇒ Column

The Spark SQL abs function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#acos(*cols) ⇒ Column

The Spark SQL acos function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#acosh(*cols) ⇒ Column

The Spark SQL acosh function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#add_months(col, months) ⇒ Object 



159
 
# File 'lib/spark_connect/functions.rb', line 159

def add_months(col, months) = Column.invoke("add_months", _col(col), lit(months))

#aggregate(col, initial, merge, finish = nil) ⇒ Column

Aggregate (fold) an array. merge combines accumulator and element; optional finish post-processes the result.

Returns:



258
259
260
261
262
 
# File 'lib/spark_connect/functions.rb', line 258

def aggregate(col, initial, merge, finish = nil)
  args = [_col(col), _col(initial), _lambda(merge)]
  args << _lambda(finish) if finish
  Column.invoke("aggregate", *args)
end

#any_value(*cols) ⇒ Column

The Spark SQL any_value function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#approx_count_distinct(col, rsd = nil) ⇒ Column

Returns approximate distinct count (optionally with relative SD).

Returns:

  • (Column) —

    approximate distinct count (optionally with relative SD).



70
71
72
 
# File 'lib/spark_connect/functions.rb', line 70

def approx_count_distinct(col, rsd = nil)
  rsd.nil? ? Column.invoke("approx_count_distinct", _col(col)) : Column.invoke("approx_count_distinct", _col(col), lit(rsd))
end

#array(*cols) ⇒ Column

Returns an array from the given columns.

Returns:

  • (Column) —

    an array from the given columns.



96
97
 
# File 'lib/spark_connect/functions.rb', line 96

def array(*cols) = Column.invoke("array", *cols.map { |c| _col(c) })
# @return [Column] a map from alternating key/value columns.

#array_append(col, value) ⇒ Object 



201
 
# File 'lib/spark_connect/functions.rb', line 201

def array_append(col, value) = Column.invoke("array_append", _col(col), lit(value))

#array_compact(*cols) ⇒ Column

The Spark SQL array_compact function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#array_contains(col, value) ⇒ Object

---- Array / map functions with value arguments -----------------------



197
 
# File 'lib/spark_connect/functions.rb', line 197

def array_contains(col, value) = Column.invoke("array_contains", _col(col), lit(value))

#array_distinct(*cols) ⇒ Column

The Spark SQL array_distinct function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#array_except(*cols) ⇒ Column

The Spark SQL array_except function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#array_insert(col, pos, value) ⇒ Object 



203
 
# File 'lib/spark_connect/functions.rb', line 203

def array_insert(col, pos, value) = Column.invoke("array_insert", _col(col), lit(pos), lit(value))

#array_intersect(*cols) ⇒ Column

The Spark SQL array_intersect function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#array_join(col, delimiter, null_replacement = nil) ⇒ Object 



205
206
207
208
209
210
211
212
 
# File 'lib/spark_connect/functions.rb', line 205

def array_join(col, delimiter, null_replacement = nil)
  if null_replacement.nil?
    Column.invoke("array_join", _col(col),
                  lit(delimiter))
  else
    Column.invoke("array_join", _col(col), lit(delimiter), lit(null_replacement))
  end
end

#array_max(*cols) ⇒ Column

The Spark SQL array_max function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#array_min(*cols) ⇒ Column

The Spark SQL array_min function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#array_position(col, value) ⇒ Object 



198
 
# File 'lib/spark_connect/functions.rb', line 198

def array_position(col, value) = Column.invoke("array_position", _col(col), lit(value))

#array_prepend(col, value) ⇒ Object 



202
 
# File 'lib/spark_connect/functions.rb', line 202

def array_prepend(col, value) = Column.invoke("array_prepend", _col(col), lit(value))

#array_remove(col, element) ⇒ Object 



199
 
# File 'lib/spark_connect/functions.rb', line 199

def array_remove(col, element) = Column.invoke("array_remove", _col(col), lit(element))

#array_repeat(col, count) ⇒ Object 



200
 
# File 'lib/spark_connect/functions.rb', line 200

def array_repeat(col, count) = Column.invoke("array_repeat", _col(col), lit(count))

#array_sort(*cols) ⇒ Column

The Spark SQL array_sort function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#array_union(*cols) ⇒ Column

The Spark SQL array_union function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#arrays_overlap(*cols) ⇒ Column

The Spark SQL arrays_overlap function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#arrays_zip(*cols) ⇒ Column

The Spark SQL arrays_zip function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#asc(col) ⇒ Column

Returns an ascending sort order for the named/given column.

Returns:

  • (Column) —

    an ascending sort order for the named/given column.



42
 
# File 'lib/spark_connect/functions.rb', line 42

def asc(col) = _col(col).asc

#asc_nulls_first(col) ⇒ Object 



44
 
# File 'lib/spark_connect/functions.rb', line 44

def asc_nulls_first(col) = _col(col).asc_nulls_first

#asc_nulls_last(col) ⇒ Object 



45
 
# File 'lib/spark_connect/functions.rb', line 45

def asc_nulls_last(col) = _col(col).asc_nulls_last

#ascii(*cols) ⇒ Column

The Spark SQL ascii function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#asin(*cols) ⇒ Column

The Spark SQL asin function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#asinh(*cols) ⇒ Column

The Spark SQL asinh function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#atan(*cols) ⇒ Column

The Spark SQL atan function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#atan2(*cols) ⇒ Column

The Spark SQL atan2 function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#atanh(*cols) ⇒ Column

The Spark SQL atanh function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#avg(*cols) ⇒ Column

The Spark SQL avg function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#base64(*cols) ⇒ Column

The Spark SQL base64 function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#bin(*cols) ⇒ Column

The Spark SQL bin function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#bit_and(*cols) ⇒ Column

The Spark SQL bit_and function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#bit_count(*cols) ⇒ Column

The Spark SQL bit_count function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#bit_length(*cols) ⇒ Column

The Spark SQL bit_length function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#bit_or(*cols) ⇒ Column

The Spark SQL bit_or function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#bit_xor(*cols) ⇒ Column

The Spark SQL bit_xor function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#bitwise_not(*cols) ⇒ Column

The Spark SQL bitwise_not function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#bool_and(*cols) ⇒ Column

The Spark SQL bool_and function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#bool_or(*cols) ⇒ Column

The Spark SQL bool_or function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#broadcast(df) ⇒ DataFrame

Mark a DataFrame for broadcast (map-side) join.

Parameters:

Returns:



269
 
# File 'lib/spark_connect/functions.rb', line 269

def broadcast(df) = df.hint("broadcast")

#bround(col, scale = 0) ⇒ Column

Returns HALF_EVEN ("banker's") rounding to scale places.

Returns:

  • (Column) —

    HALF_EVEN ("banker's") rounding to scale places.



82
 
# File 'lib/spark_connect/functions.rb', line 82

def bround(col, scale = 0) = Column.invoke("bround", _col(col), lit(scale))

#cardinality(*cols) ⇒ Column

The Spark SQL cardinality function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#cbrt(*cols) ⇒ Column

The Spark SQL cbrt function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#ceil(*cols) ⇒ Column

The Spark SQL ceil function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#ceiling(*cols) ⇒ Column

The Spark SQL ceiling function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#char_length(*cols) ⇒ Column

The Spark SQL char_length function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#character_length(*cols) ⇒ Column

The Spark SQL character_length function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#coalesce(*cols) ⇒ Column

Returns first non-null among the given columns.

Returns:

  • (Column) —

    first non-null among the given columns.



87
88
 
# File 'lib/spark_connect/functions.rb', line 87

def coalesce(*cols) = Column.invoke("coalesce", *cols.map { |c| _col(c) })
# @return [Column] `value` if `col` is NaN else `col`.

#col(name) ⇒ Column Also known as: column

A column reference by name. "*" selects all columns.

Returns:



28
 
# File 'lib/spark_connect/functions.rb', line 28

def col(name) = Column.from_name(name.to_s)

#collect_list(*cols) ⇒ Column

The Spark SQL collect_list function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#collect_set(*cols) ⇒ Column

The Spark SQL collect_set function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#concat(*cols) ⇒ Column

The Spark SQL concat function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#concat_ws(sep, *cols) ⇒ Column

Returns concatenation of columns separated by literal sep.

Returns:

  • (Column) —

    concatenation of columns separated by literal sep.



107
108
 
# File 'lib/spark_connect/functions.rb', line 107

def concat_ws(sep, *cols) = Column.invoke("concat_ws", lit(sep), *cols.map { |c| _col(c) })
# @return [Column] printf-style formatting using literal `fmt`.

#conv(col, from_base, to_base) ⇒ Column

Returns convert a number string from from_base to to_base.

Returns:

  • (Column) —

    convert a number string from from_base to to_base.



145
146
 
# File 'lib/spark_connect/functions.rb', line 145

def conv(col, from_base, to_base) = Column.invoke("conv", _col(col), lit(from_base), lit(to_base))
# @return [Column] left shift / right shift by literal bit counts.

#corr(*cols) ⇒ Column

The Spark SQL corr function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#cos(*cols) ⇒ Column

The Spark SQL cos function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#cosh(*cols) ⇒ Column

The Spark SQL cosh function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#cot(*cols) ⇒ Column

The Spark SQL cot function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#count(col) ⇒ Column

Returns count of rows (or non-null values of a column). "*" counts all rows.

Returns:

  • (Column) —

    count of rows (or non-null values of a column). "*" counts all rows.



59
60
61
 
# File 'lib/spark_connect/functions.rb', line 59

def count(col)
  col.to_s == "*" ? Column.invoke("count", lit(1)) : Column.invoke("count", _col(col))
end

#count_distinct(*cols) ⇒ Column Also known as: countDistinct

Returns count of distinct combinations of the given columns.

Returns:

  • (Column) —

    count of distinct combinations of the given columns.



64
65
66
 
# File 'lib/spark_connect/functions.rb', line 64

def count_distinct(*cols)
  Column.invoke("count", *cols.map { |c| _col(c) }, is_distinct: true)
end

#count_if(*cols) ⇒ Column

The Spark SQL count_if function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#covar_pop(*cols) ⇒ Column

The Spark SQL covar_pop function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#covar_samp(*cols) ⇒ Column

The Spark SQL covar_samp function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#crc32(*cols) ⇒ Column

The Spark SQL crc32 function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#create_map(*cols) ⇒ Column

Returns a map from alternating key/value columns.

Returns:

  • (Column) —

    a map from alternating key/value columns.



98
99
 
# File 'lib/spark_connect/functions.rb', line 98

def create_map(*cols) = Column.invoke("map", *cols.map { |c| _col(c) })
# @return [Column] a map from two array columns (keys, values).

#csc(*cols) ⇒ Column

The Spark SQL csc function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#cume_dist ⇒ Column

The Spark SQL cume_dist function (takes no arguments).

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#current_catalog ⇒ Column

The Spark SQL current_catalog function (takes no arguments).

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#current_database ⇒ Column

The Spark SQL current_database function (takes no arguments).

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#current_date ⇒ Column

The Spark SQL current_date function (takes no arguments).

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#current_schema ⇒ Column

The Spark SQL current_schema function (takes no arguments).

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#current_timestamp ⇒ Column

The Spark SQL current_timestamp function (takes no arguments).

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#current_timezone ⇒ Column

The Spark SQL current_timezone function (takes no arguments).

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#current_user ⇒ Column

The Spark SQL current_user function (takes no arguments).

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#date_add(col, days) ⇒ Object 



156
 
# File 'lib/spark_connect/functions.rb', line 156

def date_add(col, days) = Column.invoke("date_add", _col(col), lit(days))

#date_format(col, fmt) ⇒ Object

---- Date / time functions with literal arguments ---------------------



153
 
# File 'lib/spark_connect/functions.rb', line 153

def date_format(col, fmt) = Column.invoke("date_format", _col(col), lit(fmt))

#date_from_unix_date(*cols) ⇒ Column

The Spark SQL date_from_unix_date function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#date_sub(col, days) ⇒ Object 



157
 
# File 'lib/spark_connect/functions.rb', line 157

def date_sub(col, days) = Column.invoke("date_sub", _col(col), lit(days))

#date_trunc(fmt, col) ⇒ Object 



163
 
# File 'lib/spark_connect/functions.rb', line 163

def date_trunc(fmt, col) = Column.invoke("date_trunc", lit(fmt), _col(col))

#datediff(end_col, start_col) ⇒ Object 



158
 
# File 'lib/spark_connect/functions.rb', line 158

def datediff(end_col, start_col) = Column.invoke("datediff", _col(end_col), _col(start_col))

#day(*cols) ⇒ Column

The Spark SQL day function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#dayofmonth(*cols) ⇒ Column

The Spark SQL dayofmonth function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#dayofweek(*cols) ⇒ Column

The Spark SQL dayofweek function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#dayofyear(*cols) ⇒ Column

The Spark SQL dayofyear function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#degrees(*cols) ⇒ Column

The Spark SQL degrees function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#dense_rank ⇒ Column

The Spark SQL dense_rank function (takes no arguments).

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#desc(col) ⇒ Object 



43
 
# File 'lib/spark_connect/functions.rb', line 43

def desc(col) = _col(col).desc

#desc_nulls_first(col) ⇒ Object 



46
 
# File 'lib/spark_connect/functions.rb', line 46

def desc_nulls_first(col) = _col(col).desc_nulls_first

#desc_nulls_last(col) ⇒ Object 



47
 
# File 'lib/spark_connect/functions.rb', line 47

def desc_nulls_last(col) = _col(col).desc_nulls_last

#element_at(col, extraction) ⇒ Object 



214
 
# File 'lib/spark_connect/functions.rb', line 214

def element_at(col, extraction) = Column.invoke("element_at", _col(col), lit(extraction))

#every(*cols) ⇒ Column

The Spark SQL every function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#exists(col, &block) ⇒ Object 



246
 
# File 'lib/spark_connect/functions.rb', line 246

def exists(col, &block) = Column.invoke("exists", _col(col), _lambda(block))

#exp(*cols) ⇒ Column

The Spark SQL exp function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#explode(*cols) ⇒ Column

The Spark SQL explode function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#explode_outer(*cols) ⇒ Column

The Spark SQL explode_outer function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#expm1(*cols) ⇒ Column

The Spark SQL expm1 function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#expr(sql) ⇒ Column

Parse a SQL expression string into a Column.

Returns:



37
38
39
 
# File 'lib/spark_connect/functions.rb', line 37

def expr(sql)
  Column.from_expr(Proto::Expression.new(expression_string: Proto::Expression::ExpressionString.new(expression: sql)))
end

#factorial(*cols) ⇒ Column

The Spark SQL factorial function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#filter(col, &block) ⇒ Object 



248
 
# File 'lib/spark_connect/functions.rb', line 248

def filter(col, &block) = Column.invoke("filter", _col(col), _lambda(block))

#first(*cols) ⇒ Column

The Spark SQL first function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#first_value(*cols) ⇒ Column

The Spark SQL first_value function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#flatten(*cols) ⇒ Column

The Spark SQL flatten function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#floor(*cols) ⇒ Column

The Spark SQL floor function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#forall(col, &block) ⇒ Object 



247
 
# File 'lib/spark_connect/functions.rb', line 247

def forall(col, &block) = Column.invoke("forall", _col(col), _lambda(block))

#format_number(col, d) ⇒ Column

Returns number formatted to d decimal places.

Returns:

  • (Column) —

    number formatted to d decimal places.



111
112
 
# File 'lib/spark_connect/functions.rb', line 111

def format_number(col, d) = Column.invoke("format_number", _col(col), lit(d))
# @return [Column] substring of length `len` from 1-based `pos`.

#format_string(fmt, *cols) ⇒ Column

Returns printf-style formatting using literal fmt.

Returns:

  • (Column) —

    printf-style formatting using literal fmt.



109
110
 
# File 'lib/spark_connect/functions.rb', line 109

def format_string(fmt, *cols) = Column.invoke("format_string", lit(fmt), *cols.map { |c| _col(c) })
# @return [Column] number formatted to `d` decimal places.

#from_json(col, schema, options = {}) ⇒ Object

Parameters:



180
181
182
183
184
 
# File 'lib/spark_connect/functions.rb', line 180

def from_json(col, schema, options = {})
  schema_col = schema.is_a?(Types::DataType) ? lit(schema.json) : lit(schema.to_s)
  args = [_col(col), schema_col] + options.flat_map { |k, v| [lit(k.to_s), lit(v.to_s)] }
  Column.invoke("from_json", *args)
end

#from_unixtime(col, fmt = "yyyy-MM-dd HH:mm:ss") ⇒ Object 



164
 
# File 'lib/spark_connect/functions.rb', line 164

def from_unixtime(col, fmt = "yyyy-MM-dd HH:mm:ss") = Column.invoke("from_unixtime", _col(col), lit(fmt))

#from_utc_timestamp(col, tz) ⇒ Object 



170
 
# File 'lib/spark_connect/functions.rb', line 170

def from_utc_timestamp(col, tz) = Column.invoke("from_utc_timestamp", _col(col), lit(tz))

#get_json_object(col, path) ⇒ Object

---- JSON / CSV --------------------------------------------------------



176
 
# File 'lib/spark_connect/functions.rb', line 176

def get_json_object(col, path) = Column.invoke("get_json_object", _col(col), lit(path))

#greatest(*cols) ⇒ Column

The Spark SQL greatest function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#grouping(*cols) ⇒ Column

The Spark SQL grouping function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#hash(*cols) ⇒ Column

The Spark SQL hash function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#hex(*cols) ⇒ Column

The Spark SQL hex function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#hour(*cols) ⇒ Column

The Spark SQL hour function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#hypot(*cols) ⇒ Column

The Spark SQL hypot function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#initcap(*cols) ⇒ Column

The Spark SQL initcap function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#inline(*cols) ⇒ Column

The Spark SQL inline function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#inline_outer(*cols) ⇒ Column

The Spark SQL inline_outer function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#input_file_block_length ⇒ Column

The Spark SQL input_file_block_length function (takes no arguments).

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#input_file_block_start ⇒ Column

The Spark SQL input_file_block_start function (takes no arguments).

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#input_file_name ⇒ Column

The Spark SQL input_file_name function (takes no arguments).

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#instr(col, substr) ⇒ Column

Returns 1-based position of literal substr within col (0 if absent).

Returns:

  • (Column) —

    1-based position of literal substr within col (0 if absent).



117
118
 
# File 'lib/spark_connect/functions.rb', line 117

def instr(col, substr) = Column.invoke("instr", _col(col), lit(substr))
# @return [Column] 1-based position of `substr` in `col` at/after `pos`.

#isnan(*cols) ⇒ Column

The Spark SQL isnan function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#isnull(*cols) ⇒ Column

The Spark SQL isnull function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#json_tuple(col, *fields) ⇒ Object 



177
 
# File 'lib/spark_connect/functions.rb', line 177

def json_tuple(col, *fields) = Column.invoke("json_tuple", _col(col), *fields.map { |f| lit(f) })

#kurtosis(*cols) ⇒ Column

The Spark SQL kurtosis function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#lag(col, offset = 1, default = nil) ⇒ Object

---- Window / analytic functions --------------------------------------



225
 
# File 'lib/spark_connect/functions.rb', line 225

def lag(col, offset = 1, default = nil) = Column.invoke("lag", _col(col), lit(offset), lit(default))

#last(*cols) ⇒ Column

The Spark SQL last function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#last_day(*cols) ⇒ Column

The Spark SQL last_day function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#last_value(*cols) ⇒ Column

The Spark SQL last_value function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#lcase(*cols) ⇒ Column

The Spark SQL lcase function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#lead(col, offset = 1, default = nil) ⇒ Object 



226
 
# File 'lib/spark_connect/functions.rb', line 226

def lead(col, offset = 1, default = nil) = Column.invoke("lead", _col(col), lit(offset), lit(default))

#least(*cols) ⇒ Column

The Spark SQL least function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#length(*cols) ⇒ Column

The Spark SQL length function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#lit(value) ⇒ Column

A literal value column. See Column.lit for supported Ruby types.

Returns:



33
 
# File 'lib/spark_connect/functions.rb', line 33

def lit(value) = Column.lit(value)

#ln(*cols) ⇒ Column

The Spark SQL ln function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#locate(substr, col, pos = 1) ⇒ Column

Returns 1-based position of substr in col at/after pos.

Returns:

  • (Column) —

    1-based position of substr in col at/after pos.



119
120
 
# File 'lib/spark_connect/functions.rb', line 119

def locate(substr, col, pos = 1) = Column.invoke("locate", lit(substr), _col(col), lit(pos))
# @return [Column] left-padded string.

#log(*cols) ⇒ Column

The Spark SQL log function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#log10(*cols) ⇒ Column

The Spark SQL log10 function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#log1p(*cols) ⇒ Column

The Spark SQL log1p function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#log2(*cols) ⇒ Column

The Spark SQL log2 function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#lower(*cols) ⇒ Column

The Spark SQL lower function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#lpad(col, len, pad) ⇒ Column

Returns left-padded string.

Returns:

  • (Column) —

    left-padded string.



121
122
 
# File 'lib/spark_connect/functions.rb', line 121

def lpad(col, len, pad) = Column.invoke("lpad", _col(col), lit(len), lit(pad))
# @return [Column] right-padded string.

#ltrim(*cols) ⇒ Column

The Spark SQL ltrim function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#make_date(year, month, day) ⇒ Object 



172
 
# File 'lib/spark_connect/functions.rb', line 172

def make_date(year, month, day) = Column.invoke("make_date", _col(year), _col(month), _col(day))

#map_concat(*cols) ⇒ Column

The Spark SQL map_concat function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#map_contains_key(col, key) ⇒ Object 



221
 
# File 'lib/spark_connect/functions.rb', line 221

def map_contains_key(col, key) = Column.invoke("map_contains_key", _col(col), lit(key))

#map_entries(*cols) ⇒ Column

The Spark SQL map_entries function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#map_filter(col, &block) ⇒ Object 



252
 
# File 'lib/spark_connect/functions.rb', line 252

def map_filter(col, &block) = Column.invoke("map_filter", _col(col), _lambda(block))

#map_from_arrays(keys, values) ⇒ Column

Returns a map from two array columns (keys, values).

Returns:

  • (Column) —

    a map from two array columns (keys, values).



100
101
 
# File 'lib/spark_connect/functions.rb', line 100

def map_from_arrays(keys, values) = Column.invoke("map_from_arrays", _col(keys), _col(values))
# @return [Column] a named struct from alternating name/value arguments.

#map_from_entries(*cols) ⇒ Column

The Spark SQL map_from_entries function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#map_keys(*cols) ⇒ Column

The Spark SQL map_keys function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#map_values(*cols) ⇒ Column

The Spark SQL map_values function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#map_zip_with(c1, c2, &block) ⇒ Object 



253
 
# File 'lib/spark_connect/functions.rb', line 253

def map_zip_with(c1, c2, &block) = Column.invoke("map_zip_with", _col(c1), _col(c2), _lambda(block))

#max(*cols) ⇒ Column

The Spark SQL max function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#max_by(*cols) ⇒ Column

The Spark SQL max_by function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#md5(*cols) ⇒ Column

The Spark SQL md5 function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#mean(*cols) ⇒ Column

The Spark SQL mean function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#median(*cols) ⇒ Column

The Spark SQL median function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#min(*cols) ⇒ Column

The Spark SQL min function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#min_by(*cols) ⇒ Column

The Spark SQL min_by function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#minute(*cols) ⇒ Column

The Spark SQL minute function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#mode(*cols) ⇒ Column

The Spark SQL mode function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#monotonically_increasing_id ⇒ Column

The Spark SQL monotonically_increasing_id function (takes no arguments).

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#month(*cols) ⇒ Column

The Spark SQL month function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#months_between(d1, d2, round_off = true) ⇒ Object 



160
 
# File 'lib/spark_connect/functions.rb', line 160

def months_between(d1, d2, round_off = true) = Column.invoke("months_between", _col(d1), _col(d2), lit(round_off))

#named_struct(*cols) ⇒ Column

Returns a named struct from alternating name/value arguments.

Returns:

  • (Column) —

    a named struct from alternating name/value arguments.



102
 
# File 'lib/spark_connect/functions.rb', line 102

def named_struct(*cols) = Column.invoke("named_struct", *cols.map { |c| _col(c) })

#nanvl(col1, col2) ⇒ Column

Returns value if col is NaN else col.

Returns:

  • (Column) —

    value if col is NaN else col.



89
 
# File 'lib/spark_connect/functions.rb', line 89

def nanvl(col1, col2) = Column.invoke("nanvl", _col(col1), _col(col2))

#negate(*cols) ⇒ Column

The Spark SQL negate function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#negative(*cols) ⇒ Column

The Spark SQL negative function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#next_day(col, day_of_week) ⇒ Object 



161
 
# File 'lib/spark_connect/functions.rb', line 161

def next_day(col, day_of_week) = Column.invoke("next_day", _col(col), lit(day_of_week))

#now ⇒ Column

The Spark SQL now function (takes no arguments).

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#nth_value(col, offset, ignore_nulls = false) ⇒ Object 



228
 
# File 'lib/spark_connect/functions.rb', line 228

def nth_value(col, offset, ignore_nulls = false) = Column.invoke("nth_value", _col(col), lit(offset), lit(ignore_nulls))

#ntile(n) ⇒ Object 



227
 
# File 'lib/spark_connect/functions.rb', line 227

def ntile(n) = Column.invoke("ntile", lit(n))

#octet_length(*cols) ⇒ Column

The Spark SQL octet_length function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#overlay(col, replace, pos, len = -1)) ⇒ Column

Returns overlay replace into col at pos for len chars.

Returns:

  • (Column) —

    overlay replace into col at pos for len chars.



141
142
 
# File 'lib/spark_connect/functions.rb', line 141

def overlay(col, replace, pos, len = -1) = Column.invoke("overlay", _col(col), _col(replace), lit(pos), lit(len))
# @return [Column] SHA-2 hash with the given bit length (224/256/384/512).

#percent_rank ⇒ Column

The Spark SQL percent_rank function (takes no arguments).

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#pmod(*cols) ⇒ Column

The Spark SQL pmod function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#posexplode(*cols) ⇒ Column

The Spark SQL posexplode function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#posexplode_outer(*cols) ⇒ Column

The Spark SQL posexplode_outer function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#positive(*cols) ⇒ Column

The Spark SQL positive function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#pow(*cols) ⇒ Column

The Spark SQL pow function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#power(*cols) ⇒ Column

The Spark SQL power function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#product(*cols) ⇒ Column

The Spark SQL product function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#quarter(*cols) ⇒ Column

The Spark SQL quarter function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#radians(*cols) ⇒ Column

The Spark SQL radians function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#rand(seed = nil) ⇒ Object

---- Randomness --------------------------------------------------------



236
 
# File 'lib/spark_connect/functions.rb', line 236

def rand(seed = nil) = seed.nil? ? Column.invoke("rand") : Column.invoke("rand", lit(seed))

#randn(seed = nil) ⇒ Object 



237
 
# File 'lib/spark_connect/functions.rb', line 237

def randn(seed = nil) = seed.nil? ? Column.invoke("randn") : Column.invoke("randn", lit(seed))

#rank ⇒ Column

The Spark SQL rank function (takes no arguments).

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#regexp_count(col, pattern) ⇒ Object 



138
 
# File 'lib/spark_connect/functions.rb', line 138

def regexp_count(col, pattern) = Column.invoke("regexp_count", _col(col), lit(pattern))

#regexp_extract(col, pattern, idx = 0) ⇒ Column

Returns the idx-th group of pattern matched in col.

Returns:

  • (Column) —

    the idx-th group of pattern matched in col.



131
132
 
# File 'lib/spark_connect/functions.rb', line 131

def regexp_extract(col, pattern, idx = 0) = Column.invoke("regexp_extract", _col(col), lit(pattern), lit(idx))
# @return [Column] all matches of group `idx` of `pattern`.

#regexp_extract_all(col, pattern, idx = 1) ⇒ Column

Returns all matches of group idx of pattern.

Returns:

  • (Column) —

    all matches of group idx of pattern.



133
134
 
# File 'lib/spark_connect/functions.rb', line 133

def regexp_extract_all(col, pattern, idx = 1) = Column.invoke("regexp_extract_all", _col(col), lit(pattern), lit(idx))
# @return [Column] `col` with `pattern` replaced by `replacement`.

#regexp_like(col, pattern) ⇒ Column

Returns whether col matches pattern.

Returns:

  • (Column) —

    whether col matches pattern.



137
 
# File 'lib/spark_connect/functions.rb', line 137

def regexp_like(col, pattern) = Column.invoke("regexp_like", _col(col), lit(pattern))

#regexp_replace(col, pattern, replacement) ⇒ Column

Returns col with pattern replaced by replacement.

Returns:

  • (Column) —

    col with pattern replaced by replacement.



135
136
 
# File 'lib/spark_connect/functions.rb', line 135

def regexp_replace(col, pattern, replacement) = Column.invoke("regexp_replace", _col(col), lit(pattern), lit(replacement))
# @return [Column] whether `col` matches `pattern`.

#regexp_substr(col, pattern) ⇒ Object 



139
140
 
# File 'lib/spark_connect/functions.rb', line 139

def regexp_substr(col, pattern) = Column.invoke("regexp_substr", _col(col), lit(pattern))
# @return [Column] overlay `replace` into `col` at `pos` for `len` chars.

#repeat(col, n) ⇒ Column

Returns the string repeated n times.

Returns:

  • (Column) —

    the string repeated n times.



125
126
 
# File 'lib/spark_connect/functions.rb', line 125

def repeat(col, n) = Column.invoke("repeat", _col(col), lit(n))
# @return [Column] split `col` by the literal regex `pattern`.

#reverse(*cols) ⇒ Column

The Spark SQL reverse function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#rint(*cols) ⇒ Column

The Spark SQL rint function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#round(col, scale = 0) ⇒ Column

Returns HALF_UP rounding to scale decimal places.

Returns:

  • (Column) —

    HALF_UP rounding to scale decimal places.



80
81
 
# File 'lib/spark_connect/functions.rb', line 80

def round(col, scale = 0) = Column.invoke("round", _col(col), lit(scale))
# @return [Column] HALF_EVEN ("banker's") rounding to `scale` places.

#row_number ⇒ Column

The Spark SQL row_number function (takes no arguments).

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#rpad(col, len, pad) ⇒ Column

Returns right-padded string.

Returns:

  • (Column) —

    right-padded string.



123
124
 
# File 'lib/spark_connect/functions.rb', line 123

def rpad(col, len, pad) = Column.invoke("rpad", _col(col), lit(len), lit(pad))
# @return [Column] the string repeated `n` times.

#rtrim(*cols) ⇒ Column

The Spark SQL rtrim function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#schema_of_json(json, options = {}) ⇒ Object 



191
192
193
 
# File 'lib/spark_connect/functions.rb', line 191

def schema_of_json(json, options = {})
  Column.invoke("schema_of_json", _lit_or_col(json), *options.flat_map { |k, v| [lit(k.to_s), lit(v.to_s)] })
end

#sec(*cols) ⇒ Column

The Spark SQL sec function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#second(*cols) ⇒ Column

The Spark SQL second function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#sequence(start, stop, step = nil) ⇒ Object 



217
218
219
 
# File 'lib/spark_connect/functions.rb', line 217

def sequence(start, stop, step = nil)
  step.nil? ? Column.invoke("sequence", _col(start), _col(stop)) : Column.invoke("sequence", _col(start), _col(stop), _col(step))
end

#sha(*cols) ⇒ Column

The Spark SQL sha function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#sha1(*cols) ⇒ Column

The Spark SQL sha1 function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#sha2(col, num_bits) ⇒ Column

Returns SHA-2 hash with the given bit length (224/256/384/512).

Returns:

  • (Column) —

    SHA-2 hash with the given bit length (224/256/384/512).



143
144
 
# File 'lib/spark_connect/functions.rb', line 143

def sha2(col, num_bits) = Column.invoke("sha2", _col(col), lit(num_bits))
# @return [Column] convert a number string from `from_base` to `to_base`.

#shiftleft(col, num_bits) ⇒ Column

Returns left shift / right shift by literal bit counts.

Returns:

  • (Column) —

    left shift / right shift by literal bit counts.



147
 
# File 'lib/spark_connect/functions.rb', line 147

def shiftleft(col, num_bits) = Column.invoke("shiftleft", _col(col), lit(num_bits))

#shiftright(col, num_bits) ⇒ Object 



148
 
# File 'lib/spark_connect/functions.rb', line 148

def shiftright(col, num_bits) = Column.invoke("shiftright", _col(col), lit(num_bits))

#shiftrightunsigned(col, num_bits) ⇒ Object 



149
 
# File 'lib/spark_connect/functions.rb', line 149

def shiftrightunsigned(col, num_bits) = Column.invoke("shiftrightunsigned", _col(col), lit(num_bits))

#shuffle(*cols) ⇒ Column

The Spark SQL shuffle function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#signum(*cols) ⇒ Column

The Spark SQL signum function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#sin(*cols) ⇒ Column

The Spark SQL sin function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#sinh(*cols) ⇒ Column

The Spark SQL sinh function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#size(*cols) ⇒ Column

The Spark SQL size function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#skewness(*cols) ⇒ Column

The Spark SQL skewness function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#slice(col, start, length) ⇒ Object 



215
 
# File 'lib/spark_connect/functions.rb', line 215

def slice(col, start, length) = Column.invoke("slice", _col(col), _lit_or_col(start), _lit_or_col(length))

#some(*cols) ⇒ Column

The Spark SQL some function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#sort_array(col, asc = true) ⇒ Object

---- Sorting helpers ---------------------------------------------------



232
 
# File 'lib/spark_connect/functions.rb', line 232

def sort_array(col, asc = true) = Column.invoke("sort_array", _col(col), lit(asc))

#soundex(*cols) ⇒ Column

The Spark SQL soundex function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#spark_partition_id ⇒ Column

The Spark SQL spark_partition_id function (takes no arguments).

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#split(col, pattern, limit = -1)) ⇒ Column

Returns split col by the literal regex pattern.

Returns:

  • (Column) —

    split col by the literal regex pattern.



127
128
 
# File 'lib/spark_connect/functions.rb', line 127

def split(col, pattern, limit = -1) = Column.invoke("split", _col(col), lit(pattern), lit(limit))
# @return [Column] characters of `col` matching `matching` replaced per `replace`.

#sqrt(*cols) ⇒ Column

The Spark SQL sqrt function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#stddev(*cols) ⇒ Column

The Spark SQL stddev function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#stddev_pop(*cols) ⇒ Column

The Spark SQL stddev_pop function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#stddev_samp(*cols) ⇒ Column

The Spark SQL stddev_samp function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#struct(*cols) ⇒ Column

Returns a struct from the given columns.

Returns:

  • (Column) —

    a struct from the given columns.



94
95
 
# File 'lib/spark_connect/functions.rb', line 94

def struct(*cols) = Column.invoke("struct", *cols.map { |c| _col(c) })
# @return [Column] an array from the given columns.

#substring(col, pos, len) ⇒ Column

Returns substring of length len from 1-based pos.

Returns:

  • (Column) —

    substring of length len from 1-based pos.



113
114
 
# File 'lib/spark_connect/functions.rb', line 113

def substring(col, pos, len) = Column.invoke("substring", _col(col), lit(pos), lit(len))
# @return [Column] substring before the `count`-th occurrence of `delim`.

#substring_index(col, delim, count) ⇒ Column

Returns substring before the count-th occurrence of delim.

Returns:

  • (Column) —

    substring before the count-th occurrence of delim.



115
116
 
# File 'lib/spark_connect/functions.rb', line 115

def substring_index(col, delim, count) = Column.invoke("substring_index", _col(col), lit(delim), lit(count))
# @return [Column] 1-based position of literal `substr` within `col` (0 if absent).

#sum(*cols) ⇒ Column

The Spark SQL sum function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#sum_distinct(col) ⇒ Column

Returns sum of distinct values.

Returns:

  • (Column) —

    sum of distinct values.



75
 
# File 'lib/spark_connect/functions.rb', line 75

def sum_distinct(col) = Column.invoke("sum", _col(col), is_distinct: true)

#tan(*cols) ⇒ Column

The Spark SQL tan function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#tanh(*cols) ⇒ Column

The Spark SQL tanh function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#timestamp_micros(*cols) ⇒ Column

The Spark SQL timestamp_micros function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#timestamp_millis(*cols) ⇒ Column

The Spark SQL timestamp_millis function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#timestamp_seconds(*cols) ⇒ Column

The Spark SQL timestamp_seconds function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#to_date(col, fmt = nil) ⇒ Object 



154
 
# File 'lib/spark_connect/functions.rb', line 154

def to_date(col, fmt = nil) = fmt ? Column.invoke("to_date", _col(col), lit(fmt)) : Column.invoke("to_date", _col(col))

#to_json(col, options = {}) ⇒ Object 



186
187
188
189
 
# File 'lib/spark_connect/functions.rb', line 186

def to_json(col, options = {})
  args = [_col(col)] + options.flat_map { |k, v| [lit(k.to_s), lit(v.to_s)] }
  Column.invoke("to_json", *args)
end

#to_timestamp(col, fmt = nil) ⇒ Object 



155
 
# File 'lib/spark_connect/functions.rb', line 155

def to_timestamp(col, fmt = nil) = fmt ? Column.invoke("to_timestamp", _col(col), lit(fmt)) : Column.invoke("to_timestamp", _col(col))

#to_utc_timestamp(col, tz) ⇒ Object 



171
 
# File 'lib/spark_connect/functions.rb', line 171

def to_utc_timestamp(col, tz) = Column.invoke("to_utc_timestamp", _col(col), lit(tz))

#transform(col) {|element| ... } ⇒ Column

Transform each element of an array. The block receives a Column (and optionally the index) and returns a Column.

Yield Parameters:

Returns:



245
 
# File 'lib/spark_connect/functions.rb', line 245

def transform(col, &block) = Column.invoke("transform", _col(col), _lambda(block))

#transform_keys(col, &block) ⇒ Object 



250
 
# File 'lib/spark_connect/functions.rb', line 250

def transform_keys(col, &block) = Column.invoke("transform_keys", _col(col), _lambda(block))

#transform_values(col, &block) ⇒ Object 



251
 
# File 'lib/spark_connect/functions.rb', line 251

def transform_values(col, &block) = Column.invoke("transform_values", _col(col), _lambda(block))

#translate(col, matching, replace) ⇒ Column

Returns characters of col matching matching replaced per replace.

Returns:

  • (Column) —

    characters of col matching matching replaced per replace.



129
130
 
# File 'lib/spark_connect/functions.rb', line 129

def translate(col, matching, replace) = Column.invoke("translate", _col(col), lit(matching), lit(replace))
# @return [Column] the `idx`-th group of `pattern` matched in `col`.

#trim(*cols) ⇒ Column

The Spark SQL trim function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#trunc(col, fmt) ⇒ Object 



162
 
# File 'lib/spark_connect/functions.rb', line 162

def trunc(col, fmt) = Column.invoke("trunc", _col(col), lit(fmt))

#typeof(*cols) ⇒ Column

The Spark SQL typeof function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#ucase(*cols) ⇒ Column

The Spark SQL ucase function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#udf ⇒ Object

UDFs require a server-side execution environment (Python/Scala) and are not supported by the pure-Ruby client.

Raises:



273
274
275
 
# File 'lib/spark_connect/functions.rb', line 273

def udf(*)
  raise NotImplementedError, "User-defined functions are not supported by the Ruby Spark Connect client"
end

#unbase64(*cols) ⇒ Column

The Spark SQL unbase64 function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#unhex(*cols) ⇒ Column

The Spark SQL unhex function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#unix_date(*cols) ⇒ Column

The Spark SQL unix_date function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#unix_micros(*cols) ⇒ Column

The Spark SQL unix_micros function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#unix_millis(*cols) ⇒ Column

The Spark SQL unix_millis function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#unix_seconds(*cols) ⇒ Column

The Spark SQL unix_seconds function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#unix_timestamp(col = nil, fmt = "yyyy-MM-dd HH:mm:ss") ⇒ Object 



166
167
168
 
# File 'lib/spark_connect/functions.rb', line 166

def unix_timestamp(col = nil, fmt = "yyyy-MM-dd HH:mm:ss")
  col.nil? ? Column.invoke("unix_timestamp") : Column.invoke("unix_timestamp", _col(col), lit(fmt))
end

#upper(*cols) ⇒ Column

The Spark SQL upper function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#uuid ⇒ Column

The Spark SQL uuid function (takes no arguments).

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#var_pop(*cols) ⇒ Column

The Spark SQL var_pop function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#var_samp(*cols) ⇒ Column

The Spark SQL var_samp function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#variance(*cols) ⇒ Column

The Spark SQL variance function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#version ⇒ Column

The Spark SQL version function (takes no arguments).

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#weekday(*cols) ⇒ Column

The Spark SQL weekday function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#weekofyear(*cols) ⇒ Column

The Spark SQL weekofyear function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#when(condition, value) ⇒ Column

Start a CASE WHEN expression. Chain Column#when / Column#otherwise.

Returns:



51
52
53
 
# File 'lib/spark_connect/functions.rb', line 51

def when(condition, value)
  Column.invoke("when", condition, value)
end

#xxhash64(*cols) ⇒ Column

The Spark SQL xxhash64 function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#year(*cols) ⇒ Column

The Spark SQL year function. String arguments are treated as column names.

Returns:



822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
 
# File 'lib/spark_connect/functions.rb', line 822

UNIFORM = %w[
  sum avg mean max min first last stddev stddev_samp stddev_pop variance var_samp var_pop
  skewness kurtosis collect_list collect_set first_value last_value max_by min_by corr
  covar_pop covar_samp median mode any_value every some bit_and bit_or bit_xor bool_and bool_or
  product count_if grouping
  abs acos acosh asin asinh atan atanh atan2 bin cbrt ceil ceiling cos cosh cot csc degrees
  exp expm1 factorial floor hypot ln log log2 log10 log1p negative negate positive pow power
  radians rint sec signum sin sinh sqrt tan tanh hex unhex pmod isnan isnull positive
  upper lower ltrim rtrim trim length char_length character_length octet_length bit_length
  reverse ascii base64 unbase64 initcap soundex crc32 md5 sha1 sha ucase lcase
  size cardinality array_distinct array_max array_min array_compact flatten explode explode_outer
  posexplode posexplode_outer inline inline_outer map_keys map_values map_entries map_from_entries
  array_sort shuffle arrays_zip map_concat concat greatest least hash xxhash64
  array_union array_intersect array_except arrays_overlap
  year quarter month dayofmonth day dayofweek dayofyear hour minute second weekofyear last_day
  weekday unix_date unix_micros unix_millis unix_seconds timestamp_seconds timestamp_millis
  timestamp_micros date_from_unix_date
  bitwise_not bit_count typeof
].uniq.freeze

#zip_with(left, right, &block) ⇒ Object 



249
 
# File 'lib/spark_connect/functions.rb', line 249

def zip_with(left, right, &block) = Column.invoke("zip_with", _col(left), _col(right), _lambda(block))