org.apache.spark.sql.DataFrameNaFunctions
Functionality for working with missing data in a Dataset, reached via df.na. Mirrors PySpark's DataFrame.na (DataFrameNaFunctions).
df.na.drop()
df.na.fill(0)
df.na.fill(Map("name" -> "unknown", "age" -> 0))
df.na.replace("name", Map("UNKNOWN" -> "unnamed"))
Attributes
-
Graph
-
-
Supertypes
-
class Object
trait Matchable
class Any
Members list
Returns a new Dataset that drops rows containing any null values.
Returns a new Dataset that drops rows containing any null values.
Attributes
Returns a new Dataset that drops rows containing null values.
Returns a new Dataset that drops rows containing null values.
Value parameters
-
how
-
"any" drops a row if it contains any null, "all" drops a row only if every value is null.
Attributes
Returns a new Dataset that drops rows containing fewer than minNonNulls non-null values.
Returns a new Dataset that drops rows containing fewer than minNonNulls non-null values.
Attributes
Returns a new Dataset that drops rows containing any null values in the given columns.
Returns a new Dataset that drops rows containing any null values in the given columns.
Attributes
Returns a new Dataset that drops rows containing null values in the given columns.
Returns a new Dataset that drops rows containing null values in the given columns.
Value parameters
-
how
-
"any" drops a row if any of cols is null, "all" only if all of cols are null.
Attributes
Returns a new Dataset that replaces null values in all columns with value.
Returns a new Dataset that replaces null values in all columns with value.
Attributes
Returns a new Dataset that replaces null values in all columns with value.
Returns a new Dataset that replaces null values in all columns with value.
Attributes
Returns a new Dataset that replaces null values in all columns with value.
Returns a new Dataset that replaces null values in all columns with value.
Attributes
Returns a new Dataset that replaces null values in all columns with value.
Returns a new Dataset that replaces null values in all columns with value.
Attributes
Returns a new Dataset that replaces null values in cols with value.
Returns a new Dataset that replaces null values in cols with value.
Attributes
Returns a new Dataset that replaces null values in cols with value.
Returns a new Dataset that replaces null values in cols with value.
Attributes
Returns a new Dataset that replaces null values in cols with value.
Returns a new Dataset that replaces null values in cols with value.
Attributes
Returns a new Dataset that replaces null values in cols with value.
Returns a new Dataset that replaces null values in cols with value.
Attributes
Returns a new Dataset that replaces null values per column, keyed by column name.
Returns a new Dataset that replaces null values per column, keyed by column name.
Value parameters
-
valueMap
-
a column -> fill value mapping; values must be Long, Double, String or Boolean.
Attributes
Returns a new Dataset that replaces values matching keys of replacement in col.
Returns a new Dataset that replaces values matching keys of replacement in col.
Value parameters
-
col
-
the column to apply the replacement to.
-
replacement
-
an old -> new value mapping.
Attributes
Returns a new Dataset that replaces values matching keys of replacement in cols.
Returns a new Dataset that replaces values matching keys of replacement in cols.
Value parameters
-
cols
-
the columns to apply the replacement to.
-
replacement
-
an old -> new value mapping.
Attributes