Observation

org.apache.spark.sql.Observation

See theObservation companion object

class Observation(val name: String)

Holder for named aggregate metrics computed while a Dataset is being materialised, without an extra pass over the data. Pair with Dataset.observe.

 val obs = new Observation("metrics")
 df.observe(obs, count(lit(1)).as("rows"), max("id").as("max_id")).collect()
 obs.get  // Map("rows" -> 100, "max_id" -> 99)

The result of get is a map from metric column name to its value. get blocks until the metrics have been observed (i.e. until an action on the observed Dataset has run and the result code has called setMetricsFromLiterals).

Value parameters

name: a unique name for this observation.

Attributes

Companion: object
Graph
Supertypes: class Object

trait Matchable

class Any

Members list

Value members

Constructors

Creates an Observation with a random UUID name, mirroring the default in the Ruby reference client.

Attributes

Concrete methods

The observed metric values. Blocks until the metrics are available, i.e. until an action has been executed on the Dataset this Observation was attached to and the response stream has been consumed.

Attributes

Returns: a map from metric name to its observed value.

Concrete fields

In this article

Generated with