Packages

  • package root
    Definition Classes
    root
  • package org
    Definition Classes
    root
  • package apache
    Definition Classes
    org
  • package spark
    Definition Classes
    apache
  • package sql

    Allows the execution of relational queries, including those expressed in SQL using Spark.

    Allows the execution of relational queries, including those expressed in SQL using Spark.

    Definition Classes
    spark
  • package api

    Contains API classes that are specific to a single language (i.e.

    Contains API classes that are specific to a single language (i.e. Java).

    Definition Classes
    sql
  • package catalog
    Definition Classes
    sql
  • package catalyst
    Definition Classes
    sql
  • package connector
    Definition Classes
    sql
  • package dynamicpruning
    Definition Classes
    sql
  • CleanupDynamicPruningFilters
  • PartitionPruning
  • PlanDynamicPruningFilters
  • package execution

    The physical execution component of Spark SQL.

    The physical execution component of Spark SQL. Note that this is a private package. All classes in catalyst are considered an internal API to Spark SQL and are subject to change between minor releases.

    Definition Classes
    sql
  • package expressions
    Definition Classes
    sql
  • package internal

    All classes in this package are considered an internal API to Spark and are subject to change between minor releases.

    All classes in this package are considered an internal API to Spark and are subject to change between minor releases.

    Definition Classes
    sql
  • package jdbc
    Definition Classes
    sql
  • package sources

    A set of APIs for adding data sources to Spark SQL.

    A set of APIs for adding data sources to Spark SQL.

    Definition Classes
    sql
  • package streaming
    Definition Classes
    sql
  • package util
    Definition Classes
    sql
p

org.apache.spark.sql

dynamicpruning

package dynamicpruning

Ordering
  1. Alphabetic
Visibility
  1. Public
  2. All

Type Members

  1. case class PlanDynamicPruningFilters(sparkSession: SparkSession) extends Rule[SparkPlan] with PredicateHelper with Product with Serializable

    This planner rule aims at rewriting dynamic pruning predicates in order to reuse the results of broadcast.

    This planner rule aims at rewriting dynamic pruning predicates in order to reuse the results of broadcast. For joins that are not planned as broadcast hash joins we keep the fallback mechanism with subquery duplicate.

Value Members

  1. object CleanupDynamicPruningFilters extends Rule[LogicalPlan] with PredicateHelper

    Removes the filter nodes with dynamic pruning that were not pushed down to the scan.

    Removes the filter nodes with dynamic pruning that were not pushed down to the scan. These nodes will not be pushed through projects and aggregates with non-deterministic expressions.

  2. object PartitionPruning extends Rule[LogicalPlan] with PredicateHelper

    Dynamic partition pruning optimization is performed based on the type and selectivity of the join operation.

    Dynamic partition pruning optimization is performed based on the type and selectivity of the join operation. During query optimization, we insert a predicate on the partitioned table using the filter from the other side of the join and a custom wrapper called DynamicPruning.

    The basic mechanism for DPP inserts a duplicated subquery with the filter from the other side, when the following conditions are met: (1) the table to prune is partitioned by the JOIN key (2) the join operation is one of the following types: INNER, LEFT SEMI (partitioned on left), LEFT OUTER (partitioned on right), or RIGHT OUTER (partitioned on left)

    In order to enable partition pruning directly in broadcasts, we use a custom DynamicPruning clause that incorporates the In clause with the subquery and the benefit estimation. During query planning, when the join type is known, we use the following mechanism: (1) if the join is a broadcast hash join, we replace the duplicated subquery with the reused results of the broadcast, (2) else if the estimated benefit of partition pruning outweighs the overhead of running the subquery query twice, we keep the duplicated subquery (3) otherwise, we drop the subquery.

Ungrouped