SP1 - Details for Query 2504

Submitted Time: 2026/06/11 09:55:21
Duration: 0.2 s
Succeeded Jobs: 3807 3809

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; subgraph cluster1 { isCluster="true"; label="WholeStageCodegen (4)\n \nduration: 0 ms"; 2 [labelType="html" label="HashAggregate time in aggregation build: 0 ms number of output rows: 1"]; } 3 [labelType="html" label="Exchange shuffle records written: 1 local merged chunks fetched: 0 shuffle write time: 0 ms remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 1 local bytes read: 59.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 1 remote merged chunks fetched: 0 remote blocks read: 0 data size: 16.0 B local merged bytes read: 0.0 B number of partitions: 1 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written: 59.0 B"]; subgraph cluster4 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: 4 ms"; 5 [labelType="html" label="HashAggregate time in aggregation build: 4 ms number of output rows: 1"]; } 6 [labelType="html" label="CartesianProduct number of output rows: 7"]; subgraph cluster7 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: 18 ms"; 8 [labelType="html" label=" Project "]; 9 [labelType="html" label="Filter number of output rows: 7"]; 10 [labelType="html" label="ColumnarToRow number of output rows: 7 number of input batches: 1"]; } 11 [labelType="html" label="Scan parquet number of files read: 1 scan time: 3 ms metadata time: 0 ms size of files read: 20.3 KiB number of output rows: 7"]; subgraph cluster12 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: 43 ms"; 13 [labelType="html" label=" Project "]; 14 [labelType="html" label="Filter number of output rows: 1"]; } 15 [labelType="html" label="Scan csv number of output rows: 1 number of files read: 1 metadata time: 0 ms size of files read: 174.0 B"]; 2->0; 3->2; 5->3; 6->5; 8->6; 9->8; 10->9; 11->10; 13->6; 14->13; 15->14; }

AdaptiveSparkPlan isFinalPlan=true

HashAggregate(keys=[], functions=[count(1)])

WholeStageCodegen (4)

Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=488240]

HashAggregate(keys=[], functions=[partial_count(1)])

WholeStageCodegen (3)

CartesianProduct

Project

Filter (COL_64ED17E5_A991_405B_AEED_8A2977283F76#790341 <=> 011040518)

ColumnarToRow

WholeStageCodegen (1)

FileScan parquet [COL_64ED17E5_A991_405B_AEED_8A2977283F76#790341] Batched: true, DataFilters: [(COL_64ED17E5_A991_405B_AEED_8A2977283F76#790341 <=> 011040518)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/data/output/cache/parquet/datablock/00_B_ACTES_PATI_210234], PartitionFilters: [], PushedFilters: [EqualNullSafe(COL_64ED17E5_A991_405B_AEED_8A2977283F76,011040518)], ReadSchema: struct<COL_64ED17E5_A991_405B_AEED_8A2977283F76:string>

Project

Filter ((isnotnull(REGLES#790466) AND (RC-SSN-SSF = REGLES#790466)) AND (APPLICABLE#790467 <=> O))

WholeStageCodegen (2)

FileScan csv [REGLES#790466,APPLICABLE#790467] Batched: false, DataFilters: [isnotnull(REGLES#790466), (RC-SSN-SSF = REGLES#790466), (APPLICABLE#790467 <=> O)], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/data/input/depot/csv/tmp/REGLES_APPLICAB_209983/REGLES-A-TTT new..., PartitionFilters: [], PushedFilters: [IsNotNull(REGLES), EqualTo(REGLES,RC-SSN-SSF), EqualNullSafe(APPLICABLE,O)], ReadSchema: struct<REGLES:string,APPLICABLE:string>

Details

== Physical Plan ==
AdaptiveSparkPlan (21)
+- == Final Plan ==
   * HashAggregate (12)
   +- ShuffleQueryStage (11), Statistics(sizeInBytes=16.0 B, rowCount=1)
      +- Exchange (10)
         +- * HashAggregate (9)
            +- CartesianProduct Inner (8)
               :- * Project (4)
               :  +- * Filter (3)
               :     +- * ColumnarToRow (2)
               :        +- Scan parquet  (1)
               +- * Project (7)
                  +- * Filter (6)
                     +- Scan csv  (5)
+- == Initial Plan ==
   HashAggregate (20)
   +- Exchange (19)
      +- HashAggregate (18)
         +- CartesianProduct Inner (17)
            :- Project (14)
            :  +- Filter (13)
            :     +- Scan parquet  (1)
            +- Project (16)
               +- Filter (15)
                  +- Scan csv  (5)


(1) Scan parquet 
Output [1]: [COL_64ED17E5_A991_405B_AEED_8A2977283F76#790341]
Batched: true
Location: InMemoryFileIndex [file:/data/output/cache/parquet/datablock/00_B_ACTES_PATI_210234]
PushedFilters: [EqualNullSafe(COL_64ED17E5_A991_405B_AEED_8A2977283F76,011040518)]
ReadSchema: struct<COL_64ED17E5_A991_405B_AEED_8A2977283F76:string>

(2) ColumnarToRow [codegen id : 1]
Input [1]: [COL_64ED17E5_A991_405B_AEED_8A2977283F76#790341]

(3) Filter [codegen id : 1]
Input [1]: [COL_64ED17E5_A991_405B_AEED_8A2977283F76#790341]
Condition : (COL_64ED17E5_A991_405B_AEED_8A2977283F76#790341 <=> 011040518)

(4) Project [codegen id : 1]
Output: []
Input [1]: [COL_64ED17E5_A991_405B_AEED_8A2977283F76#790341]

(5) Scan csv 
Output [2]: [REGLES#790466, APPLICABLE#790467]
Batched: false
Location: InMemoryFileIndex [file:/data/input/depot/csv/tmp/REGLES_APPLICAB_209983/REGLES-A-TTT new.csv]
PushedFilters: [IsNotNull(REGLES), EqualTo(REGLES,RC-SSN-SSF), EqualNullSafe(APPLICABLE,O)]
ReadSchema: struct<REGLES:string,APPLICABLE:string>

(6) Filter [codegen id : 2]
Input [2]: [REGLES#790466, APPLICABLE#790467]
Condition : ((isnotnull(REGLES#790466) AND (RC-SSN-SSF = REGLES#790466)) AND (APPLICABLE#790467 <=> O))

(7) Project [codegen id : 2]
Output: []
Input [2]: [REGLES#790466, APPLICABLE#790467]

(8) CartesianProduct
Join type: Inner
Join condition: None

(9) HashAggregate [codegen id : 3]
Input: []
Keys: []
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#790758L]
Results [1]: [count#790759L]

(10) Exchange
Input [1]: [count#790759L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=488240]

(11) ShuffleQueryStage
Output [1]: [count#790759L]
Arguments: 0

(12) HashAggregate [codegen id : 4]
Input [1]: [count#790759L]
Keys: []
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#790755L]
Results [1]: [count(1)#790755L AS count#790756L]

(13) Filter
Input [1]: [COL_64ED17E5_A991_405B_AEED_8A2977283F76#790341]
Condition : (COL_64ED17E5_A991_405B_AEED_8A2977283F76#790341 <=> 011040518)

(14) Project
Output: []
Input [1]: [COL_64ED17E5_A991_405B_AEED_8A2977283F76#790341]

(15) Filter
Input [2]: [REGLES#790466, APPLICABLE#790467]
Condition : ((isnotnull(REGLES#790466) AND (RC-SSN-SSF = REGLES#790466)) AND (APPLICABLE#790467 <=> O))

(16) Project
Output: []
Input [2]: [REGLES#790466, APPLICABLE#790467]

(17) CartesianProduct
Join type: Inner
Join condition: None

(18) HashAggregate
Input: []
Keys: []
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#790758L]
Results [1]: [count#790759L]

(19) Exchange
Input [1]: [count#790759L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=488207]

(20) HashAggregate
Input [1]: [count#790759L]
Keys: []
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#790755L]
Results [1]: [count(1)#790755L AS count#790756L]

(21) AdaptiveSparkPlan
Output [1]: [count#790756L]
Arguments: isFinalPlan=true

SQL / DataFrame Properties

Name	Value
spark.sql.optimizer.nestedPredicatePushdown.supportedFileSources	parquet,orc,geoparquet