SP1 - Details for Query 3108

Submitted Time: 2026/09/24 14:23:27
Duration: 0.2 s
Succeeded Jobs: 4518 4519

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; 1 [labelType="html" label=" TakeOrderedAndProject "]; subgraph cluster2 { isCluster="true"; label="WholeStageCodegen (5)\n \nduration: 2 ms"; 3 [labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build: 0 ms peak memory: 16.3 MiB number of output rows: 3 number of sort fallback tasks: 0 avg hash probes per key: 1"]; } 4 [labelType="html" label="AQEShuffleRead number of partitions: 1 partition data size: 368.0 B number of coalesced partitions: 1"]; 5 [labelType="html" label="Exchange shuffle records written: 5 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 3 ms (0 ms, 1 ms, 1 ms (stage 8943.0: task 12272)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 5 local bytes read: 364.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 3 remote merged chunks fetched: 0 remote blocks read: 0 data size total (min, med, max (stageId: taskId)) 168.0 B (32.0 B, 64.0 B, 72.0 B (stage 8943.0: task 12271)) local merged bytes read: 0.0 B number of partitions: 10 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 364.0 B (71.0 B, 142.0 B, 151.0 B (stage 8943.0: task 12271))"]; subgraph cluster6 { isCluster="true"; label="WholeStageCodegen (4)\n \nduration: total (min, med, max (stageId: taskId))\n34 ms (9 ms, 12 ms, 13 ms (stage 8943.0: task 12272))"; 7 [labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build total (min, med, max (stageId: taskId)) 13 ms (3 ms, 5 ms, 5 ms (stage 8943.0: task 12271)) peak memory total (min, med, max (stageId: taskId)) 768.0 KiB (256.0 KiB, 256.0 KiB, 256.0 KiB (stage 8943.0: task 12273)) number of output rows: 5 number of sort fallback tasks: 0 avg hash probes per key: 0"]; } 8 [labelType="html" label=" Union "]; subgraph cluster9 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: 29 ms"; 10 [labelType="html" label=" Project "]; 11 [labelType="html" label="Filter number of output rows: 166"]; 12 [labelType="html" label="ColumnarToRow number of output rows: 166 number of input batches: 1"]; } 13 [labelType="html" label="Scan parquet number of files read: 1 scan time: 4 ms metadata time: 0 ms size of files read: 4.3 KiB number of output rows: 166"]; subgraph cluster14 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: 28 ms"; 15 [labelType="html" label=" Project "]; 16 [labelType="html" label="Filter number of output rows: 166"]; 17 [labelType="html" label="ColumnarToRow number of output rows: 166 number of input batches: 1"]; } 18 [labelType="html" label="Scan parquet number of files read: 1 scan time: 4 ms metadata time: 0 ms size of files read: 4.3 KiB number of output rows: 166"]; subgraph cluster19 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: 34 ms"; 20 [labelType="html" label=" Project "]; 21 [labelType="html" label="Filter number of output rows: 201"]; 22 [labelType="html" label="ColumnarToRow number of output rows: 201 number of input batches: 1"]; } 23 [labelType="html" label="Scan parquet number of files read: 1 scan time: 3 ms metadata time: 0 ms size of files read: 4.4 KiB number of output rows: 201"]; 1->0; 3->1; 4->3; 5->4; 7->5; 8->7; 10->8; 11->10; 12->11; 13->12; 15->8; 16->15; 17->16; 18->17; 20->8; 21->20; 22->21; 23->22; }

AdaptiveSparkPlan isFinalPlan=true

TakeOrderedAndProject(limit=10, orderBy=[COL_TRIGRAMME_2162722#729730L DESC NULLS LAST], output=[COL_TRIGRAMME_2162721#729721,COL_TRIGRAMME_2162722#729730L])

HashAggregate(keys=[COL_TRIGRAMME_216272#729714], functions=[count(1)])

WholeStageCodegen (5)

AQEShuffleRead coalesced

Exchange hashpartitioning(COL_TRIGRAMME_216272#729714, 10), ENSURE_REQUIREMENTS, [plan_id=444818]

HashAggregate(keys=[COL_TRIGRAMME_216272#729714], functions=[partial_count(1)])

WholeStageCodegen (4)

Union

Project [Trigramme#729424 AS COL_TRIGRAMME_216272#729714]

Filter isnotnull(Trigramme#729424)

ColumnarToRow

WholeStageCodegen (1)

FileScan parquet [Trigramme#729424] Batched: true, DataFilters: [isnotnull(Trigramme#729424)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/data/input/depot/excel/cache/TARIF_FORFAIT_S_216188_8c16894e8611..., PartitionFilters: [], PushedFilters: [IsNotNull(Trigramme)], ReadSchema: struct<Trigramme:string>

Project [Trigramme#729495 AS COL_TRIGRAMME_216272#729751]

Filter isnotnull(Trigramme#729495)

ColumnarToRow

WholeStageCodegen (2)

FileScan parquet [Trigramme#729495] Batched: true, DataFilters: [isnotnull(Trigramme#729495)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/data/input/depot/excel/cache/TARIF_FORFAIT_S_216188_40efcfa797c9..., PartitionFilters: [], PushedFilters: [IsNotNull(Trigramme)], ReadSchema: struct<Trigramme:string>

Project [Trigramme#729573 AS COL_TRIGRAMME_216272#729752]

Filter isnotnull(Trigramme#729573)

ColumnarToRow

WholeStageCodegen (3)

FileScan parquet [Trigramme#729573] Batched: true, DataFilters: [isnotnull(Trigramme#729573)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/data/input/depot/excel/cache/TARIF_FORFAIT_S_216188_808989d69c5c..., PartitionFilters: [], PushedFilters: [IsNotNull(Trigramme)], ReadSchema: struct<Trigramme:string>

Details

== Physical Plan ==
AdaptiveSparkPlan (31)
+- == Final Plan ==
   TakeOrderedAndProject (19)
   +- * HashAggregate (18)
      +- AQEShuffleRead (17)
         +- ShuffleQueryStage (16), Statistics(sizeInBytes=168.0 B, rowCount=5)
            +- Exchange (15)
               +- * HashAggregate (14)
                  +- Union (13)
                     :- * Project (4)
                     :  +- * Filter (3)
                     :     +- * ColumnarToRow (2)
                     :        +- Scan parquet  (1)
                     :- * Project (8)
                     :  +- * Filter (7)
                     :     +- * ColumnarToRow (6)
                     :        +- Scan parquet  (5)
                     +- * Project (12)
                        +- * Filter (11)
                           +- * ColumnarToRow (10)
                              +- Scan parquet  (9)
+- == Initial Plan ==
   TakeOrderedAndProject (30)
   +- HashAggregate (29)
      +- Exchange (28)
         +- HashAggregate (27)
            +- Union (26)
               :- Project (21)
               :  +- Filter (20)
               :     +- Scan parquet  (1)
               :- Project (23)
               :  +- Filter (22)
               :     +- Scan parquet  (5)
               +- Project (25)
                  +- Filter (24)
                     +- Scan parquet  (9)


(1) Scan parquet 
Output [1]: [Trigramme#729424]
Batched: true
Location: InMemoryFileIndex [file:/data/input/depot/excel/cache/TARIF_FORFAIT_S_216188_8c16894e8611e2723038563a84e8f82e/Region-PUBGUAMAR_tarifs_forfait_urgences.xlsx_28c6b5cf2e17f093396d9005fd88426d]
PushedFilters: [IsNotNull(Trigramme)]
ReadSchema: struct<Trigramme:string>

(2) ColumnarToRow [codegen id : 1]
Input [1]: [Trigramme#729424]

(3) Filter [codegen id : 1]
Input [1]: [Trigramme#729424]
Condition : isnotnull(Trigramme#729424)

(4) Project [codegen id : 1]
Output [1]: [Trigramme#729424 AS COL_TRIGRAMME_216272#729714]
Input [1]: [Trigramme#729424]

(5) Scan parquet 
Output [1]: [Trigramme#729495]
Batched: true
Location: InMemoryFileIndex [file:/data/input/depot/excel/cache/TARIF_FORFAIT_S_216188_40efcfa797c91f7b2d91dc7e15554bab/Region-PUBGUYRE_tarifs_forfait_urgences.xlsx_46d8ee25493717864f5499570cb757fd]
PushedFilters: [IsNotNull(Trigramme)]
ReadSchema: struct<Trigramme:string>

(6) ColumnarToRow [codegen id : 2]
Input [1]: [Trigramme#729495]

(7) Filter [codegen id : 2]
Input [1]: [Trigramme#729495]
Condition : isnotnull(Trigramme#729495)

(8) Project [codegen id : 2]
Output [1]: [Trigramme#729495 AS COL_TRIGRAMME_216272#729751]
Input [1]: [Trigramme#729495]

(9) Scan parquet 
Output [1]: [Trigramme#729573]
Batched: true
Location: InMemoryFileIndex [file:/data/input/depot/excel/cache/TARIF_FORFAIT_S_216188_808989d69c5c26978e67dd7f41818400/Region-PUBMET_tarifs_forfait_urgences.xlsx_85117cb97629c9c4984158ba77dd1307]
PushedFilters: [IsNotNull(Trigramme)]
ReadSchema: struct<Trigramme:string>

(10) ColumnarToRow [codegen id : 3]
Input [1]: [Trigramme#729573]

(11) Filter [codegen id : 3]
Input [1]: [Trigramme#729573]
Condition : isnotnull(Trigramme#729573)

(12) Project [codegen id : 3]
Output [1]: [Trigramme#729573 AS COL_TRIGRAMME_216272#729752]
Input [1]: [Trigramme#729573]

(13) Union

(14) HashAggregate [codegen id : 4]
Input [1]: [COL_TRIGRAMME_216272#729714]
Keys [1]: [COL_TRIGRAMME_216272#729714]
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#729753L]
Results [2]: [COL_TRIGRAMME_216272#729714, count#729754L]

(15) Exchange
Input [2]: [COL_TRIGRAMME_216272#729714, count#729754L]
Arguments: hashpartitioning(COL_TRIGRAMME_216272#729714, 10), ENSURE_REQUIREMENTS, [plan_id=444818]

(16) ShuffleQueryStage
Output [2]: [COL_TRIGRAMME_216272#729714, count#729754L]
Arguments: 0

(17) AQEShuffleRead
Input [2]: [COL_TRIGRAMME_216272#729714, count#729754L]
Arguments: coalesced

(18) HashAggregate [codegen id : 5]
Input [2]: [COL_TRIGRAMME_216272#729714, count#729754L]
Keys [1]: [COL_TRIGRAMME_216272#729714]
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#729728L]
Results [2]: [COL_TRIGRAMME_216272#729714 AS COL_TRIGRAMME_2162721#729721, count(1)#729728L AS COL_TRIGRAMME_2162722#729730L]

(19) TakeOrderedAndProject
Input [2]: [COL_TRIGRAMME_2162721#729721, COL_TRIGRAMME_2162722#729730L]
Arguments: 10, [COL_TRIGRAMME_2162722#729730L DESC NULLS LAST], [COL_TRIGRAMME_2162721#729721, COL_TRIGRAMME_2162722#729730L]

(20) Filter
Input [1]: [Trigramme#729424]
Condition : isnotnull(Trigramme#729424)

(21) Project
Output [1]: [Trigramme#729424 AS COL_TRIGRAMME_216272#729714]
Input [1]: [Trigramme#729424]

(22) Filter
Input [1]: [Trigramme#729495]
Condition : isnotnull(Trigramme#729495)

(23) Project
Output [1]: [Trigramme#729495 AS COL_TRIGRAMME_216272#729751]
Input [1]: [Trigramme#729495]

(24) Filter
Input [1]: [Trigramme#729573]
Condition : isnotnull(Trigramme#729573)

(25) Project
Output [1]: [Trigramme#729573 AS COL_TRIGRAMME_216272#729752]
Input [1]: [Trigramme#729573]

(26) Union

(27) HashAggregate
Input [1]: [COL_TRIGRAMME_216272#729714]
Keys [1]: [COL_TRIGRAMME_216272#729714]
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#729753L]
Results [2]: [COL_TRIGRAMME_216272#729714, count#729754L]

(28) Exchange
Input [2]: [COL_TRIGRAMME_216272#729714, count#729754L]
Arguments: hashpartitioning(COL_TRIGRAMME_216272#729714, 10), ENSURE_REQUIREMENTS, [plan_id=444764]

(29) HashAggregate
Input [2]: [COL_TRIGRAMME_216272#729714, count#729754L]
Keys [1]: [COL_TRIGRAMME_216272#729714]
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#729728L]
Results [2]: [COL_TRIGRAMME_216272#729714 AS COL_TRIGRAMME_2162721#729721, count(1)#729728L AS COL_TRIGRAMME_2162722#729730L]

(30) TakeOrderedAndProject
Input [2]: [COL_TRIGRAMME_2162721#729721, COL_TRIGRAMME_2162722#729730L]
Arguments: 10, [COL_TRIGRAMME_2162722#729730L DESC NULLS LAST], [COL_TRIGRAMME_2162721#729721, COL_TRIGRAMME_2162722#729730L]

(31) AdaptiveSparkPlan
Output [2]: [COL_TRIGRAMME_2162721#729721, COL_TRIGRAMME_2162722#729730L]
Arguments: isFinalPlan=true

SQL / DataFrame Properties

Name	Value
spark.sql.optimizer.nestedPredicatePushdown.supportedFileSources	parquet,orc,geoparquet