SP1 - Details for Query 6932

Submitted Time: 2026/10/27 17:18:27
Duration: 0.9 s
Succeeded Jobs: 10436 10437 10438

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; 1 [labelType="html" label=" TakeOrderedAndProject "]; subgraph cluster2 { isCluster="true"; label="WholeStageCodegen (5)\n \nduration: 228 ms"; 3 [labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build: 208 ms peak memory: 17.0 MiB number of output rows: 21,121 number of sort fallback tasks: 0 avg hash probes per key: 1.5"]; 4 [labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build: 192 ms peak memory: 16.3 MiB number of output rows: 21,121 number of sort fallback tasks: 0 avg hash probes per key: 1"]; 5 [labelType="html" label=" Project "]; 6 [labelType="html" label="SortMergeJoin number of output rows: 21,121 spill size: 0.0 B"]; } subgraph cluster7 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: 253 ms"; 8 [labelType="html" label="Sort sort time: 21 ms peak memory: 16.5 MiB spill size: 0.0 B"]; } 9 [labelType="html" label="AQEShuffleRead number of partitions: 1 partition data size: 175.9 KiB number of coalesced partitions: 1"]; 10 [labelType="html" label="Exchange shuffle records written: 21,121 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 5 ms (1 ms, 1 ms, 1 ms (stage 20849.0: task 27140)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 21,121 local bytes read: 166.3 KiB fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 3 remote merged chunks fetched: 0 remote blocks read: 0 data size total (min, med, max (stageId: taskId)) 660.0 KiB (20.7 KiB, 319.7 KiB, 319.7 KiB (stage 20849.0: task 27138)) local merged bytes read: 0.0 B number of partitions: 10 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 166.3 KiB (6.7 KiB, 79.7 KiB, 79.9 KiB (stage 20849.0: task 27138))"]; subgraph cluster11 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n357 ms (23 ms, 163 ms, 171 ms (stage 20849.0: task 27139))"; 12 [labelType="html" label=" Project "]; 13 [labelType="html" label="Filter number of output rows: 21,121"]; } 14 [labelType="html" label="Scan csv number of output rows: 21,121 number of files read: 1 metadata time: 0 ms size of files read: 8.3 MiB"]; subgraph cluster15 { isCluster="true"; label="WholeStageCodegen (4)\n \nduration: 226 ms"; 16 [labelType="html" label="Sort sort time: 5 ms peak memory: 16.5 MiB spill size: 0.0 B"]; 17 [labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build: 35 ms peak memory: 16.3 MiB number of output rows: 21,121 number of sort fallback tasks: 0 avg hash probes per key: 1"]; } 18 [labelType="html" label="AQEShuffleRead number of partitions: 1 partition data size: 159.0 KiB number of coalesced partitions: 1"]; 19 [labelType="html" label="Exchange shuffle records written: 21,122 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 4 ms (1 ms, 1 ms, 1 ms (stage 20850.0: task 27143)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 21,122 local bytes read: 150.8 KiB fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 3 remote merged chunks fetched: 0 remote blocks read: 0 data size total (min, med, max (stageId: taskId)) 660.1 KiB (5.3 KiB, 316.2 KiB, 338.6 KiB (stage 20850.0: task 27142)) local merged bytes read: 0.0 B number of partitions: 10 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 150.8 KiB (2.3 KiB, 71.5 KiB, 77.0 KiB (stage 20850.0: task 27142))"]; subgraph cluster20 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: total (min, med, max (stageId: taskId))\n281 ms (16 ms, 132 ms, 133 ms (stage 20850.0: task 27141))"; 21 [labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build total (min, med, max (stageId: taskId)) 226 ms (5 ms, 109 ms, 112 ms (stage 20850.0: task 27141)) peak memory total (min, med, max (stageId: taskId)) 32.8 MiB (256.0 KiB, 16.3 MiB, 16.3 MiB (stage 20850.0: task 27142)) number of output rows: 21,122 number of sort fallback tasks: 0 avg hash probes per key (min, med, max (stageId: taskId)): (1, 1, 1 (stage 20850.0: task 27142))"]; 22 [labelType="html" label=" Project "]; 23 [labelType="html" label="Filter number of output rows: 26,044"]; } 24 [labelType="html" label="Scan csv number of output rows: 26,044 number of files read: 1 metadata time: 0 ms size of files read: 8.0 MiB"]; 1->0; 3->1; 4->3; 5->4; 6->5; 8->6; 9->8; 10->9; 12->10; 13->12; 14->13; 16->6; 17->16; 18->17; 19->18; 21->19; 22->21; 23->22; 24->23; }

AdaptiveSparkPlan isFinalPlan=true

TakeOrderedAndProject(limit=10, orderBy=[COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D42#1650414L DESC NULLS LAST], output=[COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D41#1650387,COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D42#1650414L])

HashAggregate(keys=[COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D4#1650356], functions=[count(1)])

HashAggregate(keys=[COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D4#1650356], functions=[partial_count(1)])

Project [RSS_DECOUPAGE_237747_COL_5588E97C_F5A9_4AFC_AD58_2B0A6381409A#1649831 AS COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D4#1650356]

SortMergeJoin [COL_7948E047_29EE_472A_BFFC_723098878DAD#1649569], [RSS_DECOUPAGE_237747_COL_5588E97C_F5A9_4AFC_AD58_2B0A6381409A#1649831], Inner

WholeStageCodegen (5)

Sort [COL_7948E047_29EE_472A_BFFC_723098878DAD#1649569 ASC NULLS FIRST], false, 0

WholeStageCodegen (3)

AQEShuffleRead coalesced

Exchange hashpartitioning(COL_7948E047_29EE_472A_BFFC_723098878DAD#1649569, 10), ENSURE_REQUIREMENTS, [plan_id=1025728]

Project [str_remove_space(NO_ADMIN#1643798, ALL) AS COL_7948E047_29EE_472A_BFFC_723098878DAD#1649569]

Filter isnotnull(str_remove_space(NO_ADMIN#1643798, ALL))

WholeStageCodegen (1)

FileScan csv [NO_ADMIN#1643798] Batched: false, DataFilters: [isnotnull(str_remove_space(NO_ADMIN#1643798, ALL))], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/data/input/depot/csv/tmp/VISUALVALOSEJOU_237544/420780652.2025.1..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<NO_ADMIN:string>

Sort [RSS_DECOUPAGE_237747_COL_5588E97C_F5A9_4AFC_AD58_2B0A6381409A#1649831 ASC NULLS FIRST], false, 0

HashAggregate(keys=[COL_0FDBEAA7_683C_4647_A06F_34AC7E6418C2#1643478], functions=[])

WholeStageCodegen (4)

AQEShuffleRead coalesced

Exchange hashpartitioning(COL_0FDBEAA7_683C_4647_A06F_34AC7E6418C2#1643478, 10), ENSURE_REQUIREMENTS, [plan_id=1025753]

HashAggregate(keys=[COL_0FDBEAA7_683C_4647_A06F_34AC7E6418C2#1643478], functions=[])

Project [str_remove_space(str_extract_middle(_c0#1642899, 48, 20), ALL) AS COL_0FDBEAA7_683C_4647_A06F_34AC7E6418C2#1643478]

Filter isnotnull(str_remove_space(str_extract_middle(_c0#1642899, 48, 20), ALL))

WholeStageCodegen (2)

FileScan csv [_c0#1642899] Batched: false, DataFilters: [isnotnull(str_remove_space(str_extract_middle(_c0#1642899, 48, 20), ALL))], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/data/input/depot/csv/tmp/RSS_M12_2025_237542/RSS- 420780652.txt], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<_c0:string>

Details

== Physical Plan ==
AdaptiveSparkPlan (37)
+- == Final Plan ==
   TakeOrderedAndProject (21)
   +- * HashAggregate (20)
      +- * HashAggregate (19)
         +- * Project (18)
            +- * SortMergeJoin Inner (17)
               :- * Sort (7)
               :  +- AQEShuffleRead (6)
               :     +- ShuffleQueryStage (5), Statistics(sizeInBytes=660.0 KiB, rowCount=2.11E+4)
               :        +- Exchange (4)
               :           +- * Project (3)
               :              +- * Filter (2)
               :                 +- Scan csv  (1)
               +- * Sort (16)
                  +- * HashAggregate (15)
                     +- AQEShuffleRead (14)
                        +- ShuffleQueryStage (13), Statistics(sizeInBytes=660.1 KiB, rowCount=2.11E+4)
                           +- Exchange (12)
                              +- * HashAggregate (11)
                                 +- * Project (10)
                                    +- * Filter (9)
                                       +- Scan csv  (8)
+- == Initial Plan ==
   TakeOrderedAndProject (36)
   +- HashAggregate (35)
      +- HashAggregate (34)
         +- Project (33)
            +- SortMergeJoin Inner (32)
               :- Sort (25)
               :  +- Exchange (24)
               :     +- Project (23)
               :        +- Filter (22)
               :           +- Scan csv  (1)
               +- Sort (31)
                  +- HashAggregate (30)
                     +- Exchange (29)
                        +- HashAggregate (28)
                           +- Project (27)
                              +- Filter (26)
                                 +- Scan csv  (8)


(1) Scan csv 
Output [1]: [NO_ADMIN#1643798]
Batched: false
Location: InMemoryFileIndex [file:/data/input/depot/csv/tmp/VISUALVALOSEJOU_237544/420780652.2025.12.MCO.VisualValoSejours.csv]
ReadSchema: struct<NO_ADMIN:string>

(2) Filter [codegen id : 1]
Input [1]: [NO_ADMIN#1643798]
Condition : isnotnull(str_remove_space(NO_ADMIN#1643798, ALL))

(3) Project [codegen id : 1]
Output [1]: [str_remove_space(NO_ADMIN#1643798, ALL) AS COL_7948E047_29EE_472A_BFFC_723098878DAD#1649569]
Input [1]: [NO_ADMIN#1643798]

(4) Exchange
Input [1]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1649569]
Arguments: hashpartitioning(COL_7948E047_29EE_472A_BFFC_723098878DAD#1649569, 10), ENSURE_REQUIREMENTS, [plan_id=1025728]

(5) ShuffleQueryStage
Output [1]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1649569]
Arguments: 0

(6) AQEShuffleRead
Input [1]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1649569]
Arguments: coalesced

(7) Sort [codegen id : 3]
Input [1]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1649569]
Arguments: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1649569 ASC NULLS FIRST], false, 0

(8) Scan csv 
Output [1]: [_c0#1642899]
Batched: false
Location: InMemoryFileIndex [file:/data/input/depot/csv/tmp/RSS_M12_2025_237542/RSS- 420780652.txt]
ReadSchema: struct<_c0:string>

(9) Filter [codegen id : 2]
Input [1]: [_c0#1642899]
Condition : isnotnull(str_remove_space(str_extract_middle(_c0#1642899, 48, 20), ALL))

(10) Project [codegen id : 2]
Output [1]: [str_remove_space(str_extract_middle(_c0#1642899, 48, 20), ALL) AS COL_0FDBEAA7_683C_4647_A06F_34AC7E6418C2#1643478]
Input [1]: [_c0#1642899]

(11) HashAggregate [codegen id : 2]
Input [1]: [COL_0FDBEAA7_683C_4647_A06F_34AC7E6418C2#1643478]
Keys [1]: [COL_0FDBEAA7_683C_4647_A06F_34AC7E6418C2#1643478]
Functions: []
Aggregate Attributes: []
Results [1]: [COL_0FDBEAA7_683C_4647_A06F_34AC7E6418C2#1643478]

(12) Exchange
Input [1]: [COL_0FDBEAA7_683C_4647_A06F_34AC7E6418C2#1643478]
Arguments: hashpartitioning(COL_0FDBEAA7_683C_4647_A06F_34AC7E6418C2#1643478, 10), ENSURE_REQUIREMENTS, [plan_id=1025753]

(13) ShuffleQueryStage
Output [1]: [COL_0FDBEAA7_683C_4647_A06F_34AC7E6418C2#1643478]
Arguments: 1

(14) AQEShuffleRead
Input [1]: [COL_0FDBEAA7_683C_4647_A06F_34AC7E6418C2#1643478]
Arguments: coalesced

(15) HashAggregate [codegen id : 4]
Input [1]: [COL_0FDBEAA7_683C_4647_A06F_34AC7E6418C2#1643478]
Keys [1]: [COL_0FDBEAA7_683C_4647_A06F_34AC7E6418C2#1643478]
Functions: []
Aggregate Attributes: []
Results [1]: [COL_0FDBEAA7_683C_4647_A06F_34AC7E6418C2#1643478 AS RSS_DECOUPAGE_237747_COL_5588E97C_F5A9_4AFC_AD58_2B0A6381409A#1649831]

(16) Sort [codegen id : 4]
Input [1]: [RSS_DECOUPAGE_237747_COL_5588E97C_F5A9_4AFC_AD58_2B0A6381409A#1649831]
Arguments: [RSS_DECOUPAGE_237747_COL_5588E97C_F5A9_4AFC_AD58_2B0A6381409A#1649831 ASC NULLS FIRST], false, 0

(17) SortMergeJoin [codegen id : 5]
Left keys [1]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1649569]
Right keys [1]: [RSS_DECOUPAGE_237747_COL_5588E97C_F5A9_4AFC_AD58_2B0A6381409A#1649831]
Join type: Inner
Join condition: None

(18) Project [codegen id : 5]
Output [1]: [RSS_DECOUPAGE_237747_COL_5588E97C_F5A9_4AFC_AD58_2B0A6381409A#1649831 AS COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D4#1650356]
Input [2]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1649569, RSS_DECOUPAGE_237747_COL_5588E97C_F5A9_4AFC_AD58_2B0A6381409A#1649831]

(19) HashAggregate [codegen id : 5]
Input [1]: [COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D4#1650356]
Keys [1]: [COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D4#1650356]
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#1650421L]
Results [2]: [COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D4#1650356, count#1650422L]

(20) HashAggregate [codegen id : 5]
Input [2]: [COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D4#1650356, count#1650422L]
Keys [1]: [COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D4#1650356]
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#1650412L]
Results [2]: [COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D4#1650356 AS COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D41#1650387, count(1)#1650412L AS COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D42#1650414L]

(21) TakeOrderedAndProject
Input [2]: [COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D41#1650387, COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D42#1650414L]
Arguments: 10, [COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D42#1650414L DESC NULLS LAST], [COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D41#1650387, COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D42#1650414L]

(22) Filter
Input [1]: [NO_ADMIN#1643798]
Condition : isnotnull(str_remove_space(NO_ADMIN#1643798, ALL))

(23) Project
Output [1]: [str_remove_space(NO_ADMIN#1643798, ALL) AS COL_7948E047_29EE_472A_BFFC_723098878DAD#1649569]
Input [1]: [NO_ADMIN#1643798]

(24) Exchange
Input [1]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1649569]
Arguments: hashpartitioning(COL_7948E047_29EE_472A_BFFC_723098878DAD#1649569, 10), ENSURE_REQUIREMENTS, [plan_id=1025709]

(25) Sort
Input [1]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1649569]
Arguments: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1649569 ASC NULLS FIRST], false, 0

(26) Filter
Input [1]: [_c0#1642899]
Condition : isnotnull(str_remove_space(str_extract_middle(_c0#1642899, 48, 20), ALL))

(27) Project
Output [1]: [str_remove_space(str_extract_middle(_c0#1642899, 48, 20), ALL) AS COL_0FDBEAA7_683C_4647_A06F_34AC7E6418C2#1643478]
Input [1]: [_c0#1642899]

(28) HashAggregate
Input [1]: [COL_0FDBEAA7_683C_4647_A06F_34AC7E6418C2#1643478]
Keys [1]: [COL_0FDBEAA7_683C_4647_A06F_34AC7E6418C2#1643478]
Functions: []
Aggregate Attributes: []
Results [1]: [COL_0FDBEAA7_683C_4647_A06F_34AC7E6418C2#1643478]

(29) Exchange
Input [1]: [COL_0FDBEAA7_683C_4647_A06F_34AC7E6418C2#1643478]
Arguments: hashpartitioning(COL_0FDBEAA7_683C_4647_A06F_34AC7E6418C2#1643478, 10), ENSURE_REQUIREMENTS, [plan_id=1025705]

(30) HashAggregate
Input [1]: [COL_0FDBEAA7_683C_4647_A06F_34AC7E6418C2#1643478]
Keys [1]: [COL_0FDBEAA7_683C_4647_A06F_34AC7E6418C2#1643478]
Functions: []
Aggregate Attributes: []
Results [1]: [COL_0FDBEAA7_683C_4647_A06F_34AC7E6418C2#1643478 AS RSS_DECOUPAGE_237747_COL_5588E97C_F5A9_4AFC_AD58_2B0A6381409A#1649831]

(31) Sort
Input [1]: [RSS_DECOUPAGE_237747_COL_5588E97C_F5A9_4AFC_AD58_2B0A6381409A#1649831]
Arguments: [RSS_DECOUPAGE_237747_COL_5588E97C_F5A9_4AFC_AD58_2B0A6381409A#1649831 ASC NULLS FIRST], false, 0

(32) SortMergeJoin
Left keys [1]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1649569]
Right keys [1]: [RSS_DECOUPAGE_237747_COL_5588E97C_F5A9_4AFC_AD58_2B0A6381409A#1649831]
Join type: Inner
Join condition: None

(33) Project
Output [1]: [RSS_DECOUPAGE_237747_COL_5588E97C_F5A9_4AFC_AD58_2B0A6381409A#1649831 AS COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D4#1650356]
Input [2]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1649569, RSS_DECOUPAGE_237747_COL_5588E97C_F5A9_4AFC_AD58_2B0A6381409A#1649831]

(34) HashAggregate
Input [1]: [COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D4#1650356]
Keys [1]: [COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D4#1650356]
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#1650421L]
Results [2]: [COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D4#1650356, count#1650422L]

(35) HashAggregate
Input [2]: [COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D4#1650356, count#1650422L]
Keys [1]: [COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D4#1650356]
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#1650412L]
Results [2]: [COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D4#1650356 AS COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D41#1650387, count(1)#1650412L AS COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D42#1650414L]

(36) TakeOrderedAndProject
Input [2]: [COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D41#1650387, COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D42#1650414L]
Arguments: 10, [COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D42#1650414L DESC NULLS LAST], [COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D41#1650387, COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D42#1650414L]

(37) AdaptiveSparkPlan
Output [2]: [COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D41#1650387, COL_1D43C4C0_AE5C_43DC_ABA0_79E72CBFE6D42#1650414L]
Arguments: isFinalPlan=true

SQL / DataFrame Properties

Name	Value
spark.sql.optimizer.nestedPredicatePushdown.supportedFileSources	parquet,orc,geoparquet