SP1 - Details for Query 6975

Submitted Time: 2026/10/27 17:20:22
Duration: 0.5 s
Succeeded Jobs: 10466 10467 10468 10469 10470 10471

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; subgraph cluster1 { isCluster="true"; label="WholeStageCodegen (10)\n \nduration: 0 ms"; 2 [labelType="html" label="HashAggregate time in aggregation build: 0 ms number of output rows: 1"]; } 3 [labelType="html" label="Exchange shuffle records written: 1 local merged chunks fetched: 0 shuffle write time: 0 ms remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 1 local bytes read: 59.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 1 remote merged chunks fetched: 0 remote blocks read: 0 data size: 16.0 B local merged bytes read: 0.0 B number of partitions: 1 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written: 59.0 B"]; subgraph cluster4 { isCluster="true"; label="WholeStageCodegen (9)\n \nduration: 21 ms"; 5 [labelType="html" label="HashAggregate time in aggregation build: 20 ms number of output rows: 1"]; 6 [labelType="html" label=" Project "]; 7 [labelType="html" label="SortMergeJoin number of output rows: 21,131"]; } subgraph cluster8 { isCluster="true"; label="WholeStageCodegen (7)\n \nduration: 23 ms"; 9 [labelType="html" label="Sort sort time: 3 ms peak memory: 16.5 MiB spill size: 0.0 B"]; } 10 [labelType="html" label="AQEShuffleRead number of partitions: 1 partition data size: 148.0 KiB number of coalesced partitions: 1"]; 11 [labelType="html" label="Exchange shuffle records written: 21,127 local merged chunks fetched: 0 shuffle write time: 1 ms remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 21,127 local bytes read: 141.1 KiB fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 1 remote merged chunks fetched: 0 remote blocks read: 0 data size: 660.1 KiB local merged bytes read: 0.0 B number of partitions: 10 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written: 141.1 KiB"]; subgraph cluster12 { isCluster="true"; label="WholeStageCodegen (6)\n \nduration: 40 ms"; 13 [labelType="html" label=" Project "]; 14 [labelType="html" label="SortMergeJoin number of output rows: 21,127 spill size: 0.0 B"]; } subgraph cluster15 { isCluster="true"; label="WholeStageCodegen (4)\n \nduration: 106 ms"; 16 [labelType="html" label="Sort sort time: 7 ms peak memory: 16.5 MiB spill size: 0.0 B"]; } 17 [labelType="html" label="AQEShuffleRead number of partitions: 1 partition data size: 175.9 KiB number of coalesced partitions: 1"]; 18 [labelType="html" label="Exchange shuffle records written: 21,121 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 6 ms (1 ms, 1 ms, 3 ms (stage 20903.0: task 27178)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 21,121 local bytes read: 166.3 KiB fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 3 remote merged chunks fetched: 0 remote blocks read: 0 data size total (min, med, max (stageId: taskId)) 660.0 KiB (20.7 KiB, 319.7 KiB, 319.7 KiB (stage 20903.0: task 27176)) local merged bytes read: 0.0 B number of partitions: 10 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 166.3 KiB (6.7 KiB, 79.7 KiB, 79.9 KiB (stage 20903.0: task 27176))"]; subgraph cluster19 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n379 ms (82 ms, 145 ms, 152 ms (stage 20903.0: task 27177))"; 20 [labelType="html" label=" Project "]; } 21 [labelType="html" label="Scan csv number of output rows: 21,121 number of files read: 1 metadata time: 0 ms size of files read: 8.3 MiB"]; subgraph cluster22 { isCluster="true"; label="WholeStageCodegen (5)\n \nduration: 1.2 m"; 23 [labelType="html" label="Sort sort time: 0 ms peak memory: 16.1 MiB spill size: 0.0 B"]; 24 [labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build: 0 ms peak memory: 256.0 KiB number of output rows: 445 number of sort fallback tasks: 0 avg hash probes per key: 0"]; } 25 [labelType="html" label="AQEShuffleRead number of partitions: 1 partition data size: 5.0 KiB number of coalesced partitions: 1"]; 26 [labelType="html" label="Exchange shuffle records written: 445 local merged chunks fetched: 0 shuffle write time: 1 ms remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 445 local bytes read: 4.8 KiB fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 1 remote merged chunks fetched: 0 remote blocks read: 0 data size: 13.9 KiB local merged bytes read: 0.0 B number of partitions: 10 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written: 4.8 KiB"]; subgraph cluster27 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: 16 ms"; 28 [labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build: 11 ms peak memory: 256.0 KiB number of output rows: 445 number of sort fallback tasks: 0 avg hash probes per key: 0"]; 29 [labelType="html" label=" Project "]; } 30 [labelType="html" label="Scan csv number of output rows: 1,216 number of files read: 1 metadata time: 0 ms size of files read: 111.6 KiB"]; subgraph cluster31 { isCluster="true"; label="WholeStageCodegen (8)\n \nduration: 42 ms"; 32 [labelType="html" label="Sort sort time: 0 ms peak memory: 16.1 MiB spill size: 0.0 B"]; 33 [labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build: 0 ms peak memory: 256.0 KiB number of output rows: 464 number of sort fallback tasks: 0 avg hash probes per key: 0"]; } 34 [labelType="html" label="AQEShuffleRead number of partitions: 1 partition data size: 5.2 KiB number of coalesced partitions: 1"]; 35 [labelType="html" label="Exchange shuffle records written: 464 local merged chunks fetched: 0 shuffle write time: 2 ms remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 464 local bytes read: 5.0 KiB fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 1 remote merged chunks fetched: 0 remote blocks read: 0 data size: 14.5 KiB local merged bytes read: 0.0 B number of partitions: 10 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written: 5.0 KiB"]; subgraph cluster36 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: 14 ms"; 37 [labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build: 8 ms peak memory: 256.0 KiB number of output rows: 464 number of sort fallback tasks: 0 avg hash probes per key: 0"]; 38 [labelType="html" label=" Project "]; } 39 [labelType="html" label="Scan csv number of output rows: 486 number of files read: 1 metadata time: 0 ms size of files read: 50.8 KiB"]; 2->0; 3->2; 5->3; 6->5; 7->6; 9->7; 10->9; 11->10; 13->11; 14->13; 16->14; 17->16; 18->17; 20->18; 21->20; 23->14; 24->23; 25->24; 26->25; 28->26; 29->28; 30->29; 32->7; 33->32; 34->33; 35->34; 37->35; 38->37; 39->38; }

AdaptiveSparkPlan isFinalPlan=true

HashAggregate(keys=[], functions=[count(1)])

WholeStageCodegen (10)

Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=1028248]

HashAggregate(keys=[], functions=[partial_count(1)])

Project

SortMergeJoin [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940], [01_MOLECULES_ON_237708_COL_EE4AB0B4_F37E_40C0_9292_9E68BF166E36#1667065], FullOuter

WholeStageCodegen (9)

Sort [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940 ASC NULLS FIRST], false, 0

WholeStageCodegen (7)

AQEShuffleRead coalesced

Exchange hashpartitioning(COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940, 10), ENSURE_REQUIREMENTS, [plan_id=1028166]

Project [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940]

SortMergeJoin [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940], [01_DMI_DECOUPAG_237714_COL_1D1ABD09_1E00_4FA7_81FE_A558E22F6A6D#1666954], FullOuter

WholeStageCodegen (6)

Sort [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940 ASC NULLS FIRST], false, 0

WholeStageCodegen (4)

AQEShuffleRead coalesced

Exchange hashpartitioning(COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940, 10), ENSURE_REQUIREMENTS, [plan_id=1027958]

Project [str_remove_space(NO_ADMIN#1661280, ALL) AS COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940]

WholeStageCodegen (1)

FileScan csv [NO_ADMIN#1661280] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/data/input/depot/csv/tmp/VISUALVALOSEJOU_237544/420780652.2025.1..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<NO_ADMIN:string>

Sort [01_DMI_DECOUPAG_237714_COL_1D1ABD09_1E00_4FA7_81FE_A558E22F6A6D#1666954 ASC NULLS FIRST], false, 0

HashAggregate(keys=[COL_9D7AAEDE_94EE_40AF_8530_BC9D1F27742A#1664641], functions=[])

WholeStageCodegen (5)

AQEShuffleRead coalesced

Exchange hashpartitioning(COL_9D7AAEDE_94EE_40AF_8530_BC9D1F27742A#1664641, 10), ENSURE_REQUIREMENTS, [plan_id=1027971]

HashAggregate(keys=[COL_9D7AAEDE_94EE_40AF_8530_BC9D1F27742A#1664641], functions=[])

Project [str_remove_space(str_extract_middle(_c0#1664070, 12, 20), ALL) AS COL_9D7AAEDE_94EE_40AF_8530_BC9D1F27742A#1664641]

WholeStageCodegen (2)

FileScan csv [_c0#1664070] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/data/input/depot/csv/tmp/DMI_M12_2025_237543/DMI-M12-2025 - BIS...., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<_c0:string>

Sort [01_MOLECULES_ON_237708_COL_EE4AB0B4_F37E_40C0_9292_9E68BF166E36#1667065 ASC NULLS FIRST], false, 0

HashAggregate(keys=[COL_45066C9F_40FD_4BE1_ACAF_6139C274CECB#1663658], functions=[])

WholeStageCodegen (8)

AQEShuffleRead coalesced

Exchange hashpartitioning(COL_45066C9F_40FD_4BE1_ACAF_6139C274CECB#1663658, 10), ENSURE_REQUIREMENTS, [plan_id=1027991]

HashAggregate(keys=[COL_45066C9F_40FD_4BE1_ACAF_6139C274CECB#1663658], functions=[])

Project [str_remove_space(str_extract_middle(_c0#1663526, 12, 20), ALL) AS COL_45066C9F_40FD_4BE1_ACAF_6139C274CECB#1663658]

WholeStageCodegen (3)

FileScan csv [_c0#1663526] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/data/input/depot/csv/tmp/MOLECULES_ONERE_237545/MO-M12-2025.txt], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<_c0:string>

Details

== Physical Plan ==
AdaptiveSparkPlan (57)
+- == Final Plan ==
   * HashAggregate (34)
   +- ShuffleQueryStage (33), Statistics(sizeInBytes=16.0 B, rowCount=1)
      +- Exchange (32)
         +- * HashAggregate (31)
            +- * Project (30)
               +- * SortMergeJoin FullOuter (29)
                  :- * Sort (20)
                  :  +- AQEShuffleRead (19)
                  :     +- ShuffleQueryStage (18), Statistics(sizeInBytes=660.1 KiB, rowCount=2.11E+4)
                  :        +- Exchange (17)
                  :           +- * Project (16)
                  :              +- * SortMergeJoin FullOuter (15)
                  :                 :- * Sort (6)
                  :                 :  +- AQEShuffleRead (5)
                  :                 :     +- ShuffleQueryStage (4), Statistics(sizeInBytes=660.0 KiB, rowCount=2.11E+4)
                  :                 :        +- Exchange (3)
                  :                 :           +- * Project (2)
                  :                 :              +- Scan csv  (1)
                  :                 +- * Sort (14)
                  :                    +- * HashAggregate (13)
                  :                       +- AQEShuffleRead (12)
                  :                          +- ShuffleQueryStage (11), Statistics(sizeInBytes=13.9 KiB, rowCount=445)
                  :                             +- Exchange (10)
                  :                                +- * HashAggregate (9)
                  :                                   +- * Project (8)
                  :                                      +- Scan csv  (7)
                  +- * Sort (28)
                     +- * HashAggregate (27)
                        +- AQEShuffleRead (26)
                           +- ShuffleQueryStage (25), Statistics(sizeInBytes=14.5 KiB, rowCount=464)
                              +- Exchange (24)
                                 +- * HashAggregate (23)
                                    +- * Project (22)
                                       +- Scan csv  (21)
+- == Initial Plan ==
   HashAggregate (56)
   +- Exchange (55)
      +- HashAggregate (54)
         +- Project (53)
            +- SortMergeJoin FullOuter (52)
               :- Sort (46)
               :  +- Exchange (45)
               :     +- Project (44)
               :        +- SortMergeJoin FullOuter (43)
               :           :- Sort (37)
               :           :  +- Exchange (36)
               :           :     +- Project (35)
               :           :        +- Scan csv  (1)
               :           +- Sort (42)
               :              +- HashAggregate (41)
               :                 +- Exchange (40)
               :                    +- HashAggregate (39)
               :                       +- Project (38)
               :                          +- Scan csv  (7)
               +- Sort (51)
                  +- HashAggregate (50)
                     +- Exchange (49)
                        +- HashAggregate (48)
                           +- Project (47)
                              +- Scan csv  (21)


(1) Scan csv 
Output [1]: [NO_ADMIN#1661280]
Batched: false
Location: InMemoryFileIndex [file:/data/input/depot/csv/tmp/VISUALVALOSEJOU_237544/420780652.2025.12.MCO.VisualValoSejours.csv]
ReadSchema: struct<NO_ADMIN:string>

(2) Project [codegen id : 1]
Output [1]: [str_remove_space(NO_ADMIN#1661280, ALL) AS COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940]
Input [1]: [NO_ADMIN#1661280]

(3) Exchange
Input [1]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940]
Arguments: hashpartitioning(COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940, 10), ENSURE_REQUIREMENTS, [plan_id=1027958]

(4) ShuffleQueryStage
Output [1]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940]
Arguments: 0

(5) AQEShuffleRead
Input [1]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940]
Arguments: coalesced

(6) Sort [codegen id : 4]
Input [1]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940]
Arguments: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940 ASC NULLS FIRST], false, 0

(7) Scan csv 
Output [1]: [_c0#1664070]
Batched: false
Location: InMemoryFileIndex [file:/data/input/depot/csv/tmp/DMI_M12_2025_237543/DMI-M12-2025 - BIS.txt]
ReadSchema: struct<_c0:string>

(8) Project [codegen id : 2]
Output [1]: [str_remove_space(str_extract_middle(_c0#1664070, 12, 20), ALL) AS COL_9D7AAEDE_94EE_40AF_8530_BC9D1F27742A#1664641]
Input [1]: [_c0#1664070]

(9) HashAggregate [codegen id : 2]
Input [1]: [COL_9D7AAEDE_94EE_40AF_8530_BC9D1F27742A#1664641]
Keys [1]: [COL_9D7AAEDE_94EE_40AF_8530_BC9D1F27742A#1664641]
Functions: []
Aggregate Attributes: []
Results [1]: [COL_9D7AAEDE_94EE_40AF_8530_BC9D1F27742A#1664641]

(10) Exchange
Input [1]: [COL_9D7AAEDE_94EE_40AF_8530_BC9D1F27742A#1664641]
Arguments: hashpartitioning(COL_9D7AAEDE_94EE_40AF_8530_BC9D1F27742A#1664641, 10), ENSURE_REQUIREMENTS, [plan_id=1027971]

(11) ShuffleQueryStage
Output [1]: [COL_9D7AAEDE_94EE_40AF_8530_BC9D1F27742A#1664641]
Arguments: 1

(12) AQEShuffleRead
Input [1]: [COL_9D7AAEDE_94EE_40AF_8530_BC9D1F27742A#1664641]
Arguments: coalesced

(13) HashAggregate [codegen id : 5]
Input [1]: [COL_9D7AAEDE_94EE_40AF_8530_BC9D1F27742A#1664641]
Keys [1]: [COL_9D7AAEDE_94EE_40AF_8530_BC9D1F27742A#1664641]
Functions: []
Aggregate Attributes: []
Results [1]: [COL_9D7AAEDE_94EE_40AF_8530_BC9D1F27742A#1664641 AS 01_DMI_DECOUPAG_237714_COL_1D1ABD09_1E00_4FA7_81FE_A558E22F6A6D#1666954]

(14) Sort [codegen id : 5]
Input [1]: [01_DMI_DECOUPAG_237714_COL_1D1ABD09_1E00_4FA7_81FE_A558E22F6A6D#1666954]
Arguments: [01_DMI_DECOUPAG_237714_COL_1D1ABD09_1E00_4FA7_81FE_A558E22F6A6D#1666954 ASC NULLS FIRST], false, 0

(15) SortMergeJoin [codegen id : 6]
Left keys [1]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940]
Right keys [1]: [01_DMI_DECOUPAG_237714_COL_1D1ABD09_1E00_4FA7_81FE_A558E22F6A6D#1666954]
Join type: FullOuter
Join condition: None

(16) Project [codegen id : 6]
Output [1]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940]
Input [2]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940, 01_DMI_DECOUPAG_237714_COL_1D1ABD09_1E00_4FA7_81FE_A558E22F6A6D#1666954]

(17) Exchange
Input [1]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940]
Arguments: hashpartitioning(COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940, 10), ENSURE_REQUIREMENTS, [plan_id=1028166]

(18) ShuffleQueryStage
Output [1]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940]
Arguments: 3

(19) AQEShuffleRead
Input [1]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940]
Arguments: coalesced

(20) Sort [codegen id : 7]
Input [1]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940]
Arguments: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940 ASC NULLS FIRST], false, 0

(21) Scan csv 
Output [1]: [_c0#1663526]
Batched: false
Location: InMemoryFileIndex [file:/data/input/depot/csv/tmp/MOLECULES_ONERE_237545/MO-M12-2025.txt]
ReadSchema: struct<_c0:string>

(22) Project [codegen id : 3]
Output [1]: [str_remove_space(str_extract_middle(_c0#1663526, 12, 20), ALL) AS COL_45066C9F_40FD_4BE1_ACAF_6139C274CECB#1663658]
Input [1]: [_c0#1663526]

(23) HashAggregate [codegen id : 3]
Input [1]: [COL_45066C9F_40FD_4BE1_ACAF_6139C274CECB#1663658]
Keys [1]: [COL_45066C9F_40FD_4BE1_ACAF_6139C274CECB#1663658]
Functions: []
Aggregate Attributes: []
Results [1]: [COL_45066C9F_40FD_4BE1_ACAF_6139C274CECB#1663658]

(24) Exchange
Input [1]: [COL_45066C9F_40FD_4BE1_ACAF_6139C274CECB#1663658]
Arguments: hashpartitioning(COL_45066C9F_40FD_4BE1_ACAF_6139C274CECB#1663658, 10), ENSURE_REQUIREMENTS, [plan_id=1027991]

(25) ShuffleQueryStage
Output [1]: [COL_45066C9F_40FD_4BE1_ACAF_6139C274CECB#1663658]
Arguments: 2

(26) AQEShuffleRead
Input [1]: [COL_45066C9F_40FD_4BE1_ACAF_6139C274CECB#1663658]
Arguments: coalesced

(27) HashAggregate [codegen id : 8]
Input [1]: [COL_45066C9F_40FD_4BE1_ACAF_6139C274CECB#1663658]
Keys [1]: [COL_45066C9F_40FD_4BE1_ACAF_6139C274CECB#1663658]
Functions: []
Aggregate Attributes: []
Results [1]: [COL_45066C9F_40FD_4BE1_ACAF_6139C274CECB#1663658 AS 01_MOLECULES_ON_237708_COL_EE4AB0B4_F37E_40C0_9292_9E68BF166E36#1667065]

(28) Sort [codegen id : 8]
Input [1]: [01_MOLECULES_ON_237708_COL_EE4AB0B4_F37E_40C0_9292_9E68BF166E36#1667065]
Arguments: [01_MOLECULES_ON_237708_COL_EE4AB0B4_F37E_40C0_9292_9E68BF166E36#1667065 ASC NULLS FIRST], false, 0

(29) SortMergeJoin [codegen id : 9]
Left keys [1]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940]
Right keys [1]: [01_MOLECULES_ON_237708_COL_EE4AB0B4_F37E_40C0_9292_9E68BF166E36#1667065]
Join type: FullOuter
Join condition: None

(30) Project [codegen id : 9]
Output: []
Input [2]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940, 01_MOLECULES_ON_237708_COL_EE4AB0B4_F37E_40C0_9292_9E68BF166E36#1667065]

(31) HashAggregate [codegen id : 9]
Input: []
Keys: []
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#1667821L]
Results [1]: [count#1667822L]

(32) Exchange
Input [1]: [count#1667822L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=1028248]

(33) ShuffleQueryStage
Output [1]: [count#1667822L]
Arguments: 4

(34) HashAggregate [codegen id : 10]
Input [1]: [count#1667822L]
Keys: []
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#1667631L]
Results [1]: [count(1)#1667631L AS count#1667632L]

(35) Project
Output [1]: [str_remove_space(NO_ADMIN#1661280, ALL) AS COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940]
Input [1]: [NO_ADMIN#1661280]

(36) Exchange
Input [1]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940]
Arguments: hashpartitioning(COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940, 10), ENSURE_REQUIREMENTS, [plan_id=1027894]

(37) Sort
Input [1]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940]
Arguments: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940 ASC NULLS FIRST], false, 0

(38) Project
Output [1]: [str_remove_space(str_extract_middle(_c0#1664070, 12, 20), ALL) AS COL_9D7AAEDE_94EE_40AF_8530_BC9D1F27742A#1664641]
Input [1]: [_c0#1664070]

(39) HashAggregate
Input [1]: [COL_9D7AAEDE_94EE_40AF_8530_BC9D1F27742A#1664641]
Keys [1]: [COL_9D7AAEDE_94EE_40AF_8530_BC9D1F27742A#1664641]
Functions: []
Aggregate Attributes: []
Results [1]: [COL_9D7AAEDE_94EE_40AF_8530_BC9D1F27742A#1664641]

(40) Exchange
Input [1]: [COL_9D7AAEDE_94EE_40AF_8530_BC9D1F27742A#1664641]
Arguments: hashpartitioning(COL_9D7AAEDE_94EE_40AF_8530_BC9D1F27742A#1664641, 10), ENSURE_REQUIREMENTS, [plan_id=1027890]

(41) HashAggregate
Input [1]: [COL_9D7AAEDE_94EE_40AF_8530_BC9D1F27742A#1664641]
Keys [1]: [COL_9D7AAEDE_94EE_40AF_8530_BC9D1F27742A#1664641]
Functions: []
Aggregate Attributes: []
Results [1]: [COL_9D7AAEDE_94EE_40AF_8530_BC9D1F27742A#1664641 AS 01_DMI_DECOUPAG_237714_COL_1D1ABD09_1E00_4FA7_81FE_A558E22F6A6D#1666954]

(42) Sort
Input [1]: [01_DMI_DECOUPAG_237714_COL_1D1ABD09_1E00_4FA7_81FE_A558E22F6A6D#1666954]
Arguments: [01_DMI_DECOUPAG_237714_COL_1D1ABD09_1E00_4FA7_81FE_A558E22F6A6D#1666954 ASC NULLS FIRST], false, 0

(43) SortMergeJoin
Left keys [1]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940]
Right keys [1]: [01_DMI_DECOUPAG_237714_COL_1D1ABD09_1E00_4FA7_81FE_A558E22F6A6D#1666954]
Join type: FullOuter
Join condition: None

(44) Project
Output [1]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940]
Input [2]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940, 01_DMI_DECOUPAG_237714_COL_1D1ABD09_1E00_4FA7_81FE_A558E22F6A6D#1666954]

(45) Exchange
Input [1]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940]
Arguments: hashpartitioning(COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940, 10), ENSURE_REQUIREMENTS, [plan_id=1027903]

(46) Sort
Input [1]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940]
Arguments: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940 ASC NULLS FIRST], false, 0

(47) Project
Output [1]: [str_remove_space(str_extract_middle(_c0#1663526, 12, 20), ALL) AS COL_45066C9F_40FD_4BE1_ACAF_6139C274CECB#1663658]
Input [1]: [_c0#1663526]

(48) HashAggregate
Input [1]: [COL_45066C9F_40FD_4BE1_ACAF_6139C274CECB#1663658]
Keys [1]: [COL_45066C9F_40FD_4BE1_ACAF_6139C274CECB#1663658]
Functions: []
Aggregate Attributes: []
Results [1]: [COL_45066C9F_40FD_4BE1_ACAF_6139C274CECB#1663658]

(49) Exchange
Input [1]: [COL_45066C9F_40FD_4BE1_ACAF_6139C274CECB#1663658]
Arguments: hashpartitioning(COL_45066C9F_40FD_4BE1_ACAF_6139C274CECB#1663658, 10), ENSURE_REQUIREMENTS, [plan_id=1027899]

(50) HashAggregate
Input [1]: [COL_45066C9F_40FD_4BE1_ACAF_6139C274CECB#1663658]
Keys [1]: [COL_45066C9F_40FD_4BE1_ACAF_6139C274CECB#1663658]
Functions: []
Aggregate Attributes: []
Results [1]: [COL_45066C9F_40FD_4BE1_ACAF_6139C274CECB#1663658 AS 01_MOLECULES_ON_237708_COL_EE4AB0B4_F37E_40C0_9292_9E68BF166E36#1667065]

(51) Sort
Input [1]: [01_MOLECULES_ON_237708_COL_EE4AB0B4_F37E_40C0_9292_9E68BF166E36#1667065]
Arguments: [01_MOLECULES_ON_237708_COL_EE4AB0B4_F37E_40C0_9292_9E68BF166E36#1667065 ASC NULLS FIRST], false, 0

(52) SortMergeJoin
Left keys [1]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940]
Right keys [1]: [01_MOLECULES_ON_237708_COL_EE4AB0B4_F37E_40C0_9292_9E68BF166E36#1667065]
Join type: FullOuter
Join condition: None

(53) Project
Output: []
Input [2]: [COL_7948E047_29EE_472A_BFFC_723098878DAD#1666940, 01_MOLECULES_ON_237708_COL_EE4AB0B4_F37E_40C0_9292_9E68BF166E36#1667065]

(54) HashAggregate
Input: []
Keys: []
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#1667821L]
Results [1]: [count#1667822L]

(55) Exchange
Input [1]: [count#1667822L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=1027910]

(56) HashAggregate
Input [1]: [count#1667822L]
Keys: []
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#1667631L]
Results [1]: [count(1)#1667631L AS count#1667632L]

(57) AdaptiveSparkPlan
Output [1]: [count#1667632L]
Arguments: isFinalPlan=true

SQL / DataFrame Properties

Name	Value
spark.sql.optimizer.nestedPredicatePushdown.supportedFileSources	parquet,orc,geoparquet