SP1 - Details for Query 1998

Submitted Time: 2026/11/05 12:23:51
Duration: 2 s
Succeeded Jobs: 2838 2839 2845 2846

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; subgraph cluster1 { isCluster="true"; label="WholeStageCodegen (7)\n \nduration: 0 ms"; 2 [labelType="html" label="HashAggregate time in aggregation build: 0 ms number of output rows: 1"]; } 3 [labelType="html" label="Exchange shuffle records written: 1 local merged chunks fetched: 0 shuffle write time: 0 ms remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 1 local bytes read: 59.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 1 remote merged chunks fetched: 0 remote blocks read: 0 data size: 16.0 B local merged bytes read: 0.0 B number of partitions: 1 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written: 59.0 B"]; subgraph cluster4 { isCluster="true"; label="WholeStageCodegen (6)\n \nduration: 5 ms"; 5 [labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build: 4 ms peak memory: 0.0 B number of output rows: 1 number of sort fallback tasks: 0 avg hash probes per key: 0"]; 6 [labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build: 2 ms peak memory: 16.3 MiB number of output rows: 1,733 number of sort fallback tasks: 0 avg hash probes per key: 1"]; } 7 [labelType="html" label="AQEShuffleRead number of partitions: 1 partition data size: 28.8 KiB number of coalesced partitions: 1"]; 8 [labelType="html" label="Exchange shuffle records written: 1,733 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 5 ms (0 ms, 0 ms, 0 ms (stage 5374.0: task 5928)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 1,733 local bytes read: 27.1 KiB fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 10 remote merged chunks fetched: 0 remote blocks read: 0 data size total (min, med, max (stageId: taskId)) 67.7 KiB (6.0 KiB, 7.0 KiB, 7.4 KiB (stage 5374.0: task 5929)) local merged bytes read: 0.0 B number of partitions: 10 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 27.1 KiB (2.4 KiB, 2.8 KiB, 3.0 KiB (stage 5374.0: task 5929))"]; subgraph cluster9 { isCluster="true"; label="WholeStageCodegen (5)\n \nduration: total (min, med, max (stageId: taskId))\n12.3 s (1.2 s, 1.2 s, 1.3 s (stage 5374.0: task 5930))"; 10 [labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build total (min, med, max (stageId: taskId)) 12.2 s (1.1 s, 1.2 s, 1.3 s (stage 5374.0: task 5930)) peak memory total (min, med, max (stageId: taskId)) 2.5 MiB (256.0 KiB, 256.0 KiB, 256.0 KiB (stage 5374.0: task 5931)) number of output rows: 1,733 number of sort fallback tasks: 0 avg hash probes per key: 0"]; } 11 [labelType="html" label="CartesianProduct number of output rows: 2,139"]; subgraph cluster12 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: total (min, med, max (stageId: taskId))\n12.4 s (1.2 s, 1.2 s, 1.3 s (stage 5374.0: task 5930))"; 13 [labelType="html" label=" Project "]; } 14 [labelType="html" label="CartesianProduct number of output rows: 2,139"]; subgraph cluster15 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: total (min, med, max (stageId: taskId))\n12.6 s (1.2 s, 1.3 s, 1.3 s (stage 5374.0: task 5930))"; 16 [labelType="html" label=" Project "]; 17 [labelType="html" label="Filter number of output rows: 1,457,791"]; 18 [labelType="html" label="ColumnarToRow number of output rows: 2,199,023 number of input batches: 541"]; } 19 [labelType="html" label="Scan parquet number of files read: 10 scan time total (min, med, max (stageId: taskId)) 576 ms (51 ms, 55 ms, 69 ms (stage 5374.0: task 5930)) metadata time: 0 ms size of files read: 146.8 MiB number of output rows: 2,199,023"]; 20 [labelType="html" label="ObjectHashAggregate number of output rows: 10 time in aggregation build total (min, med, max (stageId: taskId)) 37 ms (2 ms, 3 ms, 6 ms (stage 5374.0: task 5934)) spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 5374.0: task 5931)) number of sort fallback tasks: 0"]; 21 [labelType="html" label="Exchange shuffle records written: 1 local merged chunks fetched: 0 shuffle write time: 0 ms remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 10 local bytes read total (min, med, max (stageId: taskId)) 1840.0 B (184.0 B, 184.0 B, 184.0 B (stage 5374.0: task 5931)) fetch wait time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 5374.0: task 5931)) remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 10 remote merged chunks fetched: 0 remote blocks read: 0 data size: 192.0 B local merged bytes read: 0.0 B number of partitions: 1 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written: 184.0 B"]; 22 [labelType="html" label="ObjectHashAggregate number of output rows: 1 time in aggregation build: 14 ms spill size: 0.0 B number of sort fallback tasks: 0"]; subgraph cluster23 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: 15 ms"; 24 [labelType="html" label=" Project "]; } 25 [labelType="html" label="Scan csv number of output rows: 9 number of files read: 1 metadata time: 0 ms size of files read: 50.0 B"]; subgraph cluster26 { isCluster="true"; label="WholeStageCodegen (4)\n \nduration: total (min, med, max (stageId: taskId))\n102 ms (6 ms, 10 ms, 14 ms (stage 5374.0: task 5926))"; 27 [labelType="html" label=" Project "]; 28 [labelType="html" label="Filter number of output rows: 10"]; } 29 [labelType="html" label="Scan csv number of output rows: 10 number of files read: 1 metadata time: 0 ms size of files read: 110.0 B"]; 2->0; 3->2; 5->3; 6->5; 7->6; 8->7; 10->8; 11->10; 13->11; 14->13; 16->14; 17->16; 18->17; 19->18; 20->14; 21->20; 22->21; 24->22; 25->24; 27->11; 28->27; 29->28; }

AdaptiveSparkPlan isFinalPlan=true

HashAggregate(keys=[], functions=[count(1)])

WholeStageCodegen (7)

Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=193771]

HashAggregate(keys=[], functions=[partial_count(1)])

HashAggregate(keys=[COL_F65F0EE2_FCC8_4E9F_B23F_0D97C40F54FD#470944], functions=[])

WholeStageCodegen (6)

AQEShuffleRead coalesced

Exchange hashpartitioning(COL_F65F0EE2_FCC8_4E9F_B23F_0D97C40F54FD#470944, 10), ENSURE_REQUIREMENTS, [plan_id=193453]

HashAggregate(keys=[COL_F65F0EE2_FCC8_4E9F_B23F_0D97C40F54FD#470944], functions=[])

WholeStageCodegen (5)

CartesianProduct

Project [COL_F65F0EE2_FCC8_4E9F_B23F_0D97C40F54FD#470944]

WholeStageCodegen (3)

CartesianProduct (str_if_with_rule(((((((COL_E43C6574_A3E3_4056_BB55_EA935E04502C#470940 <=> 1) AND (COL_10A975F7_B67F_438C_B8DE_9E0F00F273DC#470941 <=> 1)) AND (COL_318510CA_23E7_4678_9FCA_D34FF51ADC39#470949 <=> 0)) AND (size(str_list_intersect_fixed(COL_3377534C_2BCD_43B6_853A_869D44C151B3#470948, [MCS]), true) <=> 0)) AND (size(str_list_intersect_fixed(99_ACTES_ELIGIB_231708_COL_61C476F6_706A_4DB7_97C1_3E8CEB931839#471303, array(COL_B9A76B6D_E3EA_49F5_A0E9_DE474A8CE2B4#470947)), true) > 0)) AND (((CASE WHEN isnull(COL_414BA164_8DC3_4834_BDD7_434B2278858B#470945L) THEN false ELSE (COL_414BA164_8DC3_4834_BDD7_434B2278858B#470945L < 16) END AND NOT (COL_3931E036_315E_4AC8_A56D_37AE79FDF6B7#470934 <=> 01)) OR (CASE WHEN isnull(COL_414BA164_8DC3_4834_BDD7_434B2278858B#470945L) THEN false ELSE (COL_414BA164_8DC3_4834_BDD7_434B2278858B#470945L < 26) END AND (((COL_3931E036_315E_4AC8_A56D_37AE79FDF6B7#470934 <=> 17) OR (COL_3931E036_315E_4AC8_A56D_37AE79FDF6B7#470934 <=> 33)) OR (COL_3931E036_315E_4AC8_A56D_37AE79FDF6B7#470934 <=> 75)))) OR (COL_085E5798_40D8_4726_B01C_7D83841C9C25#470946 <=> O))), MCS, ) <=> MCS)

Project [COL_4587DB83_72BD_4F2D_BA18_AAD14178DC7F#470690 AS COL_3931E036_315E_4AC8_A56D_37AE79FDF6B7#470934, COL_346C27DC_6D06_4FCF_A022_7FF177053D81#470696 AS COL_E43C6574_A3E3_4056_BB55_EA935E04502C#470940, COL_75DE8325_2CC3_4E7A_A718_80AA4F2A9D62#470697 AS COL_10A975F7_B67F_438C_B8DE_9E0F00F273DC#470941, COL_A5765544_357B_4395_B6BC_21D7D5C2D0E0#470700 AS COL_F65F0EE2_FCC8_4E9F_B23F_0D97C40F54FD#470944, COL_969EAFA1_AD39_4285_AD05_6146CD6F9342#470701L AS COL_414BA164_8DC3_4834_BDD7_434B2278858B#470945L, COL_19CE074B_1F54_459C_ABA2_D9FCF9DC7487#470702 AS COL_085E5798_40D8_4726_B01C_7D83841C9C25#470946, COL_A4AF3049_C967_4B58_9D36_40EE5D79F6AC#470703 AS COL_B9A76B6D_E3EA_49F5_A0E9_DE474A8CE2B4#470947, COL_714AB906_7002_4ADC_AA1A_38467A3EDE10#470704 AS COL_3377534C_2BCD_43B6_853A_869D44C151B3#470948, COL_D986422B_41E3_4D4B_9F25_9D68ED522BE6#470706 AS COL_318510CA_23E7_4678_9FCA_D34FF51ADC39#470949]

Filter (NOT (COL_A4AF3049_C967_4B58_9D36_40EE5D79F6AC#470703 <=> ) AND isnotnull(COL_A4AF3049_C967_4B58_9D36_40EE5D79F6AC#470703))

ColumnarToRow

WholeStageCodegen (2)

FileScan parquet [COL_4587DB83_72BD_4F2D_BA18_AAD14178DC7F#470690,COL_346C27DC_6D06_4FCF_A022_7FF177053D81#470696,COL_75DE8325_2CC3_4E7A_A718_80AA4F2A9D62#470697,COL_A5765544_357B_4395_B6BC_21D7D5C2D0E0#470700,COL_969EAFA1_AD39_4285_AD05_6146CD6F9342#470701L,COL_19CE074B_1F54_459C_ABA2_D9FCF9DC7487#470702,COL_A4AF3049_C967_4B58_9D36_40EE5D79F6AC#470703,COL_714AB906_7002_4ADC_AA1A_38467A3EDE10#470704,COL_D986422B_41E3_4D4B_9F25_9D68ED522BE6#470706] Batched: true, DataFilters: [NOT (COL_A4AF3049_C967_4B58_9D36_40EE5D79F6AC#470703 <=> ), isnotnull(COL_A4AF3049_C967_4B58_9D3..., Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/data/output/cache/parquet/datablock/01_A_STEP_1_RSF_231581], PartitionFilters: [], PushedFilters: [Not(EqualNullSafe(COL_A4AF3049_C967_4B58_9D36_40EE5D79F6AC,)), IsNotNull(COL_A4AF3049_C967_4B58_..., ReadSchema: struct<COL_4587DB83_72BD_4F2D_BA18_AAD14178DC7F:string,COL_346C27DC_6D06_4FCF_A022_7FF177053D81:s...

ObjectHashAggregate(keys=[], functions=[collect_set(COL_B3233636_E2EB_49EB_9DD1_68C9BAA0E86F#470771, 0, 0)])

Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=193357]

ObjectHashAggregate(keys=[], functions=[partial_collect_set(COL_B3233636_E2EB_49EB_9DD1_68C9BAA0E86F#470771, 0, 0)])

Project [str_remove_space(Code-MCS#470753, ONLY_ONE) AS COL_B3233636_E2EB_49EB_9DD1_68C9BAA0E86F#470771]

WholeStageCodegen (1)

FileScan csv [Code-MCS#470753] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/data/input/depot/csv/tmp/ACTES_ELIGIBLIT_231426/Code-MCS.csv], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<Code-MCS:string>

Project

Filter ((isnotnull(REGLES#470665) AND (MCS = REGLES#470665)) AND (APPLICABLE#470666 <=> O))

WholeStageCodegen (4)

FileScan csv [REGLES#470665,APPLICABLE#470666] Batched: false, DataFilters: [isnotnull(REGLES#470665), (MCS = REGLES#470665), (APPLICABLE#470666 <=> O)], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/data/input/depot/csv/tmp/REGLES_APPLICAB_231407/REGLES-A-TTT_EXT..., PartitionFilters: [], PushedFilters: [IsNotNull(REGLES), EqualTo(REGLES,MCS), EqualNullSafe(APPLICABLE,O)], ReadSchema: struct<REGLES:string,APPLICABLE:string>

Details

== Physical Plan ==
AdaptiveSparkPlan (43)
+- == Final Plan ==
   * HashAggregate (25)
   +- ShuffleQueryStage (24), Statistics(sizeInBytes=16.0 B, rowCount=1)
      +- Exchange (23)
         +- * HashAggregate (22)
            +- * HashAggregate (21)
               +- AQEShuffleRead (20)
                  +- ShuffleQueryStage (19), Statistics(sizeInBytes=67.7 KiB, rowCount=1.73E+3)
                     +- Exchange (18)
                        +- * HashAggregate (17)
                           +- CartesianProduct Inner (16)
                              :- * Project (12)
                              :  +- CartesianProduct Inner (11)
                              :     :- * Project (4)
                              :     :  +- * Filter (3)
                              :     :     +- * ColumnarToRow (2)
                              :     :        +- Scan parquet  (1)
                              :     +- ObjectHashAggregate (10)
                              :        +- ShuffleQueryStage (9), Statistics(sizeInBytes=192.0 B, rowCount=1)
                              :           +- Exchange (8)
                              :              +- ObjectHashAggregate (7)
                              :                 +- * Project (6)
                              :                    +- Scan csv  (5)
                              +- * Project (15)
                                 +- * Filter (14)
                                    +- Scan csv  (13)
+- == Initial Plan ==
   HashAggregate (42)
   +- Exchange (41)
      +- HashAggregate (40)
         +- HashAggregate (39)
            +- Exchange (38)
               +- HashAggregate (37)
                  +- CartesianProduct Inner (36)
                     :- Project (33)
                     :  +- CartesianProduct Inner (32)
                     :     :- Project (27)
                     :     :  +- Filter (26)
                     :     :     +- Scan parquet  (1)
                     :     +- ObjectHashAggregate (31)
                     :        +- Exchange (30)
                     :           +- ObjectHashAggregate (29)
                     :              +- Project (28)
                     :                 +- Scan csv  (5)
                     +- Project (35)
                        +- Filter (34)
                           +- Scan csv  (13)


(1) Scan parquet 
Output [9]: [COL_4587DB83_72BD_4F2D_BA18_AAD14178DC7F#470690, COL_346C27DC_6D06_4FCF_A022_7FF177053D81#470696, COL_75DE8325_2CC3_4E7A_A718_80AA4F2A9D62#470697, COL_A5765544_357B_4395_B6BC_21D7D5C2D0E0#470700, COL_969EAFA1_AD39_4285_AD05_6146CD6F9342#470701L, COL_19CE074B_1F54_459C_ABA2_D9FCF9DC7487#470702, COL_A4AF3049_C967_4B58_9D36_40EE5D79F6AC#470703, COL_714AB906_7002_4ADC_AA1A_38467A3EDE10#470704, COL_D986422B_41E3_4D4B_9F25_9D68ED522BE6#470706]
Batched: true
Location: InMemoryFileIndex [file:/data/output/cache/parquet/datablock/01_A_STEP_1_RSF_231581]
PushedFilters: [Not(EqualNullSafe(COL_A4AF3049_C967_4B58_9D36_40EE5D79F6AC,)), IsNotNull(COL_A4AF3049_C967_4B58_9D36_40EE5D79F6AC)]
ReadSchema: struct<COL_4587DB83_72BD_4F2D_BA18_AAD14178DC7F:string,COL_346C27DC_6D06_4FCF_A022_7FF177053D81:string,COL_75DE8325_2CC3_4E7A_A718_80AA4F2A9D62:string,COL_A5765544_357B_4395_B6BC_21D7D5C2D0E0:string,COL_969EAFA1_AD39_4285_AD05_6146CD6F9342:bigint,COL_19CE074B_1F54_459C_ABA2_D9FCF9DC7487:string,COL_A4AF3049_C967_4B58_9D36_40EE5D79F6AC:string,COL_714AB906_7002_4ADC_AA1A_38467A3EDE10:array<string>,COL_D986422B_41E3_4D4B_9F25_9D68ED522BE6:int>

(2) ColumnarToRow [codegen id : 2]
Input [9]: [COL_4587DB83_72BD_4F2D_BA18_AAD14178DC7F#470690, COL_346C27DC_6D06_4FCF_A022_7FF177053D81#470696, COL_75DE8325_2CC3_4E7A_A718_80AA4F2A9D62#470697, COL_A5765544_357B_4395_B6BC_21D7D5C2D0E0#470700, COL_969EAFA1_AD39_4285_AD05_6146CD6F9342#470701L, COL_19CE074B_1F54_459C_ABA2_D9FCF9DC7487#470702, COL_A4AF3049_C967_4B58_9D36_40EE5D79F6AC#470703, COL_714AB906_7002_4ADC_AA1A_38467A3EDE10#470704, COL_D986422B_41E3_4D4B_9F25_9D68ED522BE6#470706]

(3) Filter [codegen id : 2]
Input [9]: [COL_4587DB83_72BD_4F2D_BA18_AAD14178DC7F#470690, COL_346C27DC_6D06_4FCF_A022_7FF177053D81#470696, COL_75DE8325_2CC3_4E7A_A718_80AA4F2A9D62#470697, COL_A5765544_357B_4395_B6BC_21D7D5C2D0E0#470700, COL_969EAFA1_AD39_4285_AD05_6146CD6F9342#470701L, COL_19CE074B_1F54_459C_ABA2_D9FCF9DC7487#470702, COL_A4AF3049_C967_4B58_9D36_40EE5D79F6AC#470703, COL_714AB906_7002_4ADC_AA1A_38467A3EDE10#470704, COL_D986422B_41E3_4D4B_9F25_9D68ED522BE6#470706]
Condition : (NOT (COL_A4AF3049_C967_4B58_9D36_40EE5D79F6AC#470703 <=> ) AND isnotnull(COL_A4AF3049_C967_4B58_9D36_40EE5D79F6AC#470703))

(4) Project [codegen id : 2]
Output [9]: [COL_4587DB83_72BD_4F2D_BA18_AAD14178DC7F#470690 AS COL_3931E036_315E_4AC8_A56D_37AE79FDF6B7#470934, COL_346C27DC_6D06_4FCF_A022_7FF177053D81#470696 AS COL_E43C6574_A3E3_4056_BB55_EA935E04502C#470940, COL_75DE8325_2CC3_4E7A_A718_80AA4F2A9D62#470697 AS COL_10A975F7_B67F_438C_B8DE_9E0F00F273DC#470941, COL_A5765544_357B_4395_B6BC_21D7D5C2D0E0#470700 AS COL_F65F0EE2_FCC8_4E9F_B23F_0D97C40F54FD#470944, COL_969EAFA1_AD39_4285_AD05_6146CD6F9342#470701L AS COL_414BA164_8DC3_4834_BDD7_434B2278858B#470945L, COL_19CE074B_1F54_459C_ABA2_D9FCF9DC7487#470702 AS COL_085E5798_40D8_4726_B01C_7D83841C9C25#470946, COL_A4AF3049_C967_4B58_9D36_40EE5D79F6AC#470703 AS COL_B9A76B6D_E3EA_49F5_A0E9_DE474A8CE2B4#470947, COL_714AB906_7002_4ADC_AA1A_38467A3EDE10#470704 AS COL_3377534C_2BCD_43B6_853A_869D44C151B3#470948, COL_D986422B_41E3_4D4B_9F25_9D68ED522BE6#470706 AS COL_318510CA_23E7_4678_9FCA_D34FF51ADC39#470949]
Input [9]: [COL_4587DB83_72BD_4F2D_BA18_AAD14178DC7F#470690, COL_346C27DC_6D06_4FCF_A022_7FF177053D81#470696, COL_75DE8325_2CC3_4E7A_A718_80AA4F2A9D62#470697, COL_A5765544_357B_4395_B6BC_21D7D5C2D0E0#470700, COL_969EAFA1_AD39_4285_AD05_6146CD6F9342#470701L, COL_19CE074B_1F54_459C_ABA2_D9FCF9DC7487#470702, COL_A4AF3049_C967_4B58_9D36_40EE5D79F6AC#470703, COL_714AB906_7002_4ADC_AA1A_38467A3EDE10#470704, COL_D986422B_41E3_4D4B_9F25_9D68ED522BE6#470706]

(5) Scan csv 
Output [1]: [Code-MCS#470753]
Batched: false
Location: InMemoryFileIndex [file:/data/input/depot/csv/tmp/ACTES_ELIGIBLIT_231426/Code-MCS.csv]
ReadSchema: struct<Code-MCS:string>

(6) Project [codegen id : 1]
Output [1]: [str_remove_space(Code-MCS#470753, ONLY_ONE) AS COL_B3233636_E2EB_49EB_9DD1_68C9BAA0E86F#470771]
Input [1]: [Code-MCS#470753]

(7) ObjectHashAggregate
Input [1]: [COL_B3233636_E2EB_49EB_9DD1_68C9BAA0E86F#470771]
Keys: []
Functions [1]: [partial_collect_set(COL_B3233636_E2EB_49EB_9DD1_68C9BAA0E86F#470771, 0, 0)]
Aggregate Attributes [1]: [buf#471650]
Results [1]: [buf#471651]

(8) Exchange
Input [1]: [buf#471651]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=193357]

(9) ShuffleQueryStage
Output [1]: [buf#471651]
Arguments: 0

(10) ObjectHashAggregate
Input [1]: [buf#471651]
Keys: []
Functions [1]: [collect_set(COL_B3233636_E2EB_49EB_9DD1_68C9BAA0E86F#470771, 0, 0)]
Aggregate Attributes [1]: [collect_set(COL_B3233636_E2EB_49EB_9DD1_68C9BAA0E86F#470771, 0, 0)#470776]
Results [1]: [collect_set(COL_B3233636_E2EB_49EB_9DD1_68C9BAA0E86F#470771, 0, 0)#470776 AS 99_ACTES_ELIGIB_231708_COL_61C476F6_706A_4DB7_97C1_3E8CEB931839#471303]

(11) CartesianProduct
Join type: Inner
Join condition: (str_if_with_rule(((((((COL_E43C6574_A3E3_4056_BB55_EA935E04502C#470940 <=> 1) AND (COL_10A975F7_B67F_438C_B8DE_9E0F00F273DC#470941 <=> 1)) AND (COL_318510CA_23E7_4678_9FCA_D34FF51ADC39#470949 <=> 0)) AND (size(str_list_intersect_fixed(COL_3377534C_2BCD_43B6_853A_869D44C151B3#470948, [MCS]), true) <=> 0)) AND (size(str_list_intersect_fixed(99_ACTES_ELIGIB_231708_COL_61C476F6_706A_4DB7_97C1_3E8CEB931839#471303, array(COL_B9A76B6D_E3EA_49F5_A0E9_DE474A8CE2B4#470947)), true) > 0)) AND (((CASE WHEN isnull(COL_414BA164_8DC3_4834_BDD7_434B2278858B#470945L) THEN false ELSE (COL_414BA164_8DC3_4834_BDD7_434B2278858B#470945L < 16) END AND NOT (COL_3931E036_315E_4AC8_A56D_37AE79FDF6B7#470934 <=> 01)) OR (CASE WHEN isnull(COL_414BA164_8DC3_4834_BDD7_434B2278858B#470945L) THEN false ELSE (COL_414BA164_8DC3_4834_BDD7_434B2278858B#470945L < 26) END AND (((COL_3931E036_315E_4AC8_A56D_37AE79FDF6B7#470934 <=> 17) OR (COL_3931E036_315E_4AC8_A56D_37AE79FDF6B7#470934 <=> 33)) OR (COL_3931E036_315E_4AC8_A56D_37AE79FDF6B7#470934 <=> 75)))) OR (COL_085E5798_40D8_4726_B01C_7D83841C9C25#470946 <=> O))), MCS, ) <=> MCS)

(12) Project [codegen id : 3]
Output [1]: [COL_F65F0EE2_FCC8_4E9F_B23F_0D97C40F54FD#470944]
Input [10]: [COL_3931E036_315E_4AC8_A56D_37AE79FDF6B7#470934, COL_E43C6574_A3E3_4056_BB55_EA935E04502C#470940, COL_10A975F7_B67F_438C_B8DE_9E0F00F273DC#470941, COL_F65F0EE2_FCC8_4E9F_B23F_0D97C40F54FD#470944, COL_414BA164_8DC3_4834_BDD7_434B2278858B#470945L, COL_085E5798_40D8_4726_B01C_7D83841C9C25#470946, COL_B9A76B6D_E3EA_49F5_A0E9_DE474A8CE2B4#470947, COL_3377534C_2BCD_43B6_853A_869D44C151B3#470948, COL_318510CA_23E7_4678_9FCA_D34FF51ADC39#470949, 99_ACTES_ELIGIB_231708_COL_61C476F6_706A_4DB7_97C1_3E8CEB931839#471303]

(13) Scan csv 
Output [2]: [REGLES#470665, APPLICABLE#470666]
Batched: false
Location: InMemoryFileIndex [file:/data/input/depot/csv/tmp/REGLES_APPLICAB_231407/REGLES-A-TTT_EXT.csv]
PushedFilters: [IsNotNull(REGLES), EqualTo(REGLES,MCS), EqualNullSafe(APPLICABLE,O)]
ReadSchema: struct<REGLES:string,APPLICABLE:string>

(14) Filter [codegen id : 4]
Input [2]: [REGLES#470665, APPLICABLE#470666]
Condition : ((isnotnull(REGLES#470665) AND (MCS = REGLES#470665)) AND (APPLICABLE#470666 <=> O))

(15) Project [codegen id : 4]
Output: []
Input [2]: [REGLES#470665, APPLICABLE#470666]

(16) CartesianProduct
Join type: Inner
Join condition: None

(17) HashAggregate [codegen id : 5]
Input [1]: [COL_F65F0EE2_FCC8_4E9F_B23F_0D97C40F54FD#470944]
Keys [1]: [COL_F65F0EE2_FCC8_4E9F_B23F_0D97C40F54FD#470944]
Functions: []
Aggregate Attributes: []
Results [1]: [COL_F65F0EE2_FCC8_4E9F_B23F_0D97C40F54FD#470944]

(18) Exchange
Input [1]: [COL_F65F0EE2_FCC8_4E9F_B23F_0D97C40F54FD#470944]
Arguments: hashpartitioning(COL_F65F0EE2_FCC8_4E9F_B23F_0D97C40F54FD#470944, 10), ENSURE_REQUIREMENTS, [plan_id=193453]

(19) ShuffleQueryStage
Output [1]: [COL_F65F0EE2_FCC8_4E9F_B23F_0D97C40F54FD#470944]
Arguments: 1

(20) AQEShuffleRead
Input [1]: [COL_F65F0EE2_FCC8_4E9F_B23F_0D97C40F54FD#470944]
Arguments: coalesced

(21) HashAggregate [codegen id : 6]
Input [1]: [COL_F65F0EE2_FCC8_4E9F_B23F_0D97C40F54FD#470944]
Keys [1]: [COL_F65F0EE2_FCC8_4E9F_B23F_0D97C40F54FD#470944]
Functions: []
Aggregate Attributes: []
Results: []

(22) HashAggregate [codegen id : 6]
Input: []
Keys: []
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#471648L]
Results [1]: [count#471649L]

(23) Exchange
Input [1]: [count#471649L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=193771]

(24) ShuffleQueryStage
Output [1]: [count#471649L]
Arguments: 2

(25) HashAggregate [codegen id : 7]
Input [1]: [count#471649L]
Keys: []
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#471645L]
Results [1]: [count(1)#471645L AS count#471646L]

(26) Filter
Input [9]: [COL_4587DB83_72BD_4F2D_BA18_AAD14178DC7F#470690, COL_346C27DC_6D06_4FCF_A022_7FF177053D81#470696, COL_75DE8325_2CC3_4E7A_A718_80AA4F2A9D62#470697, COL_A5765544_357B_4395_B6BC_21D7D5C2D0E0#470700, COL_969EAFA1_AD39_4285_AD05_6146CD6F9342#470701L, COL_19CE074B_1F54_459C_ABA2_D9FCF9DC7487#470702, COL_A4AF3049_C967_4B58_9D36_40EE5D79F6AC#470703, COL_714AB906_7002_4ADC_AA1A_38467A3EDE10#470704, COL_D986422B_41E3_4D4B_9F25_9D68ED522BE6#470706]
Condition : (NOT (COL_A4AF3049_C967_4B58_9D36_40EE5D79F6AC#470703 <=> ) AND isnotnull(COL_A4AF3049_C967_4B58_9D36_40EE5D79F6AC#470703))

(27) Project
Output [9]: [COL_4587DB83_72BD_4F2D_BA18_AAD14178DC7F#470690 AS COL_3931E036_315E_4AC8_A56D_37AE79FDF6B7#470934, COL_346C27DC_6D06_4FCF_A022_7FF177053D81#470696 AS COL_E43C6574_A3E3_4056_BB55_EA935E04502C#470940, COL_75DE8325_2CC3_4E7A_A718_80AA4F2A9D62#470697 AS COL_10A975F7_B67F_438C_B8DE_9E0F00F273DC#470941, COL_A5765544_357B_4395_B6BC_21D7D5C2D0E0#470700 AS COL_F65F0EE2_FCC8_4E9F_B23F_0D97C40F54FD#470944, COL_969EAFA1_AD39_4285_AD05_6146CD6F9342#470701L AS COL_414BA164_8DC3_4834_BDD7_434B2278858B#470945L, COL_19CE074B_1F54_459C_ABA2_D9FCF9DC7487#470702 AS COL_085E5798_40D8_4726_B01C_7D83841C9C25#470946, COL_A4AF3049_C967_4B58_9D36_40EE5D79F6AC#470703 AS COL_B9A76B6D_E3EA_49F5_A0E9_DE474A8CE2B4#470947, COL_714AB906_7002_4ADC_AA1A_38467A3EDE10#470704 AS COL_3377534C_2BCD_43B6_853A_869D44C151B3#470948, COL_D986422B_41E3_4D4B_9F25_9D68ED522BE6#470706 AS COL_318510CA_23E7_4678_9FCA_D34FF51ADC39#470949]
Input [9]: [COL_4587DB83_72BD_4F2D_BA18_AAD14178DC7F#470690, COL_346C27DC_6D06_4FCF_A022_7FF177053D81#470696, COL_75DE8325_2CC3_4E7A_A718_80AA4F2A9D62#470697, COL_A5765544_357B_4395_B6BC_21D7D5C2D0E0#470700, COL_969EAFA1_AD39_4285_AD05_6146CD6F9342#470701L, COL_19CE074B_1F54_459C_ABA2_D9FCF9DC7487#470702, COL_A4AF3049_C967_4B58_9D36_40EE5D79F6AC#470703, COL_714AB906_7002_4ADC_AA1A_38467A3EDE10#470704, COL_D986422B_41E3_4D4B_9F25_9D68ED522BE6#470706]

(28) Project
Output [1]: [str_remove_space(Code-MCS#470753, ONLY_ONE) AS COL_B3233636_E2EB_49EB_9DD1_68C9BAA0E86F#470771]
Input [1]: [Code-MCS#470753]

(29) ObjectHashAggregate
Input [1]: [COL_B3233636_E2EB_49EB_9DD1_68C9BAA0E86F#470771]
Keys: []
Functions [1]: [partial_collect_set(COL_B3233636_E2EB_49EB_9DD1_68C9BAA0E86F#470771, 0, 0)]
Aggregate Attributes [1]: [buf#471650]
Results [1]: [buf#471651]

(30) Exchange
Input [1]: [buf#471651]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=193309]

(31) ObjectHashAggregate
Input [1]: [buf#471651]
Keys: []
Functions [1]: [collect_set(COL_B3233636_E2EB_49EB_9DD1_68C9BAA0E86F#470771, 0, 0)]
Aggregate Attributes [1]: [collect_set(COL_B3233636_E2EB_49EB_9DD1_68C9BAA0E86F#470771, 0, 0)#470776]
Results [1]: [collect_set(COL_B3233636_E2EB_49EB_9DD1_68C9BAA0E86F#470771, 0, 0)#470776 AS 99_ACTES_ELIGIB_231708_COL_61C476F6_706A_4DB7_97C1_3E8CEB931839#471303]

(32) CartesianProduct
Join type: Inner
Join condition: (str_if_with_rule(((((((COL_E43C6574_A3E3_4056_BB55_EA935E04502C#470940 <=> 1) AND (COL_10A975F7_B67F_438C_B8DE_9E0F00F273DC#470941 <=> 1)) AND (COL_318510CA_23E7_4678_9FCA_D34FF51ADC39#470949 <=> 0)) AND (size(str_list_intersect_fixed(COL_3377534C_2BCD_43B6_853A_869D44C151B3#470948, [MCS]), true) <=> 0)) AND (size(str_list_intersect_fixed(99_ACTES_ELIGIB_231708_COL_61C476F6_706A_4DB7_97C1_3E8CEB931839#471303, array(COL_B9A76B6D_E3EA_49F5_A0E9_DE474A8CE2B4#470947)), true) > 0)) AND (((CASE WHEN isnull(COL_414BA164_8DC3_4834_BDD7_434B2278858B#470945L) THEN false ELSE (COL_414BA164_8DC3_4834_BDD7_434B2278858B#470945L < 16) END AND NOT (COL_3931E036_315E_4AC8_A56D_37AE79FDF6B7#470934 <=> 01)) OR (CASE WHEN isnull(COL_414BA164_8DC3_4834_BDD7_434B2278858B#470945L) THEN false ELSE (COL_414BA164_8DC3_4834_BDD7_434B2278858B#470945L < 26) END AND (((COL_3931E036_315E_4AC8_A56D_37AE79FDF6B7#470934 <=> 17) OR (COL_3931E036_315E_4AC8_A56D_37AE79FDF6B7#470934 <=> 33)) OR (COL_3931E036_315E_4AC8_A56D_37AE79FDF6B7#470934 <=> 75)))) OR (COL_085E5798_40D8_4726_B01C_7D83841C9C25#470946 <=> O))), MCS, ) <=> MCS)

(33) Project
Output [1]: [COL_F65F0EE2_FCC8_4E9F_B23F_0D97C40F54FD#470944]
Input [10]: [COL_3931E036_315E_4AC8_A56D_37AE79FDF6B7#470934, COL_E43C6574_A3E3_4056_BB55_EA935E04502C#470940, COL_10A975F7_B67F_438C_B8DE_9E0F00F273DC#470941, COL_F65F0EE2_FCC8_4E9F_B23F_0D97C40F54FD#470944, COL_414BA164_8DC3_4834_BDD7_434B2278858B#470945L, COL_085E5798_40D8_4726_B01C_7D83841C9C25#470946, COL_B9A76B6D_E3EA_49F5_A0E9_DE474A8CE2B4#470947, COL_3377534C_2BCD_43B6_853A_869D44C151B3#470948, COL_318510CA_23E7_4678_9FCA_D34FF51ADC39#470949, 99_ACTES_ELIGIB_231708_COL_61C476F6_706A_4DB7_97C1_3E8CEB931839#471303]

(34) Filter
Input [2]: [REGLES#470665, APPLICABLE#470666]
Condition : ((isnotnull(REGLES#470665) AND (MCS = REGLES#470665)) AND (APPLICABLE#470666 <=> O))

(35) Project
Output: []
Input [2]: [REGLES#470665, APPLICABLE#470666]

(36) CartesianProduct
Join type: Inner
Join condition: None

(37) HashAggregate
Input [1]: [COL_F65F0EE2_FCC8_4E9F_B23F_0D97C40F54FD#470944]
Keys [1]: [COL_F65F0EE2_FCC8_4E9F_B23F_0D97C40F54FD#470944]
Functions: []
Aggregate Attributes: []
Results [1]: [COL_F65F0EE2_FCC8_4E9F_B23F_0D97C40F54FD#470944]

(38) Exchange
Input [1]: [COL_F65F0EE2_FCC8_4E9F_B23F_0D97C40F54FD#470944]
Arguments: hashpartitioning(COL_F65F0EE2_FCC8_4E9F_B23F_0D97C40F54FD#470944, 10), ENSURE_REQUIREMENTS, [plan_id=193316]

(39) HashAggregate
Input [1]: [COL_F65F0EE2_FCC8_4E9F_B23F_0D97C40F54FD#470944]
Keys [1]: [COL_F65F0EE2_FCC8_4E9F_B23F_0D97C40F54FD#470944]
Functions: []
Aggregate Attributes: []
Results: []

(40) HashAggregate
Input: []
Keys: []
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#471648L]
Results [1]: [count#471649L]

(41) Exchange
Input [1]: [count#471649L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=193320]

(42) HashAggregate
Input [1]: [count#471649L]
Keys: []
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#471645L]
Results [1]: [count(1)#471645L AS count#471646L]

(43) AdaptiveSparkPlan
Output [1]: [count#471646L]
Arguments: isFinalPlan=true

SQL / DataFrame Properties

Name	Value
spark.sql.optimizer.nestedPredicatePushdown.supportedFileSources	parquet,orc,geoparquet