SP1 - Details for Query 2521

Submitted Time: 2026/06/11 09:55:51
Duration: 0.5 s
Succeeded Jobs: 3824 3826 3827 3828 3829

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; subgraph cluster1 { isCluster="true"; label="WholeStageCodegen (11)\n \nduration: 0 ms"; 2 [labelType="html" label="HashAggregate time in aggregation build: 0 ms number of output rows: 1"]; } 3 [labelType="html" label="Exchange shuffle records written: 1 local merged chunks fetched: 0 shuffle write time: 0 ms remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 1 local bytes read: 59.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 1 remote merged chunks fetched: 0 remote blocks read: 0 data size: 16.0 B local merged bytes read: 0.0 B number of partitions: 1 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written: 59.0 B"]; subgraph cluster4 { isCluster="true"; label="WholeStageCodegen (10)\n \nduration: 0 ms"; 5 [labelType="html" label="HashAggregate time in aggregation build: 0 ms number of output rows: 1"]; 6 [labelType="html" label=" Project "]; 7 [labelType="html" label="Filter number of output rows: 1"]; } 8 [labelType="html" label="Window spill size: 0.0 B"]; subgraph cluster9 { isCluster="true"; label="WholeStageCodegen (9)\n \nduration: 21 ms"; 10 [labelType="html" label="Sort sort time: 0 ms peak memory: 16.1 MiB spill size: 0.0 B"]; 11 [labelType="html" label=" Project "]; 12 [labelType="html" label="SortMergeJoin number of output rows: 2 spill size: 0.0 B"]; } subgraph cluster13 { isCluster="true"; label="WholeStageCodegen (7)\n \nduration: 50 ms"; 14 [labelType="html" label="Sort sort time: 0 ms peak memory: 16.1 MiB spill size: 0.0 B"]; } 15 [labelType="html" label="AQEShuffleRead number of partitions: 1 partition data size: 88.0 B number of coalesced partitions: 1"]; 16 [labelType="html" label="Exchange shuffle records written: 2 local merged chunks fetched: 0 shuffle write time: 0 ms remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 2 local bytes read: 81.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 1 remote merged chunks fetched: 0 remote blocks read: 0 data size: 64.0 B local merged bytes read: 0.0 B number of partitions: 10 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written: 81.0 B"]; subgraph cluster17 { isCluster="true"; label="WholeStageCodegen (6)\n \nduration: 1 ms"; 18 [labelType="html" label=" Project "]; 19 [labelType="html" label="Filter number of output rows: 2"]; } 20 [labelType="html" label="Window spill size: 0.0 B"]; subgraph cluster21 { isCluster="true"; label="WholeStageCodegen (5)\n \nduration: 9 ms"; 22 [labelType="html" label="Sort sort time: 0 ms peak memory: 16.1 MiB spill size: 0.0 B"]; } 23 [labelType="html" label="AQEShuffleRead number of partitions: 1 partition data size: 271.0 B number of coalesced partitions: 1"]; 24 [labelType="html" label="Exchange shuffle records written: 7 local merged chunks fetched: 0 shuffle write time: 0 ms remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 7 local bytes read: 252.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 1 remote merged chunks fetched: 0 remote blocks read: 0 data size: 512.0 B local merged bytes read: 0.0 B number of partitions: 10 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written: 252.0 B"]; subgraph cluster25 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: 5 ms"; 26 [labelType="html" label=" Project "]; } 27 [labelType="html" label="CartesianProduct number of output rows: 7"]; subgraph cluster28 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: 5 ms"; 29 [labelType="html" label=" Project "]; 30 [labelType="html" label="Filter number of output rows: 7"]; 31 [labelType="html" label="ColumnarToRow number of output rows: 7 number of input batches: 1"]; } 32 [labelType="html" label="Scan parquet number of files read: 1 scan time: 3 ms metadata time: 0 ms size of files read: 20.3 KiB number of output rows: 7"]; subgraph cluster33 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: 6 ms"; 34 [labelType="html" label=" Project "]; 35 [labelType="html" label="Filter number of output rows: 1"]; } 36 [labelType="html" label="Scan csv number of output rows: 1 number of files read: 1 metadata time: 0 ms size of files read: 174.0 B"]; subgraph cluster37 { isCluster="true"; label="WholeStageCodegen (8)\n \nduration: 31 ms"; 38 [labelType="html" label="Sort sort time: 0 ms peak memory: 16.1 MiB spill size: 0.0 B"]; } 39 [labelType="html" label="AQEShuffleRead number of partitions: 1 partition data size: 80.0 B number of coalesced partitions: 1"]; 40 [labelType="html" label="Exchange shuffle records written: 1 local merged chunks fetched: 0 shuffle write time: 0 ms remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 1 local bytes read: 79.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 1 remote merged chunks fetched: 0 remote blocks read: 0 data size: 40.0 B local merged bytes read: 0.0 B number of partitions: 10 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written: 79.0 B"]; subgraph cluster41 { isCluster="true"; label="WholeStageCodegen (4)\n \nduration: total (min, med, max (stageId: taskId))\n128 ms (4 ms, 13 ms, 24 ms (stage 8083.0: task 8757))"; 42 [labelType="html" label=" Project "]; 43 [labelType="html" label="Filter number of output rows: 1"]; 44 [labelType="html" label="ColumnarToRow number of output rows: 20,000 number of input batches: 5"]; } 45 [labelType="html" label="Scan parquet number of files read: 10 scan time total (min, med, max (stageId: taskId)) 104 ms (4 ms, 12 ms, 15 ms (stage 8083.0: task 8759)) metadata time: 0 ms size of files read: 30.5 MiB number of output rows: 20,000"]; 2->0; 3->2; 5->3; 6->5; 7->6; 8->7; 10->8; 11->10; 12->11; 14->12; 15->14; 16->15; 18->16; 19->18; 20->19; 22->20; 23->22; 24->23; 26->24; 27->26; 29->27; 30->29; 31->30; 32->31; 34->27; 35->34; 36->35; 38->12; 39->38; 40->39; 42->40; 43->42; 44->43; 45->44; }

AdaptiveSparkPlan isFinalPlan=true

HashAggregate(keys=[], functions=[count(1)])

WholeStageCodegen (11)

Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=489250]

HashAggregate(keys=[], functions=[partial_count(1)])

Project

Filter (COL_2F1A6F60_882E_44EE_A93A_AED2110E2882#794187 <=> 1)

WholeStageCodegen (10)

Window [row_number() windowspecdefinition(COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_D566CC23_E2D6_4115_8A37_046E955BBD91#794093 DESC NULLS LAST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS COL_2F1A6F60_882E_44EE_A93A_AED2110E2882#794187], [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249], [COL_D566CC23_E2D6_4115_8A37_046E955BBD91#794093 DESC NULLS LAST]

Sort [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249 ASC NULLS FIRST, COL_D566CC23_E2D6_4115_8A37_046E955BBD91#794093 DESC NULLS LAST], false, 0

Project [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, 02_A_ELIGIBLE_210376_COL_7DC2A78A_5ED5_4908_AC94_0885B83B98CD#793900 AS COL_D566CC23_E2D6_4115_8A37_046E955BBD91#794093]

SortMergeJoin [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249], [02_A_ELIGIBLE_210376_COL_F76265AB_8748_4114_92B4_00E37DE25D34#793863], Inner

WholeStageCodegen (9)

Sort [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249 ASC NULLS FIRST], false, 0

WholeStageCodegen (7)

AQEShuffleRead coalesced

Exchange hashpartitioning(COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, 10), ENSURE_REQUIREMENTS, [plan_id=489100]

Project [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249]

Filter (COL_B2D8469D_E32C_4A51_BA8A_9172B340952C#793495 <=> 1)

WholeStageCodegen (6)

Window [row_number() windowspecdefinition(COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_C97CE078_7AD5_4D54_A5FF_290B4E6B18FF#793423, COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254 ASC NULLS LAST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS COL_B2D8469D_E32C_4A51_BA8A_9172B340952C#793495], [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_C97CE078_7AD5_4D54_A5FF_290B4E6B18FF#793423], [COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254 ASC NULLS LAST]

Sort [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249 ASC NULLS FIRST, COL_C97CE078_7AD5_4D54_A5FF_290B4E6B18FF#793423 ASC NULLS FIRST, COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254 ASC NULLS LAST], false, 0

WholeStageCodegen (5)

AQEShuffleRead coalesced

Exchange hashpartitioning(COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_C97CE078_7AD5_4D54_A5FF_290B4E6B18FF#793423, 10), ENSURE_REQUIREMENTS, [plan_id=488949]

Project [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, str_extract_by_str(COL_1BD1CF4D_D1E1_4C54_94F8_0754D4126CAC#793251, *, RIGHT) AS COL_C97CE078_7AD5_4D54_A5FF_290B4E6B18FF#793423, COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254]

WholeStageCodegen (3)

CartesianProduct

Project [COL_64ED17E5_A991_405B_AEED_8A2977283F76#792816 AS COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_DC641C8D_1EAF_49BD_A02D_E74816E44C28#792858 AS COL_1BD1CF4D_D1E1_4C54_94F8_0754D4126CAC#793251, COL_7D270645_B926_42AF_A110_EB3A1A53031B#792838 AS COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254]

Filter ((COL_64ED17E5_A991_405B_AEED_8A2977283F76#792816 <=> 011040518) AND isnotnull(COL_64ED17E5_A991_405B_AEED_8A2977283F76#792816))

ColumnarToRow

WholeStageCodegen (1)

FileScan parquet [COL_64ED17E5_A991_405B_AEED_8A2977283F76#792816,COL_7D270645_B926_42AF_A110_EB3A1A53031B#792838,COL_DC641C8D_1EAF_49BD_A02D_E74816E44C28#792858] Batched: true, DataFilters: [(COL_64ED17E5_A991_405B_AEED_8A2977283F76#792816 <=> 011040518), isnotnull(COL_64ED17E5_A991_405..., Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/data/output/cache/parquet/datablock/00_B_ACTES_PATI_210234], PartitionFilters: [], PushedFilters: [EqualNullSafe(COL_64ED17E5_A991_405B_AEED_8A2977283F76,011040518), IsNotNull(COL_64ED17E5_A991_4..., ReadSchema: struct<COL_64ED17E5_A991_405B_AEED_8A2977283F76:string,COL_7D270645_B926_42AF_A110_EB3A1A53031B:s...

Project

Filter ((isnotnull(REGLES#792941) AND (RC-SSN-SSF = REGLES#792941)) AND (APPLICABLE#792942 <=> O))

WholeStageCodegen (2)

FileScan csv [REGLES#792941,APPLICABLE#792942] Batched: false, DataFilters: [isnotnull(REGLES#792941), (RC-SSN-SSF = REGLES#792941), (APPLICABLE#792942 <=> O)], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/data/input/depot/csv/tmp/REGLES_APPLICAB_209983/REGLES-A-TTT new..., PartitionFilters: [], PushedFilters: [IsNotNull(REGLES), EqualTo(REGLES,RC-SSN-SSF), EqualNullSafe(APPLICABLE,O)], ReadSchema: struct<REGLES:string,APPLICABLE:string>

Sort [02_A_ELIGIBLE_210376_COL_F76265AB_8748_4114_92B4_00E37DE25D34#793863 ASC NULLS FIRST], false, 0

WholeStageCodegen (8)

AQEShuffleRead coalesced

Exchange hashpartitioning(02_A_ELIGIBLE_210376_COL_F76265AB_8748_4114_92B4_00E37DE25D34#793863, 10), ENSURE_REQUIREMENTS, [plan_id=488985]

Project [COL_F76265AB_8748_4114_92B4_00E37DE25D34#792968 AS 02_A_ELIGIBLE_210376_COL_F76265AB_8748_4114_92B4_00E37DE25D34#793863, COL_7DC2A78A_5ED5_4908_AC94_0885B83B98CD#793005 AS 02_A_ELIGIBLE_210376_COL_7DC2A78A_5ED5_4908_AC94_0885B83B98CD#793900]

Filter (((isnotnull(COL_1DDA21A4_0127_416B_A5DB_7EDD9CC61B56#792963) AND (A = COL_1DDA21A4_0127_416B_A5DB_7EDD9CC61B56#792963)) AND (COL_F76265AB_8748_4114_92B4_00E37DE25D34#792968 <=> 011040518)) AND isnotnull(COL_F76265AB_8748_4114_92B4_00E37DE25D34#792968))

ColumnarToRow

WholeStageCodegen (4)

FileScan parquet [COL_1DDA21A4_0127_416B_A5DB_7EDD9CC61B56#792963,COL_F76265AB_8748_4114_92B4_00E37DE25D34#792968,COL_7DC2A78A_5ED5_4908_AC94_0885B83B98CD#793005] Batched: true, DataFilters: [isnotnull(COL_1DDA21A4_0127_416B_A5DB_7EDD9CC61B56#792963), (A = COL_1DDA21A4_0127_416B_A5DB_7ED..., Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/data/output/cache/parquet/datablock/02_A_ELIGIBLE_210376], PartitionFilters: [], PushedFilters: [IsNotNull(COL_1DDA21A4_0127_416B_A5DB_7EDD9CC61B56), EqualTo(COL_1DDA21A4_0127_416B_A5DB_7EDD9CC..., ReadSchema: struct<COL_1DDA21A4_0127_416B_A5DB_7EDD9CC61B56:string,COL_F76265AB_8748_4114_92B4_00E37DE25D34:s...

Details

== Physical Plan ==
AdaptiveSparkPlan (65)
+- == Final Plan ==
   * HashAggregate (38)
   +- ShuffleQueryStage (37), Statistics(sizeInBytes=16.0 B, rowCount=1)
      +- Exchange (36)
         +- * HashAggregate (35)
            +- * Project (34)
               +- * Filter (33)
                  +- Window (32)
                     +- * Sort (31)
                        +- * Project (30)
                           +- * SortMergeJoin Inner (29)
                              :- * Sort (20)
                              :  +- AQEShuffleRead (19)
                              :     +- ShuffleQueryStage (18), Statistics(sizeInBytes=64.0 B, rowCount=2)
                              :        +- Exchange (17)
                              :           +- * Project (16)
                              :              +- * Filter (15)
                              :                 +- Window (14)
                              :                    +- * Sort (13)
                              :                       +- AQEShuffleRead (12)
                              :                          +- ShuffleQueryStage (11), Statistics(sizeInBytes=512.0 B, rowCount=7)
                              :                             +- Exchange (10)
                              :                                +- * Project (9)
                              :                                   +- CartesianProduct Inner (8)
                              :                                      :- * Project (4)
                              :                                      :  +- * Filter (3)
                              :                                      :     +- * ColumnarToRow (2)
                              :                                      :        +- Scan parquet  (1)
                              :                                      +- * Project (7)
                              :                                         +- * Filter (6)
                              :                                            +- Scan csv  (5)
                              +- * Sort (28)
                                 +- AQEShuffleRead (27)
                                    +- ShuffleQueryStage (26), Statistics(sizeInBytes=40.0 B, rowCount=1)
                                       +- Exchange (25)
                                          +- * Project (24)
                                             +- * Filter (23)
                                                +- * ColumnarToRow (22)
                                                   +- Scan parquet  (21)
+- == Initial Plan ==
   HashAggregate (64)
   +- Exchange (63)
      +- HashAggregate (62)
         +- Project (61)
            +- Filter (60)
               +- Window (59)
                  +- Sort (58)
                     +- Project (57)
                        +- SortMergeJoin Inner (56)
                           :- Sort (51)
                           :  +- Exchange (50)
                           :     +- Project (49)
                           :        +- Filter (48)
                           :           +- Window (47)
                           :              +- Sort (46)
                           :                 +- Exchange (45)
                           :                    +- Project (44)
                           :                       +- CartesianProduct Inner (43)
                           :                          :- Project (40)
                           :                          :  +- Filter (39)
                           :                          :     +- Scan parquet  (1)
                           :                          +- Project (42)
                           :                             +- Filter (41)
                           :                                +- Scan csv  (5)
                           +- Sort (55)
                              +- Exchange (54)
                                 +- Project (53)
                                    +- Filter (52)
                                       +- Scan parquet  (21)


(1) Scan parquet 
Output [3]: [COL_64ED17E5_A991_405B_AEED_8A2977283F76#792816, COL_7D270645_B926_42AF_A110_EB3A1A53031B#792838, COL_DC641C8D_1EAF_49BD_A02D_E74816E44C28#792858]
Batched: true
Location: InMemoryFileIndex [file:/data/output/cache/parquet/datablock/00_B_ACTES_PATI_210234]
PushedFilters: [EqualNullSafe(COL_64ED17E5_A991_405B_AEED_8A2977283F76,011040518), IsNotNull(COL_64ED17E5_A991_405B_AEED_8A2977283F76)]
ReadSchema: struct<COL_64ED17E5_A991_405B_AEED_8A2977283F76:string,COL_7D270645_B926_42AF_A110_EB3A1A53031B:string,COL_DC641C8D_1EAF_49BD_A02D_E74816E44C28:string>

(2) ColumnarToRow [codegen id : 1]
Input [3]: [COL_64ED17E5_A991_405B_AEED_8A2977283F76#792816, COL_7D270645_B926_42AF_A110_EB3A1A53031B#792838, COL_DC641C8D_1EAF_49BD_A02D_E74816E44C28#792858]

(3) Filter [codegen id : 1]
Input [3]: [COL_64ED17E5_A991_405B_AEED_8A2977283F76#792816, COL_7D270645_B926_42AF_A110_EB3A1A53031B#792838, COL_DC641C8D_1EAF_49BD_A02D_E74816E44C28#792858]
Condition : ((COL_64ED17E5_A991_405B_AEED_8A2977283F76#792816 <=> 011040518) AND isnotnull(COL_64ED17E5_A991_405B_AEED_8A2977283F76#792816))

(4) Project [codegen id : 1]
Output [3]: [COL_64ED17E5_A991_405B_AEED_8A2977283F76#792816 AS COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_DC641C8D_1EAF_49BD_A02D_E74816E44C28#792858 AS COL_1BD1CF4D_D1E1_4C54_94F8_0754D4126CAC#793251, COL_7D270645_B926_42AF_A110_EB3A1A53031B#792838 AS COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254]
Input [3]: [COL_64ED17E5_A991_405B_AEED_8A2977283F76#792816, COL_7D270645_B926_42AF_A110_EB3A1A53031B#792838, COL_DC641C8D_1EAF_49BD_A02D_E74816E44C28#792858]

(5) Scan csv 
Output [2]: [REGLES#792941, APPLICABLE#792942]
Batched: false
Location: InMemoryFileIndex [file:/data/input/depot/csv/tmp/REGLES_APPLICAB_209983/REGLES-A-TTT new.csv]
PushedFilters: [IsNotNull(REGLES), EqualTo(REGLES,RC-SSN-SSF), EqualNullSafe(APPLICABLE,O)]
ReadSchema: struct<REGLES:string,APPLICABLE:string>

(6) Filter [codegen id : 2]
Input [2]: [REGLES#792941, APPLICABLE#792942]
Condition : ((isnotnull(REGLES#792941) AND (RC-SSN-SSF = REGLES#792941)) AND (APPLICABLE#792942 <=> O))

(7) Project [codegen id : 2]
Output: []
Input [2]: [REGLES#792941, APPLICABLE#792942]

(8) CartesianProduct
Join type: Inner
Join condition: None

(9) Project [codegen id : 3]
Output [3]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, str_extract_by_str(COL_1BD1CF4D_D1E1_4C54_94F8_0754D4126CAC#793251, *, RIGHT) AS COL_C97CE078_7AD5_4D54_A5FF_290B4E6B18FF#793423, COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254]
Input [3]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_1BD1CF4D_D1E1_4C54_94F8_0754D4126CAC#793251, COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254]

(10) Exchange
Input [3]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_C97CE078_7AD5_4D54_A5FF_290B4E6B18FF#793423, COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254]
Arguments: hashpartitioning(COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_C97CE078_7AD5_4D54_A5FF_290B4E6B18FF#793423, 10), ENSURE_REQUIREMENTS, [plan_id=488949]

(11) ShuffleQueryStage
Output [3]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_C97CE078_7AD5_4D54_A5FF_290B4E6B18FF#793423, COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254]
Arguments: 0

(12) AQEShuffleRead
Input [3]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_C97CE078_7AD5_4D54_A5FF_290B4E6B18FF#793423, COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254]
Arguments: coalesced

(13) Sort [codegen id : 5]
Input [3]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_C97CE078_7AD5_4D54_A5FF_290B4E6B18FF#793423, COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254]
Arguments: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249 ASC NULLS FIRST, COL_C97CE078_7AD5_4D54_A5FF_290B4E6B18FF#793423 ASC NULLS FIRST, COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254 ASC NULLS LAST], false, 0

(14) Window
Input [3]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_C97CE078_7AD5_4D54_A5FF_290B4E6B18FF#793423, COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254]
Arguments: [row_number() windowspecdefinition(COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_C97CE078_7AD5_4D54_A5FF_290B4E6B18FF#793423, COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254 ASC NULLS LAST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS COL_B2D8469D_E32C_4A51_BA8A_9172B340952C#793495], [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_C97CE078_7AD5_4D54_A5FF_290B4E6B18FF#793423], [COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254 ASC NULLS LAST]

(15) Filter [codegen id : 6]
Input [4]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_C97CE078_7AD5_4D54_A5FF_290B4E6B18FF#793423, COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254, COL_B2D8469D_E32C_4A51_BA8A_9172B340952C#793495]
Condition : (COL_B2D8469D_E32C_4A51_BA8A_9172B340952C#793495 <=> 1)

(16) Project [codegen id : 6]
Output [1]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249]
Input [4]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_C97CE078_7AD5_4D54_A5FF_290B4E6B18FF#793423, COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254, COL_B2D8469D_E32C_4A51_BA8A_9172B340952C#793495]

(17) Exchange
Input [1]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249]
Arguments: hashpartitioning(COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, 10), ENSURE_REQUIREMENTS, [plan_id=489100]

(18) ShuffleQueryStage
Output [1]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249]
Arguments: 2

(19) AQEShuffleRead
Input [1]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249]
Arguments: coalesced

(20) Sort [codegen id : 7]
Input [1]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249]
Arguments: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249 ASC NULLS FIRST], false, 0

(21) Scan parquet 
Output [3]: [COL_1DDA21A4_0127_416B_A5DB_7EDD9CC61B56#792963, COL_F76265AB_8748_4114_92B4_00E37DE25D34#792968, COL_7DC2A78A_5ED5_4908_AC94_0885B83B98CD#793005]
Batched: true
Location: InMemoryFileIndex [file:/data/output/cache/parquet/datablock/02_A_ELIGIBLE_210376]
PushedFilters: [IsNotNull(COL_1DDA21A4_0127_416B_A5DB_7EDD9CC61B56), EqualTo(COL_1DDA21A4_0127_416B_A5DB_7EDD9CC61B56,A), EqualNullSafe(COL_F76265AB_8748_4114_92B4_00E37DE25D34,011040518), IsNotNull(COL_F76265AB_8748_4114_92B4_00E37DE25D34)]
ReadSchema: struct<COL_1DDA21A4_0127_416B_A5DB_7EDD9CC61B56:string,COL_F76265AB_8748_4114_92B4_00E37DE25D34:string,COL_7DC2A78A_5ED5_4908_AC94_0885B83B98CD:string>

(22) ColumnarToRow [codegen id : 4]
Input [3]: [COL_1DDA21A4_0127_416B_A5DB_7EDD9CC61B56#792963, COL_F76265AB_8748_4114_92B4_00E37DE25D34#792968, COL_7DC2A78A_5ED5_4908_AC94_0885B83B98CD#793005]

(23) Filter [codegen id : 4]
Input [3]: [COL_1DDA21A4_0127_416B_A5DB_7EDD9CC61B56#792963, COL_F76265AB_8748_4114_92B4_00E37DE25D34#792968, COL_7DC2A78A_5ED5_4908_AC94_0885B83B98CD#793005]
Condition : (((isnotnull(COL_1DDA21A4_0127_416B_A5DB_7EDD9CC61B56#792963) AND (A = COL_1DDA21A4_0127_416B_A5DB_7EDD9CC61B56#792963)) AND (COL_F76265AB_8748_4114_92B4_00E37DE25D34#792968 <=> 011040518)) AND isnotnull(COL_F76265AB_8748_4114_92B4_00E37DE25D34#792968))

(24) Project [codegen id : 4]
Output [2]: [COL_F76265AB_8748_4114_92B4_00E37DE25D34#792968 AS 02_A_ELIGIBLE_210376_COL_F76265AB_8748_4114_92B4_00E37DE25D34#793863, COL_7DC2A78A_5ED5_4908_AC94_0885B83B98CD#793005 AS 02_A_ELIGIBLE_210376_COL_7DC2A78A_5ED5_4908_AC94_0885B83B98CD#793900]
Input [3]: [COL_1DDA21A4_0127_416B_A5DB_7EDD9CC61B56#792963, COL_F76265AB_8748_4114_92B4_00E37DE25D34#792968, COL_7DC2A78A_5ED5_4908_AC94_0885B83B98CD#793005]

(25) Exchange
Input [2]: [02_A_ELIGIBLE_210376_COL_F76265AB_8748_4114_92B4_00E37DE25D34#793863, 02_A_ELIGIBLE_210376_COL_7DC2A78A_5ED5_4908_AC94_0885B83B98CD#793900]
Arguments: hashpartitioning(02_A_ELIGIBLE_210376_COL_F76265AB_8748_4114_92B4_00E37DE25D34#793863, 10), ENSURE_REQUIREMENTS, [plan_id=488985]

(26) ShuffleQueryStage
Output [2]: [02_A_ELIGIBLE_210376_COL_F76265AB_8748_4114_92B4_00E37DE25D34#793863, 02_A_ELIGIBLE_210376_COL_7DC2A78A_5ED5_4908_AC94_0885B83B98CD#793900]
Arguments: 1

(27) AQEShuffleRead
Input [2]: [02_A_ELIGIBLE_210376_COL_F76265AB_8748_4114_92B4_00E37DE25D34#793863, 02_A_ELIGIBLE_210376_COL_7DC2A78A_5ED5_4908_AC94_0885B83B98CD#793900]
Arguments: coalesced

(28) Sort [codegen id : 8]
Input [2]: [02_A_ELIGIBLE_210376_COL_F76265AB_8748_4114_92B4_00E37DE25D34#793863, 02_A_ELIGIBLE_210376_COL_7DC2A78A_5ED5_4908_AC94_0885B83B98CD#793900]
Arguments: [02_A_ELIGIBLE_210376_COL_F76265AB_8748_4114_92B4_00E37DE25D34#793863 ASC NULLS FIRST], false, 0

(29) SortMergeJoin [codegen id : 9]
Left keys [1]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249]
Right keys [1]: [02_A_ELIGIBLE_210376_COL_F76265AB_8748_4114_92B4_00E37DE25D34#793863]
Join type: Inner
Join condition: None

(30) Project [codegen id : 9]
Output [2]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, 02_A_ELIGIBLE_210376_COL_7DC2A78A_5ED5_4908_AC94_0885B83B98CD#793900 AS COL_D566CC23_E2D6_4115_8A37_046E955BBD91#794093]
Input [3]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, 02_A_ELIGIBLE_210376_COL_F76265AB_8748_4114_92B4_00E37DE25D34#793863, 02_A_ELIGIBLE_210376_COL_7DC2A78A_5ED5_4908_AC94_0885B83B98CD#793900]

(31) Sort [codegen id : 9]
Input [2]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_D566CC23_E2D6_4115_8A37_046E955BBD91#794093]
Arguments: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249 ASC NULLS FIRST, COL_D566CC23_E2D6_4115_8A37_046E955BBD91#794093 DESC NULLS LAST], false, 0

(32) Window
Input [2]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_D566CC23_E2D6_4115_8A37_046E955BBD91#794093]
Arguments: [row_number() windowspecdefinition(COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_D566CC23_E2D6_4115_8A37_046E955BBD91#794093 DESC NULLS LAST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS COL_2F1A6F60_882E_44EE_A93A_AED2110E2882#794187], [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249], [COL_D566CC23_E2D6_4115_8A37_046E955BBD91#794093 DESC NULLS LAST]

(33) Filter [codegen id : 10]
Input [3]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_D566CC23_E2D6_4115_8A37_046E955BBD91#794093, COL_2F1A6F60_882E_44EE_A93A_AED2110E2882#794187]
Condition : (COL_2F1A6F60_882E_44EE_A93A_AED2110E2882#794187 <=> 1)

(34) Project [codegen id : 10]
Output: []
Input [3]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_D566CC23_E2D6_4115_8A37_046E955BBD91#794093, COL_2F1A6F60_882E_44EE_A93A_AED2110E2882#794187]

(35) HashAggregate [codegen id : 10]
Input: []
Keys: []
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#794426L]
Results [1]: [count#794427L]

(36) Exchange
Input [1]: [count#794427L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=489250]

(37) ShuffleQueryStage
Output [1]: [count#794427L]
Arguments: 3

(38) HashAggregate [codegen id : 11]
Input [1]: [count#794427L]
Keys: []
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#794423L]
Results [1]: [count(1)#794423L AS count#794424L]

(39) Filter
Input [3]: [COL_64ED17E5_A991_405B_AEED_8A2977283F76#792816, COL_7D270645_B926_42AF_A110_EB3A1A53031B#792838, COL_DC641C8D_1EAF_49BD_A02D_E74816E44C28#792858]
Condition : ((COL_64ED17E5_A991_405B_AEED_8A2977283F76#792816 <=> 011040518) AND isnotnull(COL_64ED17E5_A991_405B_AEED_8A2977283F76#792816))

(40) Project
Output [3]: [COL_64ED17E5_A991_405B_AEED_8A2977283F76#792816 AS COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_DC641C8D_1EAF_49BD_A02D_E74816E44C28#792858 AS COL_1BD1CF4D_D1E1_4C54_94F8_0754D4126CAC#793251, COL_7D270645_B926_42AF_A110_EB3A1A53031B#792838 AS COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254]
Input [3]: [COL_64ED17E5_A991_405B_AEED_8A2977283F76#792816, COL_7D270645_B926_42AF_A110_EB3A1A53031B#792838, COL_DC641C8D_1EAF_49BD_A02D_E74816E44C28#792858]

(41) Filter
Input [2]: [REGLES#792941, APPLICABLE#792942]
Condition : ((isnotnull(REGLES#792941) AND (RC-SSN-SSF = REGLES#792941)) AND (APPLICABLE#792942 <=> O))

(42) Project
Output: []
Input [2]: [REGLES#792941, APPLICABLE#792942]

(43) CartesianProduct
Join type: Inner
Join condition: None

(44) Project
Output [3]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, str_extract_by_str(COL_1BD1CF4D_D1E1_4C54_94F8_0754D4126CAC#793251, *, RIGHT) AS COL_C97CE078_7AD5_4D54_A5FF_290B4E6B18FF#793423, COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254]
Input [3]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_1BD1CF4D_D1E1_4C54_94F8_0754D4126CAC#793251, COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254]

(45) Exchange
Input [3]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_C97CE078_7AD5_4D54_A5FF_290B4E6B18FF#793423, COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254]
Arguments: hashpartitioning(COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_C97CE078_7AD5_4D54_A5FF_290B4E6B18FF#793423, 10), ENSURE_REQUIREMENTS, [plan_id=488857]

(46) Sort
Input [3]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_C97CE078_7AD5_4D54_A5FF_290B4E6B18FF#793423, COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254]
Arguments: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249 ASC NULLS FIRST, COL_C97CE078_7AD5_4D54_A5FF_290B4E6B18FF#793423 ASC NULLS FIRST, COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254 ASC NULLS LAST], false, 0

(47) Window
Input [3]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_C97CE078_7AD5_4D54_A5FF_290B4E6B18FF#793423, COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254]
Arguments: [row_number() windowspecdefinition(COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_C97CE078_7AD5_4D54_A5FF_290B4E6B18FF#793423, COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254 ASC NULLS LAST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS COL_B2D8469D_E32C_4A51_BA8A_9172B340952C#793495], [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_C97CE078_7AD5_4D54_A5FF_290B4E6B18FF#793423], [COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254 ASC NULLS LAST]

(48) Filter
Input [4]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_C97CE078_7AD5_4D54_A5FF_290B4E6B18FF#793423, COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254, COL_B2D8469D_E32C_4A51_BA8A_9172B340952C#793495]
Condition : (COL_B2D8469D_E32C_4A51_BA8A_9172B340952C#793495 <=> 1)

(49) Project
Output [1]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249]
Input [4]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_C97CE078_7AD5_4D54_A5FF_290B4E6B18FF#793423, COL_7453C4CB_1AAF_4D1F_A079_968F5F790238#793254, COL_B2D8469D_E32C_4A51_BA8A_9172B340952C#793495]

(50) Exchange
Input [1]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249]
Arguments: hashpartitioning(COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, 10), ENSURE_REQUIREMENTS, [plan_id=488866]

(51) Sort
Input [1]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249]
Arguments: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249 ASC NULLS FIRST], false, 0

(52) Filter
Input [3]: [COL_1DDA21A4_0127_416B_A5DB_7EDD9CC61B56#792963, COL_F76265AB_8748_4114_92B4_00E37DE25D34#792968, COL_7DC2A78A_5ED5_4908_AC94_0885B83B98CD#793005]
Condition : (((isnotnull(COL_1DDA21A4_0127_416B_A5DB_7EDD9CC61B56#792963) AND (A = COL_1DDA21A4_0127_416B_A5DB_7EDD9CC61B56#792963)) AND (COL_F76265AB_8748_4114_92B4_00E37DE25D34#792968 <=> 011040518)) AND isnotnull(COL_F76265AB_8748_4114_92B4_00E37DE25D34#792968))

(53) Project
Output [2]: [COL_F76265AB_8748_4114_92B4_00E37DE25D34#792968 AS 02_A_ELIGIBLE_210376_COL_F76265AB_8748_4114_92B4_00E37DE25D34#793863, COL_7DC2A78A_5ED5_4908_AC94_0885B83B98CD#793005 AS 02_A_ELIGIBLE_210376_COL_7DC2A78A_5ED5_4908_AC94_0885B83B98CD#793900]
Input [3]: [COL_1DDA21A4_0127_416B_A5DB_7EDD9CC61B56#792963, COL_F76265AB_8748_4114_92B4_00E37DE25D34#792968, COL_7DC2A78A_5ED5_4908_AC94_0885B83B98CD#793005]

(54) Exchange
Input [2]: [02_A_ELIGIBLE_210376_COL_F76265AB_8748_4114_92B4_00E37DE25D34#793863, 02_A_ELIGIBLE_210376_COL_7DC2A78A_5ED5_4908_AC94_0885B83B98CD#793900]
Arguments: hashpartitioning(02_A_ELIGIBLE_210376_COL_F76265AB_8748_4114_92B4_00E37DE25D34#793863, 10), ENSURE_REQUIREMENTS, [plan_id=488867]

(55) Sort
Input [2]: [02_A_ELIGIBLE_210376_COL_F76265AB_8748_4114_92B4_00E37DE25D34#793863, 02_A_ELIGIBLE_210376_COL_7DC2A78A_5ED5_4908_AC94_0885B83B98CD#793900]
Arguments: [02_A_ELIGIBLE_210376_COL_F76265AB_8748_4114_92B4_00E37DE25D34#793863 ASC NULLS FIRST], false, 0

(56) SortMergeJoin
Left keys [1]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249]
Right keys [1]: [02_A_ELIGIBLE_210376_COL_F76265AB_8748_4114_92B4_00E37DE25D34#793863]
Join type: Inner
Join condition: None

(57) Project
Output [2]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, 02_A_ELIGIBLE_210376_COL_7DC2A78A_5ED5_4908_AC94_0885B83B98CD#793900 AS COL_D566CC23_E2D6_4115_8A37_046E955BBD91#794093]
Input [3]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, 02_A_ELIGIBLE_210376_COL_F76265AB_8748_4114_92B4_00E37DE25D34#793863, 02_A_ELIGIBLE_210376_COL_7DC2A78A_5ED5_4908_AC94_0885B83B98CD#793900]

(58) Sort
Input [2]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_D566CC23_E2D6_4115_8A37_046E955BBD91#794093]
Arguments: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249 ASC NULLS FIRST, COL_D566CC23_E2D6_4115_8A37_046E955BBD91#794093 DESC NULLS LAST], false, 0

(59) Window
Input [2]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_D566CC23_E2D6_4115_8A37_046E955BBD91#794093]
Arguments: [row_number() windowspecdefinition(COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_D566CC23_E2D6_4115_8A37_046E955BBD91#794093 DESC NULLS LAST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS COL_2F1A6F60_882E_44EE_A93A_AED2110E2882#794187], [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249], [COL_D566CC23_E2D6_4115_8A37_046E955BBD91#794093 DESC NULLS LAST]

(60) Filter
Input [3]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_D566CC23_E2D6_4115_8A37_046E955BBD91#794093, COL_2F1A6F60_882E_44EE_A93A_AED2110E2882#794187]
Condition : (COL_2F1A6F60_882E_44EE_A93A_AED2110E2882#794187 <=> 1)

(61) Project
Output: []
Input [3]: [COL_568DEDCB_F936_41B1_A0EC_986E8205D003#793249, COL_D566CC23_E2D6_4115_8A37_046E955BBD91#794093, COL_2F1A6F60_882E_44EE_A93A_AED2110E2882#794187]

(62) HashAggregate
Input: []
Keys: []
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#794426L]
Results [1]: [count#794427L]

(63) Exchange
Input [1]: [count#794427L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=488879]

(64) HashAggregate
Input [1]: [count#794427L]
Keys: []
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#794423L]
Results [1]: [count(1)#794423L AS count#794424L]

(65) AdaptiveSparkPlan
Output [1]: [count#794424L]
Arguments: isFinalPlan=true

SQL / DataFrame Properties

Name	Value
spark.sql.optimizer.nestedPredicatePushdown.supportedFileSources	parquet,orc,geoparquet