SP1 - Details for Query 3406

Submitted Time: 2026/06/12 05:56:50
Duration: 0.4 s
Succeeded Jobs: 5295 5296 5297 5298

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; subgraph cluster1 { isCluster="true"; label="WholeStageCodegen (7)\n \nduration: 0 ms"; 2 [labelType="html" label="HashAggregate time in aggregation build: 0 ms number of output rows: 1"]; } 3 [labelType="html" label="Exchange shuffle records written: 1 local merged chunks fetched: 0 shuffle write time: 0 ms remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 1 local bytes read: 59.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 1 remote merged chunks fetched: 0 remote blocks read: 0 data size: 16.0 B local merged bytes read: 0.0 B number of partitions: 1 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written: 59.0 B"]; subgraph cluster4 { isCluster="true"; label="WholeStageCodegen (6)\n \nduration: 2 ms"; 5 [labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build: 1 ms peak memory: 0.0 B number of output rows: 1 number of sort fallback tasks: 0 avg hash probes per key: 0"]; 6 [labelType="html" label=" Project "]; 7 [labelType="html" label="SortMergeJoin number of output rows: 1 spill size: 0.0 B"]; } subgraph cluster8 { isCluster="true"; label="WholeStageCodegen (4)\n \nduration: 20 ms"; 9 [labelType="html" label="Sort sort time: 0 ms peak memory: 16.1 MiB spill size: 0.0 B"]; } 10 [labelType="html" label="AQEShuffleRead number of partitions: 1 partition data size: 88.0 B number of coalesced partitions: 1"]; 11 [labelType="html" label="Exchange shuffle records written: 1 local merged chunks fetched: 0 shuffle write time: 1 ms remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 1 local bytes read: 84.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 1 remote merged chunks fetched: 0 remote blocks read: 0 data size: 40.0 B local merged bytes read: 0.0 B number of partitions: 10 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written: 84.0 B"]; 12 [labelType="html" label="CartesianProduct number of output rows: 1"]; subgraph cluster13 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n1.2 s (21 ms, 114 ms, 169 ms (stage 12299.0: task 12810))"; 14 [labelType="html" label=" Project "]; 15 [labelType="html" label="Filter number of output rows: 1"]; } 16 [labelType="html" label="Scan csv number of output rows: 1 number of files read: 1 metadata time: 0 ms size of files read: 36.3 MiB"]; subgraph cluster17 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: total (min, med, max (stageId: taskId))\n186 ms (13 ms, 18 ms, 25 ms (stage 12299.0: task 12811))"; 18 [labelType="html" label=" Project "]; 19 [labelType="html" label="Filter number of output rows: 10"]; } 20 [labelType="html" label="Scan csv number of output rows: 10 number of files read: 1 metadata time: 0 ms size of files read: 174.0 B"]; subgraph cluster21 { isCluster="true"; label="WholeStageCodegen (5)\n \nduration: 0 ms"; 22 [labelType="html" label="Sort sort time: 0 ms peak memory: 16.1 MiB spill size: 0.0 B"]; } 23 [labelType="html" label="AQEShuffleRead number of partitions: 1 partition data size: 645.0 B number of coalesced partitions: 1"]; 24 [labelType="html" label="Exchange shuffle records written: 13 local merged chunks fetched: 0 shuffle write time: 1 ms remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 13 local bytes read: 622.0 B fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 1 remote merged chunks fetched: 0 remote blocks read: 0 data size: 432.0 B local merged bytes read: 0.0 B number of partitions: 10 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written: 622.0 B"]; subgraph cluster25 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: 4 ms"; 26 [labelType="html" label=" Project "]; 27 [labelType="html" label="Filter number of output rows: 13"]; 28 [labelType="html" label="ColumnarToRow number of output rows: 13 number of input batches: 1"]; } 29 [labelType="html" label="Scan parquet number of files read: 1 scan time: 2 ms metadata time: 0 ms size of files read: 2.5 KiB number of output rows: 13"]; 2->0; 3->2; 5->3; 6->5; 7->6; 9->7; 10->9; 11->10; 12->11; 14->12; 15->14; 16->15; 18->12; 19->18; 20->19; 22->7; 23->22; 24->23; 26->24; 27->26; 28->27; 29->28; }

AdaptiveSparkPlan isFinalPlan=true

HashAggregate(keys=[], functions=[count(1)])

WholeStageCodegen (7)

Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=650451]

HashAggregate(keys=[], functions=[partial_count(1)])

Project

SortMergeJoin [COL_9F4D6058_F196_4B17_B4B5_2C7706DF82DC#1076355], [LISTE_UF_SUN_SU_210126_SPECIALITE_ETAB_210127#1076407], LeftOuter

WholeStageCodegen (6)

Sort [COL_9F4D6058_F196_4B17_B4B5_2C7706DF82DC#1076355 ASC NULLS FIRST], false, 0

WholeStageCodegen (4)

AQEShuffleRead coalesced

Exchange hashpartitioning(COL_9F4D6058_F196_4B17_B4B5_2C7706DF82DC#1076355, 10), ENSURE_REQUIREMENTS, [plan_id=650332]

CartesianProduct

Project [str_transform_case(str_remove_accent(str_replace_extend(str_remove_space(Spécialité exécutant#1076101, ONLY_ONE), [ ], [-])), UPPER) AS COL_9F4D6058_F196_4B17_B4B5_2C7706DF82DC#1076355]

Filter (Numéro Séjour#1076097 <=> 011038404)

WholeStageCodegen (1)

FileScan csv [Numéro Séjour#1076097,Spécialité exécutant#1076101] Batched: false, DataFilters: [(Numéro Séjour#1076097 <=> 011038404)], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/data/input/depot/csv/tmp/03_ACTES_PATIEN_209974/Horodatage_actes..., PartitionFilters: [], PushedFilters: [EqualNullSafe(Numéro Séjour,011038404)], ReadSchema: struct<Numéro Séjour:string,Spécialité exécutant:string>

Project

Filter ((isnotnull(REGLES#1076165) AND (RC-SUN-AUTO = REGLES#1076165)) AND (APPLICABLE#1076166 <=> O))

WholeStageCodegen (2)

FileScan csv [REGLES#1076165,APPLICABLE#1076166] Batched: false, DataFilters: [isnotnull(REGLES#1076165), (RC-SUN-AUTO = REGLES#1076165), (APPLICABLE#1076166 <=> O)], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/data/input/depot/csv/tmp/REGLES_APPLICAB_209983/REGLES-A-TTT new..., PartitionFilters: [], PushedFilters: [IsNotNull(REGLES), EqualTo(REGLES,RC-SUN-AUTO), EqualNullSafe(APPLICABLE,O)], ReadSchema: struct<REGLES:string,APPLICABLE:string>

Sort [LISTE_UF_SUN_SU_210126_SPECIALITE_ETAB_210127#1076407 ASC NULLS FIRST], false, 0

WholeStageCodegen (5)

AQEShuffleRead coalesced

Exchange hashpartitioning(LISTE_UF_SUN_SU_210126_SPECIALITE_ETAB_210127#1076407, 10), ENSURE_REQUIREMENTS, [plan_id=650358]

Project [str_replace_extend(str_transform_case(str_remove_accent(str_remove_space(Sp%C3%A9cialit%C3%A9+etab#1076200, ONLY_ONE)), UPPER), [ ], [-]) AS LISTE_UF_SUN_SU_210126_SPECIALITE_ETAB_210127#1076407]

Filter isnotnull(str_replace_extend(str_transform_case(str_remove_accent(str_remove_space(Sp%C3%A9cialit%C3%A9+etab#1076200, ONLY_ONE)), UPPER), [ ], [-]))

ColumnarToRow

WholeStageCodegen (3)

FileScan parquet [Sp%C3%A9cialit%C3%A9+etab#1076200] Batched: true, DataFilters: [isnotnull(str_replace_extend(str_transform_case(str_remove_accent(str_remove_space(Sp%C3%A9ciali..., Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/data/input/depot/excel/cache/LISTE_UF_SUN_SU_210006_d71b6e1bc19c..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<Sp%C3%A9cialit%C3%A9+etab:string>

Details

== Physical Plan ==
AdaptiveSparkPlan (42)
+- == Final Plan ==
   * HashAggregate (25)
   +- ShuffleQueryStage (24), Statistics(sizeInBytes=16.0 B, rowCount=1)
      +- Exchange (23)
         +- * HashAggregate (22)
            +- * Project (21)
               +- * SortMergeJoin LeftOuter (20)
                  :- * Sort (11)
                  :  +- AQEShuffleRead (10)
                  :     +- ShuffleQueryStage (9), Statistics(sizeInBytes=40.0 B, rowCount=1)
                  :        +- Exchange (8)
                  :           +- CartesianProduct Inner (7)
                  :              :- * Project (3)
                  :              :  +- * Filter (2)
                  :              :     +- Scan csv  (1)
                  :              +- * Project (6)
                  :                 +- * Filter (5)
                  :                    +- Scan csv  (4)
                  +- * Sort (19)
                     +- AQEShuffleRead (18)
                        +- ShuffleQueryStage (17), Statistics(sizeInBytes=432.0 B, rowCount=13)
                           +- Exchange (16)
                              +- * Project (15)
                                 +- * Filter (14)
                                    +- * ColumnarToRow (13)
                                       +- Scan parquet  (12)
+- == Initial Plan ==
   HashAggregate (41)
   +- Exchange (40)
      +- HashAggregate (39)
         +- Project (38)
            +- SortMergeJoin LeftOuter (37)
               :- Sort (32)
               :  +- Exchange (31)
               :     +- CartesianProduct Inner (30)
               :        :- Project (27)
               :        :  +- Filter (26)
               :        :     +- Scan csv  (1)
               :        +- Project (29)
               :           +- Filter (28)
               :              +- Scan csv  (4)
               +- Sort (36)
                  +- Exchange (35)
                     +- Project (34)
                        +- Filter (33)
                           +- Scan parquet  (12)


(1) Scan csv 
Output [2]: [Numéro Séjour#1076097, Spécialité exécutant#1076101]
Batched: false
Location: InMemoryFileIndex [file:/data/input/depot/csv/tmp/03_ACTES_PATIEN_209974/Horodatage_actes_Patient_CHSF_M122025.csv]
PushedFilters: [EqualNullSafe(Numéro Séjour,011038404)]
ReadSchema: struct<Numéro Séjour:string,Spécialité exécutant:string>

(2) Filter [codegen id : 1]
Input [2]: [Numéro Séjour#1076097, Spécialité exécutant#1076101]
Condition : (Numéro Séjour#1076097 <=> 011038404)

(3) Project [codegen id : 1]
Output [1]: [str_transform_case(str_remove_accent(str_replace_extend(str_remove_space(Spécialité exécutant#1076101, ONLY_ONE), [ ], [-])), UPPER) AS COL_9F4D6058_F196_4B17_B4B5_2C7706DF82DC#1076355]
Input [2]: [Numéro Séjour#1076097, Spécialité exécutant#1076101]

(4) Scan csv 
Output [2]: [REGLES#1076165, APPLICABLE#1076166]
Batched: false
Location: InMemoryFileIndex [file:/data/input/depot/csv/tmp/REGLES_APPLICAB_209983/REGLES-A-TTT new.csv]
PushedFilters: [IsNotNull(REGLES), EqualTo(REGLES,RC-SUN-AUTO), EqualNullSafe(APPLICABLE,O)]
ReadSchema: struct<REGLES:string,APPLICABLE:string>

(5) Filter [codegen id : 2]
Input [2]: [REGLES#1076165, APPLICABLE#1076166]
Condition : ((isnotnull(REGLES#1076165) AND (RC-SUN-AUTO = REGLES#1076165)) AND (APPLICABLE#1076166 <=> O))

(6) Project [codegen id : 2]
Output: []
Input [2]: [REGLES#1076165, APPLICABLE#1076166]

(7) CartesianProduct
Join type: Inner
Join condition: None

(8) Exchange
Input [1]: [COL_9F4D6058_F196_4B17_B4B5_2C7706DF82DC#1076355]
Arguments: hashpartitioning(COL_9F4D6058_F196_4B17_B4B5_2C7706DF82DC#1076355, 10), ENSURE_REQUIREMENTS, [plan_id=650332]

(9) ShuffleQueryStage
Output [1]: [COL_9F4D6058_F196_4B17_B4B5_2C7706DF82DC#1076355]
Arguments: 0

(10) AQEShuffleRead
Input [1]: [COL_9F4D6058_F196_4B17_B4B5_2C7706DF82DC#1076355]
Arguments: coalesced

(11) Sort [codegen id : 4]
Input [1]: [COL_9F4D6058_F196_4B17_B4B5_2C7706DF82DC#1076355]
Arguments: [COL_9F4D6058_F196_4B17_B4B5_2C7706DF82DC#1076355 ASC NULLS FIRST], false, 0

(12) Scan parquet 
Output [1]: [Sp%C3%A9cialit%C3%A9+etab#1076200]
Batched: true
Location: InMemoryFileIndex [file:/data/input/depot/excel/cache/LISTE_UF_SUN_SU_210006_d71b6e1bc19c35b449324ff5fa82dd29/Info_Etab_init_v1.1 CHSF.xlsx_c4fec235cdf44be0f054527cfa6840ed]
ReadSchema: struct<Sp%C3%A9cialit%C3%A9+etab:string>

(13) ColumnarToRow [codegen id : 3]
Input [1]: [Sp%C3%A9cialit%C3%A9+etab#1076200]

(14) Filter [codegen id : 3]
Input [1]: [Sp%C3%A9cialit%C3%A9+etab#1076200]
Condition : isnotnull(str_replace_extend(str_transform_case(str_remove_accent(str_remove_space(Sp%C3%A9cialit%C3%A9+etab#1076200, ONLY_ONE)), UPPER), [ ], [-]))

(15) Project [codegen id : 3]
Output [1]: [str_replace_extend(str_transform_case(str_remove_accent(str_remove_space(Sp%C3%A9cialit%C3%A9+etab#1076200, ONLY_ONE)), UPPER), [ ], [-]) AS LISTE_UF_SUN_SU_210126_SPECIALITE_ETAB_210127#1076407]
Input [1]: [Sp%C3%A9cialit%C3%A9+etab#1076200]

(16) Exchange
Input [1]: [LISTE_UF_SUN_SU_210126_SPECIALITE_ETAB_210127#1076407]
Arguments: hashpartitioning(LISTE_UF_SUN_SU_210126_SPECIALITE_ETAB_210127#1076407, 10), ENSURE_REQUIREMENTS, [plan_id=650358]

(17) ShuffleQueryStage
Output [1]: [LISTE_UF_SUN_SU_210126_SPECIALITE_ETAB_210127#1076407]
Arguments: 1

(18) AQEShuffleRead
Input [1]: [LISTE_UF_SUN_SU_210126_SPECIALITE_ETAB_210127#1076407]
Arguments: coalesced

(19) Sort [codegen id : 5]
Input [1]: [LISTE_UF_SUN_SU_210126_SPECIALITE_ETAB_210127#1076407]
Arguments: [LISTE_UF_SUN_SU_210126_SPECIALITE_ETAB_210127#1076407 ASC NULLS FIRST], false, 0

(20) SortMergeJoin [codegen id : 6]
Left keys [1]: [COL_9F4D6058_F196_4B17_B4B5_2C7706DF82DC#1076355]
Right keys [1]: [LISTE_UF_SUN_SU_210126_SPECIALITE_ETAB_210127#1076407]
Join type: LeftOuter
Join condition: None

(21) Project [codegen id : 6]
Output: []
Input [2]: [COL_9F4D6058_F196_4B17_B4B5_2C7706DF82DC#1076355, LISTE_UF_SUN_SU_210126_SPECIALITE_ETAB_210127#1076407]

(22) HashAggregate [codegen id : 6]
Input: []
Keys: []
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#1076492L]
Results [1]: [count#1076493L]

(23) Exchange
Input [1]: [count#1076493L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=650451]

(24) ShuffleQueryStage
Output [1]: [count#1076493L]
Arguments: 2

(25) HashAggregate [codegen id : 7]
Input [1]: [count#1076493L]
Keys: []
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#1076489L]
Results [1]: [count(1)#1076489L AS count#1076490L]

(26) Filter
Input [2]: [Numéro Séjour#1076097, Spécialité exécutant#1076101]
Condition : (Numéro Séjour#1076097 <=> 011038404)

(27) Project
Output [1]: [str_transform_case(str_remove_accent(str_replace_extend(str_remove_space(Spécialité exécutant#1076101, ONLY_ONE), [ ], [-])), UPPER) AS COL_9F4D6058_F196_4B17_B4B5_2C7706DF82DC#1076355]
Input [2]: [Numéro Séjour#1076097, Spécialité exécutant#1076101]

(28) Filter
Input [2]: [REGLES#1076165, APPLICABLE#1076166]
Condition : ((isnotnull(REGLES#1076165) AND (RC-SUN-AUTO = REGLES#1076165)) AND (APPLICABLE#1076166 <=> O))

(29) Project
Output: []
Input [2]: [REGLES#1076165, APPLICABLE#1076166]

(30) CartesianProduct
Join type: Inner
Join condition: None

(31) Exchange
Input [1]: [COL_9F4D6058_F196_4B17_B4B5_2C7706DF82DC#1076355]
Arguments: hashpartitioning(COL_9F4D6058_F196_4B17_B4B5_2C7706DF82DC#1076355, 10), ENSURE_REQUIREMENTS, [plan_id=650284]

(32) Sort
Input [1]: [COL_9F4D6058_F196_4B17_B4B5_2C7706DF82DC#1076355]
Arguments: [COL_9F4D6058_F196_4B17_B4B5_2C7706DF82DC#1076355 ASC NULLS FIRST], false, 0

(33) Filter
Input [1]: [Sp%C3%A9cialit%C3%A9+etab#1076200]
Condition : isnotnull(str_replace_extend(str_transform_case(str_remove_accent(str_remove_space(Sp%C3%A9cialit%C3%A9+etab#1076200, ONLY_ONE)), UPPER), [ ], [-]))

(34) Project
Output [1]: [str_replace_extend(str_transform_case(str_remove_accent(str_remove_space(Sp%C3%A9cialit%C3%A9+etab#1076200, ONLY_ONE)), UPPER), [ ], [-]) AS LISTE_UF_SUN_SU_210126_SPECIALITE_ETAB_210127#1076407]
Input [1]: [Sp%C3%A9cialit%C3%A9+etab#1076200]

(35) Exchange
Input [1]: [LISTE_UF_SUN_SU_210126_SPECIALITE_ETAB_210127#1076407]
Arguments: hashpartitioning(LISTE_UF_SUN_SU_210126_SPECIALITE_ETAB_210127#1076407, 10), ENSURE_REQUIREMENTS, [plan_id=650285]

(36) Sort
Input [1]: [LISTE_UF_SUN_SU_210126_SPECIALITE_ETAB_210127#1076407]
Arguments: [LISTE_UF_SUN_SU_210126_SPECIALITE_ETAB_210127#1076407 ASC NULLS FIRST], false, 0

(37) SortMergeJoin
Left keys [1]: [COL_9F4D6058_F196_4B17_B4B5_2C7706DF82DC#1076355]
Right keys [1]: [LISTE_UF_SUN_SU_210126_SPECIALITE_ETAB_210127#1076407]
Join type: LeftOuter
Join condition: None

(38) Project
Output: []
Input [2]: [COL_9F4D6058_F196_4B17_B4B5_2C7706DF82DC#1076355, LISTE_UF_SUN_SU_210126_SPECIALITE_ETAB_210127#1076407]

(39) HashAggregate
Input: []
Keys: []
Functions [1]: [partial_count(1)]
Aggregate Attributes [1]: [count#1076492L]
Results [1]: [count#1076493L]

(40) Exchange
Input [1]: [count#1076493L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=650292]

(41) HashAggregate
Input [1]: [count#1076493L]
Keys: []
Functions [1]: [count(1)]
Aggregate Attributes [1]: [count(1)#1076489L]
Results [1]: [count(1)#1076489L AS count#1076490L]

(42) AdaptiveSparkPlan
Output [1]: [count#1076490L]
Arguments: isFinalPlan=true

SQL / DataFrame Properties

Name	Value
spark.sql.optimizer.nestedPredicatePushdown.supportedFileSources	parquet,orc,geoparquet