== Physical Plan ==
CollectLimit (20)
+- InMemoryTableScan (1)
+- InMemoryRelation (2)
+- * Sort (19)
+- Exchange (18)
+- * Project (17)
+- * BroadcastHashJoin Inner BuildLeft (16)
:- BroadcastExchange (10)
: +- * Project (9)
: +- * Filter (8)
: +- * ColumnarToRow (7)
: +- InMemoryTableScan (3)
: +- InMemoryRelation (4)
: +- * Project (6)
: +- Scan csv (5)
+- * Filter (15)
+- InMemoryTableScan (11)
+- InMemoryRelation (12)
+- * Project (14)
+- Scan csv (13)
(1) InMemoryTableScan
Output [7]: [sector_id#93981194, numcos#93981197, numdates#93981198, sort#93880530, description#93880532, universe#93981536, coverage#93981400]
Arguments: [sector_id#93981194, numcos#93981197, numdates#93981198, sort#93880530, description#93880532, universe#93981536, coverage#93981400]
(2) InMemoryRelation
Arguments: [sector_id#93981194, numcos#93981197, numdates#93981198, sort#93880530, description#93880532, universe#93981536, coverage#93981400], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@208e3fd9,StorageLevel(disk, memory, deserialized, 1 replicas),*(3) Sort [sort#93880530 ASC NULLS FIRST, description#93880532 ASC NULLS FIRST], true, 0
+- Exchange rangepartitioning(sort#93880530 ASC NULLS FIRST, description#93880532 ASC NULLS FIRST, 200), ENSURE_REQUIREMENTS, [id=#7503745]
+- *(2) Project [sector_id#93981194, numcos#93981197, numdates#93981198, sort#93880530, description#93880532, universe#93981536, coverage#93981400]
+- *(2) BroadcastHashJoin [sector_id#93981194], [sector_id#93880529], Inner, BuildLeft, false
:- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),false), [id=#7503738]
: +- *(1) Project [sector_id#93981194, numcos#93981197, numdates#93981198, coverage#93981400, round((cast(numcos#93981197 as double) / cast(coverage#93981400 as double)), 0) AS universe#93981536]
: +- *(1) Filter isnotnull(sector_id#93981194)
: +- *(1) ColumnarToRow
: +- InMemoryTableScan [coverage#93981400, numcos#93981197, numdates#93981198, sector_id#93981194], [isnotnull(sector_id#93981194)]
: +- InMemoryRelation [sector_id#93981194, retIC#93981195, resretIC#93981196, numcos#93981197, numdates#93981198, annual_bmret#93981199, annual_ret#93981200, std_ret#93981208, Sharpe_ret#93981211, PctPos_ret#93981215, TR_ret#93981220, IR_ret#93981223, annual_resret#93981227, std_resret#93981231, Sharpe_resret#93981235, PctPos_resret#93981238, TR_resret#93981240, IR_resret#93981254, annual_retnet#93981258, std_retnet#93981260, Sharpe_retnet#93981262, PctPos_retnet#93981263, TR_retnet#93981277, IR_retnet#93981385, ... 2 more fields], StorageLevel(disk, memory, deserialized, 1 replicas)
: +- *(1) Project [CASE WHEN ((sector_id#93980933 = NA) OR (sector_id#93980933 = null)) THEN null ELSE cast(sector_id#93980933 as int) END AS sector_id#93981194, CASE WHEN ((retIC#93980936 = NA) OR (retIC#93980936 = null)) THEN null ELSE cast(retIC#93980936 as float) END AS retIC#93981195, CASE WHEN ((resretIC#93980937 = NA) OR (resretIC#93980937 = null)) THEN null ELSE cast(resretIC#93980937 as float) END AS resretIC#93981196, CASE WHEN ((numcos#93980939 = NA) OR (numcos#93980939 = null)) THEN null ELSE cast(numcos#93980939 as float) END AS numcos#93981197, CASE WHEN ((numdates#93980941 = NA) OR (numdates#93980941 = null)) THEN null ELSE cast(numdates#93980941 as float) END AS numdates#93981198, CASE WHEN ((annual_bmret#93980942 = NA) OR (annual_bmret#93980942 = null)) THEN null ELSE cast(annual_bmret#93980942 as float) END AS annual_bmret#93981199, CASE WHEN ((annual_ret#93980944 = NA) OR (annual_ret#93980944 = null)) THEN null ELSE cast(annual_ret#93980944 as float) END AS annual_ret#93981200, CASE WHEN ((std_ret#93980946 = NA) OR (std_ret#93980946 = null)) THEN null ELSE cast(std_ret#93980946 as float) END AS std_ret#93981208, CASE WHEN ((Sharpe_ret#93980947 = NA) OR (Sharpe_ret#93980947 = null)) THEN null ELSE cast(Sharpe_ret#93980947 as float) END AS Sharpe_ret#93981211, CASE WHEN ((PctPos_ret#93980948 = NA) OR (PctPos_ret#93980948 = null)) THEN null ELSE cast(PctPos_ret#93980948 as float) END AS PctPos_ret#93981215, CASE WHEN ((TR_ret#93980950 = NA) OR (TR_ret#93980950 = null)) THEN null ELSE cast(TR_ret#93980950 as float) END AS TR_ret#93981220, CASE WHEN ((IR_ret#93980952 = NA) OR (IR_ret#93980952 = null)) THEN null ELSE cast(IR_ret#93980952 as float) END AS IR_ret#93981223, CASE WHEN ((annual_resret#93980954 = NA) OR (annual_resret#93980954 = null)) THEN null ELSE cast(annual_resret#93980954 as float) END AS annual_resret#93981227, CASE WHEN ((std_resret#93980956 = NA) OR (std_resret#93980956 = null)) THEN null ELSE cast(std_resret#93980956 as float) END AS std_resret#93981231, CASE WHEN ((Sharpe_resret#93980958 = NA) OR (Sharpe_resret#93980958 = null)) THEN null ELSE cast(Sharpe_resret#93980958 as float) END AS Sharpe_resret#93981235, CASE WHEN ((PctPos_resret#93980960 = NA) OR (PctPos_resret#93980960 = null)) THEN null ELSE cast(PctPos_resret#93980960 as float) END AS PctPos_resret#93981238, CASE WHEN ((TR_resret#93980962 = NA) OR (TR_resret#93980962 = null)) THEN null ELSE cast(TR_resret#93980962 as float) END AS TR_resret#93981240, CASE WHEN ((IR_resret#93980964 = NA) OR (IR_resret#93980964 = null)) THEN null ELSE cast(IR_resret#93980964 as float) END AS IR_resret#93981254, CASE WHEN ((annual_retnet#93980966 = NA) OR (annual_retnet#93980966 = null)) THEN null ELSE cast(annual_retnet#93980966 as float) END AS annual_retnet#93981258, CASE WHEN ((std_retnet#93980968 = NA) OR (std_retnet#93980968 = null)) THEN null ELSE cast(std_retnet#93980968 as float) END AS std_retnet#93981260, CASE WHEN ((Sharpe_retnet#93980969 = NA) OR (Sharpe_retnet#93980969 = null)) THEN null ELSE cast(Sharpe_retnet#93980969 as float) END AS Sharpe_retnet#93981262, CASE WHEN ((PctPos_retnet#93980971 = NA) OR (PctPos_retnet#93980971 = null)) THEN null ELSE cast(PctPos_retnet#93980971 as float) END AS PctPos_retnet#93981263, CASE WHEN ((TR_retnet#93980973 = NA) OR (TR_retnet#93980973 = null)) THEN null ELSE cast(TR_retnet#93980973 as float) END AS TR_retnet#93981277, CASE WHEN ((IR_retnet#93980975 = NA) OR (IR_retnet#93980975 = null)) THEN null ELSE cast(IR_retnet#93980975 as float) END AS IR_retnet#93981385, ... 2 more fields]
: +- FileScan csv [sector_id#93980933,retIC#93980936,resretIC#93980937,numcos#93980939,numdates#93980941,annual_bmret#93980942,annual_ret#93980944,std_ret#93980946,Sharpe_ret#93980947,PctPos_ret#93980948,TR_ret#93980950,IR_ret#93980952,annual_resret#93980954,std_resret#93980956,Sharpe_resret#93980958,PctPos_resret#93980960,TR_resret#93980962,IR_resret#93980964,annual_retnet#93980966,std_retnet#93980968,Sharpe_retnet#93980969,PctPos_retnet#93980971,TR_retnet#93980973,IR_retnet#93980975,... 2 more fields] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/srv/plusamp/data/default/ea-market/output/estimize_signal_histor..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<sector_id:string,retIC:string,resretIC:string,numcos:string,numdates:string,annual_bmret:s...
+- *(2) Filter isnotnull(sector_id#93880529)
+- InMemoryTableScan [sector_id#93880529, sort#93880530, description#93880532], [isnotnull(sector_id#93880529)]
+- InMemoryRelation [sector_id#93880529, sort#93880530, description#93880532, universe#93880534], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Project [CASE WHEN ((sector_id#93880497 = NA) OR (sector_id#93880497 = null)) THEN null ELSE cast(sector_id#93880497 as int) END AS sector_id#93880529, CASE WHEN (sort#93880499 = null) THEN null ELSE sort#93880499 END AS sort#93880530, CASE WHEN (description#93880501 = null) THEN null ELSE description#93880501 END AS description#93880532, CASE WHEN ((universe#93880503 = NA) OR (universe#93880503 = null)) THEN null ELSE cast(universe#93880503 as int) END AS universe#93880534]
+- FileScan csv [sector_id#93880497,sort#93880499,description#93880501,universe#93880503] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/srv/plusamp/data/default/ea-market/curate/curate_sector.csv], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<sector_id:string,sort:string,description:string,universe:string>
,None), [sort#93880530 ASC NULLS FIRST, description#93880532 ASC NULLS FIRST]
(3) InMemoryTableScan
Output [4]: [coverage#93981400, numcos#93981197, numdates#93981198, sector_id#93981194]
Arguments: [coverage#93981400, numcos#93981197, numdates#93981198, sector_id#93981194], [isnotnull(sector_id#93981194)]
(4) InMemoryRelation
Arguments: [sector_id#93981194, retIC#93981195, resretIC#93981196, numcos#93981197, numdates#93981198, annual_bmret#93981199, annual_ret#93981200, std_ret#93981208, Sharpe_ret#93981211, PctPos_ret#93981215, TR_ret#93981220, IR_ret#93981223, annual_resret#93981227, std_resret#93981231, Sharpe_resret#93981235, PctPos_resret#93981238, TR_resret#93981240, IR_resret#93981254, annual_retnet#93981258, std_retnet#93981260, Sharpe_retnet#93981262, PctPos_retnet#93981263, TR_retnet#93981277, IR_retnet#93981385, ... 2 more fields], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@208e3fd9,StorageLevel(disk, memory, deserialized, 1 replicas),*(1) Project [CASE WHEN ((sector_id#93980933 = NA) OR (sector_id#93980933 = null)) THEN null ELSE cast(sector_id#93980933 as int) END AS sector_id#93981194, CASE WHEN ((retIC#93980936 = NA) OR (retIC#93980936 = null)) THEN null ELSE cast(retIC#93980936 as float) END AS retIC#93981195, CASE WHEN ((resretIC#93980937 = NA) OR (resretIC#93980937 = null)) THEN null ELSE cast(resretIC#93980937 as float) END AS resretIC#93981196, CASE WHEN ((numcos#93980939 = NA) OR (numcos#93980939 = null)) THEN null ELSE cast(numcos#93980939 as float) END AS numcos#93981197, CASE WHEN ((numdates#93980941 = NA) OR (numdates#93980941 = null)) THEN null ELSE cast(numdates#93980941 as float) END AS numdates#93981198, CASE WHEN ((annual_bmret#93980942 = NA) OR (annual_bmret#93980942 = null)) THEN null ELSE cast(annual_bmret#93980942 as float) END AS annual_bmret#93981199, CASE WHEN ((annual_ret#93980944 = NA) OR (annual_ret#93980944 = null)) THEN null ELSE cast(annual_ret#93980944 as float) END AS annual_ret#93981200, CASE WHEN ((std_ret#93980946 = NA) OR (std_ret#93980946 = null)) THEN null ELSE cast(std_ret#93980946 as float) END AS std_ret#93981208, CASE WHEN ((Sharpe_ret#93980947 = NA) OR (Sharpe_ret#93980947 = null)) THEN null ELSE cast(Sharpe_ret#93980947 as float) END AS Sharpe_ret#93981211, CASE WHEN ((PctPos_ret#93980948 = NA) OR (PctPos_ret#93980948 = null)) THEN null ELSE cast(PctPos_ret#93980948 as float) END AS PctPos_ret#93981215, CASE WHEN ((TR_ret#93980950 = NA) OR (TR_ret#93980950 = null)) THEN null ELSE cast(TR_ret#93980950 as float) END AS TR_ret#93981220, CASE WHEN ((IR_ret#93980952 = NA) OR (IR_ret#93980952 = null)) THEN null ELSE cast(IR_ret#93980952 as float) END AS IR_ret#93981223, CASE WHEN ((annual_resret#93980954 = NA) OR (annual_resret#93980954 = null)) THEN null ELSE cast(annual_resret#93980954 as float) END AS annual_resret#93981227, CASE WHEN ((std_resret#93980956 = NA) OR (std_resret#93980956 = null)) THEN null ELSE cast(std_resret#93980956 as float) END AS std_resret#93981231, CASE WHEN ((Sharpe_resret#93980958 = NA) OR (Sharpe_resret#93980958 = null)) THEN null ELSE cast(Sharpe_resret#93980958 as float) END AS Sharpe_resret#93981235, CASE WHEN ((PctPos_resret#93980960 = NA) OR (PctPos_resret#93980960 = null)) THEN null ELSE cast(PctPos_resret#93980960 as float) END AS PctPos_resret#93981238, CASE WHEN ((TR_resret#93980962 = NA) OR (TR_resret#93980962 = null)) THEN null ELSE cast(TR_resret#93980962 as float) END AS TR_resret#93981240, CASE WHEN ((IR_resret#93980964 = NA) OR (IR_resret#93980964 = null)) THEN null ELSE cast(IR_resret#93980964 as float) END AS IR_resret#93981254, CASE WHEN ((annual_retnet#93980966 = NA) OR (annual_retnet#93980966 = null)) THEN null ELSE cast(annual_retnet#93980966 as float) END AS annual_retnet#93981258, CASE WHEN ((std_retnet#93980968 = NA) OR (std_retnet#93980968 = null)) THEN null ELSE cast(std_retnet#93980968 as float) END AS std_retnet#93981260, CASE WHEN ((Sharpe_retnet#93980969 = NA) OR (Sharpe_retnet#93980969 = null)) THEN null ELSE cast(Sharpe_retnet#93980969 as float) END AS Sharpe_retnet#93981262, CASE WHEN ((PctPos_retnet#93980971 = NA) OR (PctPos_retnet#93980971 = null)) THEN null ELSE cast(PctPos_retnet#93980971 as float) END AS PctPos_retnet#93981263, CASE WHEN ((TR_retnet#93980973 = NA) OR (TR_retnet#93980973 = null)) THEN null ELSE cast(TR_retnet#93980973 as float) END AS TR_retnet#93981277, CASE WHEN ((IR_retnet#93980975 = NA) OR (IR_retnet#93980975 = null)) THEN null ELSE cast(IR_retnet#93980975 as float) END AS IR_retnet#93981385, ... 2 more fields]
+- FileScan csv [sector_id#93980933,retIC#93980936,resretIC#93980937,numcos#93980939,numdates#93980941,annual_bmret#93980942,annual_ret#93980944,std_ret#93980946,Sharpe_ret#93980947,PctPos_ret#93980948,TR_ret#93980950,IR_ret#93980952,annual_resret#93980954,std_resret#93980956,Sharpe_resret#93980958,PctPos_resret#93980960,TR_resret#93980962,IR_resret#93980964,annual_retnet#93980966,std_retnet#93980968,Sharpe_retnet#93980969,PctPos_retnet#93980971,TR_retnet#93980973,IR_retnet#93980975,... 2 more fields] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/srv/plusamp/data/default/ea-market/output/estimize_signal_histor..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<sector_id:string,retIC:string,resretIC:string,numcos:string,numdates:string,annual_bmret:s...
,None)
(5) Scan csv
Output [26]: [sector_id#93980933, retIC#93980936, resretIC#93980937, numcos#93980939, numdates#93980941, annual_bmret#93980942, annual_ret#93980944, std_ret#93980946, Sharpe_ret#93980947, PctPos_ret#93980948, TR_ret#93980950, IR_ret#93980952, annual_resret#93980954, std_resret#93980956, Sharpe_resret#93980958, PctPos_resret#93980960, TR_resret#93980962, IR_resret#93980964, annual_retnet#93980966, std_retnet#93980968, Sharpe_retnet#93980969, PctPos_retnet#93980971, TR_retnet#93980973, IR_retnet#93980975, turnover#93980977, coverage#93980979]
Batched: false
Location: InMemoryFileIndex [file:/srv/plusamp/data/default/ea-market/output/estimize_signal_history/estimizesignal_postearnings/stats_sector_id.csv]
ReadSchema: struct<sector_id:string,retIC:string,resretIC:string,numcos:string,numdates:string,annual_bmret:string,annual_ret:string,std_ret:string,Sharpe_ret:string,PctPos_ret:string,TR_ret:string,IR_ret:string,annual_resret:string,std_resret:string,Sharpe_resret:string,PctPos_resret:string,TR_resret:string,IR_resret:string,annual_retnet:string,std_retnet:string,Sharpe_retnet:string,PctPos_retnet:string,TR_retnet:string,IR_retnet:string,turnover:string,coverage:string>
(6) Project [codegen id : 1]
Output [26]: [CASE WHEN ((sector_id#93980933 = NA) OR (sector_id#93980933 = null)) THEN null ELSE cast(sector_id#93980933 as int) END AS sector_id#93981194, CASE WHEN ((retIC#93980936 = NA) OR (retIC#93980936 = null)) THEN null ELSE cast(retIC#93980936 as float) END AS retIC#93981195, CASE WHEN ((resretIC#93980937 = NA) OR (resretIC#93980937 = null)) THEN null ELSE cast(resretIC#93980937 as float) END AS resretIC#93981196, CASE WHEN ((numcos#93980939 = NA) OR (numcos#93980939 = null)) THEN null ELSE cast(numcos#93980939 as float) END AS numcos#93981197, CASE WHEN ((numdates#93980941 = NA) OR (numdates#93980941 = null)) THEN null ELSE cast(numdates#93980941 as float) END AS numdates#93981198, CASE WHEN ((annual_bmret#93980942 = NA) OR (annual_bmret#93980942 = null)) THEN null ELSE cast(annual_bmret#93980942 as float) END AS annual_bmret#93981199, CASE WHEN ((annual_ret#93980944 = NA) OR (annual_ret#93980944 = null)) THEN null ELSE cast(annual_ret#93980944 as float) END AS annual_ret#93981200, CASE WHEN ((std_ret#93980946 = NA) OR (std_ret#93980946 = null)) THEN null ELSE cast(std_ret#93980946 as float) END AS std_ret#93981208, CASE WHEN ((Sharpe_ret#93980947 = NA) OR (Sharpe_ret#93980947 = null)) THEN null ELSE cast(Sharpe_ret#93980947 as float) END AS Sharpe_ret#93981211, CASE WHEN ((PctPos_ret#93980948 = NA) OR (PctPos_ret#93980948 = null)) THEN null ELSE cast(PctPos_ret#93980948 as float) END AS PctPos_ret#93981215, CASE WHEN ((TR_ret#93980950 = NA) OR (TR_ret#93980950 = null)) THEN null ELSE cast(TR_ret#93980950 as float) END AS TR_ret#93981220, CASE WHEN ((IR_ret#93980952 = NA) OR (IR_ret#93980952 = null)) THEN null ELSE cast(IR_ret#93980952 as float) END AS IR_ret#93981223, CASE WHEN ((annual_resret#93980954 = NA) OR (annual_resret#93980954 = null)) THEN null ELSE cast(annual_resret#93980954 as float) END AS annual_resret#93981227, CASE WHEN ((std_resret#93980956 = NA) OR (std_resret#93980956 = null)) THEN null ELSE cast(std_resret#93980956 as float) END AS std_resret#93981231, CASE WHEN ((Sharpe_resret#93980958 = NA) OR (Sharpe_resret#93980958 = null)) THEN null ELSE cast(Sharpe_resret#93980958 as float) END AS Sharpe_resret#93981235, CASE WHEN ((PctPos_resret#93980960 = NA) OR (PctPos_resret#93980960 = null)) THEN null ELSE cast(PctPos_resret#93980960 as float) END AS PctPos_resret#93981238, CASE WHEN ((TR_resret#93980962 = NA) OR (TR_resret#93980962 = null)) THEN null ELSE cast(TR_resret#93980962 as float) END AS TR_resret#93981240, CASE WHEN ((IR_resret#93980964 = NA) OR (IR_resret#93980964 = null)) THEN null ELSE cast(IR_resret#93980964 as float) END AS IR_resret#93981254, CASE WHEN ((annual_retnet#93980966 = NA) OR (annual_retnet#93980966 = null)) THEN null ELSE cast(annual_retnet#93980966 as float) END AS annual_retnet#93981258, CASE WHEN ((std_retnet#93980968 = NA) OR (std_retnet#93980968 = null)) THEN null ELSE cast(std_retnet#93980968 as float) END AS std_retnet#93981260, CASE WHEN ((Sharpe_retnet#93980969 = NA) OR (Sharpe_retnet#93980969 = null)) THEN null ELSE cast(Sharpe_retnet#93980969 as float) END AS Sharpe_retnet#93981262, CASE WHEN ((PctPos_retnet#93980971 = NA) OR (PctPos_retnet#93980971 = null)) THEN null ELSE cast(PctPos_retnet#93980971 as float) END AS PctPos_retnet#93981263, CASE WHEN ((TR_retnet#93980973 = NA) OR (TR_retnet#93980973 = null)) THEN null ELSE cast(TR_retnet#93980973 as float) END AS TR_retnet#93981277, CASE WHEN ((IR_retnet#93980975 = NA) OR (IR_retnet#93980975 = null)) THEN null ELSE cast(IR_retnet#93980975 as float) END AS IR_retnet#93981385, CASE WHEN ((turnover#93980977 = NA) OR (turnover#93980977 = null)) THEN null ELSE cast(turnover#93980977 as float) END AS turnover#93981399, CASE WHEN ((coverage#93980979 = NA) OR (coverage#93980979 = null)) THEN null ELSE cast(coverage#93980979 as float) END AS coverage#93981400]
Input [26]: [sector_id#93980933, retIC#93980936, resretIC#93980937, numcos#93980939, numdates#93980941, annual_bmret#93980942, annual_ret#93980944, std_ret#93980946, Sharpe_ret#93980947, PctPos_ret#93980948, TR_ret#93980950, IR_ret#93980952, annual_resret#93980954, std_resret#93980956, Sharpe_resret#93980958, PctPos_resret#93980960, TR_resret#93980962, IR_resret#93980964, annual_retnet#93980966, std_retnet#93980968, Sharpe_retnet#93980969, PctPos_retnet#93980971, TR_retnet#93980973, IR_retnet#93980975, turnover#93980977, coverage#93980979]
(7) ColumnarToRow [codegen id : 1]
Input [4]: [coverage#93981400, numcos#93981197, numdates#93981198, sector_id#93981194]
(8) Filter [codegen id : 1]
Input [4]: [coverage#93981400, numcos#93981197, numdates#93981198, sector_id#93981194]
Condition : isnotnull(sector_id#93981194)
(9) Project [codegen id : 1]
Output [5]: [sector_id#93981194, numcos#93981197, numdates#93981198, coverage#93981400, round((cast(numcos#93981197 as double) / cast(coverage#93981400 as double)), 0) AS universe#93981536]
Input [4]: [coverage#93981400, numcos#93981197, numdates#93981198, sector_id#93981194]
(10) BroadcastExchange
Input [5]: [sector_id#93981194, numcos#93981197, numdates#93981198, coverage#93981400, universe#93981536]
Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),false), [id=#7503738]
(11) InMemoryTableScan
Output [3]: [sector_id#93880529, sort#93880530, description#93880532]
Arguments: [sector_id#93880529, sort#93880530, description#93880532], [isnotnull(sector_id#93880529)]
(12) InMemoryRelation
Arguments: [sector_id#93880529, sort#93880530, description#93880532, universe#93880534], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@208e3fd9,StorageLevel(disk, memory, deserialized, 1 replicas),*(1) Project [CASE WHEN ((sector_id#93880497 = NA) OR (sector_id#93880497 = null)) THEN null ELSE cast(sector_id#93880497 as int) END AS sector_id#93880529, CASE WHEN (sort#93880499 = null) THEN null ELSE sort#93880499 END AS sort#93880530, CASE WHEN (description#93880501 = null) THEN null ELSE description#93880501 END AS description#93880532, CASE WHEN ((universe#93880503 = NA) OR (universe#93880503 = null)) THEN null ELSE cast(universe#93880503 as int) END AS universe#93880534]
+- FileScan csv [sector_id#93880497,sort#93880499,description#93880501,universe#93880503] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/srv/plusamp/data/default/ea-market/curate/curate_sector.csv], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<sector_id:string,sort:string,description:string,universe:string>
,None)
(13) Scan csv
Output [4]: [sector_id#93880497, sort#93880499, description#93880501, universe#93880503]
Batched: false
Location: InMemoryFileIndex [file:/srv/plusamp/data/default/ea-market/curate/curate_sector.csv]
ReadSchema: struct<sector_id:string,sort:string,description:string,universe:string>
(14) Project [codegen id : 1]
Output [4]: [CASE WHEN ((sector_id#93880497 = NA) OR (sector_id#93880497 = null)) THEN null ELSE cast(sector_id#93880497 as int) END AS sector_id#93880529, CASE WHEN (sort#93880499 = null) THEN null ELSE sort#93880499 END AS sort#93880530, CASE WHEN (description#93880501 = null) THEN null ELSE description#93880501 END AS description#93880532, CASE WHEN ((universe#93880503 = NA) OR (universe#93880503 = null)) THEN null ELSE cast(universe#93880503 as int) END AS universe#93880534]
Input [4]: [sector_id#93880497, sort#93880499, description#93880501, universe#93880503]
(15) Filter
Input [3]: [sector_id#93880529, sort#93880530, description#93880532]
Condition : isnotnull(sector_id#93880529)
(16) BroadcastHashJoin [codegen id : 2]
Left keys [1]: [sector_id#93981194]
Right keys [1]: [sector_id#93880529]
Join condition: None
(17) Project [codegen id : 2]
Output [7]: [sector_id#93981194, numcos#93981197, numdates#93981198, sort#93880530, description#93880532, universe#93981536, coverage#93981400]
Input [8]: [sector_id#93981194, numcos#93981197, numdates#93981198, coverage#93981400, universe#93981536, sector_id#93880529, sort#93880530, description#93880532]
(18) Exchange
Input [7]: [sector_id#93981194, numcos#93981197, numdates#93981198, sort#93880530, description#93880532, universe#93981536, coverage#93981400]
Arguments: rangepartitioning(sort#93880530 ASC NULLS FIRST, description#93880532 ASC NULLS FIRST, 200), ENSURE_REQUIREMENTS, [id=#7503745]
(19) Sort [codegen id : 3]
Input [7]: [sector_id#93981194, numcos#93981197, numdates#93981198, sort#93880530, description#93880532, universe#93981536, coverage#93981400]
Arguments: [sort#93880530 ASC NULLS FIRST, description#93880532 ASC NULLS FIRST], true, 0
(20) CollectLimit
Input [7]: [sector_id#93981194, numcos#93981197, numdates#93981198, sort#93880530, description#93880532, universe#93981536, coverage#93981400]
Arguments: 1000000