Project [CASE WHEN (date#94010336 = null) THEN null ELSE cast(date#94010336 as date) END AS date#94010641, CASE WHEN ((overall#94010337 = NA) OR (overall#94010337 = null)) THEN null ELSE cast(overall#94010337 as int) END AS overall#94010642, CASE WHEN ((ret#94010338 = NA) OR (ret#94010338 = null)) THEN null ELSE cast(ret#94010338 as float) END AS ret#94010647, CASE WHEN ((resret#94010339 = NA) OR (resret#94010339 = null)) THEN null ELSE cast(resret#94010339 as float) END AS resret#94010651, CASE WHEN ((retnet#94010340 = NA) OR (retnet#94010340 = null)) THEN null ELSE cast(retnet#94010340 as float) END AS retnet#94010655, CASE WHEN ((turnover#94010341 = NA) OR (turnover#94010341 = null)) THEN null ELSE cast(turnover#94010341 as float) END AS turnover#94010658, CASE WHEN ((numcos#94010342 = NA) OR (numcos#94010342 = null)) THEN null ELSE cast(numcos#94010342 as float) END AS numcos#94010663, CASE WHEN (benchmark#94010343 = null) THEN null ELSE benchmark#94010343 END AS benchmark#94010665, CASE WHEN (excess_ret#94010344 = null) THEN null ELSE excess_ret#94010344 END AS excess_ret#94010668, CASE WHEN (excess_resret#94010345 = null) THEN null ELSE excess_resret#94010345 END AS excess_resret#94010672, CASE WHEN (excess_retnet#94010346 = null) THEN null ELSE excess_retnet#94010346 END AS excess_retnet#94010675]
Filter (isnotnull(cap#94010646) AND (cast(cap#94010646 as string) = 0))
WholeStageCodegen (1)
InMemoryTableScan [cap#94010646, date#94010643, numcos#94010669], [isnotnull(cap#94010646), (cast(cap#94010646 as string) = 0)]
Project [CASE WHEN (date#94010359 = null) THEN null ELSE cast(date#94010359 as date) END AS date#94010643, CASE WHEN ((cap#94010360 = NA) OR (cap#94010360 = null)) THEN null ELSE cast(cap#94010360 as float) END AS cap#94010646, CASE WHEN ((ret#94010361 = NA) OR (ret#94010361 = null)) THEN null ELSE cast(ret#94010361 as float) END AS ret#94010650, CASE WHEN ((resret#94010362 = NA) OR (resret#94010362 = null)) THEN null ELSE cast(resret#94010362 as float) END AS resret#94010656, CASE WHEN ((retnet#94010363 = NA) OR (retnet#94010363 = null)) THEN null ELSE cast(retnet#94010363 as float) END AS retnet#94010660, CASE WHEN ((turnover#94010364 = NA) OR (turnover#94010364 = null)) THEN null ELSE cast(turnover#94010364 as float) END AS turnover#94010664, CASE WHEN ((numcos#94010365 = NA) OR (numcos#94010365 = null)) THEN null ELSE cast(numcos#94010365 as float) END AS numcos#94010669, CASE WHEN ((coverage#94010366 = NA) OR (coverage#94010366 = null)) THEN null ELSE cast(coverage#94010366 as float) END AS coverage#94010673, CASE WHEN (benchmark#94010367 = null) THEN null ELSE benchmark#94010367 END AS benchmark#94010677, CASE WHEN (excess_ret#94010368 = null) THEN null ELSE excess_ret#94010368 END AS excess_ret#94010681, CASE WHEN (excess_resret#94010369 = null) THEN null ELSE excess_resret#94010369 END AS excess_resret#94010683, CASE WHEN (excess_retnet#94010370 = null) THEN null ELSE excess_retnet#94010370 END AS excess_retnet#94010698]
== Physical Plan ==
CollectLimit (16)
+- InMemoryTableScan (1)
+- InMemoryRelation (2)
+- * Sort (15)
+- Exchange (14)
+- Union (13)
:- InMemoryTableScan (3)
: +- InMemoryRelation (4)
: +- * Project (6)
: +- Scan csv (5)
+- * Project (12)
+- * Filter (11)
+- InMemoryTableScan (7)
+- InMemoryRelation (8)
+- * Project (10)
+- Scan csv (9)
(1) InMemoryTableScan
Output [2]: [date#94010641, numcos#94010663]
Arguments: [date#94010641, numcos#94010663]
(2) InMemoryRelation
Arguments: [date#94010641, numcos#94010663], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@208e3fd9,StorageLevel(disk, memory, deserialized, 1 replicas),*(2) Sort [date#94010641 ASC NULLS FIRST], true, 0
+- Exchange rangepartitioning(date#94010641 ASC NULLS FIRST, 200), ENSURE_REQUIREMENTS, [id=#7506150]
+- Union
:- InMemoryTableScan [date#94010641, numcos#94010663]
: +- InMemoryRelation [date#94010641, overall#94010642, ret#94010647, resret#94010651, retnet#94010655, turnover#94010658, numcos#94010663, benchmark#94010665, excess_ret#94010668, excess_resret#94010672, excess_retnet#94010675], StorageLevel(disk, memory, deserialized, 1 replicas)
: +- *(1) Project [CASE WHEN (date#94010336 = null) THEN null ELSE cast(date#94010336 as date) END AS date#94010641, CASE WHEN ((overall#94010337 = NA) OR (overall#94010337 = null)) THEN null ELSE cast(overall#94010337 as int) END AS overall#94010642, CASE WHEN ((ret#94010338 = NA) OR (ret#94010338 = null)) THEN null ELSE cast(ret#94010338 as float) END AS ret#94010647, CASE WHEN ((resret#94010339 = NA) OR (resret#94010339 = null)) THEN null ELSE cast(resret#94010339 as float) END AS resret#94010651, CASE WHEN ((retnet#94010340 = NA) OR (retnet#94010340 = null)) THEN null ELSE cast(retnet#94010340 as float) END AS retnet#94010655, CASE WHEN ((turnover#94010341 = NA) OR (turnover#94010341 = null)) THEN null ELSE cast(turnover#94010341 as float) END AS turnover#94010658, CASE WHEN ((numcos#94010342 = NA) OR (numcos#94010342 = null)) THEN null ELSE cast(numcos#94010342 as float) END AS numcos#94010663, CASE WHEN (benchmark#94010343 = null) THEN null ELSE benchmark#94010343 END AS benchmark#94010665, CASE WHEN (excess_ret#94010344 = null) THEN null ELSE excess_ret#94010344 END AS excess_ret#94010668, CASE WHEN (excess_resret#94010345 = null) THEN null ELSE excess_resret#94010345 END AS excess_resret#94010672, CASE WHEN (excess_retnet#94010346 = null) THEN null ELSE excess_retnet#94010346 END AS excess_retnet#94010675]
: +- FileScan csv [date#94010336,overall#94010337,ret#94010338,resret#94010339,retnet#94010340,turnover#94010341,numcos#94010342,benchmark#94010343,excess_ret#94010344,excess_resret#94010345,excess_retnet#94010346] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/srv/plusamp/data/default/ea-market/output/rankviews_history/rank..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<date:string,overall:string,ret:string,resret:string,retnet:string,turnover:string,numcos:s...
+- *(1) Project [date#94010643, numcos#94010669]
+- *(1) Filter (isnotnull(cap#94010646) AND (cast(cap#94010646 as string) = 0))
+- InMemoryTableScan [cap#94010646, date#94010643, numcos#94010669], [isnotnull(cap#94010646), (cast(cap#94010646 as string) = 0)]
+- InMemoryRelation [date#94010643, cap#94010646, ret#94010650, resret#94010656, retnet#94010660, turnover#94010664, numcos#94010669, coverage#94010673, benchmark#94010677, excess_ret#94010681, excess_resret#94010683, excess_retnet#94010698], StorageLevel(disk, memory, deserialized, 1 replicas)
+- *(1) Project [CASE WHEN (date#94010359 = null) THEN null ELSE cast(date#94010359 as date) END AS date#94010643, CASE WHEN ((cap#94010360 = NA) OR (cap#94010360 = null)) THEN null ELSE cast(cap#94010360 as float) END AS cap#94010646, CASE WHEN ((ret#94010361 = NA) OR (ret#94010361 = null)) THEN null ELSE cast(ret#94010361 as float) END AS ret#94010650, CASE WHEN ((resret#94010362 = NA) OR (resret#94010362 = null)) THEN null ELSE cast(resret#94010362 as float) END AS resret#94010656, CASE WHEN ((retnet#94010363 = NA) OR (retnet#94010363 = null)) THEN null ELSE cast(retnet#94010363 as float) END AS retnet#94010660, CASE WHEN ((turnover#94010364 = NA) OR (turnover#94010364 = null)) THEN null ELSE cast(turnover#94010364 as float) END AS turnover#94010664, CASE WHEN ((numcos#94010365 = NA) OR (numcos#94010365 = null)) THEN null ELSE cast(numcos#94010365 as float) END AS numcos#94010669, CASE WHEN ((coverage#94010366 = NA) OR (coverage#94010366 = null)) THEN null ELSE cast(coverage#94010366 as float) END AS coverage#94010673, CASE WHEN (benchmark#94010367 = null) THEN null ELSE benchmark#94010367 END AS benchmark#94010677, CASE WHEN (excess_ret#94010368 = null) THEN null ELSE excess_ret#94010368 END AS excess_ret#94010681, CASE WHEN (excess_resret#94010369 = null) THEN null ELSE excess_resret#94010369 END AS excess_resret#94010683, CASE WHEN (excess_retnet#94010370 = null) THEN null ELSE excess_retnet#94010370 END AS excess_retnet#94010698]
+- FileScan csv [date#94010359,cap#94010360,ret#94010361,resret#94010362,retnet#94010363,turnover#94010364,numcos#94010365,coverage#94010366,benchmark#94010367,excess_ret#94010368,excess_resret#94010369,excess_retnet#94010370] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/srv/plusamp/data/default/ea-market/output/rankviews_history/rank..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<date:string,cap:string,ret:string,resret:string,retnet:string,turnover:string,numcos:strin...
,None), [date#94010641 ASC NULLS FIRST]
(3) InMemoryTableScan
Output [2]: [date#94010641, numcos#94010663]
Arguments: [date#94010641, numcos#94010663]
(4) InMemoryRelation
Arguments: [date#94010641, overall#94010642, ret#94010647, resret#94010651, retnet#94010655, turnover#94010658, numcos#94010663, benchmark#94010665, excess_ret#94010668, excess_resret#94010672, excess_retnet#94010675], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@208e3fd9,StorageLevel(disk, memory, deserialized, 1 replicas),*(1) Project [CASE WHEN (date#94010336 = null) THEN null ELSE cast(date#94010336 as date) END AS date#94010641, CASE WHEN ((overall#94010337 = NA) OR (overall#94010337 = null)) THEN null ELSE cast(overall#94010337 as int) END AS overall#94010642, CASE WHEN ((ret#94010338 = NA) OR (ret#94010338 = null)) THEN null ELSE cast(ret#94010338 as float) END AS ret#94010647, CASE WHEN ((resret#94010339 = NA) OR (resret#94010339 = null)) THEN null ELSE cast(resret#94010339 as float) END AS resret#94010651, CASE WHEN ((retnet#94010340 = NA) OR (retnet#94010340 = null)) THEN null ELSE cast(retnet#94010340 as float) END AS retnet#94010655, CASE WHEN ((turnover#94010341 = NA) OR (turnover#94010341 = null)) THEN null ELSE cast(turnover#94010341 as float) END AS turnover#94010658, CASE WHEN ((numcos#94010342 = NA) OR (numcos#94010342 = null)) THEN null ELSE cast(numcos#94010342 as float) END AS numcos#94010663, CASE WHEN (benchmark#94010343 = null) THEN null ELSE benchmark#94010343 END AS benchmark#94010665, CASE WHEN (excess_ret#94010344 = null) THEN null ELSE excess_ret#94010344 END AS excess_ret#94010668, CASE WHEN (excess_resret#94010345 = null) THEN null ELSE excess_resret#94010345 END AS excess_resret#94010672, CASE WHEN (excess_retnet#94010346 = null) THEN null ELSE excess_retnet#94010346 END AS excess_retnet#94010675]
+- FileScan csv [date#94010336,overall#94010337,ret#94010338,resret#94010339,retnet#94010340,turnover#94010341,numcos#94010342,benchmark#94010343,excess_ret#94010344,excess_resret#94010345,excess_retnet#94010346] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/srv/plusamp/data/default/ea-market/output/rankviews_history/rank..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<date:string,overall:string,ret:string,resret:string,retnet:string,turnover:string,numcos:s...
,None)
(5) Scan csv
Output [11]: [date#94010336, overall#94010337, ret#94010338, resret#94010339, retnet#94010340, turnover#94010341, numcos#94010342, benchmark#94010343, excess_ret#94010344, excess_resret#94010345, excess_retnet#94010346]
Batched: false
Location: InMemoryFileIndex [file:/srv/plusamp/data/default/ea-market/output/rankviews_history/rankviews/longshort_overall.csv]
ReadSchema: struct<date:string,overall:string,ret:string,resret:string,retnet:string,turnover:string,numcos:string,benchmark:string,excess_ret:string,excess_resret:string,excess_retnet:string>
(6) Project [codegen id : 1]
Output [11]: [CASE WHEN (date#94010336 = null) THEN null ELSE cast(date#94010336 as date) END AS date#94010641, CASE WHEN ((overall#94010337 = NA) OR (overall#94010337 = null)) THEN null ELSE cast(overall#94010337 as int) END AS overall#94010642, CASE WHEN ((ret#94010338 = NA) OR (ret#94010338 = null)) THEN null ELSE cast(ret#94010338 as float) END AS ret#94010647, CASE WHEN ((resret#94010339 = NA) OR (resret#94010339 = null)) THEN null ELSE cast(resret#94010339 as float) END AS resret#94010651, CASE WHEN ((retnet#94010340 = NA) OR (retnet#94010340 = null)) THEN null ELSE cast(retnet#94010340 as float) END AS retnet#94010655, CASE WHEN ((turnover#94010341 = NA) OR (turnover#94010341 = null)) THEN null ELSE cast(turnover#94010341 as float) END AS turnover#94010658, CASE WHEN ((numcos#94010342 = NA) OR (numcos#94010342 = null)) THEN null ELSE cast(numcos#94010342 as float) END AS numcos#94010663, CASE WHEN (benchmark#94010343 = null) THEN null ELSE benchmark#94010343 END AS benchmark#94010665, CASE WHEN (excess_ret#94010344 = null) THEN null ELSE excess_ret#94010344 END AS excess_ret#94010668, CASE WHEN (excess_resret#94010345 = null) THEN null ELSE excess_resret#94010345 END AS excess_resret#94010672, CASE WHEN (excess_retnet#94010346 = null) THEN null ELSE excess_retnet#94010346 END AS excess_retnet#94010675]
Input [11]: [date#94010336, overall#94010337, ret#94010338, resret#94010339, retnet#94010340, turnover#94010341, numcos#94010342, benchmark#94010343, excess_ret#94010344, excess_resret#94010345, excess_retnet#94010346]
(7) InMemoryTableScan
Output [3]: [cap#94010646, date#94010643, numcos#94010669]
Arguments: [cap#94010646, date#94010643, numcos#94010669], [isnotnull(cap#94010646), (cast(cap#94010646 as string) = 0)]
(8) InMemoryRelation
Arguments: [date#94010643, cap#94010646, ret#94010650, resret#94010656, retnet#94010660, turnover#94010664, numcos#94010669, coverage#94010673, benchmark#94010677, excess_ret#94010681, excess_resret#94010683, excess_retnet#94010698], CachedRDDBuilder(org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer@208e3fd9,StorageLevel(disk, memory, deserialized, 1 replicas),*(1) Project [CASE WHEN (date#94010359 = null) THEN null ELSE cast(date#94010359 as date) END AS date#94010643, CASE WHEN ((cap#94010360 = NA) OR (cap#94010360 = null)) THEN null ELSE cast(cap#94010360 as float) END AS cap#94010646, CASE WHEN ((ret#94010361 = NA) OR (ret#94010361 = null)) THEN null ELSE cast(ret#94010361 as float) END AS ret#94010650, CASE WHEN ((resret#94010362 = NA) OR (resret#94010362 = null)) THEN null ELSE cast(resret#94010362 as float) END AS resret#94010656, CASE WHEN ((retnet#94010363 = NA) OR (retnet#94010363 = null)) THEN null ELSE cast(retnet#94010363 as float) END AS retnet#94010660, CASE WHEN ((turnover#94010364 = NA) OR (turnover#94010364 = null)) THEN null ELSE cast(turnover#94010364 as float) END AS turnover#94010664, CASE WHEN ((numcos#94010365 = NA) OR (numcos#94010365 = null)) THEN null ELSE cast(numcos#94010365 as float) END AS numcos#94010669, CASE WHEN ((coverage#94010366 = NA) OR (coverage#94010366 = null)) THEN null ELSE cast(coverage#94010366 as float) END AS coverage#94010673, CASE WHEN (benchmark#94010367 = null) THEN null ELSE benchmark#94010367 END AS benchmark#94010677, CASE WHEN (excess_ret#94010368 = null) THEN null ELSE excess_ret#94010368 END AS excess_ret#94010681, CASE WHEN (excess_resret#94010369 = null) THEN null ELSE excess_resret#94010369 END AS excess_resret#94010683, CASE WHEN (excess_retnet#94010370 = null) THEN null ELSE excess_retnet#94010370 END AS excess_retnet#94010698]
+- FileScan csv [date#94010359,cap#94010360,ret#94010361,resret#94010362,retnet#94010363,turnover#94010364,numcos#94010365,coverage#94010366,benchmark#94010367,excess_ret#94010368,excess_resret#94010369,excess_retnet#94010370] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/srv/plusamp/data/default/ea-market/output/rankviews_history/rank..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<date:string,cap:string,ret:string,resret:string,retnet:string,turnover:string,numcos:strin...
,None)
(9) Scan csv
Output [12]: [date#94010359, cap#94010360, ret#94010361, resret#94010362, retnet#94010363, turnover#94010364, numcos#94010365, coverage#94010366, benchmark#94010367, excess_ret#94010368, excess_resret#94010369, excess_retnet#94010370]
Batched: false
Location: InMemoryFileIndex [file:/srv/plusamp/data/default/ea-market/output/rankviews_history/rankviews/longshort_cap.csv]
ReadSchema: struct<date:string,cap:string,ret:string,resret:string,retnet:string,turnover:string,numcos:string,coverage:string,benchmark:string,excess_ret:string,excess_resret:string,excess_retnet:string>
(10) Project [codegen id : 1]
Output [12]: [CASE WHEN (date#94010359 = null) THEN null ELSE cast(date#94010359 as date) END AS date#94010643, CASE WHEN ((cap#94010360 = NA) OR (cap#94010360 = null)) THEN null ELSE cast(cap#94010360 as float) END AS cap#94010646, CASE WHEN ((ret#94010361 = NA) OR (ret#94010361 = null)) THEN null ELSE cast(ret#94010361 as float) END AS ret#94010650, CASE WHEN ((resret#94010362 = NA) OR (resret#94010362 = null)) THEN null ELSE cast(resret#94010362 as float) END AS resret#94010656, CASE WHEN ((retnet#94010363 = NA) OR (retnet#94010363 = null)) THEN null ELSE cast(retnet#94010363 as float) END AS retnet#94010660, CASE WHEN ((turnover#94010364 = NA) OR (turnover#94010364 = null)) THEN null ELSE cast(turnover#94010364 as float) END AS turnover#94010664, CASE WHEN ((numcos#94010365 = NA) OR (numcos#94010365 = null)) THEN null ELSE cast(numcos#94010365 as float) END AS numcos#94010669, CASE WHEN ((coverage#94010366 = NA) OR (coverage#94010366 = null)) THEN null ELSE cast(coverage#94010366 as float) END AS coverage#94010673, CASE WHEN (benchmark#94010367 = null) THEN null ELSE benchmark#94010367 END AS benchmark#94010677, CASE WHEN (excess_ret#94010368 = null) THEN null ELSE excess_ret#94010368 END AS excess_ret#94010681, CASE WHEN (excess_resret#94010369 = null) THEN null ELSE excess_resret#94010369 END AS excess_resret#94010683, CASE WHEN (excess_retnet#94010370 = null) THEN null ELSE excess_retnet#94010370 END AS excess_retnet#94010698]
Input [12]: [date#94010359, cap#94010360, ret#94010361, resret#94010362, retnet#94010363, turnover#94010364, numcos#94010365, coverage#94010366, benchmark#94010367, excess_ret#94010368, excess_resret#94010369, excess_retnet#94010370]
(11) Filter [codegen id : 1]
Input [3]: [cap#94010646, date#94010643, numcos#94010669]
Condition : (isnotnull(cap#94010646) AND (cast(cap#94010646 as string) = 0))
(12) Project [codegen id : 1]
Output [2]: [date#94010643, numcos#94010669]
Input [3]: [cap#94010646, date#94010643, numcos#94010669]
(13) Union
(14) Exchange
Input [2]: [date#94010641, numcos#94010663]
Arguments: rangepartitioning(date#94010641 ASC NULLS FIRST, 200), ENSURE_REQUIREMENTS, [id=#7506150]
(15) Sort [codegen id : 2]
Input [2]: [date#94010641, numcos#94010663]
Arguments: [date#94010641 ASC NULLS FIRST], true, 0
(16) CollectLimit
Input [2]: [date#94010641, numcos#94010663]
Arguments: 1000000