alezinhacris

I need to creeate an new Spark DF MapType Column based on the existing columns where column name is… Read more Pyspark: Create Maptype Column From Existing Columns

I need to read contiguous files in pySpark. The following works for me. from pyspark.sql import SQ… Read more Read Range Of Files In Pyspark

My problem is based on the similar question here PySpark: Add a new column with a tuple created fro… Read more Create A Tuple Out Of Two Columns - Pyspark

I wrote a pyspark script that reads two json files, coGroup them and sends the result to an elastic… Read more Pyspark: Ship Jar Dependency With Spark-submit

I'm currently working with Spark 2.1 and have a main script that calls a helper module that con… Read more Pyspark 2.1: Importing Module With Udf's Breaks Hive Connectivity

The following is my RDD, there are 5 fields [('sachin', 200, 10,4,True), ('Raju', 4… Read more How To Get Specific Values From Rdd In Spark With Pyspark

Did my research, but didn't find anything on this. I want to convert a simple pandas.DataFrame … Read more Typeerror Converting A Pandas Dataframe To Spark Dataframe In Pyspark

My data.csv file has three columns like given below. I have converted this file to python spark dat… Read more Add Column To Pyspark Dataframe Based On A Condition

I'm stuck in a process where I need to perform some action for each column value in my Datafram… Read more Pyspark Best Alternative For Using Spark Sql/df Withing A Udf?

I have line data in .gz compressed format. I have to read it in pyspark Following is the code snipp… Read more How To Read Gz Compressed File By Pyspark

There are lots of online examples of reading json from Kafka (to write to parquet) - but I cannot f… Read more Spark Streaming: Read Csv String From Kafka, Write To Parquet

I am trying to understand as to what happens when we run the collectAsMap() function in spark. As p… Read more How Does The Collectasmap() Function Work For Spark Api

I could run PySpark from the terminal line and everything works fine. ~/spark-1.0.0-bin-hadoop1/bin… Read more Running Pyspark On And Ide Like Spyder?

I get this weird error message 15/01/26 13:05:12 INFO spark.SparkContext: Created broadcast 0 from … Read more Spark Can't Pickle Method_descriptor

I'd like to be able to have Spark group by a step size, as opposed to just single values. Is th… Read more Pyspark Numeric Window Group By

I have a project that uses java, scala and Apache Spark to do distributed computations on genomic d… Read more Connecting Log4j To Ipython Notebook Stderr In A Java/python Project Using Py4j

I want to remove the words in main data frame from secondary data frame. This is the main data fram… Read more Remove Words From Pyspark Dataframe Based On Words From Another Pyspark Dataframe

What I have so far is: lookup = sc.textFile('/user/myuser/lookup.asv') lookup.map(lambda r… Read more Turn Rdd Into Broadcast Dictionary For Lookup

How to create a new column in PySpark and fill this column with the date of today? This is what I t… Read more Assertionerror: Col Should Be Column

I have been able to load this MongoDB database before, but am now receiving an error I haven't … Read more Mongodb Spark Connector Py4j.protocol.py4jjavaerror: An Error Occurred While Calling O50.load