alezinhacris

I am not sure is this possible in pyspark. I believe it should be just that i am not winning here :… Read more Pyspark Dataframe - How To Pass String Variable To Df.where() Condition

I need to creeate an new Spark DF MapType Column based on the existing columns where column name is… Read more Pyspark: Create Maptype Column From Existing Columns

I need to read contiguous files in pySpark. The following works for me. from pyspark.sql import SQ… Read more Read Range Of Files In Pyspark

My problem is based on the similar question here PySpark: Add a new column with a tuple created fro… Read more Create A Tuple Out Of Two Columns - Pyspark

I wrote a pyspark script that reads two json files, coGroup them and sends the result to an elastic… Read more Pyspark: Ship Jar Dependency With Spark-submit

I'm currently working with Spark 2.1 and have a main script that calls a helper module that con… Read more Pyspark 2.1: Importing Module With Udf's Breaks Hive Connectivity

The following is my RDD, there are 5 fields [('sachin', 200, 10,4,True), ('Raju', 4… Read more How To Get Specific Values From Rdd In Spark With Pyspark

I am writing data into s3 bucket and creating parquet files using pyspark . MY bucket structure loo… Read more Dynamically Folder Creation In S3 Bucket From Pyspark Job