Pyspark Launch Issue Windows 10 Wxith Python 3.6

I am unable to launch Pyspark in windows after installing Python 3.x with Anaconda. Getting below error - Python 3.6.0 |Anaconda 4.3.0 (64-bit)| (default, Dec 23 2016, 11:57:41) [

Solution 1:

PySpark 2.1 currently does not work with Python 3.6.0. This issue has been reported here. It has been resolved on the 17th of January 2017, but has not been released as of today (17 March 2017). However, looking at the committed changes, you should be able to fix this yourself by downloading the following two Python files:

https://github.com/apache/spark/blob/master/python/pyspark/serializers.pyhttps://github.com/apache/spark/blob/master/python/pyspark/cloudpickle.py

and saving them to the following location (overwriting the existing files):

C:\Users\prudra\Desktop\Udemy\spark-2.1.0-bin-hadoop2.7\python\pyspark

or more generic the files should be saved to the python\pyspark subfolder of your Spark installation.

Solution 2:

Spark 2.1.1 just released on May 4th. It's now working with Python 3.6, you can see the release note here.