PicklingError: Could not serialize object: IndexError: tuple index out of range

[Srirangan02]

When i try to run a Pyspark code getting the error while creating data frame.this same code runs in intellij and also i used colab for three months without any issue with the same versions of spark and python.

This should run without any issues

I'm using Chrome

https://colab.research.google.com/drive/1-s_pjQqKOWxWzzFjkypePfZ76jQU2f7M#scrollTo=cZLFdCJahMUl&line=21&uniqifier=1

[cperry-goog]

can you grant open access to the notebook?

[Srirangan02]

can you grant open access to the notebook?

Could you please specify what you need to check? I’ve already attached the screenshot and issue details.

[epsi10nvn]

try to use spark-3.5.0

!sudo apt update
!apt-get install openjdk-8-jdk-headless -qq > /dev/null
!wget -q http://archive.apache.org/dist/spark/spark-3.5.0/spark-3.5.0-bin-hadoop3.tgz
!tar xf spark-3.5.0-bin-hadoop3.tgz
!pip install -q findspark
`
````
import os
import findspark

os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["SPARK_HOME"] = "/content/spark-3.5.0-bin-hadoop3"

findspark.init()
`

[Srirangan02]

try to use spark-3.5.0

!sudo apt update
!apt-get install openjdk-8-jdk-headless -qq > /dev/null
!wget -q http://archive.apache.org/dist/spark/spark-3.5.0/spark-3.5.0-bin-hadoop3.tgz
!tar xf spark-3.5.0-bin-hadoop3.tgz
!pip install -q findspark
`

import os
import findspark

os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["SPARK_HOME"] = "/content/spark-3.5.0-bin-hadoop3"

findspark.init()
`

Thank you so much its working why its not working in spark 3.1?