-
Notifications
You must be signed in to change notification settings - Fork 45
Description
The error occurs when running the following code:
from pyspark.sql import SparkSession
import pyrasterframes
from pyrasterframes.rasterfunctions import *
from pyspark.sql.functions import lit
from pyspark.sql.functions import col
import os
from pyrasterframes.utils import create_rf_spark_session
java8_location = 'C:/Program Files/Java/jdk1.8.0_311'
os.environ['JAVA_HOME'] = java8_location
os.environ['PYSPARK_PYTHON'] = "E:/conda3/envs/pyGIS/python.exe"
spark = create_rf_spark_session()
df1 = spark.read.raster("F:/qx_data/mishi/out8/rainfall_20080102.tif")
df2 = spark.read.raster("F:/qx_data/mishi/out8/rainfall_20080103.tif")
combined_df = df1.crossJoin(df2)
The error message displays:
25/02/26 16:09:09 WARN Shell: Did not find winutils.exe: java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
25/02/26 16:09:09 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Error setting up SparkSession; cannot find the pyrasterframes assembly jar
'JavaPackage' object is not callable
Traceback (most recent call last):
File "E:\conda3\envs\pyGIS\lib\code.py", line 90, in runcode
exec(code, self.locals)
File "", line 1, in
File "E:\pycharm2022\PyCharm 2022.2.1\plugins\python\helpers\pydev_pydev_bundle\pydev_umd.py", line 198, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
File "E:\pycharm2022\PyCharm 2022.2.1\plugins\python\helpers\pydev_pydev_imps_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "F:/testproject/rasteroverlaytest.py", line 18, in
df1 = spark.read.raster("F:/qx_data/mishi/out8/rainfall_20080102.tif")
AttributeError: 'NoneType' object has no attribute 'read'