why all fields are nullable

[yuryn]

When I read simple avro file with all fileds non-nullable, resulted dataframe schema has all fields nullable.

avro file schema:

{
  "type" : "record",
  "name" : "RobotDetection",
  "namespace" : "cz.search.robotdetection",
  "fields" : [ {
    "name" : "sessionId",
    "type" : "string"
  }, {
    "name" : "robotDetectionResult",
    "type" : "int"
  } ]
}

dataframe schema:

val a = spark.read.format("com.databricks.spark.avro").load("avrofile.avro")
a.schema
res1: org.apache.spark.sql.types.StructType = StructType(StructField(sessionId,StringType,true), StructField(robotDetectionResult,IntegerType,true))

[gengliangwang]

This is for null safety in data sources:
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L379