pyspark - Can't open GeoJson with Python Sedona GeoJsonReader

Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.
Learn more about Collectives
Teams
Q&A for work
Connect and share knowledge within a single location that is structured and easy to search.
Learn more about Teams
I am using Python's Apache Sedona to open a GeoJson file. I followed this guide . I follow every step for opening a GeoJson, but for the sake of clarity, this is what I did:
spark = SparkSession.\
    builder.\
    master("local[*]").\
    appName("Sedona App").\
    config("spark.serializer", KryoSerializer.getName).\
    config("spark.kryo.registrator", SedonaKryoRegistrator.getName) .\
    config("spark.jars.packages", "org.apache.sedona:sedona-python-adapter-3.0_2.12:1.1.0-incubating,org.datasyslab:geotools-wrapper:1.1.0-25.2") .\
    getOrCreate()
SedonaRegistrator.registerAll(spark)
sc = spark.sparkContext
amenity_file = 'example/2/amenity.geojson'
geojson_file = GeoJsonReader.readToGeometryRDD(sc, amenity_file)
The last line spits this:
22/03/25 16:52:17 WARN FormatMapper: [Sedona] The GeoJSON file doesn't have feature properties
However, I continued with the following line (just like in the example):
Adapter.toDf(geojson_file, spark).show()
But I got an error:
22/03/25 16:52:26 ERROR FormatMapper: [Sedona] com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (String)"{"; line: 1, column: 1])
 at [Source: (String)"{"; line: 1, column: 3]
22/03/25 16:52:26 ERROR Executor: Exception in task 0.0 in stage 8.0 (TID 8)
java.lang.RuntimeException: com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (String)"{"; line: 1, column: 1])
 at [Source: (String)"{"; line: 1, column: 3]
    at org.wololo.geojson.GeoJSONFactory.create(GeoJSONFactory.java:31)
    at org.wololo.jts2geojson.GeoJSONReader.read(GeoJSONReader.java:20)
    at org.wololo.jts2geojson.GeoJSONReader.read(GeoJSONReader.java:16)
    at org.apache.sedona.core.formatMapper.FormatMapper.readGeoJSON(FormatMapper.java:206)
    at org.apache.sedona.core.formatMapper.FormatMapper.readGeometry(FormatMapper.java:304)
    at org.apache.sedona.core.formatMapper.FormatMapper.call(FormatMapper.java:377)
    at org.apache.sedona.core.formatMapper.FormatMapper.call(FormatMapper.java:52)
    at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitions$1(JavaRDDLike.scala:153)
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:837)
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:837)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:127)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:462)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:465)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (String)"{"; line: 1, column: 1])
 at [Source: (String)"{"; line: 1, column: 3]
    at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:664)
    at com.fasterxml.jackson.core.base.ParserBase._handleEOF(ParserBase.java:486)
    at com.fasterxml.jackson.core.base.ParserBase._eofAsNextChar(ParserBase.java:498)
    at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._skipWSOrEnd(ReaderBasedJsonParser.java:2354)
    at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextFieldName(ReaderBasedJsonParser.java:905)
    at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:249)
    at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:68)
    at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:15)
    at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4254)
    at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2711)
    at org.wololo.geojson.GeoJSONFactory.create(GeoJSONFactory.java:21)
    ... 32 more
22/03/25 16:52:26 WARN TaskSetManager: Lost task 0.0 in stage 8.0 (TID 8, works-mbp.lan, executor driver): java.lang.RuntimeException: com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (String)"{"; line: 1, column: 1])
 at [Source: (String)"{"; line: 1, column: 3]
    at org.wololo.geojson.GeoJSONFactory.create(GeoJSONFactory.java:31)
    at org.wololo.jts2geojson.GeoJSONReader.read(GeoJSONReader.java:20)
    at org.wololo.jts2geojson.GeoJSONReader.read(GeoJSONReader.java:16)
    at org.apache.sedona.core.formatMapper.FormatMapper.readGeoJSON(FormatMapper.java:206)
    at org.apache.sedona.core.formatMapper.FormatMapper.readGeometry(FormatMapper.java:304)
    at org.apache.sedona.core.formatMapper.FormatMapper.call(FormatMapper.java:377)
    at org.apache.sedona.core.formatMapper.FormatMapper.call(FormatMapper.java:52)
    at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitions$1(JavaRDDLike.scala:153)
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:837)
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:837)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:127)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:462)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:465)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (String)"{"; line: 1, column: 1])
 at [Source: (String)"{"; line: 1, column: 3]
    at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:664)
    at com.fasterxml.jackson.core.base.ParserBase._handleEOF(ParserBase.java:486)
    at com.fasterxml.jackson.core.base.ParserBase._eofAsNextChar(ParserBase.java:498)
    at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._skipWSOrEnd(ReaderBasedJsonParser.java:2354)
    at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextFieldName(ReaderBasedJsonParser.java:905)
    at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:249)
    at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:68)
    at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:15)
    at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4254)
    at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2711)
    at org.wololo.geojson.GeoJSONFactory.create(GeoJSONFactory.java:21)
    ... 32 more
22/03/25 16:52:26 ERROR TaskSetManager: Task 0 in stage 8.0 failed 1 times; aborting job
EDIT: The file contains geometries... but it is inside an Array, then Struct. This is how it looks:
    "type": "FeatureCollection",
    "name": "amenity",
    "features": [
        "type": "Feature",
        "feature_type": "amenity",
        "id": "1231312323f",
        "properties": {
          "accessibility": null,
          "address_id": "1231232312",
          "alt_name": null,
          "category": "elevator",
          "correlation_id": null,
          "hours": null,
          "name": null,
          "phone": null,
          "unit_ids": [
            "1232312",
            "123212"
          "website": null
        "geometry": {
          "type": "Point",
          "coordinates": [
            -121.8888997,
            37.3285715
UPDATE:
I tried also making up a very simple GeoJson and still throws the same error... I believe this is not to do with the GeoJsonReader:
  "type": "Point",
  "coordinates": [
      -105.01621,
      39.57422
                Hi Paul, thanks for the help... yes it is... I've updated the post with GeoJson and error in text
– user18140022
                Mar 25, 2022 at 17:21
                That's right. I made a mistake copying the geojson content... I copied only the first child as the file is quite long. I edited the geojson in the post and shows as valid in geojsonlint.com...
– user18140022
                Mar 25, 2022 at 20:10
        Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.