What a benchmark of nine models taught me about spatial questions, and the open-source engine I built from it.

Disclosure: I built GeoSpark, the tool this article describes. It is open source (Apache 2.0): https://github.com/Maz2580/geospark. The benchmark figures below come from its repository, where the datasets, code and per-model results are published so anyone can re-run them. The full analysis is in a manuscript under peer review.

GeoSpark's website: ask a spatial question, and the answer comes with its method, source and units.

A question with one right answer

How far is the Eiffel Tower from the Louvre?

That question has one correct answer. On the WGS84 ellipsoid, between those two points, it is 3,172 metres. Ask a language model, though, and the answer depends on which model you picked.

We tested that with GeoSpark Bench: nine models, three ways of asking. The models were five open-weight models from 3.8B to 9B parameters, gpt-oss 20B, and three frontier APIs (GPT-5.4, Claude Sonnet 4.5 and Gemini 2.5 Pro). Each one answered the bare question, answered again with step-by-step reasoning, and answered a third time with spatial tools attached. On numeric distance questions, an answer counted as correct within 10% of the true geodesic distance. There were 80 questions per model.

On the bare question:

The small open-weight models were wrong about 95% of the time.
The frontier models ranged from 25% correct (Gemini 2.5 Pro) to 95% (GPT-5.4).

Tools level the field

With the tools attached, every model that made valid tool calls scored between 76% and 96%.

Model	Bare question	Step by step	With tools
Qwen 2.5 7B	5.0%	20.0%	76.2%
Gemini 2.5 Pro	25.0%	55.0%	85.0%
Claude Sonnet 4.5	73.8%	92.5%	96.2%
GPT-5.4	95.0%	90.0%	95.0%

Qwen 2.5 7B gained 71 points: a 7B model on an ordinary CPU, given the right tools, performs like a frontier API. GPT-5.4 gained nothing, because it was already at 95%. The tools don't make a model smarter. They take the part a model is bad at, arithmetic on an ellipsoid, and hand it to software that can't get it wrong.

The benchmark in one line, as shown on geospark.terrascout.app.

Four things the numbers taught me

1. Aggregate scores hide bias. Qwen scored 56% on topology questions, which sounds middling. Underneath, it was mostly saying "yes": 93% on intersects, where the true answer is usually yes, and 3% on disjoint. GPT-5.4 scored 0% on touches. With a geometry engine computing the predicate, every relationship scored 100%. If you evaluate spatial AI, break your results down by predicate.

2. Step-by-step prompting is not a fix. It helped some models and hurt others. For Gemma 2 9B, distance accuracy fell from 37.5% to 6.2%, and its correct bare answers were well-known long-distance pairs, which is recall of training data rather than computation.

3. Having tools is not the same as using them. Llama 3.1 8B had the tools but mixed up kilometres and metres in the results it got back, and finished at 7.5%. Mistral 7B ignored the tools and answered in plain text. Tool calling is a skill of its own, and it should be measured separately.

4. Not every question needs a tool. Knowledge questions, such as whether the Amazon lost forest between 2015 and 2023, were answered well at every model size without tools. The rule is to compute what can be computed and let the model handle the rest.

What GeoSpark does

GeoSpark is the engine I built from these results. The model reads the question; the engine computes the answer.

Computation: geodesic distance, area and length on the WGS84 ellipsoid (pyproj), exact topology through Shapely (contains, within, intersects, touches, crosses, overlaps, disjoint), buffers, and CRS transformation of every input.
Places and terrain: geocoding through OpenStreetMap Nominatim, elevation from the Copernicus DEM GLO-30, routing, isochrones and travel-time matrices.
Open data with no key: Overture Maps places, buildings, roads and admin areas, read directly from GeoParquet with DuckDB; weather from Open-Meteo.
Your own files: GeoPackage, Shapefile, GeoParquet, FlatGeobuf, CSV and more.

A live isochrone: the area reachable on foot in 10 minutes from Federation Square, Melbourne, measured at 1.52 km², from Valhalla routing on OpenStreetMap.

Every answer shows its working

A correct number isn't enough if nobody can check it. Every GeoSpark result carries:

its provenance: the source, licence, attribution, the time it was retrieved and the method used;
its units;
its warnings.

The Eiffel Tower to Louvre distance, from the hosted API, comes back as 3,172.01 m. It names the method (pyproj's geodesic inverse, Karney's algorithm) and the exact versions of GeoSpark, PROJ and GEOS that computed it. An elevation answer includes the Copernicus attribution and the model's stated vertical accuracy. Results export as GeoJSON that opens straight in QGIS.

Mount Kosciuszko from the Copernicus DEM GLO-30. The Proof panel names the dataset, its licence and how the height was read, and the result downloads as GeoJSON.

It refuses rather than guesses

This is the design choice I care about most. When GeoSpark can't answer reliably, it says so, with the reason, instead of returning a confident wrong number:

a self-intersecting polygon;
coordinates out of range;
a likely swap of latitude and longitude;
an ambiguous place name (there are a lot of Springfields).

A self-intersecting polygon is refused, with the point where it crosses itself and how to repair it.

"Springfield" returns a result, but the caveat says three Springfields are almost equally likely and far apart.

It never guesses a CRS. A Shapefile without a .prj file is refused until you say which CRS it uses, and a projected file, such as MGA zone 55, is reprojected with its original CRS reported.

Analyses you can replay

geospark record saves a run: every data request, every computation, and the library versions, each with a fingerprint. geospark replay runs it again and reports one of three outcomes:

REPRODUCED;
DATA CHANGED, naming the source and the step;
CODE CHANGED, naming the library whose version moved.

For anyone who has tried to reproduce a GIS analysis from a year ago, that distinction is the whole point.

What it doesn't do yet

Slope, viewshed, raster band maths and change detection are planned but not built. Ask for them today and you get a clear "not implemented yet" error, not an answer.

Try it

In Claude or any MCP client: add https://geospark.terrascout.app/mcp as a custom connector. It is open, with no key, and has 15 read-only tools.
On your own machine: pip install "geospark-ai[mcp]", then run geospark-mcp. The local server can also read your own files.
In a browser: https://geospark.terrascout.app
The code and the benchmark: https://github.com/Maz2580/geospark. Run python -m geospark.bench run to reproduce the results.

GeoSpark already powers part of Terrascout. When you ask Terrascout's Copilot how high a mountain is, the answer comes from GeoSpark, and the dataset, its licence and its stated accuracy are named in the reply.

If you work in GIS or geospatial AI, I'd like to know where it breaks. Open an issue, or tell me which question it should refuse but doesn't.

Language models shouldn't do geometry. They should call it.