Foss4g NL: late afternoon sessions (web components + railway API + long-living data)¶

Railmaps: Prorail open source GIS viewer based on generic web components (GeoBlocks)¶

Warning: the title mentioned it as an open source GIS viewer, but it is not open source. After the presentation I asked where I could find it, but they said it was internal-only. They used open source, but the project itself isn’t.

Note: the actual “Railmaps” website is only accessible for the company itself. I’ll mention the URL, but it isn’t that useful: https://www.railmaps.nl . ProRail is the Dutch national railway infrastructure maintainer.

Railmaps used GeoWeb previously. They wanted to move away to open source, but wanted to keep the existing functionality. They started with interviews with users. Then a UX designer made designs in Figma. Then they started iteratively building it with typescript/lit. Last thursday they went live.

One thing they wanted to solve was redundant code. They had many repositories with maps and there was a lot of duplication. The various map websites also were also not consistent regarding UI/UX.

They created web components to handle it all (they called it geoblocks). The three main components:

• Some central map.

• Map layers (some standard toggle system for switching on/off various layers and adding them).

• Sidebar tools (user actions, zooming, search tools, etc.)

What they used: Openlayers (for the actual map), WebAwesome (robust component foundation), Lit Elements (lightweight web components). And lots of open source tools. They made sure it works with most javascript frameworks.

He mentions Storybook as a fantastic documentation tool for interactive components.

NS (=national railway company) API portal - Niek van Ruler¶

NS means Nederlandse Spoorwegen, the main railway company in the Netherlands.

He noticed that the NS had a nice API. You only have to request a key manually, and then you can access quite a lot of data. Prices, public bike info, data about stations, station floor plans, live train location info, disruptions, all details of every individual train trip, geojson with the train tracks, etc.

(He showed a couple of API responses and the demo website he build with it.)

His demo: https://geodienst.xyz/ns/

A Python wrapper for the NS API: https://github.com/aquatix/ns-api

Core flow: sovereign nature data, readable in 2075 - Joris Röling¶

(Nice detail: the conference is being held in the university’s “Röling building”, which is named after Joris’ grandfather, Bert Röling, one of the judges at the WW2 Tokyo trial.)

His aim is to have nature data not only usable today, but also in fifty years’ time. Nature data? The Dutch nature is monitored: vegetation, species, administrative geographies.

Core Flow is the core of a larger data platform. Its promise is the data must outlive every tool we used to make it.

Public data rarely dies on purpose. It dies in boring ways: a license expires; someone switches off a server; the format is only readable by one vendor. Nature data often spans 50 to 100 years, but our tooling lasts only between 3-5 years.

Their solution:

• Storage is just files: they chose Parquet on S3-compatible storage. This means there’s no database system that might not be available.

• Querying is with DuckDB now, but Parquet should be queryable in 50 years, whatever the tool.

• They use stable identifiers. Every file is named with a UUID and is immutable. There’s a folder structure, too, but that’s only for convenience. Immutable: it means you can look at various versions. You can re-discover what we thought at an earlier date.

The approach is almost serverless. It helps that nothing needs to be running: the files are “just” stored. You access the files with DuckDB when you need to, the current API and website are only for convenience.

Tasks that need to be done are run through GitLab’s regular pipeline. So they’re just using regular CI tools: stuff you can easily run with other system. You could use one of the current beautiful workflow systems (prefect, airflow, etc.), but how long will those be around?

So: data stored in files. On top, DuckDB as access tool. On top of that some works-for-now tools like an API and a website.

Note that they store both the original data and their converted-to-parquet version.

Unrelated photo: we have two offices in the center of Utrecht. As a handy connection, we’re using a radio link (“straalverbinding”) between the two. We have line of sight. This view is from our second building. Noticable is the “city castle” Oudaen: city politics in Utrecht could get a bit lively in the middle ages.