Map Arlington 1: Extracting OSM Data for a Localized Map Server

The Problem

OpenStreetMap (OSM) is collaborative map of the world, freely available to anyone to use and contribute to its development. Planet.osm has information on downloading the database, which at the time I’m writing this was 33.6GB for the data in PBF format.

I wanted to create a map server for my local community of Arlington County, Virginia. In the first part of this case study I’ll document how I got OSM map data into PostGIS.

Creating the Database

I’m working on an Ubuntu 18.04 server with PostgreSQL 10 and PostGIS 2.4 installed. These tools can be installed with the following command:

$ sudo apt install postgis

To allow external connections to the database, I edited /etc/postgresql/10/main/pg_hba.conf and changed:

host    all             all             127.0.0.1/32            md5

to:

host    all             all             0.0.0.0/0               md5

and then edited /etc/postgresql/10/main/postgresql.conf and changed:

#listen_addresses = 'localhost'

to:

listen_addresses = '*'

and then finally ran the following command to restart the database server and have these changes take effect:

$ sudo service postgresql restart

Next I gave my user database superuser privileges and set its postgress password with:

$ sudo su - postgres
$ createuser --superuser [user]
$ psql -c "ALTER ROLE [user] PASSWORD '[password]'"
$ exit

so I could create a database with:

$ createdb map_arlington
$ psql -d map_arlington -c 'CREATE EXTENSION postgis'

I installed this postgres server on a KVM using virt-manager. On the 192.168.122.x network that virt-manager creates, this server had address 192.168.122.19. To verify I could connect to it I ran:

$ psql -h 192.168.122.19 -p 5432 -U jelkner -d map_arlington

When greeted with:

map_arlington=#

I knew I was ready to go.

With a spatial database created, now it’s time to get the data.

Getting the Data

The German company Geofabrik provides a mirror of Planet.osm data broken down into localized parts, including countries and states in the U.S.. Their Virginia page has a link to 215 MB file (at the time of this writing) containing recent data for the state of Virginia.

I downloaded this with:

$ wget http://download.geofabrik.de/north-america/us/virginia-latest.osm.pbf

While 215 MB isn’t unmanageable, for this project I’m only interested in the small part of that data that contains Arlington County. Osmosis is a command line Java application that can extract data from a PBD file within a specified bounding box.

To find the bounding box for Arlington County, I used an web application appropriately named BoundingBox at http://boundingbox.klokantech.com.

With the bounding box determined, I installed osmosis and used it to the PBF data for Arlington County:

$ sudo apt install osmosis
$ osmosis --read-pbf file=virginia-latest.osm.pbf --bounding-box top=38.9342803955 left=-77.1723251343 bottom=38.8272895813 right=-77.032081604 --write-pbf file=arlington.osm.pbf

81607 milliseconds later on my little atom processor server and I had a 2.4MB file with Arlington County PBF data.

The next step is loading this data into the PostGIS database.

Loading the PBF Data into PostGIS

OSM provides another tool, imposm, for importing XML or PBF data into PostGIS.:

$ sudo apt install imposm

After installing imposm, loading the data into PostGIS is a two step process:

$ imposm --read arlington.osm.pbf
$ imposm --write -d map_arlington -U [username]

The first step creates a collection of files ending in .cache. The next step writes these to the database.

Viewing the Database in QGIS

QGIS provides a quick and easy way to view the spatial data in a PostGIS database. After loading the data with imposm, I connected QGIS to it:

Map Arlington database in QGIS

and took a look:

Map Arlington database in QGIS

Note

The osm_new_roads layer reported an error when I tried to load it. This didn’t happen when I originally wrote this a few years ago.