Difference between revisions of "Working with OpenStreetMap data"
(Created page with '>> return to Documentation ---- by Jukka Rahkonen == Using OpenStreetMap XML data with OpenJUMP == Since August 2013 OpenJUMP Plus version has a native support for opening …') |
m |
||
(9 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
>> return to [[Documentation]] | >> return to [[Documentation]] | ||
---- | ---- | ||
− | by Jukka Rahkonen | + | by Jukka Rahkonen, http://latuviitta.org - 14. Aug. 2013 |
== Using OpenStreetMap XML data with OpenJUMP == | == Using OpenStreetMap XML data with OpenJUMP == | ||
Line 7: | Line 7: | ||
Since August 2013 OpenJUMP Plus version has a native support for opening OpenStreetMap XML data. OpenJUMP can open .osm files either as uncompressed or from zip or bzip archive files. | Since August 2013 OpenJUMP Plus version has a native support for opening OpenStreetMap XML data. OpenJUMP can open .osm files either as uncompressed or from zip or bzip archive files. | ||
− | The OpenJUMP OSM driver is | + | The OpenJUMP OSM driver is utilizing JOSM code and therefore the input data must contain the same metadata tags which are needed by JOSM. |
− | OpenJUMP is a Java program which is running totally in memory. Therefore the amount of available memory is limiting how big OSM-XML files can be opened. OpenJUMP needs roughly 2-3 as much memory than the size of the OSM-XML file to be opened. Conversion is quite fast if the OSM dataset is not too big. For example berlin-latest.osm from geofabrik.de (455 MB, 529234 map features) opens in 50 seconds with laptop running 64-bit JRE 1.7.0 (Oracle) on Windows 7 with 8 GB of memory and Intel Core i7 2760QM processor. Sachsen-latest.osm (1.7 GB, 1.52 million map features) takes 3 minutes and 30 seconds with the same computer. | + | OpenJUMP is a Java program which is running totally in memory. Therefore the amount of available memory is limiting how big OSM-XML files can be opened. OpenJUMP needs roughly ''2-3 times'' as much memory than the size of the OSM-XML file to be opened. Conversion is quite fast if the OSM dataset is not too big. For example berlin-latest.osm from www.geofabrik.de (455 MB, 529234 map features) opens in 50 seconds with laptop running 64-bit JRE 1.7.0 (Oracle) on Windows 7 with 8 GB of memory and Intel Core i7 2760QM processor. Sachsen-latest.osm (1.7 GB, 1.52 million map features) takes 3 minutes and 30 seconds with the same computer. |
− | + | Note that OSM data are in geographic coordinates (WGS84, EPSG code: 4326), so the display of rectangular roads and buildings seems to be skewed. OpenJUMPs [[CTS Extension]] can help you to bring the data into you preferred cartographic projection. | |
− | === | + | == Practical examples on reading OSM-XML with OpenJUMP == |
+ | |||
+ | === Getting a suitable OpenJUMP version === | ||
At the moment a development version of OpenJUMP Plus is needed and the revision should be r3662 or higher. Snapshot versions of OpenJUMP can be downloaded from | At the moment a development version of OpenJUMP Plus is needed and the revision should be r3662 or higher. Snapshot versions of OpenJUMP can be downloaded from | ||
http://sourceforge.net/projects/jump-pilot/files/OpenJUMP_snapshots/ | http://sourceforge.net/projects/jump-pilot/files/OpenJUMP_snapshots/ | ||
− | === | + | === Getting some OpenStreetMap XML data === |
Small extracts of OpenStreetMap XML data can be downloaded through the map window of the main OpenStreetMap site http://www.openstreetmap.org | Small extracts of OpenStreetMap XML data can be downloaded through the map window of the main OpenStreetMap site http://www.openstreetmap.org | ||
[[File:Gettingosmdata.png|thumb|center|550px]] | [[File:Gettingosmdata.png|thumb|center|550px]] | ||
− | It is also possible to use JOSM editor for downloading data and saving osm files on disk. However, for bigger data extracts and for reading filtered data it is recommended to use an API service like OverPASS API http://wiki.openstreetmap.org/wiki/Overpass_API | + | It is also possible to use JOSM editor for downloading data and saving osm files on disk. However, for bigger data extracts and for reading filtered data it is recommended to use an API service like OverPASS API: http://wiki.openstreetmap.org/wiki/Overpass_API |
− | Advanced features of Overpass API are described in the above mentioned wiki page and in the main site http://www.overpass-api.de/. However, simple queries can be created by using this | + | Advanced features of Overpass API are described in the above mentioned wiki page and in the main site http://www.overpass-api.de/. However, simple queries can be created by using this web application: http://harrywood.co.uk/maps/uixapi/xapi.html |
Example: Draw a bounding box which covers Finland and create an Overpass API request template. | Example: Draw a bounding box which covers Finland and create an Overpass API request template. | ||
Line 32: | Line 34: | ||
This request must be edited before it can be used for capturing data for OpenJUMP because OSM driver needs some additional metadata which can be obtained by adding parameter [@meta]. | This request must be edited before it can be used for capturing data for OpenJUMP because OSM driver needs some additional metadata which can be obtained by adding parameter [@meta]. | ||
− | Request for getting all OSM data inside a small bounding box | + | * Request for getting all OSM data inside a small bounding box |
+ | *: <big><code>http://www.overpass-api.de/api/xapi?*[bbox=27,61,27.1,61.1][@meta]</code></big> | ||
− | + | * Request for getting all OSM data with tag ”highway_bus_stop” from a box covering whole Finland | |
− | + | *: <big><code>http://www.overpass-api.de/api/xapi?*[highway=bus_stop][bbox=19.39087,59.72041,32.39868,70.14596][@meta]</code></big> | |
− | Request for getting all OSM data with tag ”highway_bus_stop” from a box covering whole Finland | ||
− | |||
− | <code>http://www.overpass-api.de/api/xapi?*[highway=bus_stop][bbox=19.39087,59.72041,32.39868,70.14596][@meta]</code> | ||
Advanced users can use OSM extracts from Geofabrik and filter data for example with Osmosis or Osmfilter for preparing OSM datasets for OpenJUMP. | Advanced users can use OSM extracts from Geofabrik and filter data for example with Osmosis or Osmfilter for preparing OSM datasets for OpenJUMP. | ||
Line 44: | Line 44: | ||
* http://wiki.openstreetmap.org/wiki/Osmosis | * http://wiki.openstreetmap.org/wiki/Osmosis | ||
* http://wiki.openstreetmap.org/wiki/Osmfilter | * http://wiki.openstreetmap.org/wiki/Osmfilter | ||
+ | |||
+ | === Reading OSM data with OpenJUMP Plus === | ||
+ | OSM data files are opened with OpenJUMP as any other supported file formats through File – Open menu. | ||
+ | |||
+ | [[File:Loadingosm.png|thumb|center|550px]] | ||
+ | |||
+ | OpenJUMP imports all the OSM features on one map layer. The schema of the layer is fixed and hardcoded in the OSM driver code. | ||
+ | |||
+ | [[File:Ojosmschema.png|thumb|center|550px]] | ||
+ | |||
+ | Attribute window is listing the attribute values and small geometry type icons show that layer contains geometries of types Point, LineString, Polygon, and GeometryCollection. | ||
+ | |||
+ | [[File:Ojosmattributeview.png|thumb|center|550px]] | ||
+ | |||
+ | The '''''osm_tags''''' column contains all the OSM tags concatenated into one text string. Column '''''lu_type''''' contains the result of classification. The classification rules are hardcoded. Special tag '''''part_of_OSM_relation''''' holds two pieces of information. First one tells if the feature is acting as a member in some relation. The other part is filled only for features which are relations themselves and it shows if all the members of the relation have been found or not. | ||
+ | |||
+ | [[File:Ojosmtagsuse.png|thumb|center|550px]] | ||
+ | |||
+ | Reading all types of geometries suits well for internal data model of OpenJUMP but otherwise it may not be very practical. For example it may be impossible to define rendering rules which make sense for such mixed layer. However, OpenJUMP has some tools which can be used for filtering and categorizing OSM data once they have been imported. | ||
+ | |||
+ | === Working with the OSM data layer - Example 1 === | ||
+ | |||
+ | OSM layer can be divided into separate layers for each geometry type with the '''Edit>Extract>Extract Layers by Geometry Type''' tool. | ||
+ | |||
+ | [[File:Extractlayers.png|thumb|center|550px]] | ||
+ | |||
+ | By default this tools is also exploding all the geometry collections into geometry primitives (points, lines, and polygons). If user wants to study OSM relations this option can be unchecked and relations will be written as geometry collections into a separate layer. | ||
+ | |||
+ | [[File:Extractlayerbygeom_dialog.png|thumb|center|450px]] | ||
+ | |||
+ | Layer category '''''Result''''' has now OSM features on '''four''' distinct layers. | ||
+ | |||
+ | [[File:Extractbygeomresultlayers.png|thumb|center]] | ||
+ | |||
+ | This example shows the OSM bus stops which were fetched from XAPI on top of a WMS base map. | ||
+ | |||
+ | [[File:Osmfibusstops.png|thumb|center|550px]] | ||
+ | |||
+ | === Working with the OSM data layer - Example 2 === | ||
+ | |||
+ | OpenJUMP can extract layers also by distinct values of any attribute ('''Edit>Extract>Extract Layer by Attribute'''). In this example a new layer will be created for each OSM username that appears in the data. | ||
+ | |||
+ | [[File:Osmextractbyattribute.png|thumb|center|450px]] | ||
+ | |||
+ | Layers in the catagory can be sorted by the number of features they contain. We want to check the most active OSM bus stop mappers now. | ||
+ | |||
+ | [[File:Sortcategorybynumfeatures.png|thumb|center|550px]] | ||
+ | |||
+ | As can be seen from the following screenshot: OSM contributor ”Pikari” edited 4701 bus stops. | ||
+ | |||
+ | [[File:Osmmostfeatureschamp.png|thumb|center|550px]] | ||
+ | |||
+ | == Conclusions == | ||
+ | |||
+ | OpenJUMP OSM driver offers a direct way for using OSM data in a desktop GIS program. However, OSM data are converted rather brute-force into the [http://en.wikipedia.org/wiki/Simple_Features OGC simple features] (geometry types) with a fixed attribute schema. While all the tags included in the OSM data are preserved as a concatenated text string, it is not easy to utilize them in a productive way. It is best to regard the initial release of the driver in August 2013 as something that can lead to more practical applications in the future. Because the OpenJUMP OSM driver is using the well-known [http://en.wikipedia.org/wiki/JTS_Topology_Suite JTS library] and also parts of the well known [http://josm.openstreetmap.de/ JOSM] code base it should give a solid base for other Java programmers to go on with development. | ||
+ | |||
+ | == References == | ||
+ | * Plugin: | ||
+ | ** http://sourceforge.net/projects/jump-pilot/files/OpenJUMP_plugins/Drivers/OSM%20Driver/ | ||
+ | |||
+ | * Driver Announcement on the OpenJUMP users mailing list: | ||
+ | ** http://groups.google.com/forum/#!topic/openjump-users/Y-VkVnEABlM | ||
+ | |||
+ | * Source Code: | ||
+ | ** http://sourceforge.net/p/jump-pilot/code/3677/tree/plug-ins/OsmFileReaderPlugin/trunk/ |
Latest revision as of 00:11, 20 August 2013
>> return to Documentation
by Jukka Rahkonen, http://latuviitta.org - 14. Aug. 2013
Using OpenStreetMap XML data with OpenJUMP
Since August 2013 OpenJUMP Plus version has a native support for opening OpenStreetMap XML data. OpenJUMP can open .osm files either as uncompressed or from zip or bzip archive files.
The OpenJUMP OSM driver is utilizing JOSM code and therefore the input data must contain the same metadata tags which are needed by JOSM.
OpenJUMP is a Java program which is running totally in memory. Therefore the amount of available memory is limiting how big OSM-XML files can be opened. OpenJUMP needs roughly 2-3 times as much memory than the size of the OSM-XML file to be opened. Conversion is quite fast if the OSM dataset is not too big. For example berlin-latest.osm from www.geofabrik.de (455 MB, 529234 map features) opens in 50 seconds with laptop running 64-bit JRE 1.7.0 (Oracle) on Windows 7 with 8 GB of memory and Intel Core i7 2760QM processor. Sachsen-latest.osm (1.7 GB, 1.52 million map features) takes 3 minutes and 30 seconds with the same computer.
Note that OSM data are in geographic coordinates (WGS84, EPSG code: 4326), so the display of rectangular roads and buildings seems to be skewed. OpenJUMPs CTS Extension can help you to bring the data into you preferred cartographic projection.
Practical examples on reading OSM-XML with OpenJUMP
Getting a suitable OpenJUMP version
At the moment a development version of OpenJUMP Plus is needed and the revision should be r3662 or higher. Snapshot versions of OpenJUMP can be downloaded from http://sourceforge.net/projects/jump-pilot/files/OpenJUMP_snapshots/
Getting some OpenStreetMap XML data
Small extracts of OpenStreetMap XML data can be downloaded through the map window of the main OpenStreetMap site http://www.openstreetmap.org
It is also possible to use JOSM editor for downloading data and saving osm files on disk. However, for bigger data extracts and for reading filtered data it is recommended to use an API service like OverPASS API: http://wiki.openstreetmap.org/wiki/Overpass_API
Advanced features of Overpass API are described in the above mentioned wiki page and in the main site http://www.overpass-api.de/. However, simple queries can be created by using this web application: http://harrywood.co.uk/maps/uixapi/xapi.html
Example: Draw a bounding box which covers Finland and create an Overpass API request template.
This request must be edited before it can be used for capturing data for OpenJUMP because OSM driver needs some additional metadata which can be obtained by adding parameter [@meta].
- Request for getting all OSM data inside a small bounding box
http://www.overpass-api.de/api/xapi?*[bbox=27,61,27.1,61.1][@meta]
- Request for getting all OSM data with tag ”highway_bus_stop” from a box covering whole Finland
http://www.overpass-api.de/api/xapi?*[highway=bus_stop][bbox=19.39087,59.72041,32.39868,70.14596][@meta]
Advanced users can use OSM extracts from Geofabrik and filter data for example with Osmosis or Osmfilter for preparing OSM datasets for OpenJUMP.
- http://download.geofabrik.de/europe.html
- http://wiki.openstreetmap.org/wiki/Osmosis
- http://wiki.openstreetmap.org/wiki/Osmfilter
Reading OSM data with OpenJUMP Plus
OSM data files are opened with OpenJUMP as any other supported file formats through File – Open menu.
OpenJUMP imports all the OSM features on one map layer. The schema of the layer is fixed and hardcoded in the OSM driver code.
Attribute window is listing the attribute values and small geometry type icons show that layer contains geometries of types Point, LineString, Polygon, and GeometryCollection.
The osm_tags column contains all the OSM tags concatenated into one text string. Column lu_type contains the result of classification. The classification rules are hardcoded. Special tag part_of_OSM_relation holds two pieces of information. First one tells if the feature is acting as a member in some relation. The other part is filled only for features which are relations themselves and it shows if all the members of the relation have been found or not.
Reading all types of geometries suits well for internal data model of OpenJUMP but otherwise it may not be very practical. For example it may be impossible to define rendering rules which make sense for such mixed layer. However, OpenJUMP has some tools which can be used for filtering and categorizing OSM data once they have been imported.
Working with the OSM data layer - Example 1
OSM layer can be divided into separate layers for each geometry type with the Edit>Extract>Extract Layers by Geometry Type tool.
By default this tools is also exploding all the geometry collections into geometry primitives (points, lines, and polygons). If user wants to study OSM relations this option can be unchecked and relations will be written as geometry collections into a separate layer.
Layer category Result has now OSM features on four distinct layers.
This example shows the OSM bus stops which were fetched from XAPI on top of a WMS base map.
Working with the OSM data layer - Example 2
OpenJUMP can extract layers also by distinct values of any attribute (Edit>Extract>Extract Layer by Attribute). In this example a new layer will be created for each OSM username that appears in the data.
Layers in the catagory can be sorted by the number of features they contain. We want to check the most active OSM bus stop mappers now.
As can be seen from the following screenshot: OSM contributor ”Pikari” edited 4701 bus stops.
Conclusions
OpenJUMP OSM driver offers a direct way for using OSM data in a desktop GIS program. However, OSM data are converted rather brute-force into the OGC simple features (geometry types) with a fixed attribute schema. While all the tags included in the OSM data are preserved as a concatenated text string, it is not easy to utilize them in a productive way. It is best to regard the initial release of the driver in August 2013 as something that can lead to more practical applications in the future. Because the OpenJUMP OSM driver is using the well-known JTS library and also parts of the well known JOSM code base it should give a solid base for other Java programmers to go on with development.
References
- Driver Announcement on the OpenJUMP users mailing list: