神刀安全网

Drawing every street in Romania

Ben Fry’s project All Streets left quite the impression on me back in 2007 and it’s stayed in the back of my head ever since. About a year ago I started to wonder if I could pull off something similar with tools I am comfortable with, namely JavaScript for data processing and SVG for drawing things.

And this happened:

Drawing every street in Romania

Here’s a step-by-step account of how I got there.

1. Get the data

Geofabrik thoughtfully packages OpenStreetMap data for every country, so I grabbed the .osm.pbf for Romania. PBF is an alternative to the XML format in which OSM data is usually kept.

OSM works with just three data types:

  • nodes define points in space (through latitude & longitude);
  • ways are collections of nodes which define linear features (yay, streets!) and area boundaries;
  • relations are sometimes used to explain how other elements work together – e.g. multiple ways that define a longer route.

For our modest purposes, we only need:

  1. Ways that are labeled as streets ;
  2. The nodes that comprise those ways.

2. Extracting street data from the PBF

Time to brush off our Node.js skills and extract the data from the PBF file.

After a naïve attempt at loading all the data in memory, it became apparent that data at this volume needs streaming – a technique in which we read data item-by-item, with only fraction in memory at any given time. osm-pbf-parser is a streaming parser for PBF data which outputs these structures:

{   type: 'node',   id: 122321,   lat: 53.527972600000005,   lon: 10.0241143,   tags: {...},   info: {...} } 
{   type: 'way',   id: 108,   tags: {     created_by: 'Potlatch 0.8',     highway: 'living_street',     name: 'Kitzbühler Straße',     postal_code: '01217'    },   refs: [ 442752, 231712390, 442754 ],   info: {...} } 

Traversing the PBF file, we can do these checks to find the items we need:

  function isNode(item) {     return item.type === 'node';   }    function isStreet(item) {     return item.type === 'way' && item.tags.highway;   } 

With streams in mind, let’s extract nodes and streets into plain-text files, with one item per line.

extract-nodes.js

var fs = require('fs'); var through2 = require('through2'); var osm_parser = require('osm-pbf-parser'); var JSONStream = require('JSONStream');  var INPUT_FILE = 'data/data.osm.pbf'; var OUTPUT_FILE = 'output/nodes.txt';  function isNode(item) {     return item.type === 'node'; }  console.log('Extracting nodes from data file: ' + INPUT_FILE); fs.createReadStream(INPUT_FILE)     .pipe(new osm_parser())     .pipe(         through2.obj(function(items, enc, next) {             var nodes = items.filter(isNode).map(function(item) {                 return item.id + ',' + item.lat + ',' + item.lon;             });             if (nodes.length) {                 this.push(nodes.join('/n'));             }             next();         })     )     .pipe(         fs.createWriteStream(OUTPUT_FILE)     )     .on('finish', function() {         console.log('Finished extracting nodes onto file: ' + OUTPUT_FILE);     }); 

extract-streets.js

var fs = require('fs'); var through2 = require('through2'); var osm_parser = require('osm-pbf-parser'); var JSONStream = require('JSONStream');  var INPUT_FILE = 'data/data.osm.pbf'; var OUTPUT_FILE = 'output/streets.txt';  function isStreet(item) {     return item.type === 'way' && item.tags.highway; }  console.log('Extracting roads from data file: ' + INPUT_FILE); fs.createReadStream(INPUT_FILE)     .pipe(new osm_parser())     .pipe(         through2.obj(function(items, enc, next) {             var roads = items.filter(isStreet).map(function(item) {                 return item.refs;             });             if (roads.length) {                 this.push(roads.map(function(road) {                     return road.join(',');                 }).join('/n'));             }             next();         })     )     .pipe(         fs.createWriteStream(OUTPUT_FILE)     ).on('finish', function() {         console.log('Finished extracting roads onto file: ' + OUTPUT_FILE);     }); 

Which outputs:

nodes.txt

360714,44.493699500000005,26.0854494 360853,44.467436600000006,26.0771428 537912,44.425765000000006,26.123137900000003 546140,44.47436450000001,26.123994300000003 ... 

( id,latitude,longitude )

streets.txt

656951,2260664460,3227352565,656952, ... 256700851,2152136723,659642,256705252,2152144026, ... 304797001,2382014755,310215524,255848765 ... 

( node1,node2,node3,... )

Note: We’re using through2.obj() to simplify the pipework.

3. Mapping node IDs to their coordinates

We now have a huge set of node coordinates and another huge set of node IDs. In order to map the IDs to the coordinates, we need to do two things:

  1. Load the nodes into some sort of database
  2. Query the database to look up the coordinates for a given ID

For storage I’ve turned to LevelDB which is a pretty straightforward, file-based database. You use it in Node through leveldown and levelup .

load-nodes.js

var fs = require('fs'); var through2 = require('through2'); var split2 = require('split2'); var levelup = require('level');  var DATABASE_NAME = 'everystreet'; var INPUT_FILE = 'output/nodes.txt';  var i = 0;  console.log('creating levelDB database ' + DATABASE_NAME); levelup(DATABASE_NAME, function(err, db) {     var write_stream = db.createWriteStream();     fs.createReadStream(INPUT_FILE, { encoding: 'utf8' })         .pipe(split2())         .pipe(through2.obj(function(line, enc, next){             var parts = line.split(',');             this.push({                  key: parts[0],                  value: parts[1] + "," + parts[2]             });              // Prevent memory leak             // See: https://github.com/rvagg/node-levelup/issues/298             if (i++ > 999) {                 setImmediate(next);                 i = 0;             } else {                 next();             }         }))         .pipe(write_stream)         .on('finish', function() {             console.log('Finished importing nodes into the database ' + DATABASE_NAME);         }); }); 

This creates a LevelDB database with the name everystreet (which in turn creates an everystreet folder where the data is kept), and adds all nodes with key=ID and value=lat,lon .

Note: While attempting this I ran into some memory troubles to which the easy solution is to delay every 1000th next() call. There’s also level-bulk-load which attempts to optimize bulk writing in LevelDB, so that might be something to look into.

Next, let’s map the node IDs to their coordinates in our street definitions.

apply-nodes.js

var fs = require('fs'); var through2 = require('through2'); var split2 = require('split2'); var levelup = require('level'); var async = require('async');  var DATABASE_NAME = 'everystreet'; var INPUT_FILE = 'output/streets.txt'; var OUTPUT_FILE = 'output/streets-with-coordinates.txt';  console.log('Applying node data from database ' + DATABASE_NAME + ' to street data from file: ' + INPUT_FILE); levelup(DATABASE_NAME, function(err, db) {      var write_stream = fs.createWriteStream(OUTPUT_FILE);      fs.createReadStream(INPUT_FILE, { encoding: 'utf8' })         .pipe(split2())         .pipe(through2.obj(function(line, enc, next){             async.mapSeries(line.split(','), function(node_id, callback) {                 db.get(node_id, function(err, coords) {                     callback(err, coords);                 });             }, function(err, result) {                 this.push(result.join(',') + '/n');                 next();             }.bind(this));         }))         .pipe(write_stream)         .on('finish', function() {             console.log('Finished applying node data into file: ' + OUTPUT_FILE);         }); }); 

We’re streaming through each street in the data file, querying the database for the coordonates – using async.mapSeries to make sure we get back the node data in the correct order – and serializing them into a plain-text file.

streets-with-coordinates.js

44.469672200000005,26.093109000000002,44.469469600000004,26.093366600000003,... 44.46975080000001,26.092981700000003,44.4696756,26.092841000000004,... 

At this point we’re done with extracting all the data we need but we still need to convert it from geographical coordinates to screen coordinates. *Takes deep breath* . Onwards!

4. Mapping geographical coordintes to screen coordinates

There are many different ways to project the Earth’s surface onto 2D space. Many maps are laid out based on the spherical Mercator projection . Assuming λ is the longitude and φ is the latitude, both expressed in radians, the formula is simple:

{   type: 'way',   id: 108,   tags: {     created_by: 'Potlatch 0.8',     highway: 'living_street',     name: 'Kitzbühler Straße',     postal_code: '01217'    },   refs: [ 442752, 231712390, 442754 ],   info: {...} } 

0

Before we dive into it, let’s see what we need to do:

  1. Find the bounding box of all our coordinates and its aspect ratio;
  2. Transform the geographical coordinates into screen coordinates, based on the bounding box.

The bounding box of our map is the smallest rectangle that contains all the nodes. We can find it by identifying the minimum/maximum longitude and latitude in our dataset. If we transform the points that define the bounding box using the Mercator projection, we can also obtain our final map’s aspect ratio , which we’re going to use later. The script below computes both:

bbox.js

{   type: 'way',   id: 108,   tags: {     created_by: 'Potlatch 0.8',     highway: 'living_street',     name: 'Kitzbühler Straße',     postal_code: '01217'    },   refs: [ 442752, 231712390, 442754 ],   info: {...} } 

1

The script outputs the following information:

bbox.json

{   type: 'way',   id: 108,   tags: {     created_by: 'Potlatch 0.8',     highway: 'living_street',     name: 'Kitzbühler Straße',     postal_code: '01217'    },   refs: [ 442752, 231712390, 442754 ],   info: {...} } 

2

Let’s now take our geographical coordinates transform them using the Mercator projection; afterwards, we express them as percentages within the map’s bounding box which will make it easy for us to draw at any scale.

map-coordinates.js

{   type: 'way',   id: 108,   tags: {     created_by: 'Potlatch 0.8',     highway: 'living_street',     name: 'Kitzbühler Straße',     postal_code: '01217'    },   refs: [ 442752, 231712390, 442754 ],   info: {...} } 

3

All nodes should be now in the [0,1] range:

strets-with-coordinates-mapped.txt

{   type: 'way',   id: 108,   tags: {     created_by: 'Potlatch 0.8',     highway: 'living_street',     name: 'Kitzbühler Straße',     postal_code: '01217'    },   refs: [ 442752, 231712390, 442754 ],   info: {...} } 

4

5. Drawing the map in SVG

We now have everything we need to start drawing some SVG paths. This is the structure we’re aiming for:

{   type: 'way',   id: 108,   tags: {     created_by: 'Potlatch 0.8',     highway: 'living_street',     name: 'Kitzbühler Straße',     postal_code: '01217'    },   refs: [ 442752, 231712390, 442754 ],   info: {...} } 

5

Remember the map ratio we computed earlier? We can use that to derive the height of our map based on a width of our choice. And to transform our points from the [0,1] range to SVG-ready coordinates, we just need to factor in the map’s dimensions:

{   type: 'way',   id: 108,   tags: {     created_by: 'Potlatch 0.8',     highway: 'living_street',     name: 'Kitzbühler Straße',     postal_code: '01217'    },   refs: [ 442752, 231712390, 442754 ],   info: {...} } 

6

Note: The 1 - latitude is to account for the fact that the origin of SVG coordinates is at the top left corner while our coordinates assume an origin in the bottom left corner.

generate-svg.js

{   type: 'way',   id: 108,   tags: {     created_by: 'Potlatch 0.8',     highway: 'living_street',     name: 'Kitzbühler Straße',     postal_code: '01217'    },   refs: [ 442752, 231712390, 442754 ],   info: {...} } 

7

And voilá! We have our all streets in Romania drawn up in SVG, which you can open in your browser ( as opposed to other tools ).

Here it is in all its glory:streets.svg (Warning: 262MB file!)

Final thoughts

Making it printable. Loading or converting a 262MB SVG is no easy feat, but ImageMagick somehow miraculously created a wall-sized PNG image:

{   type: 'way',   id: 108,   tags: {     created_by: 'Potlatch 0.8',     highway: 'living_street',     name: 'Kitzbühler Straße',     postal_code: '01217'    },   refs: [ 442752, 231712390, 442754 ],   info: {...} } 

8

Another idea worth pursuing is making electron print out a PDF.

Optimizing the map. Taking into account the output resolution, one could simplify the paths with simplify.js to eliminate details without affecting the appearance (e.g. points that are very close together):

{   type: 'way',   id: 108,   tags: {     created_by: 'Potlatch 0.8',     highway: 'living_street',     name: 'Kitzbühler Straße',     postal_code: '01217'    },   refs: [ 442752, 231712390, 442754 ],   info: {...} } 

9

And considering the browser needs to build the DOM tree along with drawing the millions of paths in our file, one easy fix to have fewer DOM nodes is to batch the path data into multipaths of, say, a thousand paths:

  function isNode(item) {     return item.type === 'node';   }    function isStreet(item) {     return item.type === 'way' && item.tags.highway;   } 

0

I hope you’ve enjoyed this short foray into mapping! You can find all the scripts discussed here on Github: danburzo/every-street . If you have any idea on how to make this workflow better, I’d love to hear it !

原文  http://danburzo.ro/every-street/

转载本站任何文章请注明:转载至神刀安全网,谢谢神刀安全网 » Drawing every street in Romania

分享到:更多 ()

评论 抢沙发

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址
分享按钮