Create a service and API from a dataset

About a month ago, I put up a repo with data and code that takes National Centers for Environmental Information (NCEI) data (formerly the National Climatic Data Center AKA NCDC), parses it, puts in a database and serves it out via a node app. Said another way: using text files from a NOAA FTP server, it creates an API. Note that this is different from the ThreadEx data stuff I did earlier this year. I like temperature datasets (and there are plenty).

To see how it all works, start by looking at the scripts section in package.json to see how the data is parsed into something more useful and then loaded into Mongo. From there, peek into app.js to see the express app that serves out the data and exposes a REST-like API to pull out temperature avearages based on latitude, longitude and date. Thanks to Mongo's geoNear operator, this is cake. That's about it. Deploy to a cheap VPS and anyone you tell about the service can use it.

The hardest part of this whole thing is the couple of hours or days it takes to wrangle the data. Perseverance is probably the most constrained resource in this process. We've got free data, open code and cheap hardware. No matter what anyone says about framework fatigue, boilerplate fatigue or fatigue fatigue, it's a great time to be working on and with the web.