Just be VERY careful when using any of these open data sets. Datasets will undoubtedly be filtered by some unwritten criteria. It's best to use these datasets as a sanity check for some things, with follow-up FOIA requests for the serious work. And even that will be filled with similar filtering. Source: spent the last ten years working with open data and FOIA in Chicago.
Also, here's a tweet-fitting one liner I wrote that downloads all Socrata CSV datasets across all gov instances:
It's not just the filtering. It is also that many datasets are full of errors. I wrote a data analytics tool that I would test out on a bunch of open data sets like this one. One of my favorites was the Chicago crime data. The tool could quickly find (and fix) a bunch of errors. I tried contacting the people who published the crime data and even offered to show them how to use my free tool to clean up the data. They could not have cared less that their data set had large numbers of errors in it.
Each crime report would specify (among other things) what crime was reported and what city block it had occurred. Using my tool, it was easy to analyze specific crimes over time for specific areas. One of the problems was their database would have multiple spellings for the same location which made it much harder to do the analysis since you had to account for that.
This is a common problem, especially for "crime" datasets and very emblematic of the kind of data capturing processes for each reported "crime" (really, they're some of the worst datasets because of the extent of imperfect information and incentives to capture some "crimes" some vs others). Send it through a geocoder like geocodio to resolve most of that problem. It won't be perfect.
This strikes me as a Wikipedia Article type caveat. Sure the warning applies to certain categories of article/data, namely the ones where people are jockeying to have the authority of an encyclopaedia / data to reinforce their political stance, but on any sort of non political topic neutrality is a non issue. I fully anticipate that say, police and crime data by demographic shouldn't be trusted. Other data, like say pounds of recycled garbage collected per census tract, is probably fine.
I mean, sure? I'm very much saying this for police datasets, where for example, a "crime" database might not include homicide information.
And no, I don't think it's fair to say that the amount of pounds of recycled garbage collected per census tract would be remotely accurate. Too many private companies with private incentives. Just a taste, for example: https://www.wbez.org/stories/bga-investigation-reveals-major...
What FOIA request methodologies would you recommend to ensure you’re getting the most comprehensive data possible out of an agency or government body response?
One of the most productive things I've seen is to request for "data sufficient to show" rather than describing the data you're looking for. That way you have a bit more wiggle room in the space of interpretation. Another way is to just look up forms ("filetype:pdf site:yourmunicipality.gov" in google works wonders) that are online -- if filled out reports have text filled in by a computer, then that information was likely filled into a database, and the form you're looking at is a product of a multitude of SQL queries. In Illinois, Hites v. Waubonese compels an agency to run a SQL query if data is represented in a database, so being explicit about the existence of that law can be helpful. I'm sure similar can be said about other states. Also, be sure to explicitly request your data in an excel compatible format (some states legally require it).
Just curious: for a municipality with limited resources and very minimal understanding of data governance, why should a city government follow in the footsteps of Boulder?
Do the missteps of Boulder make the case for other municipalities to not pursue open data in a similar way, given FOIA is sufficient?
My local city government also recently published a detailed interactive online GIS map of the city, including all lot boundaries, buried utilities, etc. It's great to feel like the public can easily access what is meant to be public record, rather than relying on for-profit third parties to make it available to us.
I really like the ESRI API's (much faster than Socrata). This is a crazy url, but they do allow for different SQL operations (including spatial operations) right in the API. Here is top 10 crime street segments in Raleigh for example:
I live in Boulder. It's certainly expensive, although more reasonable than some of the other popular tech cities. I honestly don't think of it as crowded though. There's tourist spots to avoid, but plenty of open space for all.
I found it crowded. I guess it depends on where you're coming from.
On weekends, It takes 3 hours to make (what should be) a 1 hour drive at 5AM going to the mountain to ski, very difficult to find parking at most trailheads, camping spots are pretty much swamped on the peak to peak...etc. etc.
But I guess if you're coming from L.A. or somewhere that kind of traffic isn't too bad lol
And that's not all Boulder traffic, I'm sure there's a lot of Denver and surrounding city population crush, was just miserable on the weekends I found.
I don't know why our city council and planning boards waste so much time on "affordable housing" when it's clearly not fixable.
The economics don't make sense to have a highly desirable place to live with a highly constrained housing supply and also make it affordable. The affordable housing plans here basically boil down to a lottery.
CAGR is misleading because it tracks population when what is more important is household growth. Household size has fallen from 2.63 in 1993 to 2.51 today, which doesn’t sound like a lot, but means you need 5% more houses to hold the same number of people.
Yes, and there is a general misunderstanding, mostly a willful misunderstanding among older, established homeowners, of the fact that you need to build more housing just to keep the population the same, especially when older people are squatting on empty family-sized homes.
I lived there for 6 years and halfheartedly agree. It's a very nice town in terms of general quality of life - definitely the best I've ever experienced, and I've lived all over America. But there's a pretentiousness in the air that can be utterly nauseating at times. The "Boulder bubble" is real.
I enjoy some of the pretentiousness, namely the physical fitness and cycling fanaticism. There are some really, really tremendous athletes there.
But Boulder lost me at the rock worshipping yogis that appropriate "brown people cultures" into their daily routines while chastising white people who celebrate Cinco de Mayo. They're fucking everywhere.
It's pretty, and it would be nice to be that close to the mountains, but I can't stand the political climate in Boulder. Denver is fairly blue but still diverse enough that I feel comfortable here even though I don't personally share a lot of those views. Boulder, though... it's like people are trying to build the Berkeley of Colorado there. Which is fine, live and let live. But I would never want to live there.
I don't know, but based on this entire thread, I think this was one of the places that was popular for people to move from cities during the pandemic, when everyone was working remotely.
Its fascinating to me that for decades people have been moving to the same cities from places like Boulder (and many such all over the country) for opportunities/careers/lives and now a few chose to move from the cities to these small towns. While I understand that people do not like change but how is this any different, its still people making a choice of moving to a different place in the country which they think is better for their lives? Isn't this NIMBY, its fine as long as everyone moves to NY, but not here.
Boulder is more accurately described as a neighborhood or region of Denver.
If that’s not obvious today it certainly will be when the staggering construction in Arvada, Broomfield, and so on, has filled in all of the areas that could plausibly be used to argue otherwise.
Part of the reason it's so expensive here, though, is the swath of city and county owned open space around the outskirts of town. Maybe some day it'll get developed, but I think it'll be a long time before that can happen.
Boulder probably has enough of a unique character that thinking of it as its own thing rather than just a commuter suburb is probably reasonable. But there are certainly places within 25+ miles of Boston, say, that are reasonably described as suburbs and probably exurbs out to about 50 miles. (It's how ESRI would generally describe them.)
It sort of depends. To use another Boston area example, no one would reasonably consider Portsmouth or Nashua in NH as suburbs even though people will go into Boston for a night's entertainment and probably fly out of the airport.
From my limited perspective there isn't much space.
I'm a bit of a stranger to the area- right now I am in a Starbucks in (I think?) Boulder. But I came out to work as a tech with an audio company I contract for to do concerts up at some ski resort. And I am sleeping in my truck front of the Warehouse in Broomfield. But the guy who hires me has a house in Boulder where he keeps a lot of our equipment... and I am often not super clear about when I am just crossing a large green space and when I am crossing into different cities.
So you are probably right, but that's not how I've been experiencing this over the last summer's worth of gigs.
Of course, to me the drive from Boulder to FoCo seems like one continuous city.
Contextually, it might be the fact that I come out for work from Durango, and crossing the san luis valley is what "open space" looks like to me.
So maybe I'm just not sufficiently calibrated for the city yet.
So…? Arlington and Fairfax are different counties than DC. Heck, they’re in a different state and across a river too. But nobody would argue they aren’t suburbs of DC.
25 miles is well within commuting distance for many major metro areas.
Boulder may arguably be a suburb or not. But an arbitrary 25 mile limit isn’t what defines suburb or not.
Brooklyn and Manhattan are in different counties too.
Ask yourself questions like what airport do you fly to and where would you take your kids to the children’s museum if you lived there and you’ll realize what city Boulder is a part of.
>Brooklyn and Manhattan are in different counties too.
You're going to use the most densely populated city in the US for your example?
>Ask yourself questions like what airport do you fly to
Colorado has one international airport - are you saying ALL of Colorado is Denver? Are you saying Newark NJ is in the same city as Brooklyn? I don't know where you're heading, but it's just logical gymnastics.
My old boss when I worked in Boulder (commuting from Arvada) claimed he hadn't been to Denver proper in at least a decade, not counting DIA or the interstate to it. I believe him. Boulder is definitely its own thing.
Also, here's a tweet-fitting one liner I wrote that downloads all Socrata CSV datasets across all gov instances: