Reading 800,000 comments about housing in Massachusetts

By Jeff Reichman

Housing • July 13, 2026

This is a story about using AI carefully. We tested it, caught what it got wrong, and used it to go deeper in our research. This project wouldn’t happen without AI, but that doesn’t make it a model for every consulting project. I’ve been candid about my optimism and anxiety about this technology. That hasn’t changed. There were enough bumps along the way to underscore that disclosure matters more than ever.

qq – how do ppl talk about housing?

Late last year I heard from Jennifer Gilbert, a friend, client, and housing advocate. Jennifer was in a new role at CHAPA, a nonprofit that has worked on affordable housing in Massachusetts, and she came to us with an idea.

She wanted to understand how people across Massachusetts were actually talking about housing, and especially about the MBTA Communities Act. That law asks cities and towns near transit to allow more homes. CHAPA was a big advocate to get it passed. Jennifer wanted to know how people talked about this law online. We called it a “social listening” project.

Testing the idea before scoping it

When Jennifer described the idea, I didn’t even know if it was possible. What if we expected all sorts of interesting results, but no one was talking about housing? I didn’t want to write a scope of work for a study that failed to launch. I wanted an idea of what we were going to find. So I built an agent and pointed it at the question. I treated it as a proof of concept.

In an hour, I learned that it was possible. The agent had pulled together more than 7,000 Reddit posts and comments, and 73 news articles from 16 news outlets. That was enough for a first real look, and it turned up some interesting signals: coverage of the MBTA Communities Act in the news and from officials was strongly positive, but the conversation on Reddit was far more mixed. There was a gap.

I’ve done this kind of data collection work before. I’ve written data scrapers and performed sentiment analysis. But this time, the agent made it faster and more comprehensive. It found data sources I didn’t know about. It helped me understand what we could collect, what we could rely on, and ways we could parse the data. It was so easy to go farther and do another round of research, just to see if we could make the results a little bit better.

What we collected and what we did with it

Most of that conversation isn’t about the MBTA Communities Act. Most of it isn’t about any law at all. So we sorted the corpus into three tiers, by how close each item sat to the Act itself:

Tier 3 dwarfs the other two. Most of what people say about housing is personal and immediate: a rent increase, a landlord, an eviction notice, a voucher that won’t stretch. Keeping the tiers apart mattered, because the policy conversation and the everyday one answer different questions, and you can’t read one as if it were the other. 

That split is what turned one engagement into two deliverables: Jennifer used the tiered research for two separate CHAPA reports, one for policy debate and one for the lived-experience conversation. You can find both reports on CHAPA’s website.

Knowing what to keep and what to scrap

We approached this engagement with a lot of optimism and a healthy dose of skepticism about AI’s accuracy. After all, when AI is wrong, it’s often confidently wrong. So we questioned everything.

This was exceptionally valuable, because as we dug deeper, we found some flaws. After auditing the first results, we learned where it was including too much content in some places, and too little in others, so we adapted our collection and recalibrated the results. 

There were two big areas that had a meaningful impact on the results.

The first was conversation in the wrong geography. Our first pass pulled in communities that share a name with a Massachusetts town but sit somewhere else: Salem, Oregon. Plymouth, England. And so on. Ultimately, we audited and removed about 31,000 items from communities we couldn’t confirm were in Massachusetts, and added several that we missed the first time around. What remained came from 62 subreddits that we checked, with our human eyes and human hands, one by one.

The second was fake citations. We wanted to know how people use citations to back up their assertions. In other words, what sources do people link to when they talk about housing? But the initial counts were inflated by things that look like citations and aren’t: automated bot posts, a weather widget embedded in a recurring thread, image-preview links, even a script people post when they delete their account. We removed about 13,000 of these “fake” citations before the 30,000 remaining “real” citations could be trusted.

Whatever survived the cleaning, we analyzed in ways a person can check:

• We tagged frames like “missing middle,” “local control,” and “developer greed” by matching words and phrases, so every tagged item traces back to the exact term that tagged it.
• We scored sentiment with VADER, a published, rule-based lexicon. The same text always gets the same score. This has known limitations, so we only looked at sentiment in the aggregate to determine trends.
• Where we used AI to read text, it only flagged relevant passages for a person to judge and review.

Most of all, we made sure that every number in the final report traces to a row in a database. And every line of code is audited and replicable.

What we did with the time saved

At the beginning of this project, there was a lot of general research and coding needed to build the database. Writing scrapers. Collecting data. Writing the scripts for initial analysis and cleaning. This can be both frustrating and deeply satisfying work, but here it just wasn’t necessary. It’s exactly what the agent can do well. So it saved us a lot of time, and we spent that time on more work.

We spent it with Jennifer. We met every week, sometimes more often. Each meeting looked at what the latest round of analysis had found, and each one shaped what we did next. For an hour, we talked like human beings and brainstormed ideas, then I would synthesize my notes and go a few rounds with the agent. It kept the project moving fast and allowed us to quickly dig deeper and explore the unexpected.

We spent it on quality assurance. That was slow and unglamorous. Some early ideas had us excited about new results, but they turned out to be false alarms. Yet with every small disappointment came more clarity about what we were seeing and more confidence in the work.

We spent it on research cycles, and this is the part that surprised me most. You’re working on a topic, and you learn something that sparks a whole line of thinking. Big questions like, does immigration come up in Massachusetts housing arguments? (It didn’t.)  

These questions used to take a lot of effort to answer, if we even followed up at all. With the agent, it takes a few minutes to get preliminary results. So we asked more questions. Parking. Luxury housing. The way “missing middle” can be a rallying cry in one thread and a complaint in the next. None of it was in the scope of work. All of it came from being able to follow a thread right after it came up in conversation.

And we spent it back on the agent itself. A force multiplier still needs a hand on it. You have to know where to press, how to interrogate a source, when a line of inquiry is done. It took time to make an agent that produced work that we could trust.

So the agent takes over some of the labor and saves us time. We spend what it saves on more client meetings, quality assurance, refining our questions, doing more work, and making the agent better.

How we think about AI in our work

This has shifted how we think about AI in our work. When we can center it in our work with the client, we’ve found it valuable as a way to do more: ask more questions, talk to the client more often, chase the idea that wasn’t in the scope. It amplifies the skills we already have. But it doesn’t replace our judgment, and it isn’t magic. We are still responsible for the outcomes. 

Here are a few of the lessons we learned along the way:

• Speed creates room for follow-up. Yes, technical tasks were more efficient, but we spent that time elsewhere: checking work, talking with experts, meeting with Jennifer. We asked more questions that we never could have imagined up front.
  
• AI doesn’t check itself. While the agent found massive troves of data, it didn’t do a good job at validation. Every efficiency gain came with a corresponding need for more scrutiny. We couldn’t automate away the need for critical thinking and frequent human reviews.
  
• Time saved on labor gets reinvested in the relationship. We met with Jennifer more often, for longer, and let each conversation reshape the next round of analysis.
  
• There is a new craft emerging: knowing where to press, how to interrogate a source, and when a line of inquiry is actually done. We spent a lot of time just making the agent produce work we could trust. 

It’s not for every project. But when used carefully, AI lets our small team do ambitious work, and in turn, we can be good thought partners to people who know their world better than any model ever will.