Talk to the City

Product overview

Talk to the City is an open-source LLM interface for improving collective deliberation and decision-making by analyzing detailed, qualitative data. It aggregates responses and arranges similar arguments into clusters.

What makes Talk to the City unique is the combination of the following features:

  • It leverages the power of frontier LLMs to analyze large qualitative datasets

  • It is designed to summarize human opinions, as opposed to just factual data

  • It automatically prepares summaries, visualizations and reports 

  • It is developed as a non-profit initiative 

Talk to the City’s data processing pipeline starts by processing a variety of data types, then uses LLMs to extract key arguments, and finally arranges similar arguments into clusters and subclusters. Users can navigate through a map of opinions and drill down to the subclusters they find most interesting. 

For more detail on our vision for Talk to the City and what we’ve learned so far, see our recent blog post: Talk to the City: an open-source AI tool for scaling deliberation

Background

The fast pace of technological development in the AI space is both a source of concern and a source of hope for democracy. On the pessimistic side, many are worried that our existing democratic processes and institutions are too slow and inefficient to address the many crises humanity is facing. The more we lose trust in the efficacy of our democratic processes, the more we lose the broad participation and agreement on legitimacy of outcomes that make democracy functional, which in turn may further damage the levels of trust that people have in democratic institutions. 

On the more optimistic side, however, there is hope that AI technologies may help us escape the spiral of declining trust by allowing us to consult the public in much larger numbers, at a much faster pace, and in much more inclusive and transparent ways that capture diversity and nuance of opinion. Making this a reality won’t be an easy task, and building sufficient public trust in AI technologies will also take some time, but now that a growing number of AI pioneers have been popularizing this idea (e.g. here and here),  we believe that, with the appropriate AI safety precautions, it is time to accelerate the design and development of open-source prototypes and begin testing them in the wild.

Research questions

We're building this tool as a proof of concept and testing it with a variety of datasets, including data extracted from twitter, blog posts, pol.is consultations, and video interviews.

Our primary goal with these first experimental deployments has been to improve our understanding of the risks and benefits of using LLMs in the context of democratic consultations, and to identify where LLMs are helpful in creating intuitive interfaces for complex datasets. In particular, we wanted to answer the following questions: 

  • Are modern LLMs already reliable enough to prove helpful? 

  • Which interfaces best help users understand large corpora of opinion data?

  • How can we mitigate natural safety and quality concerns inherent in the use of current LLMs? 

  • How can we collaborate with and bring value to the broader researcher community?

Current and Upcoming Experiments

WhatsApp Integration

A simple LLM-based WhatsApp bot to elicit responses will drastically reduce the barrier to entry:

  • Interactive, conversation-based elicitation can engage respondents in expressing more reflective responses, or guiding them back to core research questions

  • A bot can encourage responding with voice messages, further decreasing barriers to respondents expressing their perspectives

  • WhatsApp is usage is high in locations with limited or price-prohibitive internet access, allowing us to poll respondents who may otherwise be inaccessible

The WhatsApp bot can also ask about clusters that the respondent has not yet mentioned, and can be deployed in group conversations, where respondents can prompt each other. With the transparency and expectation setting that the conversation over the thread will be used to create an AI-driven report,) all participants can share their desires, preferences, and suggestions accordingly.

Smart Surveying

We can foster richer civic discourse by encouraging people to discuss topics overlooked by a majority of participants, but which minorities believe are important to the matter at hand. Refocusing on such topics can help create explicit common ground, or clearer articulation of disagreements:

  • Agreement: “I didn’t think of a road from A to B, but I agree that this would be a good use of the budget”

  • Disagreement: “I didn’t think of building a road from A to B, as it is not a good use of limited resources.”

Rather than leaving these overlooked topics to the discretion of the centralized entity, we can design processes that use LLMs to guide new submissions to touch on unaddressed topics, or reach out to previous submitted responses for further elucidation. This approach can help groups reach higher levels of epistemic coherence, by:

  • Giving all contributors the opportunity to react to each other’s topics

  • Enabling the community to reach mutual understanding

  • Creating an active space for consideration of opinions that would be explicitly disagreed with, rather than ignored.

Geographic and Demographic Metadata

Especially in cases where it is important to have an evergreen dataset, or a specific event that might influence the responses has taken place, it will be important to make sure incoming data is not skewed towards specific demographics. Future case studies will solicit and visualize the geographic and demographic breakdown of survey respondent populations, and of specific topics presented in Talk to the City reports.

It is possible that this implementation might introduce path dependencies or reinforce biases, however, we believe that encouraging discourse on clusters that have been identified by the AI while a significant portion of the submitters haven’t yet touched on the subject, is likely to be a net positive on the civic discourse health.