Requisite Variety: The How
We’ve explained the why of Requisite Variety. This post focuses on the how.
Requisite Variety is a GitHub repository. It is built on Quarto and manifests as a static site hosted through GitHub Pages. We’ve created Claude and Codex skills that allow researchers to turn source material (e.g., blog posts, papers, “Iotas” entries, or other site updates) into live web pages or elements. For posts, we’ve developed skills that automatically assess legal implications as well as potential communication issues or opportunities for improvement (one of our skills explicitly pushes human-written posts through Pangram to provide some confidence that the post is, in fact, human written!).
Users do not need to know much about Quarto to contribute. Our skills handle layout and design as well as GitHub mechanics. This allows users to focus on the content to be published, what source material it is based on, and who should review it.
Our skills also help users optionally install a local server so that they can preview how content looks before sending it along for comms and legal review. AI reviewer agents perform an initial screen for legal and communication issues, and human experts review flagged and sensitive material before anything goes live.
AI Reporter
The AI Reporter has been taken offline while TRI evaluates its policy on AI-assisted writing.
If Requisite Variety only hosted written content, we could’ve used a third-party platform like Substack. But we want this site to be a “conversation with materials”, where the material here is web-based interactive technology. As such, we firmly believe that variety means variety of media. After all, if the medium is the message, and our goal is variety, then it stands to reason that we need a variety of media!
We deliver this variety through a set of agents that run regularly to update the site. One such agent is AI Reporter, which every week gathers candidate research material from configured sources, chooses a human-AI topic, writes a short article or Jupyter Notebook-style post, creates supporting media when needed, and formats the result as a Quarto page.
Importantly, the AI Reporter makes its own decisions about what to write or build. It also cycles through different models to use to do the writing or code building. Variety at every turn!
We are not convinced that the pieces AI Reporter produces will always be good (in fact, we suspect the hit rate to be pretty low), but its posts will give us insight into how well different LLMs can choose topics, build arguments, and communicate a point-of-view.
AI Artist
“I think of art, at its most significant, as a DEW line, a Distant Early Warning system that can always be relied on to tell the old culture what is beginning to happen to it.”
— M. McLuhan
“Art is how a culture feels the future before it can explain it.”
— Codex 5.5 extra high
Ted Chiang says that AI isn’t going to make art. AI Artist puts that thesis to the test.
AI Artist is an agent that generates a new piece of dynamic web art on a daily cadence. It reads a creative guideline, asks an LLM to produce a self-contained HTML/JS artwork, archives the result by date, and updates the current published artwork. The agent can select from a variety of tools to generate art, including different LLMs and selected media creation models available through platforms such as fal.ai. Its harness prompts it to consider different styles of art. It also explicitly tries to combine conventions and styles or generate entirely novel styles on its own. (It creates pieces in a portrait orientation so that they can be shown on a display in the TRI lab space with a similar orientation.)
The agent also keeps track of what it has made in the past so that it doesn’t get repetitive. Human curators regularly push particularly impressive creations to a Hall of Fame list; each Wednesday the AI Artist serves up a Hall of Fame entry, advancing one slot through the list each week, rather than creating a new piece.
Artworks generated by AI Artist and selected by human curators for the Hall of Fame.
In his piece, Chiang writes that, “An artist—whether working digitally or with paint—implicitly makes far more decisions during the process of making a painting than would fit into a text prompt of a few hundred words.” We agree, which is why the “text prompt” is only a small aspect of AI Artist: the agent also reasons through stylistic choices and experiments with dynamic DOM elements, video, image editing, and other media.
Another common critique of AI-based creative tools is that they replace work that humans would have done, de-skilling people and reducing their agency. But that critique is zero sum and not relevant: The fact that AI Reporter creates a new post does not prevent humans from writing their own posts (like this one!). AI Artist is on even stronger ground here: no one in our group (indeed, very few people ever) has been doing this kind of experimentation with dynamic, web-based art.
AI Critics
What gets measured gets better! Separate AI critics evaluate both AI Reporter and AI Artist and write feedback to files that both agents read when creating new content. Look for clickable scales of justice (⚖️) icons to see the critic’s feedback. Of course, measurement has its problems, so the agents take the critics’ feedback with a grain of salt.
Benchmarking Tool
We are still developing Diplomacy, but we have released benchmarks on a related set of behavioral games.
Model benchmarks are vital information for any user-facing system builder. They also guide model development.
Our AI benchmarking tool is another independent agent. It gathers publicly available results from regularly updated benchmark leaderboards that researchers on our team have found relevant to human-centered concerns. The tool normalizes the top results into a shared format, merges new results with the previous data, and writes the updated benchmark data file back into the Requisite Variety repo.
So far, all benchmarks are third-party, but we are hard at work building our own benchmarks to better understand how best to integrate AI into our workaday tasks.
The first of these will be a Diplomacy benchmark. Diplomacy is a game of strategy and negotiation. It is often used in studies of human behavior to study bounded rationality, cognitive limitations under time pressure, and trust dynamics. Diplomacy rewards negotiation and alliance-building as much as tactics, which makes it a way to observe agent behavior. We plan to score several capabilities, including strategy, social inference, negotiation, coordination, resilience, and execution, from replayed critical game states, and to publish the formula, uncertainty, and source evidence behind every number. Look for this and other benchmarks in the coming months!