Invented by DESGARENNES; Gabriel A., DOLAN; William B., RAO; Sudha, BROCKETT; Christopher J., XU; Yun Hui, XU; Weijia, LOBB; Ken, Microsoft Technology Licensing, LLC

Understanding how digital agents think and act is the next big step in making video games, simulations, and smart apps feel alive. A new patent application introduces a clever way to give these agents more lifelike behaviors, all while using computer resources wisely. Let’s explore how this works, why it matters, and what makes this invention special.

Background and Market Context

For years, video games and simulations have used digital agents—like non-player characters (NPCs) or self-driving cars—to make virtual worlds more interesting and interactive. These agents make decisions, move around, and react to players or to each other. But getting them to act smartly and realistically has always been tricky.

Early agents were simple. Their behavior was hard-coded, meaning programmers wrote every possible action and reaction. This worked for basic tasks, but if anything in the game changed, developers had to go back and rewrite code. It was time-consuming and rigid.

As games and applications grew in size and complexity, there was a need for agents that could adapt on their own. The rise of machine learning, especially reinforcement learning, offered some hope. With reinforcement learning, agents could try things out, learn from rewards, and adjust their actions over time. This made them better at handling surprises, but still limited them to the situations their reward systems anticipated.

In recent years, generative language models like GPT and others have shown that AI can handle much more complicated scenarios, even those never seen before. These AIs can chat with people, summarize stories, and even pass professional exams. They learn by studying huge amounts of text and can answer all sorts of questions, making them powerful tools for agent behaviors. However, they have a big downside: they need lots of computer power and memory. Running them for every agent in a game or simulation just isn’t practical for most systems.

This disconnect created a challenge. Developers wanted agents that were both smart and efficient, but existing tools meant choosing one or the other. If you made all agents smart, the system slowed down. If you made them efficient, they acted predictably and sometimes unrealistically. There was a clear need for a new approach that could give the best of both worlds.

The market for more lifelike and adaptable agents is growing fast. Players expect smarter NPCs in games. Scientists and engineers want more realistic simulations for training or research. Smart apps need to respond to users in ways that feel natural. All these needs pointed toward a solution that could blend powerful AI models with lighter, faster ones, so every agent could act just right for its role.

This patent application answers that call. It introduces a way to organize agents in a hierarchy, assigning more advanced (and resource-heavy) AI models to high-level agents, while using simpler, faster models for those lower down. This lets software designers get the benefits of advanced AI where it matters most, but still keep everything running smoothly.

Scientific Rationale and Prior Art

To see why this new approach stands out, let’s look at what came before and the science behind agent behavior models.

The most basic agent model is hard-coded logic. Programmers decide in advance how an agent reacts in every situation. This works for simple games—think of a ghost in Pac-Man that always chases the player in a set way. But as soon as something unexpected happens, or the environment changes, these agents can’t adapt. Changing their behavior means rewriting and redeploying code.

Reinforcement learning improved things. Here, agents learn by trying different actions and getting rewarded for the good ones. Over time, they build up a set of policies—rules about what to do in different situations. Reinforcement learning is more flexible than hard-coded logic. It allows agents to learn from experience, and it doesn’t always need new code for every change. But these agents are still limited by their reward functions. If the designer didn’t think of a new situation or objective, the agent can’t handle it well. Also, reinforcement learning models usually have a limited set of actions and states, making them less effective in truly open-ended environments.

Generative language models are a recent breakthrough. Models like GPT, PaLM, and BLOOM are trained on massive collections of text and can generate human-like responses or make complex decisions. They are not limited to a small set of actions or states. Instead, they can handle new and unexpected inputs and produce creative solutions. They’re great for scenarios that require flexible thinking and adaptation.

But there are costs. These models are huge, sometimes with billions or trillions of parameters. Running them requires lots of memory and fast processors, often specialized chips like GPUs. They can also be slow to respond, making them less suitable for agents that need to react instantly. This makes it impractical to use generative models for every agent in a large-scale game or simulation.

Some systems tried to mix and match models, but usually only in a simple way. For example, a game might use a generative model for a main character and basic logic for everyone else. Or it might use reinforcement learning for all agents, regardless of their role. These approaches didn’t fully solve the resource and realism trade-off.

The science behind agent hierarchies borrows ideas from how organizations or animal groups work. In real life, leaders make strategic decisions, while lower-level members carry out simple tasks. By applying this idea to digital agents, we can use heavy-duty AI power only where it makes the biggest difference—at the top of the chain. Lower-level agents just need to follow orders or handle simple tasks, so they can use lightweight models.

Before this invention, there was no clear way to organize and coordinate agents using different kinds of behavior models based on their role and available resources. There was also no system for managing communication between these agents, summarizing information to save bandwidth, or dynamically switching models based on what’s happening in the environment.

This patent application brings all these ideas together. It explains how to:

– Assign behavior models to agents based on their place in a hierarchy and how much computer power they need.
– Set up the models with the right information about the environment, rules, and user preferences.
– Make sure agents talk to each other efficiently, summarizing details as information moves up the chain.
– Let agents control the application through standard interfaces, and even change their own model type if the situation calls for it.

In short, the invention fills a big gap in both the science and practice of agent-based systems, making it possible to build smarter, more efficient, and more lifelike agents in games, simulations, and apps.

Invention Description and Key Innovations

Now let’s break down how this invention works, using simple terms and examples.

Imagine a video game with a team of monsters: a “witch” at the top, two “ghosts” in the middle, and four “mummies” at the bottom. The witch gives orders to the ghosts, who manage the mummies. Players try to win prizes while these monsters try to stop them.

Each agent in this team gets a behavior model suited to its job:

– The witch uses a generative language model. She makes big decisions and needs to understand lots of details, so she gets the smartest AI.
– The ghosts use generative models too, but theirs might be smaller than the witch’s. They manage their teams and pass on summaries of what’s happening.
– The mummies use simple, hard-coded models. They just follow orders or chase players they see.

The system works like an organization. The mummies report what they see to their ghost leader. The ghosts condense this information into short summaries and send them to the witch. The witch doesn’t need every tiny detail, just the big picture. This way, the system saves on bandwidth and computer power. If every agent sent every detail to the top, it would be slow and wasteful.

This is more than just smart model assignment. The invention introduces several clever ideas:

1. Hierarchical Assignment Based on Resources and Role: The system assigns behavior models (generative, reinforcement learning, or hard-coded) based on where the agent sits in the hierarchy and how much computer power is available. High-level agents get smarter, more demanding models. Lower-level agents get lighter, faster ones.

2. Configurable Models with Contextual Prompts: Each agent’s model is set up with the right information—like the rules of the game, what actions are possible, and user preferences—using prompts. This helps the models know what to do and keeps their decisions on track.

3. Efficient Communication and Summarization: When agents communicate, lower levels send detailed information upwards, but each level summarizes it before passing it on. For example, instead of sending every mummy’s exact location, a ghost might just report, “Two mummies in the northeast sector, no players seen.” This saves time and computer resources while still giving leaders enough information to make decisions.

4. API Control and Message Parsing: The generative models can output messages that are parsed to identify specific instructions for other agents or application programming interface (API) calls. For example, the witch might say, “Ghost 1, defend the southwest sector. Ghost 2, scout the northwest.” The system splits these orders and sends them to the right ghosts.

5. Dynamic Model Upgrading: If something important happens—like players moving into a new area—the system can “promote” an agent to use a smarter, more powerful model. For example, if a mummy finds a player near a big prize, the system can temporarily give it a generative model to handle the situation better.

6. User Feedback Integration: The system can take feedback from players (like ratings, quitting early, or comments) and use it to adjust agent behavior. If the game is too hard, it can make things easier. If it’s too easy, it can add challenges. This keeps the experience fun and fair.

7. Multi-Modal Support: The invention supports agents that can use not just text, but also images, sounds, or data structures as input and output. This means agents can “see” and “hear” the environment, making them even smarter.

8. Flexible Deployment Across Devices: The system can run agents on different types of devices—some local (like a game console) and some remote (like cloud servers). It can choose where to run each model for the best mix of speed and intelligence.

In practice, this setup leads to more lifelike and fair experiences. Players can’t accuse the game of “cheating” because high-level agents only know what their subordinates report. The system feels more natural and responsive, and it runs efficiently even as the number of agents grows.

This invention is not limited to games. It can be used in training simulations (like self-driving car tests), smart apps, or any digital world where agents need to act and react in believable ways. By mixing powerful and lightweight models in a smart, organized way, it offers a blueprint for the next generation of digital agents.

Conclusion

The invention described in this patent application represents a big step forward in making digital agents smarter, faster, and more lifelike. By organizing agents into a hierarchy and giving each the right kind of behavior model, it brings together the best of both worlds: the creativity and adaptability of advanced AI, and the speed and efficiency of simpler models.

This approach doesn’t just improve games—it opens the door for better simulations, smarter apps, and more engaging virtual environments. For developers and companies, it means creating richer experiences without overwhelming their systems. For users, it means a more natural, fair, and fun interaction with digital worlds.

As AI continues to evolve, ideas like this will be key to making digital experiences feel truly alive. If you are developing next-generation games, simulations, or smart applications, understanding and using these concepts will help you stay ahead of the curve.

Click here https://ppubs.uspto.gov/pubwebapp/ and search 20250217194.