NPC Behavior: State Machines, Behavior Trees, GOAP & AI-Driven NPCs

NPC behavior is the logic that turns a model, a health bar, and a collision capsule into something that appears to want things. It answers one question on a loop, usually every frame or on a fixed tick: given what I know about the world, what should I do next? Everything else on this page — finite state machines, behavior trees, GOAP, utility AI, LLM-driven intent — is a different answer to that question, with different costs and different failure modes.

We build NPCs that run in a browser tab inside Neta Studio's 3D game builder. That constraint shapes the advice below. There is no dedicated server, no separate physics thread, and no budget for a planner that searches for 40 milliseconds every tick. If a technique cannot be made cheap enough to run next to rendering on a mid-range laptop, we either downgrade it or move it off the hot path. The sections that follow describe what we actually ship, and where each approach breaks.

Quick Answer: How an NPC Decides

An NPC decides by running a decision function on a schedule, reading a small slice of world state (distance to target, line of sight, current health, last known position), and returning an action the movement and animation systems can execute. The schedule matters as much as the logic. Decisions on a 100 ms to 250 ms tick, with movement and collision every frame, cost a fraction of what per-frame decision logic costs and are indistinguishable to a player in most cases.

The four classic models differ in where the complexity lives. A state machine encodes decisions as transitions between named states. A behavior tree encodes them as a hierarchy of selector and sequence nodes. GOAP encodes them as a goal plus a set of actions with preconditions and effects, and searches for a plan. Utility AI encodes them as a scoring function over every candidate action. LLM-driven NPCs encode high-level intent in natural language and hand the result back to one of the cheaper models.

In practice you combine them. A utility scorer picks the goal, a behavior tree or state machine executes it, a navmesh moves the body, and the language model only touches dialogue and intent. The rest of this page takes each layer in turn and names the specific failure mode you will hit.

State Machines (FSM)

A finite state machine is the cheapest useful model and the right first choice for most NPCs. You define a set of states — Idle, Patrol, Chase, Attack, Flee, Stunned — and a set of transition conditions between them. Each tick you evaluate the transitions out of the current state and, if one fires, switch. Cost is proportional to the number of transitions you check, which is small. State is a single variable, which makes it trivial to serialize, debug, and display in an editor overlay.

The failure mode is oscillation. An NPC at 5.1 meters from the player transitions from Patrol to Chase, moves to 4.9 meters, and a symmetric transition sends it back to Patrol. It vibrates at the boundary forever. The fix is hysteresis: enter Chase at 5.0 meters and leave it at 6.5 meters. The same problem appears with line-of-sight checks that flicker as the player strafes behind a pillar, and the same fix applies — require the condition to hold for N consecutive ticks, or add a memory timer that keeps the NPC in Chase for two seconds after the last sighting.

The second failure mode is transition tangle. Five states need ten transitions; ten states need closer to forty-five. Each new state must be wired to every state it can be reached from, and the conditions overlap in ways that are hard to reason about. When your transition table stops fitting on one screen, that is the signal to move to a behavior tree.

A concrete FSM for a patrol guard: Idle (wait 2 s) → Patrol (walk to next waypoint) → Alert (turn toward noise, 1.5 s) → Chase (move to last known position at run speed) → Attack (if within 2 m and cooldown expired) → Search (wander near last known position for 8 s) → Patrol. Seven states, roughly a dozen transitions, and the memory timer in Search is what makes it feel deliberate rather than jittery.

Behavior Trees

A behavior tree replaces the transition table with a hierarchy. Leaf nodes are actions (MoveTo, PlayAnimation, Wait) and conditions (IsPlayerVisible, HealthBelow). Two composite node types do most of the work. A selector tries its children in order and returns success as soon as one succeeds — this is your priority list. A sequence runs its children in order and fails as soon as one fails — this is your step-by-step procedure.

The scaling win is composability. Adding a new behavior means adding a subtree under a selector, not wiring a new state to every state it can be reached from. Priorities are explicit in child order, which is far easier to read than a set of overlapping boolean conditions. Sub-trees can be reused: the same 'investigate noise' branch works for a guard, a dog, and a civilian.

Trees are not automatically faster. A naive implementation re-evaluates the whole tree every time it is ticked, which is why you keep ticking on the 100–250 ms decision schedule and let the currently running action drive movement every frame. Long-running actions need a running state that the tree remembers, otherwise an action restarts from scratch each tick and the NPC slides back to the start of its path. Most engines that ship a behavior tree — Unity's Behavior package, Unreal's built-in tree, or a small custom one in TypeScript — handle this with a running return value, but it is the thing to verify first when an NPC stutters.

A selector structure for a combat NPC, in priority order: (1) if health is below 25 percent, sequence [Flee to cover, Heal]; (2) if player is visible and within 2 m, sequence [Face player, Attack]; (3) if player is visible, sequence [MoveTo strafe position, Attack]; (4) if last known position is set, Search; (5) Patrol. The selector picks the first branch whose condition passes. Reordering those children is how you tune aggression without touching code.

GOAP (Goal-Oriented Action Planning)

GOAP flips the model. Instead of describing when to transition, you describe what actions do and what they require, and a planner searches for a sequence that reaches a goal. Each action has preconditions, effects, and a cost. 'Attack' requires HasWeapon and InRange, and sets TargetDead. 'GetWeapon' requires WeaponAvailable and sets HasWeapon. 'MoveToTarget' requires TargetKnown and sets InRange. The planner starts from the goal, works backwards, and finds the cheapest chain of actions whose effects satisfy it.

This pays off when the world state is rich and the correct sequence is not obvious. An NPC that needs to break line of sight, reload, reposition, and then engage produces a plan the designer never explicitly wrote, because the planner discovered the ordering from the precondition graph. It also pays off in production because adding a new item or ability means adding one action definition, not editing every state machine that should know about it.

The cost is search. A naive planner explores action combinations until it finds a plan, which can blow up as you add actions and state variables. The standard mitigation is a heuristic — A* over the state graph with a distance estimate to the goal — plus a strict expansion limit. In our own browser contexts, plans are computed on the decision tick with a cap of a few hundred node expansions, and anything that exceeds it falls back to a cached plan or a simple behavior tree branch. The honest limit is this: GOAP is worth the search cost when the number of meaningful action sequences is large, and wasted overhead when the NPC only ever does three things.

Utility AI

Utility AI scores every option and picks the highest. Each candidate action has a function that maps world state to a number, usually normalized to zero through one. 'Flee' scores high when health is low and enemies are near. 'Attack' scores high when the target is in range and ammo is above zero. 'Wander' scores moderately when nothing else is pressing. The NPC takes the argmax, optionally with a small random jitter so ties resolve unpredictably.

This is the model that produces the most surprising-looking characters, because behavior emerges from the interaction of curves rather than from explicit rules. Two NPCs with slightly different scoring weights behave noticeably differently without any branching code, which makes it the natural fit for personality variation. It is also the easiest model to tune at runtime: expose the weights in a debug panel and drag them while the game runs.

The failure mode is flat or noisy utility. If every action scores 0.4, the NPC looks indecisive and switches constantly. If the highest score flips every tick between two close values, you get the same oscillation as a bad state machine. The standard fix is a commitment window: once an action is chosen, minimum-commit for a fixed duration — 500 ms is a reasonable default — unless the highest score exceeds the committed action's score by a meaningful margin. Curves also matter. Linear response curves on health produce an NPC that flees at exactly the same moment every fight. Exponential or logistic curves produce thresholds that move with the situation.

Navigation & Collision

Decision logic picks a destination. Navigation gets the body there. A navmesh is a triangulated walkable surface; pathfinding runs A* over its polygon graph, and the result is a list of polygon edges the NPC follows. We build ours at load time from the level geometry and cache it, because rebuilding on every frame is the single easiest way to tank frame rate in a browser game.

Steering handles the last few meters and the moment-to-moment movement. Seek and arrive produce the forward vector, separation pushes the NPC away from nearby agents, and obstacle avoidance bends the path around dynamic blockers like a door or another character. Run steering every frame; run pathfinding only when the destination changes or the path is invalidated. A path recomputed every frame for every NPC is a common and avoidable cost.

Collision is the pass that stops an NPC walking through a wall. The navmesh keeps the path on walkable ground, but steering and physics can still push a character into geometry, so the character controller resolves penetration against colliders after movement. In our isometric worlds — Mini World and The Cyclops' Island — this is where most visible NPC bugs come from: a path that threads a gap narrower than the character radius, or a steering vector that pushes an agent into a wall at a corner. Setting agent radius correctly on the navmesh, and keeping separation force below the controller's push-out strength, resolves the majority of it.

LLM-Driven NPCs

A language model changes the picture at the intent and dialogue layer, not the navigation layer. Pathfinding, steering, and collision remain exactly what they were, because a model that takes hundreds of milliseconds to respond cannot drive a body that needs to move every frame. What the model can do is decide which goal makes sense given a player's words, generate response text in character, and choose between high-level options when the situation is ambiguous and the hand-authored rules would produce something flat.

A workable architecture separates the two speeds. Immediate reactions run on the state machine or utility scorer — attack, flee, take cover — because they need to happen inside a frame or two. Deliberate intent runs through the model on a much slower cadence, maybe once per interaction or every few seconds, and its output is a structured command the fast layer executes: set goal to Follow, set attitude toward player to Hostile, set dialogue line. The model never touches position directly.

The honest limits are latency, cost, and determinism. Response times are long enough that the NPC must acknowledge the input immediately with a gesture or a short canned line, or it reads as broken. Cost scales with how often you call, so you batch and cache. And because output is non-deterministic, you must validate it against a schema before it reaches the game — a free-text response that fails to parse should fall back to a default line, not crash the dialogue system. Treat the model as an unreliable service behind a typed interface, and the integration holds up.

Walk into the world it belongs to

A sprite is a character with nowhere to stand. The last step is the one nobody else offers: take the sheet you just built and put it in a world someone can walk around in — a small island, a planet, a rainy street corner. Every world on this site was made from one written description and published as a page you can play.