If you want to find every dark corner where an old online game can surprise you, do not send one party into a dungeon. Send twenty-eight parties into seven different dungeons at three in the morning, and leave them alone for five hours.
That was our push today on the test realm. The closed beta is running out in the open world, but behind the curtain we are trying to get every dungeon on the board running reliably with AI squads: from the early call-up runs to Frostcovered City, the three acts of Twilight Temple, Gui Mu, Lunar Glade, and the twin level-99 dungeons of Seat of Torment and Abaddon.
When six players walk into an instance and something goes wrong, someone laughs or swears in party chat, adjustments are made, and life moves on. When six autonomous characters hit an edge case, they follow their instructions with absolute, terrifying precision until the server runs out of memory or everyone dies ninety times.
Here is what the logs gave us today.
Twelve rooms with five coats on chairs
Before we could fix what the parties were doing inside the instances, we had to solve why the test realm kept grinding to a halt before they even made it through the doors.
We have rotating farm squads that swap characters between duties so test rosters stay varied. Each squad definition had 24 candidate accounts registered to it. But in this game, an instance party has a strict limit: six characters. Only six people ever log in and step through the portal together.
Our bot host supervisor, however, did not know that. When a farm task launched, it reserved seats for all 24 registered accounts on the worker process. The worker dutifully allocated network buffers, tracking tables, and process heap for 24 accounts, even though 18 of them never logged in. It was the digital equivalent of one person walking into a busy tavern, throwing 24 coats over empty chairs, and telling everyone else the room was full.
The result was absurd: workers were chewing through memory while running an average of only five active bots per worker. Across the test fleet, those phantom seats held onto 5,784 MB of idle RAM.
We added a --places 6 parameter to the scenario dispatcher. The worker now reserves only the six seats that actually enter the instance. That single change allowed us to pack four to six full squads onto a single worker process and raised our test capacity to 28 concurrent party tasks: 168 active characters running across five workers on a 10 GB virtual machine without a single memory stall.
Once 28 parties could actually get into the instances at once, the real entertainment began.
The Frostcovered City revolving door
Frostcovered City is a sprawling, sub-zero palace with 21 bosses stretching from the first Shade encounter all the way to Holeen. Clearing it is a long, disciplined march.
In almost every dungeon in the game, dying and clicking "Revive in town" drops your character back outside the entrance portal in the overland snow. The standard death recovery routine is straightforward: if you revive in town, you are outside, so you walk back up to the entrance portal and step through to rejoin your team.
Frostcovered City does not work like that. The original 2008 map design has a quirk: when you revive in town inside Frostcovered City, you do not spawn outside in the blizzard. You spawn inside the heated entrance sanctuary at coordinates (286, 247, -211).
The party leader died during an early test pull. The leader woke up at the entrance sanctuary, checked its state machine, and saw: "Revived in town. Action: return to dungeon." It dutifully plotted a route to the entrance portal coordinates at (305.7, -211.5).
The trouble was that from inside the entrance hall, walking into those coordinates steps directly into the exit portal.
The leader trotted over to the glowing arch, stepped straight through, and found itself standing outside in the mountain snowdrifts. It stopped on the snow, puzzled that the dungeon was suddenly missing. Meanwhile, the remaining five teammates inside the dungeon regrouped and engaged the next boss without a tank, fighting on until the party wiped.
The fix was a single condition: check the map tag before deciding you need to enter. If your map tag says you are already inside Frostcovered City, do not touch the portal. Regroup at the staging hall at (286, 247, -211) and run forward down the corridor.
The 300-metre errand in Gui Mu
Gui Mu is not an ordinary dungeon run. It is a wave defense against level-150 monsters assaulting the spirit of the Widow. You do not kill level-150 monsters with raw blade damage; you collect Celebeans, buy elemental auras from the Revelation oracles at five hundred beans an aura, and let the God's Eye turrets blast the waves with 15-metre area-of-effect strikes.
Two things went wrong here, both magnificent in their own way.
The first was the errand boy. When the first wave boss dies, it drops a special treasure chest called "God's Bestowal" right at the defensive circle near the Widow. The chest has a strict despawn window: exactly 150 seconds. If nobody opens it within two and a half minutes, it vanishes.
Our code had a routine called chest_actor() to decide who should break off combat to open the box. Because the role check simply looked for an unassigned utility damage dealer without checking where anyone was standing, it assigned chest duty to the Venomancer.
The Venomancer was three hundred metres away down in the lower valley, digging up beans and running laps around the Garden.
The Venomancer dutifully stopped what she was doing and began the long jog back up the valley path. Meanwhile, five heavily armed defenders—the Barbarian, the Blademaster, the Cleric, the Wizard, and the Archer—stood exactly two metres from the glowing chest with their weapons drawn. For 150 seconds they stood shoulder to shoulder, guarding the chest with their lives, watching the clock tick down until the box dissolved into thin air. Five seconds later, the Venomancer arrived at an empty patch of dirt.
The second problem was economic priorities. Once the parties started bringing back beans consistently, they reached Tier 3. But their aura upgrade queue had been configured to buy Mana and Chi auras before Defense and Health.
The squad stood in the defensive circle with completely full blue bars, glowing with three sparks of Chi, radiating confidence. Five seconds later, the first wave of physical melee monsters arrived and flattened both the Barbarian and the Cleric in a single hit. Having unlimited mana does not help when your physical defense is zero.
The priority list is now firmly locked: Defense, Health, Attack, Chi, and then Mana. In Tier 3, survival comes first; the fancy sparks can wait.
Voidweaver's bodyguards and the thousand-metre cave
In the level-99 celestial dungeon Seat of Torment, the final boss is Voidweaver. He stands on an elevated dais surrounded by six Level-99 elite guards.
The game uses bitmasks to determine monster loyalty. Voidweaver has a monster faction call-for-help mask of 128. The six elite guards also belong to faction 128. Because 128 & 128 is non-zero, our tank's combat evaluation checked the mask and concluded: "If we pull any bodyguard, the boss will wake up."
Instead of thinking, "Then we must be very careful," the tank's logic took a darker turn. It reasoned: "Since pulling an add will pull the boss anyway, clearing trash is pointless. Charge the boss."
The Barbarian leaped straight past the bodyguards and tackled Voidweaver and all six Level-99 elites simultaneously. The logs recorded between 70 and 91 deaths per run. Six level-99 elites and an endgame boss hit hard enough to turn a squad into mist before the Cleric can finish channeling an Ironheart blessing.
The reality of the engine is that the call-for-help only triggers within the boss's 15-metre sight radius. Any bodyguard pulled 16 metres or further away from Voidweaver can be safely lured out and killed one by one without disturbing the boss. We taught lair_pullable to measure distance instead of panicking over the bitmask, and the bloodbath stopped.
Meanwhile, over in Abaddon, the sister level-99 dungeon, the problem was not fighting but geography.
The final boss, Mountain's Finger, waits at coordinates (125.5, 163.1, -245.4) at the end of a twisting subterranean cavern. The path from the entry staging point at (-138.6, 165.0, 309.1) is exactly 1,108 metres of underground tunnels.
When the bot tried to calculate a direct pathfinding route from the start of the cavern directly to the boss, the A* algorithm exploded. It burned through 64,159 pathfinding nodes, hit its emergency iteration limit, and gave up 275 metres short of the boss room. The squad would march three-quarters of the way through the dungeon and then freeze solid in a dark corridor, staring blankly down the tunnel.
We broke the 1,108-metre march into eight distinct corridor waypoints (A24_MOUNTAIN_HOPS), each under 160 metres. The pathfinder now solves each short hop in a few milliseconds, and the squad walks straight into Mountain's Finger's lair without hesitating.
The hate wipe at 50 % and the 33-metre ledge
In Lunar Glade, the Humorum valley has an unpleasant surprise waiting in its trash packs.
Under Policy 1468, every trash monster in the valley has a trigger set at 50 % health. The moment its hit points drop below half, it completely wipes its hate table.
In a textbook pull, the Barbarian generates heavy threat, the damage dealers unload, and the Cleric stacks heals. The instant the monster hits 50 % health, the Barbarian's threat drops to zero. At that exact microsecond, whose name is at the top of the threat list? The Cleric, who is actively channeling healing over time onto the party.
Every half-dead monster in the pack would immediately turn around, ignore the tank, and sprint across the room to pulverize the healer. The Cleric would drop in three seconds, and without a healer the pull fell apart.
The fix was to teach the Barbarian to watch for the 50 % HP mark during Policy 1468 fights. The moment a mob reaches half health, the tank fires Flesh Ream to force the monster's attention back onto itself before it can take a step toward the cloth armor in the back.
Over in Twilight Temple, the obstacle was vertical.
In Acts I and II, bosses Dimentora and Shooting Aur wait on the sunken stone floor of the Golden Chamber at an elevation of 94.4 metres. The access corridor runs along an upper balcony at 128.2 metres. The difference between the balcony and the chamber floor is exactly 33.6 metres.
Because it was a vertical drop with no walkable slope, the navmesh marked the chamber floor as an unreachable disconnected island. Test squads would march up to the balcony railing, look down 33.6 metres at the boss standing in the pit, log "NoPath to target", and stand there for forty minutes until the dungeon timer expired.
We built dedicated vertical transition links. The squad now steps off the balcony onto the ramp path, drops into the sunken chamber, clears the boss, and uses Transport Box 9 to teleport back up to the upper halls.
The two loops in the workshop
It is tempting to imagine dungeon automation as a team of engineers watching live video feeds or feeding prompts into a language model. In practice, that approach would fail on day one. Running 168 characters in real-time combat through external language models would produce astronomical token bills and latency measured in seconds. And having engineers manually read through 6,000 lines of combat logs for every failed run would grind development to a crawl.
Instead, our workshop runs on two interlocking loops that feed each other.
The first loop is the continuous sandbox supervisor. It runs entirely on deterministic rules, protocol packets, and spatial math. It controls 28 party lanes around the clock on our internal test realm with zero language-model cost. It runs hundreds of dungeon attempts a day, cataloging every pull, every skill interrupt, every death coordinate, and every timer.
The second loop is our five-hour engineering shift. When a run fails, the test supervisor does not dump raw log files. It compiles the failure into a compact scoreboard packet—packet.md and packet.json—measuring under 4 kilobytes. That briefing strips out the noise and highlights only the essential facts: what stage failed, what the threat tables looked like, which path stalled, and where the bodies fell.
An engineer reads the 4 KB summary, finds the logic bug, writes a twenty-line fix, verifies it against regression tests, and lands it. Within minutes, the continuous runner picks up the new code and tests it across the next hundred runs.
That is how you turn a board full of seventy-death wipeouts into reliable clears. We will keep tuning until every dungeon on the board meets the project standard: three deaths or fewer per run.