Sovereign Intelligence, Localized Edge.

1. The tax nobody escapes

Ask why a long chat slows to a crawl, or why a browser agent falls over an hour into a session, and the answer is almost always the same three letters: the KV cache. Attention gives a transformer exact access to every earlier token, but it pays for that access with a cache that grows linearly with the length of the context. The arithmetic is rarely the wall. The memory is.

For an 8B transformer the key-value cache costs about 0.52 MB per token. The 4.8 GB left free on an 8 GB card holds roughly 9,155 tokens of context, and then it stops, no matter how fast the silicon underneath it is.

Linear recurrent models (S4, Mamba and their relatives) remove that growth by carrying a fixed-size state instead of a growing cache. RD-S4 sits in that family, so it inherits the constant-memory property. It is worth being precise about the three families, because they are easy to blur together and they fail in different places:

• Transformers keep every past token and attend over all of them. That buys exact recall and pays with a key-value cache that grows with every token, which is the wall above.
• State-space models (SSMs) such as S4 and Mamba throw the cache away and compress all of history into one fixed-size state. Memory stops growing, but a fixed state cannot hold everything, so recall of a specific far-away detail is lossy and tends to be biased toward recent context.
• RD-S4 starts from the SSM state and then changes how the state is parameterised, how it is read, and how the weights are formed. The constant memory is shared with SSMs; the three additions below are what is ours.

2. What RD-S4 actually is, and how it differs from an SSM

RD-S4 stands for Resonant Dynamic Subspace model. It replaces attention with a linear recurrence and layers three ideas of its own on top. None of them requires a cache that grows with context, and each one targets a specific weakness of a plain SSM.

• A resonant state. Each channel is written as a damped oscillator in polar form, a radius and an angle, so the layer stores history as a bank of resonances rather than a growing list of past tokens, and rather than the raw decaying memory of a vanilla SSM. The state is a fixed number of complex numbers. It does not get bigger when the context does.
• A phase-aligned read-out. Information is read back by matching the phase of a query against those resonances, which makes retrieval content-based instead of recency-based. This is the direct answer to the SSM recall weakness: a detail is addressed by what it is, not by how long ago it arrived. It is the Koopman view of a dynamical system: choose the right coordinates and a hard nonlinear sequence becomes a simple linear one you can summarise with a handful of modes.
• An operator dictionary (EFR). A transformer and a plain SSM both carry one frozen weight tensor. RD-S4 does not. It keeps a small dictionary of low-rank generators and, for each window of text, synthesises a custom operator from them with a truncated exponential map. A dense shared base handles the common case; a sparse routed tail specialises. Simple spans light up a few operators, dense spans recruit more.

The honest way to read these three claims is through what separates and what does not. At our pilot scale the constant-memory result is shared with any SSM and is not special to RD-S4. Of our own additions, the shared-generator base and stationary-state training both separate from the plain state-space baseline under a strict multi-seed test, while the question of whether the full evolved dictionary beats a plain SSM on quality is still open at this scale. We put the comparison in one place rather than hide it:

3. The part that reinvents itself

EFR stands for Evolutionary Functional Reconstruction, and the word evolutionary is literal. The dictionary is not fixed at design time. While the network trains, an in-network evolutionary loop (a MAP-Elites illuminator, the same family of search that powers AlphaEvolve and FunSearch) proposes new operators, tests each one on held-out probes against a measured noise floor, and installs the survivors without touching the backbone. We call an operator that survives that test an invented neuron, or ineuron.

The bar is deliberately brutal. A candidate has to beat a random operator of the same shape on a third, fresh batch of text that played no part in choosing it. That is the difference between a genuine discovery and a lucky fit to noise, and most architectures never check it.

Across our replication runs, 25 of 25 installed operators beat their random-operator null on text they were never scored against, at p < 0.001. They are discoveries, not noise. The honest caveat: at pilot scale the yield is low and high-variance, and we say so plainly in the paper.

4. Dreaming instead of fine-tuning

Two more pieces come from how brains consolidate rather than how transformers train. First, borrowing from JEPA (Joint Embedding Predictive Architecture), RD-S4 does not learn from next-token prediction alone. A second branch predicts the model's own future embeddings in latent space, regularised so the representation cannot collapse, which hands the backbone a far richer target than a one-hot token. Second, a day-dreaming loop: between exposures to new material the model replays and consolidates, acquiring the new passage while a variance-covariance anchor keeps it from overwriting the old one.

5. Reading several tokens at once

Generation is one token per step unless you do something about it. RD-S4 carries four lightweight self-drafting heads that propose several tokens per step, which a single verification pass then accepts or rejects. Because the model is recurrent, a rejected guess is undone by copying back a small state, not by truncating a giant cache, which is structurally cheaper than the same trick on a transformer. At pilot scale we measure a mean acceptance of about 2.35 tokens per pass.

6. The honest results

Here is where house style matters. We report the numbers that embarrass us next to the ones that do not, because an architecture claim you cannot reproduce is marketing. The constant-state result is the one we stand behind without qualification.

On quality we are deliberately cautious. On a small, saturated Shakespeare corpus RD-S4 does not separate from a plain state-space model at four seeds; it does at eight. Removing the shared generators costs a measurable amount, and stationary-state training earns its keep for no inference cost. An earlier version of this work carried a headline we have since retracted, because a training trick we added also helped the baseline. We would rather publish the protocol and the retraction than the headline.

Then we changed one thing: the amount of data. On TinyStories, a corpus 40x larger, the same model improved by 38 percent and perplexity fell from 5.6 to 2.9. That told us the pilot ceiling was the data, not the architecture, which is exactly the result that justifies scaling up.

7. Why this matters for Hayai

Hayai, our browser-automation models, lives or dies on this one property. A point worth making plainly first: Zeupiter agents do not drive the DOM. They look at the screen. Our Sight-Driven Agents perceive an interface the way a person does, through our own Hayai Vision detector, and they act from what they actually see rather than from CSS selectors or XPath that break the moment a site ships a redesign. We already built that vision stack; RD-S4 is the sequence engine underneath it.

That design makes the input a stream that never ends: a screenshot, a visual grounding pass, an action, another screenshot, for as long as the session runs. A transformer backbone on a stream like that hits the memory wall and has to start forgetting. RD-S4 does not, because the state is the same size at the first frame and at the hundred-thousandth. An agent can watch and act across a long session on an edge GPU at flat memory and flat latency.

The EFR dictionary lets a single model specialise on the fly, spending a fraction of its parameters when the viewport is a sparse login screen and recruiting more when it is a dense dashboard of controls, and the day-dreaming loop lets a deployed agent pick up a new site overnight without a fine-tuning pipeline. We measured RD-S4 on a single laptop-class RTX 5050 on purpose: if constant memory holds on close to the weakest card an 8B model is plausibly served on, it holds on everything above it. Enterprise accelerators are the next step, where we scale the same core through the billion-parameter range toward the trillion-parameter regime that sparse routing is built for.

RD-S4 is early, and we are the first to list its weak spots. But the core promise, an agent whose memory does not grow with how long it has been looking at the web, is measured and it is real.

1. The assumption nobody audits

Ask an engineer why their web agent broke and you will get one of four answers, every time:

• The selector changed.
• The element was in a shadow root / cross-origin iframe / canvas.
• The DOM said the button was there; it was covered by a modal.
• The site shipped a redesign.

These read like four bugs. They are one bug, wearing four coats.

Every one of them is a perception failure disguised as an integration failure. The agent did not misunderstand the task. It did not reason badly. It was handed a description of the page that did not correspond to the page, and it acted faithfully on a lie.

This is the part the industry has not internalized: the DOM is not a description of the interface. It is a description of the document that the interface was built from. Those two things diverged around 2014 and have not spoken since.

Consider what a modern interface actually is by the time a human sees it. It is a React tree that has been through a compiler, a bundler, and a CSS-in-JS runtime that generated a class name like .css-1x7fj2p. It is a stack of composited layers where paint order, not DOM order, decides what is clickable. It contains web components with closed shadow roots that are, by specification, unreadable from the outside. It contains canvas and WebGL surfaces where the “button” is a rectangle drawn by a fragment shader and exists in no tree at all. It contains cross-origin iframes for checkout widgets and OAuth flows and consent walls, where same-origin policy makes DOM traversal not merely difficult but illegal. It contains virtualized lists where the row you want is not in the DOM until you scroll, and where the DOM is actively lying about its own length.

A human handles all of that without noticing, because a human never had DOM access in the first place. A human has always been solving the pixel problem. The DOM is a crutch we handed to machines because it was easier than solving vision, and then we built an entire industry on the crutch and called the limp a platform.

Earlier this year we decided that was the wrong trade, and we went and built the other thing. This is what happened.

2. Sight-Driven Agents: perception before planning

A Sight-Driven Agent, which is what we call an SDA, inverts the standard pipeline. The conventional agent loop extracts the DOM, serializes it to text, hands the text to an LLM, gets a selector back, asks an executor to resolve that selector, and then hopes. Every stage in that chain is lossy and the last stage is a prayer. Our vision lane takes a rendered frame, runs dense control detection over it, produces a spatially grounded set of numbered markers, lets the planner pick a marker, and acts on the geometry behind it. Then it looks at the next frame to find out what actually happened.

We should be precise about where that lane sits, because we have seen enough vendor material in this category to know how easily “we have a vision model” slides into “everything runs through the vision model.” Magine Sandbox runs several execution lanes and an operator picks one per agent. The classic lane is still selector-driven and remains the default. The vision lane, which is the subject of this paper, is opt-in today and is where we are putting the work, because we believe it is where the category ends up. Everything below describes what the vision lane does and what we have measured of it, not a claim that every agent on the platform runs it right now.

Within that lane, the consequences of the inversion are not cosmetic.

The planner never emits a selector. It picks a marker from a set of numbered, spatially located controls, each carrying a class and a confidence score. It cannot emit a selector that fails to resolve because it is not composing selectors at all, and that removes the largest failure class in web automation rather than handling it more gracefully.

The perception layer has no privileged access to anything. It reads the rendered frame, which is the same artifact that arrives at a human retina. This sounds like a limitation until you sit with it for a moment, at which point it becomes the whole point: a perception layer that can only see what a human can see cannot be defeated by anything a human can defeat. Closed shadow DOM is invisible to querySelector and perfectly visible to a camera. Canvas is opaque to the accessibility tree and completely legible as pixels. Cross-origin iframes are a security boundary for scripts and not a boundary at all for light.

And the action space becomes geometric rather than symbolic. “Act on marker 14, whose box sits at (412, 887), which the detector scored 0.91 as a button” is a claim you can verify against the very next frame. “Click button.css-1x7fj2p” is a claim that either throws an exception or, far worse, silently does the wrong thing to the wrong element and carries on as though nothing happened.

Here is an SDA working a dense commercial page that has no bespoke handling anywhere in our stack:

The point worth making is that there is no Amazon integration. No recorded flow for this site, no MCP server, no selector library anyone maintains, no entry in a config file. The vision lane resolves controls here the way it resolves them anywhere else, because the input is pixels and pixels are the same everywhere.

One caveat we will make rather than let you discover it: Magine does contain site-specific code, just not for Amazon. There is a LinkedIn integration that talks to LinkedIn's private API directly and falls back to driving the UI when that fails, and it exists because a customer needed it to work reliably before the vision lane was ready. It is exactly the kind of per-site maintenance burden this architecture is meant to make unnecessary, and we would rather tell you it is there than have you find it in the repository and wonder what else we rounded off.

Now the adversarial case, which is the one that taught us the most about our own blind spots:

Wikipedia is also where the honest limitations of our current detector are most visible, and we will get to exactly why in section six rather than quietly walking past it.

3. Hayai Vision, and why interface perception is not object detection

There is a reflex among engineers hearing this described for the first time, which is to say that it sounds solved. It's object detection. Grab a standard detector, point it at some screenshots, done by Friday. That reflex is wrong in several specific and expensive ways, and each of those ways cost us an entire training run to learn properly.

The classes are defined by function, not by appearance. A cat is a cat. Every cat shares a visual signature, some combination of fur and ears and a characteristic silhouette, and a detector can learn that manifold because the manifold genuinely exists in the pixels. A button has no such signature. A button might be a filled rounded rectangle with a drop shadow, or four words of underlined blue text buried in the middle of a paragraph, or a bare SVG glyph with a forty-four pixel touch target and no visible boundary at all, or an entire product card, or a table row. These share essentially zero pixels of common structure. What makes them one class is that clicking them does something, which is a functional property and therefore invisible in any single frame. This is the central difficulty of the domain and it is a labeling design problem long before it is a modeling problem, which is a sentence we earned rather than deduced.

Scale variance is pathological. Objects in natural-image benchmarks occupy a reasonable fraction of the frame. Interface controls span four orders of magnitude of area within a single image: a sixteen by sixteen icon button and a full-width hero call to action, present simultaneously, both requiring precise localization. Anchor and assignment strategies tuned on photographs are simply not tuned for that distribution, and nobody tells you this because nobody publishes on it.

Density is brutal. A natural image contains a handful of objects. A news aggregator front page or a reference article carries several hundred interactive controls in a single viewport. This is not a hypothetical constraint for us: our own selector-driven lane caps its element listing at eighty entries per frame precisely because enumerating more than that in text becomes unusable, and that cap is one of the concrete reasons we want perception carrying this load instead. Detection stacks with fixed per-image proposal caps have the same problem in a less visible form, because truncation stays invisible in aggregate metrics while being catastrophic in production when the control you needed fell off the end of the list.

And standard augmentation is not merely unhelpful here, it is actively harmful. This one is counterintuitive enough that we now state it as a rule for anyone working in the domain: do not use photographic augmentation on interface data. A horizontally flipped form is not a form. Navigation lives on the left, submit lives bottom right, close lives top right, and those are learned spatial priors carrying genuine signal that mirroring destroys while teaching the model a layout that exists nowhere on the web. Rotation has the same problem, because interfaces are axis-aligned and a fifteen-degree rotated checkout page is not a harder example, it is a wrong one. Hue shift is worse still, because color carries real semantics in this domain: primary action, destructive action, disabled state, validation error. Randomizing hue teaches the model that a red delete button and a green confirm button are interchangeable, which is a genuinely dangerous thing to teach a system that clicks things on your behalf. We zeroed all of them and kept only scale jitter, translation, mosaic composition, and random erasing, because those simulate viewport variation and occlusion, which are things that actually happen to interfaces in the wild. The general lesson underneath is that augmentation has to encode a real invariance of the domain, and copying a natural-image recipe into a synthetic-image domain injects false invariances that quietly cap your ceiling forever without ever showing up as an error.

4. AGP: the layer between seeing and doing

Detection on its own is necessary and nowhere near sufficient, and the gap between a bounding box and a reliable click is where a surprising amount of the interesting engineering lives. Closing that gap is what the Autonomous GUI Pilot, which we call AGP, exists to do.

The loop runs like this. AGP captures a clean frame, hands it to Hayai Vision, and gets back a set of detections. It draws those detections onto the frame as numbered markers and builds a mapping from each marker to the geometry behind it. That marked-up frame goes to a vision model, whose entire job is to answer one narrow question: which marker, and what action. The answer comes back as a marker index, AGP resolves it through the mapping table, and the action lands on real coordinates.

That design deserves an honest accounting, because it is easy to describe it in a way that overstates what it does. There is still a vision-language model in the per-step loop. What changes is what we ask of it. An unassisted screenshot agent has to do perception and planning in the same forward pass, which means localising every candidate and describing it in tokens before it can choose. AGP has already done the localisation, so the model is choosing from an enumerated list rather than reading coordinates off an image. The detector absorbs the part that scales with page complexity, and the language model handles the part that scales with task complexity. Those two things grow independently, which is the whole reason for splitting them.

There is a second honest caveat. When the trained detector is unavailable or returns nothing usable, AGP falls back to a classical computer vision and OCR heuristic, and if that also fails the step falls through to the selector-driven lane. The step log tells you which one ran on every step: the trained model, a mix, or the heuristic. We built it this way because a perception system that is right ninety percent of the time and catastrophic the other ten percent is worse than useless in production, but it does mean that not every AGP step is a Hayai Vision step, and any number we quote has to say which.

AGP is also where the reward signal gets captured, which turns out to matter enormously and which we will come back to properly in section seven. Every single time an agent acts on a detection, AGP records whether that action succeeded or whether it had to fall back to something else. That one bit, attached to a specific bounding box on a specific real page at a specific moment, is the thing that no static dataset in the world can give you.

5. Three catastrophes, in order

I want to walk through what went wrong rather than what went right, partly because the failures are where all the transferable content lives, and partly because any team telling you their first vision model worked is a team you should be asking considerably harder questions.

5.1 False-negative poisoning, which is a recall bomb with a long fuse

Our first trained detector scored mAP50 of 0.147 with recall sitting at 0.169. Completely unusable, and the diagnosis remains the most instructive bug we have hit to date.

The original harvest wrote a label file containing only the control the agent had clicked, while the screenshot itself contained dozens of other controls. Object detectors treat unlabelled regions as background. So on every single frame, we were explicitly teaching the model that this thing which is unmistakably a button is in fact not a button, it is background, and so is that one, and so is that one over there in the corner. We were not under-labeling the data. We were actively training a hard negative on the exact class we most wanted to detect. Recall collapsed because we had, in effect, sat down and asked it to.

The fix was to harvest full-page labels always, and to record the clicked control separately as a reward sidecar rather than as the label set itself. The transferable lesson, for anyone whose labels come from interaction logs of any kind, is to verify that your unlabelled regions are genuinely empty. Interaction logs are inherently sparse because they record what was touched rather than what existed, and feeding that sparsity straight into a detector will quietly destroy your recall while every other part of your pipeline continues to look completely fine.

5.2 Container flooding

The next problem was the mirror image of the first one. Generic control heuristics started matching layout wrappers, producing as many as fifty-seven “buttons” in a frame where a human would count six. Nested divs carrying click handlers, wrapper elements with a stray tabindex, container elements whose class names happened to contain “btn”, each one generating a box and most of them enclosing all the others.

The fixes were geometric and cheap. Drop any candidate that contains another candidate, so you keep leaves rather than wrappers. Drop any box exceeding thirty percent of viewport area, because nothing that large is a control. Deduplicate anything overlapping above 0.85 IoU, keeping the most specific class. Measured against legacy-shaped data this took 1040 boxes down to 360, which means sixty-five percent of everything we had been training on was noise.

The second-order property here turned out to be the genuinely valuable one. Because the repair is purely geometric and needs no DOM access at all, it applies retroactively to already-harvested data, so we recovered our entire historical pool instead of throwing it out and starting again from zero.

5.3 The measurement catastrophe, where we nearly shipped a fantasy

This is the section I would most want another ML team to read, because we came within a single training run of publishing a number that was pure fiction, and we caught it only because somebody insisted on testing an assumption that essentially nobody in the field ever tests.

The tell was a training run reporting that dropdown, our rarest class with 291 training instances, had scored mAP50 of 0.946 with recall at 0.958. Sit with how wrong that is for a second. The rarest class in the entire dataset, with two orders of magnitude less data than the common class, was comfortably the best performing class we had. That is not a result. That is a symptom.

The mechanism was that our harvester screenshots every few seconds while an agent works, so a single page yields dozens of nearly identical frames differing only by a cursor position or a hover state or an image that finished lazy-loading. Our split was randomizing over individual frames, which meant frame N landed in train and frame N+1 landed in validation. We were validating on training data wearing a false moustache, and every metric we had was measuring memorization of near-copies while confidently reporting it back to us as generalization.

The obvious remedy was to perceptual-hash the frames, group the near-duplicates, and split by group. So we did exactly that, using a 64-bit difference hash at eight by eight with a Hamming threshold of six, which is a completely standard configuration that appears in essentially every tutorial written on the subject. The dataset collapsed from 1735 samples to 248, destroying 85.7 percent of it, and mAP50 fell from 0.739 to 0.320.

At which point we finally did the thing we should have done first, which was to measure the hash instead of trusting it. We generated pairs of genuinely different application pages that share common chrome, meaning the same header and nav and footer with entirely different content, because that is precisely what a real crawl produces. Then we measured the distance distribution between them.

An 8×8 dHash at threshold 6 merged 6,840 out of 6,840 different-page pairs. The median inter-page Hamming distance was 1.

Not “over-merged”. Merged everything, without a single exception across nearly seven thousand trials. A 64-bit hash downsamples the frame to an eight by eight grid, and two pages sharing a header and a nav bar and a footer are simply identical at that resolution. The content differences, which is to say the entire signal we cared about, live in exactly the pixels that get averaged away. This is a load-bearing finding for anybody deduplicating UI screenshots and it appears in no tutorial we could find anywhere: the perceptual hash configuration that works beautifully for photographs fails catastrophically on interfaces, because interfaces share vastly more low-frequency structure than natural images do.

At 256 bits, meaning sixteen by sixteen, different pages never scored below a Hamming distance of 7 and we finally had real separation to work with. Then came the insight that took us a third run to see, which is that deduplication and train/val grouping are different jobs with opposite failure costs and must never share a threshold:

Two further findings arrived the way most of ours do, which is by breaking something. The clustering algorithm matters far more than we expected, because leader clustering is order-dependent and split a single site across two groups, leaking it straight through the wall we had just built; single-linkage chaining only ever produces fewer and larger groups, which is the safe direction to fail in. And splits have to be size-aware, because assigning a fixed fraction of groups produced a 12-train / 43-validation split when the pool happened to contain only two groups.

Size-aware greedy assignment is now only the fallback. What a production run uses is a stable content hash: the group key goes through CRC32 into a thousand buckets and the lowest fifth is held out, so a site lands on the same side of the wall in every run, permanently, and the exam cannot quietly change underneath a comparison. Every run prints a holdout signature, and our training dashboard refuses to difference two runs whose signature or validation population do not match, which it exercises more often than we would like. That single rule is what makes any of the numbers in the next section mean anything at all.

Final verification against synthetic data with known ground truth, six sites by four pages by five frames, gave us 59 samples kept in a 47/12 split with zero pages and zero sites spanning both sides.

The meta-lesson cost us roughly three training runs, and the expensive part was never the compute. It was that for several weeks every decision we made was informed by metrics that were fiction, so we were tuning against noise and quietly congratulating ourselves for the improvements. Before you trust a metric, verify that your validation set is genuinely held out, and by verify we do not mean “we called train_test_split”. Construct adversarial cases and measure the separation empirically. And if a rare class ever outperforms a common class, stop whatever you are doing immediately, because you do not have a good model, you have a leak.

6. Where we actually are, and the ceiling we found

Here are the real numbers from real runs against a real site-held-out split, current as of our last training run in late September 2026. Several of them are bad. We are publishing them anyway, because the interesting content of an ML paper is the diagnosis, and you cannot diagnose a number you invented.

Run 3 is worth pausing on, because the dataset was finally correct and the number went down anyway. Batch size had been hardcoded to 4, which on a large-memory accelerator leaves the device mostly idle and makes gradients extremely noisy. The loss curve bounced around (.029, .087, .152, .121, .122, .144, .209, .178) and never converged at all. The epoch count was far too low as well, and it is worth saying plainly that eight epochs of fine-tuning does not produce a converged model, which means every historical number in our repository before this point was read off an unconverged run and was therefore meaningless.

The last six runs are the ones that changed what we work on. Across them the harvest grew from 7,073 samples to 8,482, a twenty percent increase, and the class mean went 0.547, 0.543, 0.545, 0.545, 0.539, 0.535, with mAP50-95 moving 0.422 to 0.413 over the same span. That is flat, and slightly down. Two of those runs were the same pool trained twice and returned identical numbers to the decimal, which is how we learned the pipeline is deterministic end to end: same data, same detector. So the flatness is not sampling noise between runs, it is the honest answer. More screenshots of the sites we already cover buy nothing. We had assumed sample count was the binding constraint, and it measurably is not, which is the most useful negative result we have produced so far.

The per-class breakdown is the entire story

Read that right-hand column twice, because it is still the most useful thing in this entire paper. Performance tracks how tightly the class is defined far more closely than it tracks instance count. Definitional coherence is the first-order term. It is not the only one, and dropdown is the exception that taught us the second: the definition there is tight and the score is 0.010, because a clean definition cannot rescue a class the training set barely contains. Coherence is necessary. It is not sufficient.

The button class still dominates the aggregate completely: the current exam holds 30,563 button boxes against 68 checkbox and 35 dropdown, so the headline is very close to a button score wearing a class mean as a disguise. What has changed is that the union is now deliberate. We split link semantics into their own class, measured it, and found the model scored a separated link at 0.118 while button sat at 0.312 beside it, and that the agent never once needed the distinction, because a link and a button are the same affordance to something whose output is a click. So production folds them back together from stored provenance, and on the current exam that one class reads 0.634 against the 0.373 we published last time. Those two numbers come from different exams at different input sizes, so read the direction as evidence and the magnitude as not.

So our accuracy ceiling is still a taxonomy and data problem rather than a capacity problem, and that distinction is the entire difference between a hard open research programme and a known piece of work with a known shape. The taxonomy half is now done. Class ids are append-only, which is why link, image and media could be appended without a single historical label changing meaning, and the trainer takes a profile flag: five classes for production, eight when we want to interrogate a distinction. The catch-all is the piece still open. It sits at 0.380 with recall of 0.359, which is a recall problem rather than a confusion problem, and the honest description of text remains that it is whatever was left over.

Which is exactly why we store per-box provenance, meaning the originating tag and role and type for every single label, kept alongside the label itself. Taxonomy changes become a recomputation over evidence we already hold rather than a re-crawl of the open web. That decision cost about a day of engineering and converted our single largest foreseeable data liability into a batch job, and we have since spent the credit twice over: once to append three classes, and once to correct every control carrying a combobox role, which our original rule had been labelling as a dropdown even when the element underneath was a plain text input. Both were a re-derivation over stored evidence. Neither cost us a crawl.

One more honest limit while we are here. Our rare classes are not rare by a 200:1 margin as we reported last time, they are rare by closer to 870:1 in the current exam, and oversampling cannot fix that, because repeating an image to boost a rare control also repeats the forty buttons sitting in that same image. Measured, input went from 0.4 percent to 2.6 percent and dropdown from 0.1 percent to 0.7 percent. Better, and nowhere close to balanced.

And there was a quieter bug sitting underneath that, which is worth stating because it is the kind of thing every team writes without noticing. Our oversampling floor counted boxes. dropdown cleared it comfortably with 725 training boxes and was therefore never oversampled at all, in any run, while those 725 boxes lived in a few dozen screenshots. A detector does not learn from boxes, it learns from layouts, and a class present in thirty images has thirty layouts no matter how many boxes you count in them. There is now an image-count floor alongside the box floor, capped at four repeats so a rare control cannot drag its forty button neighbours along eight times over. The real fix is still targeted harvest, deliberately routing agents through forms and checkouts and settings panels where rare controls are dense, which the data engine makes almost free because harvesting is a byproduct of work we are already being paid to do.

What the detector is worth on live pages, which is a different number entirely

mAP is an exam score. The question that pays the bills is whether the click landed on the control the planner chose, and we did not measure that at all when this paper was first published. We do now. Before each move AGP records the rectangle of the element the planner addressed, arms a mutation probe on the page, clicks, and then records four things: whether the point fell inside that rectangle, whether the detector had boxed it, whether the predicted class matched, and whether the page reacted at all. Every row is stored against the SHA of the detector that produced it, so the measurement belongs to a specific checkpoint rather than to the model in general.

Two things fell out of that immediately, and the first one is embarrassing. Screenshots were being captured in device pixels while clicks were issued in CSS pixels, so on any browser profile with a scale factor above one every single glide landed short by exactly that factor. It had been doing this for as long as the lane had existed. It never once showed up in mAP, because mAP never leaves the exam hall. The second, after the fix, is that in the first window of 165 recorded clicks roughly ninety-nine percent landed inside the intended control, and 41 of those 165 had no detector box behind them at all and were placed from the element rectangle instead. Both halves of that matter. The grounding layer is carrying more of the load than the detector is, which is a perfectly reasonable thing for it to do and a completely unreasonable thing to not know, and before we instrumented it the lane had been recording every step as a success unconditionally. One window on a handful of sites is not a verdict, and we do not treat it as one.

7. The flywheel, and why a fork of it doesn't work

Here is the part that should make an investor sit forward, and it is not the model.

Every SDA run is a labeled data collection event. When an agent operates a page, the system captures the rendered frame and simultaneously derives full-page control labels from the interaction substrate at that exact instant, covering every control in the viewport rather than only the one that happened to get clicked. Then AGP attaches the outcome: did acting on this detection succeed, or did it fail and require a fallback path?

That outcome is a reward signal bound to a specific bounding box on a specific real page at a specific moment in time. Failures get boosted in the training mix, which is hard-example mining that costs precisely nothing at inference because it only changes which samples the trainer sees.

Read that again in terms of unit economics, because this is the whole argument: our training data is a byproduct of revenue-generating work. We do not run a labeling operation. We do not buy annotations. Every customer task is simultaneously a task completion and a training sample drawn from the real distribution, meaning real sites with real layouts and real dark patterns and real A/B test variants and real regional differences that no annotation vendor would ever think to collect.

Three properties make this a moat rather than a nice-to-have. It is distributionally correct by construction, because these are not synthetic pages or a scraped screenshot corpus from 2022 but the exact pages users actually need automated, at the exact moment they need them, including the checkout flow that shipped yesterday afternoon. It is reward-labeled rather than merely annotated, and a standard detection dataset can tell you where things are but can never tell you which detections led to successful action, because obtaining that requires having actually tried. And it compounds against usage rather than against capital: more agents produce more pages, which produce more labeled failures, which produce a better detector, which produces more reliable agents, which produces more usage. A competitor can fork the architecture in an afternoon. They cannot fork four quarters of interaction traces from real production workloads.

8. What surrounds the model, and why demos die without it

A perception model is necessary and nowhere near sufficient. The things that separate a compelling demo from something a business will actually run unattended are mostly unglamorous, and they are exactly where most stacks quietly fall over.

Agents loop, and it is the most expensive failure mode in the category. An agent that has entered a two-state cycle will burn its entire token budget with total confidence and produce absolutely nothing. We maintain a directed state graph keyed on a normalized route hash, recording per-state facts and failed actions with cycle detection running over the transition path. When a cycle is detected the planner receives a hard topological constraint forbidding the repeated action. Every node in that graph is bounded, with hard caps on facts and failed actions and edges and history, because an unbounded memory graph is a memory leak wearing a research paper's clothes and it silently inflates every prompt for the remainder of the run.

Prompt text does not hold, and enforcement has to be mechanical. This was genuinely surprising to us. We repeatedly observed agents inventing reasons to stop, not failing but quitting, with fluent and confident and entirely fabricated justifications: citing terms-of-service restrictions that do not apply to a user operating their own account, claiming a sandbox or domain restriction that does not exist anywhere in our system, or declaring a task complete when a quantitative target like “visit at least fifty sites” was demonstrably unmet. No amount of system-prompt engineering fixed any of it. What fixed it was refusing the terminal action at the executor level, pattern-matching the stated justification against known fabrication classes, rejecting the completion, injecting a corrective message, and continuing, with a hard cap so a genuine dead end can still end the run.

That produced a corollary we now treat as invariant: any time an agent response is discarded, you must tell the agent why. Silently dropping a response and continuing guarantees an identical-output loop, because the model sees its own rejected text with no explanation attached and re-emits it forever. That single rule eliminated a class of infinite loop we had been misdiagnosing as a model quality problem for weeks.

Runs have to survive the infrastructure they run on. Full run state, meaning conversation history and step record and token accounting and the memory graph, is checkpointed after every step. If the process is OOM-killed or the platform redeploys mid-run, the agent resumes from the next step with its context intact. Not restarts the task. Resumes it.

And when an agent hits a wall it should not climb, it hands over. When a provider blocks automated sign-in, the correct answer is to hand the session to the human or to import the human's existing authenticated session, and never to defeat the check itself. This is a deliberate stance and worth stating explicitly, because the incentives in this market point firmly the other way: we do not build authentication bypasses. An agent platform that solves auth walls through evasion is a liability that simply has not been discovered yet.

Most agent demos, meanwhile, are one-shot. Ours run indefinitely on a schedule:

Making that work requires per-agent persistent browser profiles with stable fingerprints, credential material encrypted at rest with envelope encryption, one-time codes given short TTLs so a stale OTP is never replayed into a login form, and a scheduler that survives control-plane restarts. None of it is interesting to write about and all of it is load-bearing.

9. The competitive landscape, structurally

I want to be careful here, because the interesting comparison is not a leaderboard, it is architecture, and architecture is what determines whether something is fixable at all.

A benchmark number is a snapshot; an architectural property is a destiny. If your perception substrate is the DOM, no amount of model scaling will ever fix canvas. If the cost of a step grows with how complicated the page is, no amount of inference optimization changes that shape. These are not engineering backlogs, they are load-bearing walls, and the distinction matters enormously when you are deciding who to bet on.

9.1 Perception substrate, which is the axis that actually matters

The MCP row deserves its own paragraph, because MCP is currently absorbing an enormous amount of ecosystem energy and we are not remotely dismissing it. It is a genuinely good protocol. But it is worth understanding precisely what it is, which is a standardized way to call an integration that somebody else has already built. It is a connector standard. It does not perceive anything. If the service you need has an MCP server then MCP is excellent and you should absolutely use it. If it does not, which describes the overwhelming majority of the web including essentially every internal enterprise tool and legacy admin panel and government portal and every SaaS product that has not prioritized an agent story, then MCP offers you nothing whatsoever. The long tail of the web is never going to ship you a server, and that tail is where all the work actually is.

9.2 Latency architecture

This is the axis on which we think the category gets re-sorted next, and almost nobody is positioned for it.

Here is the actual argument, stated carefully. A vision-language model working from a raw screenshot has to decode tokens in order to tell you where a button is, and that decode is autoregressive, which means inherently sequential with cost scaling in how much it has to say. Ask it to enumerate the controls on a dense page and you pay for that enumeration, sequentially, on every step.

AGP does not remove the language model from the loop, and we said so plainly in section four. What it removes is the enumeration. The detector emits every box in a single forward pass at a cost that is flat with respect to how many controls are on the page, so a dense catalogue page costs the detector exactly what a sparse login form costs it. The model downstream then produces a marker index and an action, which is a handful of tokens regardless of whether the page had six controls or six hundred.

So the per-step cost stops scaling with page complexity and starts scaling only with task complexity. That is a real structural difference and it is the one worth betting on, but it is a difference in what grows, not a claim that we have eliminated the model or that we are running at video frame rate today. We are not, and section ten says exactly why.

9.3 Where the SDA model specifically pulls away

Against general-purpose agent platforms, with Manus being the most visible of them, the honest comparison is not about which system is smarter. Those platforms are genuinely impressive at open-ended task decomposition and it would be silly to pretend otherwise. The gap is in operational substrate:

That auditability row is the one enterprises actually sign on, and it is badly underrated across the whole category. When an agent does something unexpected inside a regulated workflow, “here is a text log of what the model said it did” is not evidence of anything, because the model is precisely the component whose reliability is in question. “Here is the frame the agent was looking at in the moment it clicked, and here is the box it clicked” is evidence. We keep both, for every step, of every run.

9.4 On honest failure

One more frame, and it is the one I would put in front of a technical diligence team first:

We include this deliberately. Image-grid CAPTCHAs are an adversarial perception problem engineered specifically against the class of model we run, by an organization with considerably more vision research capacity than we have. We do not solve them reliably, and any vendor telling you they do is either lying to you or is going to have an interesting conversation with a fraud team at some point. What matters architecturally is what the system does with the failure, which is to label it, record it as a hard negative, and route around it. The failure becomes training data. That is the flywheel from section seven closing its loop in public, on a frame we are choosing to show you.

10. What we are not claiming

A section that should be mandatory in posts like this and almost never is.

We have not solved web perception. Our current detector sits at a class-mean mAP50 of 0.535 on a site-held-out exam, and that headline is worth decomposing rather than quoting. Weighted by how many validation boxes each class actually carries it is roughly 0.62, and with the starved dropdown class removed the four-class mean is 0.667. Those are the same model, on the same exam, scored three defensible ways, and the spread between them is entirely an artifact of one class with 35 boxes counting as much as one class with 30,563. We quote the lowest of the three as the headline, because presenting an inflated figure to people who will eventually run diligence is a spectacularly bad trade. As section six lays out in detail, the number is low for identifiable and fixable reasons, and the per-class breakdown is completely unambiguous about which ones.

We do not reliably defeat adversarial CAPTCHAs, as shown above.

We are not at frame rate in the production lane today, though the shape of that gap has changed since we first wrote this. The detector itself ran inference in roughly a hundred milliseconds at the 640-pixel input we originally measured; production now runs a 960-pixel input, which is 2.25 times the pixels, because small controls were being lost at the lower resolution. The expensive part was never the inference. It was that the lane spawned a fresh process per step, and interpreter startup plus framework import plus model load cost somewhere between 0.4 and 1.0 seconds before a single pixel got examined, or one to three seconds and between 600MB and 1.2GB resident if the lightweight runtime was missing and the full framework loaded instead. We were spending four to ten times our inference cost on process startup, which was never a model problem.

That fix has shipped. A worker in our mesh now runs the detector as a resident service holding a warm interpreter keyed on the model file, so a step posts a frame and gets boxes back with no process to start. Routing is decided per step: a worker gets the frame only if it is serving the same model checkpoint and answering, there is a hard six-second ceiling, and a worker whose round trip runs more than fifteen percent slower than doing the work locally is benched for five minutes. When no worker qualifies the control plane falls back to the one-shot process and pays the startup cost described above, so the honest statement is that the daemon exists, it is doing the work when the mesh is healthy, and the fallback path still has the old shape.

The detector is early. The pool is 8,482 harvested frames, which collapse to 3,319 distinct views and 2,810 training images once near-duplicates are grouped, drawn from 169 sites with 33 of them held out as the exam. That is thousands, not millions. And the slope, which we said last time was the interesting question rather than the current checkpoint, is currently flat: a twenty percent larger pool bought nothing measurable, because the frames it added were more of the sites we already had. The flywheel still matters, and it matters more than the checkpoint does, but what it has to produce is site diversity rather than frame count, which makes the next move a routing problem inside the agent fleet rather than a storage problem.

Sites can still block us. Bot detection is real and adversarial, we route around it with legitimate techniques, and we hand off to humans when we should not proceed.

What we do claim is this: the perception substrate is architecturally correct, the failure modes that dominate DOM-based automation are eliminated by construction rather than mitigated by effort, the data engine compounds against usage rather than capital, and the latency architecture has a real-time ceiling that vision-language-model-in- the-loop designs cannot reach through optimization at all.

11. Why now

Three curves crossed at roughly the same time, and that timing is most of the reason this is buildable in 2026 and simply was not in 2023.

Edge inference got good enough that a quantized dense detector runs at interactive rates on commodity hardware. Two years ago this needed an accelerator per concurrent agent and the unit economics were frankly absurd.

The web got less parseable rather than more. Every year brings more canvas, more shadow DOM, more compiled class names, more client-side rendering. The DOM-based approach is not holding steady, it is actively degrading, and every framework release makes selector-based automation slightly worse than it was the month before. We are betting against a curve that is already falling, which is a considerably more comfortable position than betting against a stable one.

And the market discovered that agents need to actually work. The 2024 and 2025 cycle produced spectacular demos whose production reliability did not survive first contact with a redesign. Buyers have now been burned once and are asking the correct question, which is not “can it do this once, on stage” but “what happens when the page changes”. We have an answer to that question, and the answer is not “we'll update the selectors”.

12. Closing

The industry spent a decade building increasingly elaborate machinery to extract structure from a medium that had stopped exposing structure. Scrapers, then headless browsers, then accessibility-tree serializers, then MCP servers, each layer a heavier abstraction over the same eroding foundation, and each one requiring per-site human labor that never amortizes into anything.

Humans never needed any of it. Humans look at the screen.

The Zeupiter Research Lab is preparing for the release of an incoming research paper detailing our next-generation visual foundation models: Hayai 4B & 8B (早い - meaning fast in Japanese).

Running full visual screen analysis at 30-second intervals can be cost-prohibitive on public endpoints. Hayai is our solution: a suite of visual LLMs designed to run locally on consumer-grade edge GPUs while maintaining performance parity with models 5x their size.

By dynamically allocating weights, a Hayai 8B model can reduce active parameter usage down to 1.8B for simple visual tasks (like finding a login button) and scale up to its full capacity for dense, text-heavy spreadsheets.

Breaking the Transformer Constraint

To achieve high-frequency visual loops, we constructed a novel hybrid architecture designed to directly compete with Transformer-based models and Mamba/State Space Models (SSM). This hybrid approach eliminates the quadratic complexity of standard attention mechanisms, allowing the model to handle rapid screenshot streaming with flat performance latency.

Technical Capabilities:

• Latency: Targeting sub-20ms visual inference.
• Throughput: Over 5,000 tokens/sec on consumer-grade GPUs (e.g., RTX 4090/3090).
• Footprint Expansion: Engineered specifically to support autonomous, selector-free navigation across more than 100 million websites.

We are currently a small, highly aligned team of 5 people. Right now, every line of code represents a conscious choice about how we want intelligent software to interact with the world.

As a quick company update, Zeupiter has recently undergone a significant zoomed-out pivot. We originally started by solving DevOps and cloud infrastructure challenges. However, as we engineered those lower-layer routing nodes, we realized the underlying sandboxing and browser orchestration technology had massive applications far beyond DevOps workflows.

The pivot expanded our addressable market from a niche audience of systems engineers to a much broader category of internet users, developers, marketers, government bodies, and businesses looking to automate complex digital workflows. Since launching SDAs, we have already started seeing strong early user adoption, organic revenue generation, and product validation that reinforces our deepest technical convictions.

Key Architectural Milestones (Roadmap 2026):

• Fear Recall Intelligence & KTO-inspired RL: Our agents utilize an experimental, visual Kahneman-Tversky Optimization (KTO) reinforcement learning model. They learn from past interface failures ("fear recall"), preventing loop traps and navigating anti-bot screens with self-healing decision steps.
• Execution Across 20 Million Websites: Scaling visual thread clusters to safely map, isolate, and orchestrate automated tasks across a broad footprint of legacy portals and SaaS pages.
• Coding Agents: Integrating dedicated browser sandboxed IDE & harnesses that compile, run, and self-correct software tasks, outperforming standard static coding harnesses.
• Sub-20ms Latency & 1,000 tokens/sec on Consumer GPUs: Porting lightweight visual weights directly onto localized edge containers.