EgoShift

What EgoShift measures

A teapot sits on the table in front of you. Turn 90° to the right and it is on your left — the teapot has not moved, you have. Robots and augmented-reality assistants do this constantly, and they must also tell whether an object has left the camera's view or is hidden behind something else. EgoShift puts that to vision–language models in simulated indoor rooms, one stated turn at a time.

Task C, switched live

The first view boxes the target. Pick what happens next — each is a real second view from the same room.

visible occluded out of view

The chair is still drawn in the second view, so the answer is visible.

View 1 — target boxed

View 2 — visible

What the three tasks show

Each task answers a different question and the three are never combined into one score. Two of them come out well, and they are what make the third one readable.

Tasks A and C state no action, so no stated turn can narrow their answers; the answer balancing described further down protects them instead. Only Task B names an action, which is why only Task B has a bound to be read against — and why the sections that follow are about what happens there.

One rule covers almost every frontier error

The failure is not many different spatial mistakes. It is one mistake made repeatedly, and it is specific enough to state in a sentence.

What this does not settle. The straight-ahead rule is an interpretation of the evidence, not an established mechanism. Three tests converge on it, but the matched counterfactuals that would settle it — varying the stated turn while holding the picture fixed — have not been run. A model behaving this way for another reason would produce the same evidence.

People find those turns harder too, and never fall that far

Nine paid lab annotators per task returned 27 sessions, scoring 98.1%, 84.6% and 89.6% on the three tasks, with Fleiss' κ of 0.960, 0.681 and 0.820. Their accuracy drops at the same two turns, so part of the difficulty belongs to the questions rather than to the models. It never approaches the no-image score.

The pooled score makes the two look level

On the pooled Task-B number the best configuration reaches 83.6% against a trained-human mean of 84.6%. Read one turn at a time and the two stop matching: the best model beats the panel by 6.9, 8.6 and 11.4 percentage points at 30°, 60° and 120°, and trails it by 24.5 and 27.8 at 90° and 180°.

Raters stay 41 to 42 percentage points above the action-only bound at both hard turns. The model mean comes within 5.0 percentage points of it at 90° and within 1.1 at 180°.

How to read the human number. This is a trained-human reference, never a human ceiling. It estimates what a trained panel does on these questions, not a bound on what people can do, and three departures from the written protocol limit it further. On the 120 shared questions the panel tied or disagreed with the simulator on four Task-B and ten Task-C questions; setting those aside raises rater accuracy from 84.6 to 86.3% and from 89.6 to 94.8%, so the gap reported here is the conservative one. The institution's review board determined before collection that the study does not constitute human subjects research requiring oversight.

Why you can believe the finding

A question that states how far the camera turned can be answered from that turn alone, with no image. EgoShift computes what that is worth — from camera geometry, before any model runs — and holds every turn to that floor, so a score can be read against what the question gave away.

Where the bound comes from

Every Task-B image is taken at standing height with a fixed tilt and field of view, and the stated turn is a rotation about the vertical axis from that same spot. Only the heading changes, so the set of answers a turn can produce is finite and can be computed exactly.

The construction requires the object's centre to start within 66.5868° of straight ahead — the horizontal half-angle of a 640×480 frame at a 120° vertical field of view, written into the build before any pose was replayed. Two independent implementations agree on all ten turn cells, and the held-out set meets the bound in every one, holding the shortcut to 45.90%.

Why balancing is not enough

An earlier build passed every conventional balance check, yet a system that saw only the stated turn scored 72.3%. The imbalance that matters sits inside individual turn cells, not across the question set. That build differs from the released one in other ways, so the number shows what the test catches, not a controlled measure of what the construction changed.

The recipe is not specific to EgoShift. Any benchmark whose questions state an action can compute the score reachable from that action alone, and build against it before balancing.

What the bound covers. Exactly one shortcut: answering from the stated turn with no image. A build that meets it has shown nothing about wording, metadata, or a cue that some future model finds and we did not anticipate. Learned attacks on the questions are a separate check, with a protocol fixed in advance and not yet run.

Run it

The public artifact holds the questions for every split, the development answers, the construction and audit tools, and every raw model response behind the paper's tables. A reader can recompute the numbers instead of trusting them. Only the held-out answer key is withheld, which keeps the benchmark usable as a test.

git clone https://github.com/UTSA-VIRLab/EgoShift.git
cd EgoShift && pip install -e ".[full]"
egoshift audit-taskb --questions taskb_questions.jsonl

If you report results

• Report Task B one turn at a time and Task C one answer class at a time.
• Compare each score against the random rate, that turn's action-only bound, and the trained-human reference.
• A score above the bound is necessary but not sufficient — a system can reach it without using the image.
• Progress should appear as higher scores at 90° and 180°. A system that raises its pooled score while leaving those at the bound has not improved at what the benchmark measures.
• Treat questions from the same room as dependent, and do not tune on the held-out questions.

@inproceedings{ahmed2027egoshift,
  title     = {EgoShift: Bounding the Action-Only Shortcut in Egocentric
               Spatial Updating for Vision-Language Models},
  author    = {Ahmed, Faisal and Quarles, John and Desai, Kevin},
  booktitle = {Proceedings of the IEEE/CVF Winter Conference on
               Applications of Computer Vision (WACV)},
  year      = {2027}
}