Published · CVPR 2026 VidLLM Workshop 3D Scene Graphs Counterfactual Reasoning Physics-Grounded

CausalScene: Typed Causal Scene Graphs for Counterfactual Physical Reasoning

Vision-language systems describe what is visible but struggle to reason about what will happen. We argue the gap is structural, not a matter of scale — and close it with typed, physically grounded causal edges.

Noor Islam S. Mohammad1  ·  Uluğ Bayazıt2
1 Department of Computer Science  ·  2 Department of Computer Engineering  ·  Istanbul Technical University
{islam23, ulugbayazit}@itu.edu.tr
Paper (PDF) OpenReview Code CausalBench
Abstract

Reasoning about what will happen, not just what is visible.

Scene graphs encode where objects are but never how acting on one changes another. We present CausalScene, the first framework to equip 3D scene graphs with typed, physically grounded causal edges — encoding physical affordances, force-consequence relations, temporal ordering, and co-occurrence dependencies — predicted from 3D geometry, PyBullet physics priors, and LLM commonsense distillation.

Unlike free-form prompting, a Causal Query Language constrains an LLM to traverse the resulting Causal 3D Scene Graph (C3SG) along explicit causal paths, so every answer to a counterfactual query — "Will object A fall if object B is removed?" — is inspectable and verifiable rather than an opaque guess.

On CausalBench (8,543 counterfactual QA pairs over 1,247 real ScanNet scenes), CausalScene reaches 71.4% accuracy, surpassing GPT-4V+Projection by +15.3 and ConceptGraphs by +19.7 points, with the largest gains on physical stability. The advantage is backbone-agnostic: it holds across LLM families and scales, indicating that the gains come from causal structure rather than any particular model. The same graph lifts RLBench manipulation success from 47.2% to 63.1%, and sim-to-real transfer holds without domain adaptation.

71.4%
CausalBench accuracy
+19.7
pts over ConceptGraphs
8,543
counterfactual QA pairs
+20.2
pts from causal edges
Contributions

Four interconnected contributions.

A single structured representation that supports both spatial queries and counterfactual causal inference — grounded in physics priors and queryable in natural language.

i · C3SG

Causal 3D Scene Graph

We augment standard 3D scene graphs with four categories of causal edges — physical affordances, force-consequence, temporal orderings, and co-occurrence — predicted via 3D geometry analysis, PyBullet physics priors, and LLM commonsense distillation. Reasoning becomes inspectable: answers are supported by explicit causal paths, not opaque generation.

ii · CQL

Causal Query Language

A structured query interface that lets an LLM traverse the C3SG and answer counterfactual questions such as "Will object A fall if object B is removed?" in a principled, reproducible manner — constraining generation to causal paths licensed by geometry and physics.

iii · CausalBench

A new benchmark

8,543 counterfactual QA pairs spanning 1,247 real indoor ScanNet scenes, with typed causal ground truth from PyBullet simulation (stability, force propagation) and human annotation (temporal ordering, co-occurrence). The first benchmark for typed physical counterfactual reasoning at scale over real RGB-D 3D scenes.

iv · Generalization

Backbone-agnostic & sim-to-real

The C3SG advantage holds across four LLM backbones of differing scale and family, isolating causal structure — not model scale — as the source of improvement. Explicit sim-to-real experiments confirm reliable transfer to real-world RGB-D observations without domain adaptation.

The Pipeline

From RGB-D observation to inspectable counterfactual answer.

Three complementary sources predict typed causal edges over segmented object nodes. At inference, CQL hands the relevant causal subgraph to an LLM that answers via deterministic YES/NO log-probabilities.

INPUT RGB-D Scene point cloud SEGMENT 3D Instance Segmentation Mask3D · nodes + geometry Causal Edge Prediction PyBullet Physics Sim. force-consequence + affordance Geometric Analysis affordance (surface normals) LLM Commonsense co-occurrence + functional (GPT-4) C3SG A B C D E CQL · REASON Causal Query Language → LLM Qwen2.5-7B YES / NO log-prob True/False/Uncertain
Affordance Force-consequence Temporal Co-occurrence
The CausalScene pipeline. From an RGB-D observation, Mask3D recovers object nodes with geometric attributes. Three complementary sources predict typed causal edges; together with spatial edges they form the C3SG. At inference, CQL hands the relevant causal subgraph to an LLM backbone, which answers via deterministic YES/NO log-probabilities and a 3-way decision rule. Because every answer is grounded in an explicit causal path, reasoning remains inspectable rather than opaque.

Causal 3D Scene Graph G = (V, Eₛ, E_c)

A force-consequence edge exists iff PyBullet simulation of removing or displacing object oᵢ drives object oⱼ past a displacement threshold εd = 5 cm within horizon Ts = 3 s. Four typed edge categories — affordance, force-consequence, temporal, co-occurrence — capture complementary, non-overlapping causal signals.

Single-token elicitation 2 forward passes

For each query the LLM emits exactly one token. We softmax over {YES, NO}, query both consequence φ and its negation ¬φ, and apply a 3-way decision rule with thresholds (τ, δ) = (0.60, 0.10). This eliminates decoding temperature, hallucinated rationales, and format variance at once.

# 3-way decision rule p_phi = softmax(YES,NO | Q_φ) p_negphi = softmax(YES,NO | Q_¬φ) if p_phi ≥ τ and p_phi ≥ p_negphi+δ: TRUE elif p_negphi ≥ τ and p_negphi ≥ p_phi+δ: FALSE else: UNCERTAIN
Results

Structure, not scale, is the bottleneck.

CausalScene substantially and consistently outperforms all baselines across every question category, with statistically significant margins throughout (non-overlapping 95% bootstrap CIs).

CausalBench results — accuracy on all counterfactual categories (test split, n = 250 scenes). Brackets: 95% bootstrap CIs (B = 1,000, seed = 42).
MethodOverall ↑Phys. Stability ↑Force Prop. ↑Temporal ↑Co-occur. ↑Macro Avg. ↑
GPT-4V + Projection0.561.538–.5830.4730.5910.6040.5770.561
ConceptGraphs0.517.494–.5400.4210.5430.5610.5430.517
SpatialVLM0.534.511–.5570.4470.5630.5780.5490.534
CausalScene (ours)0.714.692–.7360.7410.7180.7030.6940.714
All differences between CausalScene and any baseline have non-overlapping 95% bootstrap CIs on every metric. The largest gain is on physical stability (+26.8 over GPT-4V+Projection), where physics-grounded force-consequence edges supply support- and mass-geometry signal that free-form VLM prompting cannot recover from images alone.
Backbone-agnostic analysis — the causal-structure gain Δ_C3SG (tens of points) dwarfs the spread across backbones (a few points). Default backbone in bold.
LLM BackboneParamsSpatial-only+ C3SG (full)Δ C3SG
Mistral-7B7B0.4980.701+20.3
LLaMA-3-8B8B0.5150.709+19.4
Qwen2.5-7B7B0.5120.714+20.2
GPT-4—0.5310.726+19.5
Spread across backbones—3.32.50.9
Adding causal structure is roughly an order of magnitude larger in effect than changing the language model — including the move to the substantially larger proprietary GPT-4. The ranking of representations is preserved under every backbone, ruling out a single favorable model as the source of the gain.
Sim-to-real transfer — edge-prediction pipeline trained on simulation only; no domain adaptation at test time.
CategorySimulationReal (ScanNet)Δ
Overall0.7270.683−0.044
Physical Stability0.7560.719−0.037
Force Propagation0.7310.660−0.071
Temporal Ordering0.7160.698−0.018
Co-occurrence0.7060.675−0.031
The 4.4-point overall gap is well within the +12.2-point real-world margin over the strongest baseline. The largest gap is on force propagation (−7.1), where real-world contact geometry deviates most from rigid-body simulation. Temporal and co-occurrence gaps are smallest, since those edges rely on LLM distillation, which is unaffected by the sim-to-real shift.
47.2→63.1
RLBench manipulation success (%)
4.4 pt
Sim-to-real gap (overall)
+26.8
pts on physical stability
κ = 0.81
Inter-annotator agreement
Ablations

Causal edges are the primary driver.

Adding causal edge types incrementally to the base spatial scene graph raises overall accuracy from 51.2% to 71.4% — a +20.2-point gain. Force-consequence edges drive the largest single jump.

Spatial edges only
51.2
+ Affordance
54.9+3.7
+ Force-consequence
61.0+6.1
+ Temporal ordering
63.7+2.7
+ Co-occurrence (full)
71.4+7.7
Spatial-only → full C3SG: +20.2 points overall  ·  +26.7 on physical stability (47.4% → 74.1%)

Removing physics priors costs 8.4 points on physical stability — the largest single-component gap — confirming that PyBullet simulation priors are irreplaceable for force-consequence reasoning. No single component is substitutable without category-specific degradation; only the full system achieves strong performance across all four question types simultaneously.

Outlook

A path to Video LLMs.

Videos are inherently causal: actions produce consequences, events follow temporal orderings, and object interactions unfold over time. The C3SG paradigm extends naturally to monocular video — without architectural redesign.

Temporal C3SG

We extend the C3SG to a temporal graph GT = (VT, EsT, EcT), where nodes are (object, time) pairs and temporal edges span across frames. The CQL elicitation protocol applies without modification.

  • Replace RGB-D segmentation with monocular depth + video instance segmentation
  • The causal edge taxonomy is modality-agnostic — all theoretical properties are preserved
  • Plugs into existing Video LLM pipelines without depth sensors
  • Enables online C3SG updates as object relations evolve over time
Qφ : Does action a on oᵢ at time t
cause φ(oⱼ) at time t′?

Q¬φ : Does action a on oᵢ at time t
prevent φ(oⱼ) at time t′?

— Video Causal Query Pair (Def. 10.1)
Citation

BibTeX

@inproceedings{mohammad2026causalscene,
  title     = {CausalScene: Typed Causal Scene Graphs for Counterfactual
               Physical Reasoning with a Path to Video LLMs},
  author    = {Mohammad, Noor Islam S. and Bayaz{\i}t, Ulu{\u{g}}},
  booktitle = {CVPR 2026 Workshop on Video Large Language Models (VidLLM)},
  year      = {2026},
  url       = {https://openreview.net/forum?id=WP6hAk9ani},
  organization = {Istanbul Technical University}
}