Does Point Cloud Boost Spatial Reasoning of Large Language Models?

Computer Science > Computer Vision and Pattern Recognition arXiv:2504.04540 (cs) [Submitted on 6 Apr 2025] Title:The Point, the Vision and the Text: Does Point Cloud Boost Spatial Reasoning of Large Language Models? Authors:Weichen Zhang, Ruiying Peng, Chen Gao, Jianjie Fang, Xin Zeng, Kaiyuan Li, Ziyou Wang, Jinqiang Cui, Xin Wang, Xinlei Chen, Yong Li View a PDF of the paper titled The Point, the Vision and the Text: Does Point Cloud Boost Spatial Reasoning of Large Language Models?, by Weichen Zhang and 10 other authors View PDF HTML (experimental) Abstract:3D Large Language Models (LLMs) leveraging spatial information in point clouds for 3D spatial reasoning attract great attention. Despite some promising results, the role of point clouds in 3D spatial reasoning remains under-explored. In this work, we comprehensively evaluate and analyze these models to answer the research question: \textit{Does point cloud truly boost the spatial reasoning capacities of 3D LLMs?} We first evaluate the spatial reasoning capacity of LLMs with different input modalities by replacing the point cloud with the visual and text counterparts. We then propose a novel 3D QA (Question-answering) benchmark, ScanReQA, that comprehensively evaluates models' understanding of binary spatial relationships. Our findings reveal several critical insights: 1) LLMs without point input could even achieve competitive performance even in a zero-shot manner; 2) existing 3D LLMs struggle to comprehend the binary spatial relationships; 3) 3D LLMs exhibit limitations in exploiting the structural coordinates in point clouds for fine-grained spatial reasoning. We think these conclusions can help the next step of 3D LLMs and also offer insights for foundation models in other modalities. We release datasets and reproducible codes in the anonymous project page: this https URL. Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) Cite as: arXiv:2504.04540 [cs.CV] (or arXiv:2504.04540v1 [cs.CV] for this version) https://doi.org/10.48550/arXiv.2504.04540 Focus to learn more arXiv-issued DOI via DataCite Submission history From: Weichen Zhang [view email] [v1] Sun, 6 Apr 2025 16:38:48 UTC (3,593 KB) Full-text links: Access Paper: View a PDF of the paper titled The Point, the Vision and the Text: Does Point Cloud Boost Spatial Reasoning of Large Language Models?, by Weichen Zhang and 10 other authorsView PDFHTML (experimental)TeX Source view license Current browse context: cs.CV < prev | next > new | recent | 2025-04 Change to browse by: cs cs.AI References & Citations NASA ADSGoogle Scholar Semantic Scholar export BibTeX citation Loading... BibTeX formatted citation × loading... Data provided by: Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Code, Data and Media Associated with this Article alphaXiv Toggle alphaXiv (What is alphaXiv?) Links to Code Toggle CatalyzeX Code Finder for Papers (What is CatalyzeX?) DagsHub Toggle DagsHub (What is DagsHub?) GotitPub Toggle Gotit.pub (What is GotitPub?) Huggingface Toggle Hugging Face (What is Huggingface?) ScienceCast Toggle ScienceCast (What is ScienceCast?) Demos Demos Replicate Toggle Replicate (What is Replicate?)…

This excerpt is published under fair use for community discussion. Read the full article at arXiv.org.