ROSInfer
Finding Architectural Bugs in ROS-based Robotics Systems
Description
Robotics systems built on the Robot Operating System (ROS) are assembled from hundreds of independently developed software components that interact at run time by passing messages. When those components are composed inconsistently, the system can fail in ways that are hard to debug — a component waits forever for a message that no other component sends, two components deadlock, or messages are silently dropped because a component is in the wrong state. ROSInfer automatically recovers behavioral component models directly from ROS source code so that these behavioral architecture composition bugs can be found before a robot ever moves.
Problem: Finding Behavioral Architecture Composition Bugs in ROS-based Systems
Robotics systems are complex, safety-critical, component-based systems. Their components are typically reactive (they respond to incoming messages), periodic (they publish data at a fixed frequency), and state-dependent (what they do depends on what they have already received). Composing and evolving so many components is error-prone, because each one makes undocumented assumptions about its environment — for example, that it will receive a set of initialization messages before it starts working.
When those assumptions are violated, the behavior of the assembled system diverges from what the developers expected: a component indefinitely waits for a required input, the system reaches a deadlock, or inputs are ignored because the system is in an incorrect state. These bugs are especially hard to find because components often fail silently and the fault is spread across many different locations in the system rather than sitting in one place.
Approach: Automatically Inferring Behavioral Component Models from Code
Model-based analysis is a well-proven way to catch incorrect behavioral composition — but it needs formal models, and in practice robotics teams rarely have them. Writing and maintaining state-machine models for hundreds of components by hand is labor-intensive and error-prone, and the models would have to be rewritten every time the code changes.
This project makes that analysis practical by inferring the models automatically. The key insight is that only a small slice of a component's behavior is architecturally relevant (what causes it to send messages: its triggers, state variables, and transitions), and that ROS developers implement that behavior using a small, recognizable vocabulary of framework API calls and idioms. By exploiting assumptions about how the ROS API and ecosystem are used, the project recovers a state-machine model of each component's architecturally-relevant behavior by combining static and dynamic analysis.
Contributions
The project contributes four tools that take a system from source code to checkable models and readable diagrams:
- ROSInfer — static recovery
- Statically reconstructs per-component state machines by locating the ROS API calls that define architecturally-relevant behavior (subscriber callbacks for receiving messages, publish calls for sending them, and rate/sleep calls for periodic behavior) and following the control and data flow between them. Values that cannot be determined statically are marked as an explicit “unknown” so the models are honest about their gaps.
- ROSInstrument — dynamic completion
- Uses the partial static models to drive targeted experiments: it deploys components and systematically sends messages to them, instrumenting the unknown source locations to observe their real values at run time (such as publishing frequencies or values loaded dynamically), filling in what static analysis alone could not.
- ROSFindBugs — model checking
- Translates the inferred models into PlusCal/TLA+ and checks properties that matter in the robotics domain — for example, that every required input is eventually produced by some component, that an input reliably causes its expected output, and that message-publishing frequencies stay within safe bounds — to flag architecture-misconfiguration bugs.
- ROSView — visualization
- Automatically turns the inferred models into medium-grained diagrams that combine the component-port-connector structure with behavior (initial state, periodic triggers and their frequency, and conditional reactions), so developers can understand a component's architecturally-relevant behavior at a glance.
Results
- Accurate static inference at scale. ROSInfer was evaluated on five large, real-world ROS systems (Autoware.AI, AutoRally, Fetch, Husky, and Turtlebot) totaling more than 500 components. Against 155 hand-built ground-truth models — released as a dataset for other researchers — it inferred the causal trigger for about 88% of architecturally-relevant behaviors, with a recall of 95% (periodic), 86% (reactive), and 75% (state) and a precision of 91–100% across behavior types. The analysis scales to whole systems and supports incremental re-analysis as code changes.
- Finding real bugs other tools miss. ROSFindBugs found real behavioral architecture composition bugs in open-source systems that earlier purely-structural recovery could not detect. The project also contributed datasets of real-world architecture misconfiguration bugs (29 across five systems) and behavioral architecture composition bugs (20 mined from open-source ROS projects) to support further research.
- Diagrams that help developers. In a controlled study with 24 practicing roboticists and robotics graduate students, participants given ROSView's automatically generated diagrams answered questions about component behavior more accurately. For a seeded composition bug, everyone with the diagram located it (versus half of those without) and 92% fixed it (versus 42%) — without taking any more time.
For ROS Developers
We are looking for ROS developers who would like to apply our tools to their projects. Currently our tools only support ROS 1 projects written in C++ (we might add ROS 2 and Python later). If you are interested, please use the form below.
We are also very interested in learning about real-world bugs that developers experience in their projects, and challenges in composing architectures. If you would like to share your experience with us, please send us a message! This will help us to build tools that are more usable in your practical setting.
Talks
Video description: This recorded conference talk presents ROSDiscover, a static analysis approach for recovering ROS run-time architecture from code and launch files. The visual track primarily shows presentation slides with the paper title, architecture-recovery workflow, examples of ROS component connections, and evaluation results; the spoken talk explains the same research claims and examples.
Publications related to this Project
2024
ROSInfer: Statically Inferring Behavioral Component Models for ROS-based Robotics Systems
Abstract, BibTeX, Tool, Paper (Open Access)
Robotics systems are complex, safety-critical systems that can consist of hundreds of software components that interact with each other dynamically during run time. Software components of robotics systems often exhibit reactive, periodic, and state-dependent behavior. Incorrect component composition can lead to unexpected behavior, such as components passively waiting for initiation messages that never arrive. Model-based software analysis is a common technique to identify incorrect behavioral composition by checking desired properties of given behavioral models that are based on component state machines. However, writing state machine models for hundreds of software components manually is a labor-intensive process. This motivates work on automated model inference. In this paper, we present an approach to infer behavioral models for systems based on the Robot Operating System (ROS) using static analysis by exploiting assumptions about the usage of the ROS API and ecosystem. Our approach is based on searching for common behavioral patterns that ROS developers use for implementing reactive, periodic, and state-dependent behavior using the ROS framework API.We evaluate our approach and our tool ROSInfer on five complex real-world ROS systems with a total of 532 components. For this purpose we manually created 155 models of components from the source code to be used as a ground truth and available data set for other researchers. ROSInfer can infer causal triggers for 87 % of component architectural behaviors in the 532 components.
@inproceedings{Duerschmid2024ROSInfer,
author = {D\"{u}rschmid, Tobias and Timperley, Christopher Steven and Garlan, David and {Le~Goues}, Claire},
title = {ROSInfer: Statically Inferring Behavioral Component Models for ROS-based Robotics Systems},
year = {2024},
isbn = {9798400702174},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3597503.3639206},
doi = {10.1145/3597503.3639206},
booktitle = {Proceedings of the IEEE/ACM 46th International Conference on Software Engineering},
articleno = {144},
numpages = {13},
location = {Lisbon, Portugal},
series = {ICSE '24}
}
2022
ROSDiscover: Statically Detecting Run-Time Architecture Misconfigurations in Robotics Systems
Abstract, BibTeX, DOI, Tool, Talk, Replication Package
Robot systems are growing in importance and complexity. At the same time, it is becoming more important for them to be free of errors. Ecosystems for robot software, such as the Robot Operating System (ROS), provide libraries of reusable software components that can be configured and composed into larger systems. To support compositionality, ROS uses late binding and architecture configuration via “launch files” that describe how to initialize the components in a system. However, late binding often leads to systems failing silently due to misconfiguration, for example by misrouting or dropping messages entirely. In this paper we present ROSDiscover, which statically recovers the run-time architecture of ROS systems to find such architecture misconfiguration bugs. First, ROSDiscover constructs component level architectural models (ports, parameters) from source code. Second, architecture configuration files are analyzed to compose the system from these component models and derive the connections in the system. Finally, the reconstructed architecture is checked against architectural rules described in first-order logic to identify potential misconfigurations. We present an evaluation of ROSDiscover on real world, off-the-shelf robotic systems, measuring the accuracy, effectiveness, and practicality of our approach. To that end, we collected the first data set of architecture configuration bugs in ROS from popular open-source systems and measure how effective our approach is for detecting configuration bugs in that set.
@inproceedings{Timperley2022ROSDiscover,
author = {Timperley, Christopher S. and D\"{u}rschmid, Tobias and Schmerl, Bradley and Garlan, David and {Le~Goues}, Claire},
title = {ROSDiscover: Statically Detecting Run-Time Architecture Misconfigurations in Robotics Systems},
booktitle = {Proceedings of the 19th IEEE International Conference on Software Architecture.},
series = {ICSA '22},
year = {2022},
numpages = {12},
location = {Honolulu, HI, USA},
publisher = {IEEE},
pages={112-123},
doi={10.1109/ICSA53651.2022.00019}
}