AI Notes
Master robot perception. Sensors, computer vision. Robotics 2024.
What is Robot Perception?
Robot perception is the ability to interpret sensor data into meaningful representations of the environment. Raw sensor readings—laser distances, pixel intensities, force measurements—are transformed into semantic understanding: "there's a table 2 meters ahead with a cup on it." This transformation from raw data to actionable knowledge is what enables robots to interact intelligently with unstructured environments.
Perception bridges the gap between the physical world and the robot's computational decision-making. Without accurate perception, even the most sophisticated planning algorithms are useless—garbage in, garbage out. Modern robot perception combines multiple sensor modalities with deep learning to achieve robust environmental understanding.
Sensor Modalities
Vision (Cameras)
Types
RGB cameras: Color images (cheap, rich information)
Depth cameras: Per-pixel distance (Intel RealSense, Kinect)
Stereo cameras: Two cameras → depth from disparity
Event cameras: Detect brightness changes (ultra-fast, low power)
What vision provides
Object recognition: "That's a coffee mug"
Semantic segmentation: Pixel-level classification
Pose estimation: 6-DOF object position and orientation
Visual odometry: Motion estimation from image sequence
Limitations: Affected by lighting, occlusion, requires computation
LiDAR (Light Detection and Ranging)
| Principle: Emit laser pulse, measure return time | distance |
| 2D scanning LiDAR | Single plane, 360° (indoor robots) |
| 3D spinning LiDAR | 64-128 channels (self-driving cars) |
| Solid-state LiDAR | No moving parts (cheaper, smaller) |
| 3D point clouds | Precise geometry of environment |
| Accurate distance | ±2cm at 100m range |
| Works in dark | Active sensing, no ambient light needed |
| Limitations | Expensive, sparse data, affected by rain/fog |
Other Sensors
| Radar | Long range (200m+), works in all weather |
| Accelerometer + gyroscope | orientation and acceleration |
| Force/Torque sensors | Contact forces during manipulation |
| Tactile sensors | Surface texture, slip detection |
Sensor Fusion
Combining multiple sensors for robust perception:
| Camera | rich semantics, poor depth |
| LiDAR | precise depth, no color/texture |
| Radar | works in rain, low resolution |
| IMU | high rate, drifts over time |
| Together | compensate each other's weaknesses |
| Early fusion | Combine raw data (project LiDAR onto image) |
| Late fusion | Detect objects separately, merge detections |
| Deep fusion | Neural network learns optimal combination |
| Example | Self-driving car object detection |
| Fused | "Car at (10m, 3m, 0m), 2m×4m, moving at 30km/h" |
3D Perception
Point Cloud Processing
| Raw LiDAR output | Set of (x, y, z) points (100K-1M per scan) |
| 1. Ground removal: RANSAC plane fitting | remove road |
| 2. Clustering: DBSCAN | group points into objects |
| 3. Classification: PointNet/PointNet++ | identify object type |
| 4. Bounding box | Oriented 3D box around each cluster |
| Per-point features | max pooling (order invariant) → classification |
Depth Estimation from Images
| Monocular depth: Single image | depth map |
| Deep learning | trained on stereo pairs or LiDAR supervision |
| MiDaS, DPT | relative depth from any image |
| Stereo depth | Two cameras (known baseline) |
| Match features between images | disparity → depth |
| Structured light | Project known pattern, observe deformation |
Object Recognition and Detection
Detection pipeline for robots
Input: RGB image or point cloud
Output: [(class, bounding_box, confidence), ...]
2D Detection (images):
YOLO (You Only Look Once): Single-shot, real-time
Input image → grid → each cell predicts boxes + classes
YOLOv8: 80 FPS on GPU, 50% AP on COCO
Faster R-CNN: Two-stage, more accurate
Region Proposal Network → ROI features → classification
3D Detection (point clouds):
PointPillars: Encode points as vertical columns
VoxelNet: Voxelize + 3D convolution
CenterPoint: Detect object centers in bird's-eye view
Instance Segmentation
Mask R-CNN: Per-object pixel masks
Critical for: grasping (know exact object shape)
Perception for Manipulation
Grasp detection
Input: RGB-D image of objects
Output: Grasp pose (position + orientation of gripper)
Approaches
- Analytical: Compute force-closure grasps from 3D model
- Learned: CNN predicts grasp quality for candidate poses
- Sampling: Generate candidates, evaluate with learned scorer
6-DOF pose estimation:
"Where exactly is this object and how is it oriented?"
PoseCNN, DenseFusion: estimate from RGB-D
Critical for: assembly tasks, tool use
Category-level pose
"General mug pose" without CAD model of specific mug
Enables grasping novel objects of known category
Challenges in Robot Perception
| 1. Real-time constraint | Must process at sensor rate (10-30Hz) |
| Solution | Efficient architectures, GPU acceleration |
| 2. Domain gap | Training data ≠ deployment environment |
| Solution | Domain randomization, continual learning |
| 3. Occlusion | Objects hidden behind other objects |
| Solution | Multi-view reasoning, shape completion |
| 4. Lighting variation | Shadows, reflections, darkness |
| Solution | Multi-sensor fusion, active illumination |
| 5. Novel objects | Never-seen-before items |
| Solution | Zero-shot detection, foundation models |
Interview Questions
Q: Why use sensor fusion instead of the best single sensor? A: No single sensor excels in all conditions. Cameras fail in darkness, LiDAR fails in heavy rain, radar has low resolution. Fusion provides redundancy (safety-critical), complementary information (camera semantics + LiDAR geometry), and graceful degradation (if one sensor fails, others compensate). Self-driving cars use 6+ sensor types precisely because no single one is sufficient.
Q: How do robots handle novel objects they haven't been trained on? A: Through generalization at multiple levels: (1) Shape primitives—decompose unknown objects into known geometric primitives (cylinder, box, sphere) for grasping. (2) Foundation models—CLIP and SAM provide zero-shot recognition and segmentation. (3) Category-level reasoning—grasp "mug-like objects" without knowing this specific mug. (4) Active perception—move to get better viewpoints when uncertain.
Q: What is the role of semantic understanding vs. geometric understanding? A: Geometric understanding (depth, shape, position) enables collision avoidance and grasping. Semantic understanding (object identity, function, category) enables task-level reasoning: "pick up the tool" requires knowing which object IS a tool. Modern systems need both: geometry for safe motion, semantics for intelligent behavior.
Exam Focus
Revise definitions, diagrams, examples, and short-answer points for Robot Perception - Sensing & Vision.
Interview Use
Prepare one clear explanation, one practical example, and one common mistake for this Artificial Intelligence topic.
Search Terms
artificial-intelligence, artificial intelligence, artificial, intelligence, robotics, robot, perception, robot perception - sensing & vision
Related Artificial Intelligence Topics