Scenario 1: Class-based Grasping
RGB-D image with 1 to 5
objects of certain class
(approx. 100 categories)
Segmentation
(Walter, TUW)
segmented point clouds
Differently scaled
object model DB
(Walter, TUW)
Classification and pose estimation
(Walter, Aitor TUW, Marianna, KTH)
Categorized models with
estimated pose and scale
Deformable object registration/fitting
(Chavo, TUM)
Models fitted to the point cloud
in a deformable manner
Grasp hypotheses DB
(Beatriz, UJI)
Grasp selection
(Beatriz, UJI)
Robot model
(Otto-Bock, KIT)
Select a different
object in the scene
Best ranked grasp
hypothesis
Path planning
(Beatriz, UJI)
Trajectory
yes
Execute grasp and task
(TUW, Kuka, Otto-Bock)
no
Reachable?
SCENARIO 1
- How should the systems be adapted to vienna hw?
– backup platforms? Arm and hand models?
-- Finish working with ARMAR and UJI;
--task-based adaptation in Vienna UNLIKELY due to time constraints (robot
models unavailable yet)
-- Different grasps will be executed on the Michelangelo category-based, not taskbased
-- Can a database be collected by Vienna for training Marianna’s system? (kinect
and plane-based segmentation)
- Include task into the flow chart
- Who is giving the task? Through what channel (speech, keyboard, etc)?
- If possible, Task-based execution coming from Scenario3
- SPEED?
- Walter is the interfacing/integration responsible
Scenario 2: Empty the Basket
Input 1: RGB-D image
of scene with basket
and unknown objects
Basket detection and attention points
(TUW(Kate))
Attention points
Basket Pose
(under)Segmentation of object(s) in basket
(TUW(Kate), KTH?)
Region hypotheses
3D part detection
(TUW(Karthik))
Superquadric
Superquadric mesh
& Pile mesh
Grasp Hypothesis Generation
(TUW(Karthik))
Ranked list of
Grasp Point +
Approach vector
Online path planning
(UJI(Beatriz))
Output: Execute on
TUW Amtec robot
and old OB hand
Trajectories
in Joint-Space
Reachable?
Yes
No
Select next region
in image
SCENARIO 2
- robustness of attention point detection and basket detection
– Should be OK as soon as basket points are filtered out. Detection of points slightly out of the pile due to randomness will be solved. Work will be put in speeding it up. ROS
package
- should there be a simple milestone before the demo?
- plan B:
- empty a basket of known objects?
- part-based grasping of easily segmentable unknown objects
- use attention points (TUW) to provide approach vectors to UJI emptying-box approach?
•
Kate will send Javier (UJI) code (ROS package) for the attention points
- SEGMENTATION WON'T WORK ROBUSTLY. Is robust segmentation critical for part detection?
– Segmentation actually looks reasonably good. Plus multiple view could improve robustness, human could enter the scene and rotate the basket.
- Optimization (~20s right now) and integration (ROS-Matlab) required
- 3DPart detection seems to be working in not too crowded scenes, not tested in extra-crowded boxes (probably not needed)
-Open-loop grasping? Calibration.
- ??
- SPEED?
- some modules need to be sped up
- UJI still has to provide the path planning based on the pile mesh and superquadric
- Vienna should know by 30/11 more about the feasibility of this scenario
- Software interface of Trajectory from OpenRAVE and physical robot
TECHNICAL ANNEX:”Grasping any object by building up relations between task setting, embodied hand actions, object attributes, and contextual knowledge such that learned
grasps are extendable towards new, never seen objects in new situations […] Progress will be measured along two benchmarks as exemplars of industrial and home tasks:
–
Emptying a box of industrial parts and
–
Emptying a basket of grocery item
- David (TUW) is integration responsible
Task Recognition on Human Demo
Possible objects: hammer, knife, screwdriver, mug, bottle
Input:
Scene with known objects
(as many as possible)
Possible tasks: pouring, tool-use, dishwashing, playing
Object Recognition & Pose Estimation
(Chavo, TUM)
Which objects,
where placed
in the scene
Object Model DB
with Object Features
Object index
Hand/object Tracking
(Iason, Niko, FORTH)
Object-centered
hand pose
(position/orientation)
Object category,
Size, Convexity,
etc
Task Recognition
(Dan, KTH)
Output:
Task for which the observed
grasp is good for
SCENARIO 3
-(Chavo-FORTH)Hand initialization?
-Multi-object tracking? Should we go for single object? Multi-hand?
- Dan needs object and hand pose. FORTH-KTH different hand models?
- Maybe use task information for scenario 1.
- Which sensors will be used? Chavo prefers KINECT, Forth might work with
KINECT, but has worked previously with portable multicam setup.
- Iason is responsible for integration
Video-Demos
- Ville: adaptive grasping?
– Combined LUT and UJI? not implemented in ARMAR
– Collaboration with Vienna?
- UJI and ARMAR task and category-based grasping videos.
- Video on surprise grasping
– Update scenario knowledge, collaboration between FORTH and TUM
- Videos from individual partners
- Benchmarking environment (KIT)
–
Different physics engines for dynamic grasping environments
–
Object reconstruction from incomplete pointclouds
–
Grasp planners on incomplete pointclouds