News and Blog¶
2021.10.01 Best Paper Award for a RoboSherlock paper
Our paper entitled “Imagination-enabled Robot Perception” won this years IROS Best Paper Award on Cognitive Robotics. In this work we are proposing a game engine based belief state extension to RoboSherlock, which enables robots to reason about perception beliefs by providing methods to compare a realistic rendering of the estimated belief with the observed sensor data.
2020.10.06 EASE Milestone Demonstration
In an extensive demo scenario, the PR2 robot of the EASE CRC demonstrated a large variety of manipulation actions in a household environment. A commented video of the demonstration can be seen here: https://www.youtube.com/watch?v=pv_n9FQRoZQ . RoboSherlock and RobotVQA provide the perception methods for the demo to enable the detection and pose estimation of objects in even challenging configurations.
2018.08.11 Implement Python Annotators GSoC’18: Shingo Kitagawa
Introduction
This project is Google Summer of Code(GSoC) 2018 program by Shingo Kitagawa. The GSoC project and proposal page is here.
Two thing below is done in the second period:
Implement Mask-RCNN instance segmentation annotator with Python library Chainer.
Train various deep learning model with large dataset.
This project is completed as GSoC project as belows:
Run Python code with boost:python and boost:numpy from C++ annotator.
Implement various annotators with Python library Chainer.
Train models for kitchen task with large dataset (Epic Kitchen Dataset, COCO).
Tutorials
Tutorial page of my GSoC project: Use Deep learning Python Annotator.
2018.08.01 Final blog of GSOC’18: An T. Le
Summary
The project “Flexible perception pipeline manipulation for RoboSherlock” is at its final stage.
Two things have been completed:
Finalize RSAggregatedAnalysisEngine (can be operated in linear or parallel) and port it to current pipeline execution implementation of Robosherlock.
Refactoring symmetry segmentation pipeline code from GSOC 2017, add some failsafe mechanisms and make the symmetry pipeline work in parallel.
All implementation so far does not require new dependencies.
Future development
These are points that need to be designed and implemented in the near future for RSAggregatedAnalysisEngine to be more reliable:
Implement failsafe mechanisms in all annotators of Robosherlock so that the pipeline will not crash and better handling errors.
Scrub dependency chain of annotators of robosherlock_knowrob for the RSParallelPipelinePlanner to plan correct parallel pipeline structure
symmetry segmentation pipeline needs redesign to work with the rest of Robosherlock as it currently only work on non-NAN clouds.
Some important bugs has discovered so far:
weird callbackKey issues in Visualizer of RoboSherlock
unit tests shutdown incorrectly (deadlock issues)
Results
Please clone my repo: https://github.com/anindex/robosherlock/tree/parallelism-dev and compile it.
Then run this command roslaunch robosherlock rs.launch ae:=demo wait:=false parallel:=true.
The parallel pipeline execution has successfully reduce the runtime of any pipelines: high frequency or expensive computation. It bypasses runtime of annotators in the same ordering,
the total time of each ordering is the longest process time of annotators in that ordering.
2018.07.12 Implement Python Annotators GSoC’18: Shingo Kitagawa
Introduction
This project is Google Summer of Code(GSoC) 2018 program by Shingo Kitagawa. The GSoC project and proposal page is here.
Two thing below is done in the second period:
Implement SSD detection annotator with Python library Chainer.
Implement FCIS instance segmentation annotator with Python library Chainer.
Annotate deep learning network output properly.
Approaches
The project is on progress as the flow below:
Run Python code with boost:python and boost:numpy from C++ annotator.
Implement various annotators with Python library Chainer.
Collect dataset and train models for kitchen task with PR2.
Formatize python template and make build system for Python annotators.
Run kitchen demo with PR2 in the real environment.
Now, I’m on 3rd and 4th stage of the flow. I trained SSD model with Epic Kitchens Dataset, but the dataset is captured mainly from first person view and contains small variety of object items (My training repository). Therefore, I need to train with another dataset for kitchen task. I will make a template and build system for python annotators and execute kitchen experiments with PR2.
Results
I implemented SSD detection annotator and FCIS instance segmentation annotator. The result are shown as below.
Tutorials
Tutorial page of my GSoC project: Use Deep learning Python Annotator.
2018.07.09 GSoC’18 - Shixin Li - Second Month Work
For the second month as a GSoCer in the Institute for Artificial Intelligence , I worked on the second phase of the project, which is adding proper error handler into uimacpp_ros so as to make debugging easier when something’s wrong. Original uimacpp usually just crashes and gives segfault, which is almost useless for programmers.
Firstly I tried to figure out some pretty frequent errors, including name typo of Analysis Engine, name typo of methods in fixed flow, missed annotator implementation name, mismatched parameter type and name clash of parameters. I finished the first four error checkings and tested if uimacpp or robosherlock already gives enough information. For name typo in fixed flow, it only gives segfault so I looked into uimacpp side and located where the program crashes, added explicit error messages to indicate the absence. For name clash, there is more work to be done.
Also, I further completed the migration of uimacpp to the ROS version, as well as for robosherlock. I re-investigated the original autotool configurations and made the new CMake configurations consistent as the counterpart. By doing this, I solved the process crash problem when operating with database. This pull request is finally merged into robosherlock master branch, and ready to be released in the hopefully near future.
2018.07.08 RSParallelAnalysisEngine GSOC’18: An T. Le
Introduction
The project is going on track as required of second evaluation of GSOC2018.
Two things have been completed:
Implement RSParallelAnalysisEngine that executes parallel pipeline based on the execution model as RSParallelPipelinePlanner outputs.
Port RSParallelAnalysisEngine to RSControledAnalysisEngine including setPipelineOrdering, resetPipelineOrdering, etc functionalities.
All implementation so far does not require new dependencies.
Approaches
RSParallelAnalysisEngine not only reserves core pipeline handling features by inheriting from uima::AggregatedEngine, but also expands pipeline handler to parallelism. Everytime pipeline xml file is loaded or setPipelineOrdering function in RsPipelineManager is called, it invokes RSParallelPipelinePlanner to query annotator dependencies from knowrob and then uses uses parallel pipeline orderings to begin pipeline execution. The parallel pipeline execution is implemented as following algorithm:
L - list of orderings
function parallelOrderingProcess
begin
for each ordering O in list L:
for each annotator Ai in O:
Ti = start thread calling PrimitiveEngine process (return error Ti);
lock until all Ti of O has been returned;
for each Ti in O:
if Ti != UIMA_ERROR_NONE:
print stack traces;
system exit with error;
end
RSParallelAnalysisEngine uses std::async and std::promise as its core components of implementation.
Results
The parallel pipeline execution result is hard to demonstrate. Please clone my repo: https://github.com/anindex/robosherlock/tree/parallelism-dev and compile it.
Then run this command roslaunch robosherlock rs.launch ae:=demo wait:=false parallel:=true.
I have noticed the performance issues of parallel pipeline execution due to std::promise bottleneck (avg execute time is worse than linear execution), I am actively investigating the root causes.
2018.07.03 RoboSherlock at IROS’18
Our paper [2018-1] on learning to recognize new objects through generating variations of scenes using the integration between the Unreal Engione and RoboSherlock has been accepted at this years IROS. We are exited to present our work at this venue, and share our insights on trainign perception algorithms for robtic applications through the use of syntetic data generated from virtual reality.
2018.06.26 Implement Python Annotators GSoC’18: Shingo Kitagawa
Introduction
This project is Google Summer of Code(GSoC) 2018 program by Shingo Kitagawa. The GSoC project and proposal page is here.
Two thing below is done:
Run Python code with boost:python and boost:numpy from C++ annotator.
Implement Faster-RCNN annotator with Python library Chainer.
Approaches
The project is on progress as the flow below:
Run Python code with boost:python and boost:numpy from C++ annotator.
Implement various annotators with Python library Chainer.
Collect dataset and train models for kitchen task with PR2.
Formatize python template and make build system for Python annotators.
Run kitchen demo with PR2 in the real environment.
Now, I’m on the 2nd and 3rd stage of the flow. The trained model with VOC dataset is not suitable for kitchen task with PR2, so I decided to train new model with kitchen dataset. After the training the model, I will make a template and build system for python annotators and execute kitchen experiments with PR2.
Results
I implemented Faster-RCNN object detection annotator.
2018.06.22 GOSC’18 - Shixin Li - First Month Work
For the first month as a GSoCer in the Institute for Artificial Intelligence , my work is mainly in converting the existing uima-cpp project , which is built with autotools, to a CMake/ROS-build project.
To be more specific, I wrote new CMake-related scripts and ros-related configuration files, such as package.xml for uima-ros project and got rid of dependency of all the autotools scripts. Meanwhile, I also modified the Robosherlock CMake-related files so as to integrate with the new UIMA-ros project, to make sure the new RS can include and link to uima-ros headers and libraries as before. After test, this part is regarded as completed, and ready to be released as a debian package.
2018.06.22 RSParallelPipelinePlanner GSOC’18: An T. Le
Introduction
The project is going on track as required by the scope of RoboSherlock on this year GSOC.
Two things have been completed:
Implement RSParallelPipelinePlanner that is able to mark execution orderings of annotators based on their required inputs and outputs.
Examine UIMACPP code repository, gain a deep level of how a AnalysisEngine execution calls annotators’ process.
All implementation so far does not require new dependencies.
Approaches
RSParallelPipelinePlanner planning approach is according to the flow:
Query annotator input output data from knowrob and store as std::map of annotator name and pair of input output list.
Build dependency graph from input output requirements of each annotator as Directed Graph
Generate annotator orderings marker based on the designed graph traveling algorithm.
Refine orderings to treat standalone annotators to execute last.
At current state, RSParallelPipelinePlanner works closely with RSControledAnalysisEngine. When a query is posted, the planner is triggered to query input output requirements of new pipeline annotator list. It then build dependency graph and run ordering marker, the pipeline structure is finally extracted from the markers. This model is foundation for next phase implementation of parallel annotators’ execution.
Results
Executing query {"detect":{"color":"red", "shape":"flat"}} will produce the result below:
2018.05.18 GOSC’18
We are happy to announce that the Institute for Artificial Intelligence has been accepted as a mentoring organization for this years Google Summer of Code. Three students will be actively working on improoving the RoboSherlock framework. An Le Thai, will implement functionalities for CAS splitting and merging as well as different strategies for these, while also looking into paralelle execution of AEs. Shingo Kitagawa will be investigating ways of implementing annotators in native Python code, and integrating deep learning approaches. Shixin Li will look into modernizing the uimacpp library and releasing it as a ROS package, paving the way for a future release of the RoboSherlock framework as a binary package. We wish them all success for the upcomming months.
2017.05.28 GOSC’17
RoboSherlock is participating in this years Google Summer of Code, under the umbrealla of the Institute for Artificial Intelligence. Mykola Dolhyi and An Le Thai are going to be working on implementing/integrating algorithms for hypotheses generation. We hope to have a successful collaboration during the upcomming months.
2016.04.21 RoboSherlock at ICRA16
Our paper [2016-1] on taskability of perception has been accepted at this years ICRA, in Stockholm, Sweeden. The paper presents an extension to RoboSherlock that allows a tighter coupling between knowledge processing and perception.
2016.01.17 RoboSherlock at ERF2016
We are happy to announce that the latest advances of RoboSherlock will be persented at the European Robotics Forum 2016 in Ljubljana, Slovenia, as part of the ‘Believing in Robot’s Eyes - Robust Observation of Task Relevant Cues’ Workshop.
2015.12.01 - RoboSherlock Released
The core implementation of RoboSherlock is now available as an open-source ROS package. Details about installation and getting started on the documentations web-page.
2015.05.28 - RoboSherlock at ICRA15
The reference paper presenting the ideas behind RoboSherlock [2015-2] received the Best Service Robotics Paper Award and was finalist for the Best conferenc Paper Award at ICRA’15 in Seattle, USA.
