Add new type to the TypeSystem

The type-system is there to assure the communication between the individual annotators. New types are to be defined in the packages’ descriptors/typesystem/ folder in dedicated xml files.

Adding the new type

Go to the newly created rs_tutorial package and navigate to the upper mentioned folder:

roscd rs_tutorial
cd descriptors/typesystem

Here you will find a file called all_types.xml. Don’t pay attention to this file yet. We’ll get back to it later. Create a new xml file where your new types are going to reside:

touch test_types.xml

We will create a new type of annotation called MyFirstAnnotation and an atomic type called Centroid having the x,y,z as the parameters. Open the test_types.xml file with your favorite editor and add the following content to it:

<?xml version="1.0" encoding="UTF-8"?>
<typeSystemDescription xmlns="http://uima.apache.org/resourceSpecifier">
<name>test</name>
<description/>
<version>1.0</version>
<vendor/>
<imports>
</imports>
<types>
  <typeDescription>
    <name>rs_tutorial.test.Centroid</name>
    <description/>
    <supertypeName>uima.cas.TOP</supertypeName>
    <features>
      <featureDescription>
        <name>x</name>
        <description/>
        <rangeTypeName>uima.cas.Float</rangeTypeName>
        <multipleReferencesAllowed>false</multipleReferencesAllowed>
      </featureDescription>
      <featureDescription>
        <name>y</name>
        <description/>
        <rangeTypeName>uima.cas.Float</rangeTypeName>
        <multipleReferencesAllowed>false</multipleReferencesAllowed>
      </featureDescription>
      <featureDescription>
        <name>z</name>
        <description/>
        <rangeTypeName>uima.cas.Float</rangeTypeName>
        <multipleReferencesAllowed>false</multipleReferencesAllowed>
      </featureDescription>
    </features>
  </typeDescription>

  <typeDescription>
    <name>rs_tutorial.test.MyFirstAnnotation</name>
    <description>Centorid of a cluster</description>
    <supertypeName>rs.core.Annotation</supertypeName>
    <features>
      <featureDescription>
        <name>centroid</name>
        <description></description>
        <rangeTypeName>rs_tutorial.test.Centroid</rangeTypeName>
      </featureDescription>
      <featureDescription>
        <name>clusterId</name>
        <description></description>
        <rangeTypeName>uima.cas.Integer</rangeTypeName>
      </featureDescription>
    </features>
  </typeDescription>
</types>
</typeSystemDescription>

Call catkin_make in order to generate the container classes for the type-sytem, and resolve the dependencies of defined types (e.g. rs.core.Annotation). Notice that after compilation terminates, the following lines appear in your xml:

<!-- THESE IMPORTS WILL BE AUTOMATICALLY GENERATED BY A SCRIPT -->
<import location="../../../robosherlock/robosherlock/descriptors/typesystem/core_types.xml"/>

These imports get generated by one of the helper scripts during compilation, and import the definitions of other types, in our case core_types.xml, where the type rs.core.Annotation is defined. The script also edits the all_types.xml (this was the xml the newly created annotator imported in the previous tutorial used), adding the path to our new type descriptor. You can also have a look at the container classes that got generated in include/rs_tutorial/types/.

Note

When modifying this example, make sure that you are following the naming conventions for the new type. Otherwise your type might not get generated. More info about this at the end of this page.

Using it in the code

Now that a new type has been created, you can start using it from the annotators. For this purpose we will edit the source code of MyFirstAnnotator. Add these lines to the process function:

rs::Scene scene = cas.getScene();
std::vector<rs::ObjectHypothesis> clusters;
scene.identifiables.filter(clusters);
int idx = 0;
for (auto cluster:clusters)
{
    rs::ObjectHypothesis &c = cluster;
    pcl::PointIndices indices;
    rs::conversion::from(((rs::ReferenceClusterPoints)c.points()).indices(),indices);
    outInfo("ObjectHypothesis has "<<indices.indices.size()<<" points");

    Eigen::Vector4d pCentroid;
    pcl::compute3DCentroid(*cloud_ptr,indices, pCentroid);
    rs_tutorial::MyFirstAnnotation annotation  = rs::create<rs_tutorial::MyFirstAnnotation>(tcas);
    rs_tutorial::Centroid centroid = rs::create<rs_tutorial::Centroid>(tcas);
    centroid.x.set(pCentroid[0]);
    centroid.y.set(pCentroid[1]);
    centroid.z.set(pCentroid[2]);
    annotation.centroid.set(centroid);
    annotation.clusterId.set(idx++);
    c.annotations.append(annotation);
}

Don’t forget to include the necessary header files:

#include <rs_tutorial/types/all_types.h>
#include <pcl/common/centroid.h>

Now Compile and run.

If you have not modified your analysis engine nothing will happen. That is because the MyFirstAnnotator is placed right after the ImagePreprocessor in my_demo.xml. Move it down right after the ClusterMerger node. Run it now. You will see it outputting the number of points in each cluster it found. The annotator will store each clusters centroid as an annotation of our new type, allowing the retrieval of it from other components. For the sake of simplicity we will retrieve it from within the same annotator. Before the for loop ends add the following lines and recompile:

std::vector<rs_tutorial::MyFirstAnnotation> testAnnotations;
c.annotations.filter(testAnnotations);
if(testAnnotations.empty())
  continue;
outInfo("ObjectHypothesis "<<idx-1<<" has "<<testAnnotations.size()<<" annotation of type MyFirstAnnotation");
outInfo("ID is: "<<testAnnotations[0].clusterId() );
outInfo("x="<<testAnnotations[0].centroid().x());
outInfo("y="<<testAnnotations[0].centroid().y());
outInfo("z="<<testAnnotations[0].centroid().z());

Compile and run it again. You will see in the output the cluster IDs and their respective centroids.

This is of course a very simple example. The purpose of the type system is mainly to have higher level annotations that can come from different sources(annotators) represented in a unique manner, so that the results from multiple similar experts can be easily compared, ranked etc.

If you want to see how the rs_tutorial package should look like after following the tutorials until here, you can check out this repository: https://github.com/RoboSherlock/rs_tutorial .

Conventions for the naming of files and types

The script that is generating the concrete cpp classes from the type descriptions makes certain assumptions about the naming of the new typesystem and the individual types. If your definition is not following these conventions, it might happen that your type is not generated while no errors are printed. In order to avoid that, please keep the following things in mind when defining your types:

  • In the example above, we’ve created a type with the name rs_tutorial.test.Centroid. The scheme behind this is PACKAGENAME.TYPESYSTEMNAME.TYPE . You should always have all three of these different variables/names with a ‘.’ between them.

  • PACKAGENAME can either be rs or it should be the name of the RoboSherlock ROS package you are defining the type in. In the example above, this was rs_tutorial.

  • TYPESYSTEMNAME should match the value that you’ve set in <name> in the typeSystemDescription block (line 3 in the example above).

  • The filename of the XML should be TYPESYSTEMNAME_types.xml