Proposed approach to plant point cloud registration

Our approach operates on a time-series of 3D point clouds of plants. Our registration procedure starts with extracting a skeleton along with the organ level semantics for each point cloud. The skeletons are undirected acyclic graphs, which represent the topology or the inner structure of the plant. Each node contains the x, y, z coordinates of its position, a 4x4 affine transformation matrix T to describe the local transformation, and a semantic class label as attributes. The skeletons are extracted from the point cloud data and are often imperfect. This fact is taken into consideration during the registration procedure. We also operate directly on the ordered point clouds and do not require a mesh structure or other cues such as the normals providing the inside-outside information of the surface. The skeleton structures allow us to compute data associations between temporally separated 3D scans and use these correspondences to perform an iterative procedure which registers the plant scans obtained at different times. Finally, the registered skeletons can be used to deform the complete point cloud, e.g., the point cloud from time step t₁ deformed to time step t₂. The overall registration scheme is summarized in Alg. 1 and will be explained in detail in the following sections.

Algorithm 1 Skeleton-driven iterative non-rigid registration procedure

1: $$ {\mathcal{P}}_1,{\mathcal{P}}_2$$ ⊳ Input point clouds

2: $$ {\mathcal{C}}_{12}^{t-1},{\mathcal{C}}_{12}^t=\varnothing $$ ⊳ Initialization

3: $$ {\mathcal{O}}_1\leftarrow \text{performInstanceSegmentation}\left({\mathcal{P}}_1\right)$$ ⊳ Segment $$ {\mathcal{P}}_1$$

4: $$ {\mathcal{O}}_2\leftarrow \text{performInstanceSegmentation}\left({\mathcal{P}}_2\right)$$ ⊳ Segment $$ {\mathcal{P}}_2$$

5: $$ {\mathcal{S}}_1\leftarrow \text{computeSemanticSkeleton}({\mathcal{P}}_1,{\mathcal{O}}_1)$$ ⊳ Compute skeleton $$ {\mathcal{S}}_1$$

6: $$ {\mathcal{S}}_2\leftarrow \text{computeSemanticSkeleton}({\mathcal{P}}_2,{\mathcal{O}}_2)$$ ⊳ Compute skeleton $$ {\mathcal{S}}_2$$

7: while $$ ({\mathcal{C}}_{12}^t\backslash {\mathcal{C}}_{12}^{t-1})\cup ({\mathcal{C}}_{12}^{t-1}\backslash {\mathcal{C}}_{12}^t)=\varnothing $$ do ⊳ Repeat until matches are same

8:  $$ {\mathcal{C}}_{12}^{t-1}={\mathcal{C}}_{12}^t$$

9:  $$ {\mathcal{C}}_{12}^t\leftarrow \text{findSkeletalCorrespondences}({\mathcal{S}}_1,{\mathcal{S}}_2)$$ ⊳ Compute matches

10:  $$ {\mathcal{T}}_{12}\leftarrow \text{compSkeletalDeformation}({\mathcal{S}}_1,{\mathcal{S}}_2,{\mathcal{C}}_{12}^t)$$ ⊳ Compute deformation

11: $$ \widehat{{\mathcal{P}}_1}\leftarrow \text{applyDeformation}({\mathcal{P}}_1,{\mathcal{T}}_{12})$$ ⊳ Apply deformation to $$ {\mathcal{P}}_1$$

Extracting the skeletal structure along with semantics

The first step of our approach is to extract the skeleton structure $$ \mathcal{S}$$ of the plant from the input point cloud $$ \mathcal{P}$$. To achieve such goal, we first perform a segmentation step aiming at grouping together points belonging to the same plant organ, namely one leaf instance or the stem. To do this, we start by classifying each point of the point cloud $$ \mathcal{P}$$ as a point belonging to either the stem or leaf category. We use a standard support vector machine classifier with the x, y, z coordinates along with with the fast point feature histograms [71] as a feature vector. The fast point feature histograms technique computes a histogram of directions around a point using the neighborhood information, thereby capturing the local surface properties in a compact form. With these feature vectors as inputs, the support vector machine can classify each point of a plant point cloud into stem and leaf points. We train the support vector machine model using the scikit-learn library [72] in a supervised manner by providing labels for two randomly sampled scans of the temporal sequence as the training set.

After the model is trained, we use it to predict the semantics (category) for all the remaining point clouds of the sequence. Once the classification step is complete, we perform a clustering step in order to find the individual leaves or the stem as unique instances. We perform the clustering using the density-based spatial clustering algorithm [73]. It uses the x, y, z coordinates of the points to obtain an initial segmentation, which is then refined by discarding small clusters and assigning each discarded point to one of the remaining clusters based on a k-nearest neighbor approach.

At this stage, each point in the point cloud is assigned to an organ, namely to the stem or to one leaf instance. We then learn keypoints for each organ using self-organizing maps [74]. These keypoints would form the nodes of the skeleton structure. Self organizing maps are unsupervised neural networks using the concept of competitive learning instead of back-propagation. They take as input a grid that organizes itself to capture the topology of the input data. Given an input grid $$ \mathcal{G}$$ and the input set of points $$ \mathcal{P}$$, the self organizing map defines a fully-connected layer between $$ \mathcal{G}$$ and $$ \mathcal{P}$$. This network learns a transformation the grid $$ \mathcal{G}$$ points in manner to cluster the data $$ \mathcal{P}$$ effectively. The learning process is composed of two alternating steps until convergence. First, the winning unit is computed as:

Fig 2

Extracting skeletal structure for using semantics of the plant.

The figure illustrates the skeletonization pipeline for a maize (top) and tomato (bottom) plant scan. Note that for the tomato plant, we classify individual leaflets (green + yellow + light-blue) as separate instances rather than as an individual leaf. The leaflets can be combined into a single leaf in case this distinction is not desired/required for the application.

Estimating skeletal correspondences

Before data from any 3D objects can be aligned, we need to establish the data associations between the sensor readings, i.e, estimating which part of cloud $$ {\mathcal{P}}_1$$ corresponds to which parts of cloud $$ {\mathcal{P}}_2$$. Establishing this data association is especially hard for objects that change their appearance and automatic processes are likely to contain data association outliers, which will affect the subsequent alignment. For registering temporally separated plant scans, we propose to obtain these data associations by matching the corresponding skeleton structures and not work directly on the raw point clouds.

In this data association step, we estimate correspondences between two skeletons by exploiting their geometric structure and semantics, which are computed using the approach described in the previous section. As the skeleton structure and the semantics are estimated from sensor measurements, it might suffer from several imperfections. To cope with these imperfections in the individual skeletons and inconsistencies between them, we use a probabilistic approach to associate the skeleton parts as opposed to graph matching approaches, which typically do not tolerate such errors well. We, therefore, formulate the problem of finding correspondences between the skeleton pair ($$ {\mathcal{S}}_1$$,$$ {\mathcal{S}}_2$$) using a hidden Markov model formulation [75]. The HMM model provides the flexibility to encode different cues, define constraints for the correspondences, as well as include prior information about the skeleton structure. This allows us to track several correspondence candidates and choose the best correspondences between the skeleton pair.

The unknowns or the hidden states of the HMM model represent all the potential correspondences between the nodes of the two skeletons. In addition, we also add a so-called “not matched state” for each node to the HMM to account for the situations in which the node may have no correspondences at all. This happens, for example, when nodes belong to new organs that were not present before or when new nodes emerge on the curve skeleton due to the plant growth. As required in a HMM formulation, we need to define the emission cost Z and the transition cost Γ. The emission cost Z describes the cost for a given hidden state (correspondence) to produce a certain observation. In our case, the observations are the sequence of nodes of the first skeleton $$ {\mathcal{S}}_1$$ arranged in depth first manner starting from the node at the base of the stem. We define this cost for a correspondence $$ c_{ij}\in {\mathcal{C}}_{12}$$ between node n_i of $$ {\mathcal{S}}_1$$ and node n_j of $$ {\mathcal{S}}_2$$ as:

The transition cost Γ describes the cost involved in transitioning from one hidden state c_ij to another c_kh. This can be treated as the cost involved in having c_kh as a valid match given that c_ij is a valid match as well. We define this cost as:

Once the emission and transition costs are defined, we compute the correspondences between the skeletons by performing inference on the HMM. The result is the most likely sequence of hidden variables, i.e., the set of correspondences between $$ {\mathcal{S}}_1$$ and $$ {\mathcal{S}}_2$$. We perform this inference using the Viterbi algorithm [76]. In case a node has more than one correspondence, we choose the correspondence with the smaller Euclidean distance to ensure a one-to-one correspondence. As an illustration, Fig 3 (left) shows an example skeleton pair for which we want to estimate the correspondences $$ {\mathcal{C}}_{12}$$. Fig 3 (right) depicts the HMM for the example pair where the red path indicates the set of correspondences estimated by the Viterbi algorithm. The HMM model only shows a subset of the connections between the hidden states, where in practice each state is connected to every other state.

Fig 3

Left: Skeletal matching for an example pair of plant point clouds with the variables involved. Right: Hidden Markov model (HMM) used for correspondence estimation. We only show a subset of the hidden variables, i.e. the potential correspondences, in the HMM. The red line depicts the sequence of best correspondence estimated by the Viterbi algorithm. This produces the correspondences between $$ {\mathcal{S}}_1$$ and $$ {\mathcal{S}}_2$$ visualized with the dash-lined arrows on the left.

Computing skeletal deformation parameters

In this step, we compute the registration parameters between $$ {\mathcal{S}}_1$$ and $$ {\mathcal{S}}_2$$ given the set of correspondences $$ {\mathcal{C}}_{12}$$. While registering temporally separated plant scans, we need to take into account the plant growth, which manifests in the form of change in shape and the topology of the plant. Therefore, to capture these changes we need to forego the usual assumption of rigidity, often used in point cloud registration [8]. Our goal is to capture the non-rigid changes by computing sets of deformation parameters between skeleton parts of the respective plant scans. We estimate these deformation parameters through a non-linear least-squares optimization procedure based on the correspondences obtained from the procedure described in the previous section.

To model the deformations between the plant scans, we attach an affine transformation T_i to each node n_i of the skeleton $$ {\mathcal{S}}_1$$ as illustrated in Fig 4 (left). The intuition behind such a model is that the skeleton may be deformed differently at different locations along the skeleton. By modeling the deformations through a 3D affine transformation with 12 unknown parameters per node, we are able to capture the growth as well as bending of the plant via the scaling, shearing, and rotation parameters.

Fig 4

Left: Registering the skeleton pair involves estimating the deformation parameters attached to the nodes of the source skeleton $$ {\mathcal{S}}_1$$. Right: Transferring the deformation results to the entire point cloud.

We define the objective function of the optimization problem as a combination of the three energy terms. The first term E_corresp is defined as:

The second energy term E_rot captures how close the estimated affine transformation is to a pure rotation and it determines the smoothness of the deformation. We define E_rot as:

We also define a regularization term E_reg as:

We use the weights w_corresp = 100, w_rot = 10, and w_reg = 1 for all the example in our datasets. The weights have been chosen such that the cost due to each component of the loss is in the same order of magnitude. We employ the Gauss-Newton algorithm to solve the unconstrained non-linear least squares problem [77]. We use the Cauchy robust kernel [78] for the error residuals belonging to E_corresp as this prevents incorrect correspondences to influence the optimization process. The robust kernel automatically downweights potentially wrong correspondences, which have large residuals. The approach is related to the formulation by Sumner et al. [63] for estimating deformation parameters for surfaces parametrized as triangular meshes. In our case, we adapt the energy terms to reflect the constraints valid for deformation between curve skeletons as opposed to surfaces. Furthermore, the approach by Sumner et al. [63] is not able to fully constrain the nodes which have a degree smaller than 3, but is essential for registration of curve skeletons.

Point cloud deformation

Traditional approaches to point cloud registration assume rigid objects. In this case, the alignment results in the execution of a 6 degree of freedom transformation consisting of rotations and translations. This, however, is substantially different in our case. To obtain the final registered point cloud $$ {\widehat{\mathcal{P}}}_1$$ of a growing plant, we need to apply the deformation parameters estimated for the skeleton nodes to all the 3D points of the scan. This means, that the individual data points will be affected by individual affine transformation to obtain the aligned cloud. An example of the point cloud deformation is visualized in Fig 4 (right).

For each point $$ p\in {\mathcal{P}}_1$$, we obtain the deformed point $$ \widehat{p}$$ as a weighted sum of affine transformations corresponding to the two nearest nodes to the point p as

Iterative non-rigid registration procedure

We use the steps from the previous sections to formulate an iterative approach to register the point cloud pair $$ ({\mathcal{P}}_1,{\mathcal{P}}_2)$$ as summarized in Alg. 1. Similar to the popular ICP approach [8], we alternate between correspondence estimation steps and registration steps given the correspondences. We start out by computing the organ level instance segmentation and skeleton structure with semantic information (lines 3-6 of Alg. 1). We then start the iterative procedure, which alternates between estimating the correspondences $$ {\mathcal{C}}_{12}$$ (line 9) and the registration parameters, i.e., the 3D affine transformations attached to each node (line 10). By iterating through these steps multiple times, we can obtain new correspondences, which might not have been captured due to the large distance between the skeletons given their initial configuration. Finally, we exit the iterative scheme when there is no change in the estimated correspondence set $$ {\mathcal{C}}_{12}^t$$. After computing the registration parameters $$ {\mathcal{T}}_{12}$$ between the nodes of the skeletons $$ {\mathcal{S}}_1$$ and $$ {\mathcal{S}}_2$$, we apply these parameters to the entire point cloud $$ {\mathcal{P}}_1$$, which results in the registered point cloud $$ \widehat{{\mathcal{P}}_1}$$ (line 11).

Interpolating point clouds of plants

In addition to registering the plant scans recorded at different times, we would also like to interpolate how the plant may be deformed at an intermediate time in between the actual acquisition times. This may allow for increasing the time between the recordings and doing further analysis on the interpolated point clouds. We compute the deformed point cloud by interpolating the deformation parameters T estimated between the two registered scans. To obtain a smooth interpolation, we first decompose of the estimated affine transformation T into scale/shear transformation T_s, pure rotation T_R, and a pure translation T_t using the polar decomposition approach described by Shoemake [79] and is given by:

We then linearly interpolate T_s and T_t to obtain the transformation at time t. For interpolating T_R, we use the spherical linear interpolation described by Shoemake [80].

Computing phenotypic traits

In this work, we compute different phenotypic traits for both the stem and leaf instances. In particular, for the stem class, we compute stem diameter and stem length, whereas for the leaf class, we compute the leaf length and area. We choose these traits as examples, and other geometric traits that use the area or shape of the organs can be computed and tracked over time as well.

The stem and leaf length can be easily computed from the geometric information of the semantic skeleton: The stem/leaf length is just the sum of the lengths of the edges of all nodes of a particular instance.

For computing the stem diameter s_d, we first assign each point in the point cloud classified as stem to the closest node on the skeleton. Then, we compute the main axis of the data distribution in the neighboring region of the selected node using a standard singular value decomposition (SVD) approach. We can then compute the stem diameter with respect to the main axis of the considered neighborhood. The stem diameter s_d is obtained as:

We compute the leaf area l_a by exploiting the SVD to estimate the main plane A of the points associated with each node. This gives us the leaf area l_a as:

Dataset description

We evaluate our approach on time-series 3D point cloud data of three sample plants of maize (Zea mays) and tomato plants (Solanum lycopersicum). The scans were recorded using a robotic arm (Romer Absolute Arm) [81] equipped with a high precision laser scanner. The dataset was recorded daily over a period of 10 days. The plants have been scanned in manner so as to minimize self occlusions whenever possible. The point cloud data undergoes a pre-processing step where all the points which do not belong to the plant are removed, for example the points belonging to the soil and the pot. The datasets cover substantial growth of the plants, starting from the 2 cotyledons (i.e. seed leaves) at the start to around 8 leaflets (2 cotyledons + 2 leaves) for the tomato plants, and 1 to 4 leaves for the maize plants. The plants undergo substantial leaf and stem growth, and includes several branching events till the end of the dataset acquisition period as illustrated in Fig 1. The height of the tomato plants reaches up to 150 mm and the maize plant reaches over 400 mm. The data used in the paper is available at: https://www.ipb.uni-bonn.de/data/4d-plant-registration/.

Semantic classification of plant point clouds

In the first experiment, we evaluate the performance of our approach for organ level semantic classification. The classification system has been trained on two randomly selected point clouds from each datasets. All the remaining point clouds in the sequence are used as test datasets. The ground truth information for both the training and test sets have been generated manually by human users. We show the qualitative results computed by our approach by visualizing the semantic instances for some point clouds from the two datasets in Fig 5. Each stem and leaf instance is visualized with a different color. We can visually inspect the stem and the leave instances throughout the temporal sequence, and see that the classification is successful for instances despite their size and shape changing over time. The colors of same leaf instances do not persist over the temporal sequence since the data associations between them have not been computed at this stage.

Fig 5

Semantic classification of maize (top) and tomato (bottom) point clouds.

Each stem and leaf (or leaflet) instance is visualized with a different color. Note that the colors of same leaf instances do not correspond over time, as data associations have not been computed at this stage.

We also perform a quantitative evaluation of the classification performance of our classification approach by computing standard metrics [82] such as precision $$ p=\frac{\mathit{\text{tp}}}{\mathit{\text{tp}}+\mathit{\text{fp}}}$$ (Table 1), recall $$ r=\frac{\mathit{\text{tp}}}{\mathit{\text{tp}}+\mathit{\text{fn}}}$$ (Table 2), and intersection over union (IoU) $$ IoU=\frac{\mathit{\text{tp}}}{\mathit{\text{tp}}+\mathit{\text{fp}}+\mathit{\text{fn}}}$$ (Table 3), for the maize and tomato plant datasets. For each metrics, we show the minimum and maximum values over the dataset as well as the standard deviation. In the definitions above, tp stands for true positive, fp for false positive, and fn for false negative. In all tables, the SVM is responsible for the stem and the leaf class, while instance refers to the unsupervised clustering of individual leaves. We obtain over 90% precision and recall for leaf point in both the datasets, whereas they are around 85% for the stem points. Regarding the leaves instances, both metrics are around 90%.

Table 1

Precision values for class-wise and instance segmentation on our datasets.

Dataset	Stem				Leaf				Instances
	mean	min	max	std	mean	min	max	std	mean	min	max	std
Maize	86.3	74.5	99.6	8.6	95.5	93.6	99.4	2.2	94.4	91.9	99.6	2.7
Tomato	89.6	68.6	99.2	9.6	97.9	96.9	99.0	0.9	83.4	59.8	99.7	15.4

Table 2

Recall values for class-wise and instance segmentation on our datasets.

Dataset	Stem				Leaf				Instances
	mean	min	max	std	mean	min	max	std	mean	min	max	std
Maize	85.7	48.6	99.1	16.3	92.9	89.1	99.8	3.3	94.7	91.3	99.6	3.1
Tomato	92.2	60.6	99.4	11.4	96.5	74.0	99.6	8.3	78.2	66.6	99.4	11.2

Table 3

Intersection over union (IoU) score for our datasets.

Dataset	Stem				Leaf				Instances
	mean	min	max	std	mean	min	max	std	mean	min	max	std
Maize	80.0	47.8	94.6	12.7	94,6	88.9	97.4	2.5	94.0	91.0	97.6	2.3
Tomato	82.6	60.6	92.2	10.4	93.4	73.9	99.0	7.8	69.1	51.4	98.7	15.7

Despite these accurate results, it is worth noticing that the ability of the SVM to classify stem points is lower on the maize dataset than on the tomato dataset. This is due to a smoother transition between stem and leaves in the maize plants. In contrast, the performance of the clustering is higher on the maize dataset. This behavior can be explained by looking at how leaves develop in the two species. While for the maize plants there is a clear separation between individual leaves, this separation is not as clear for the leaves in the tomato plants. For reference, we have shown the ground truth labels for two example plants used in the evaluation of semantic classification in Fig 6.

Fig 6

Example ground truth labels used in the evaluation of semantic classification of maize (left) and tomato (right) point clouds.

The instance wise labels for each plant organ have been manually annotated by a human user.

4D registration of plant point clouds

In the second experiment, we demonstrate that our approach successfully registers time-series point cloud data for two plant types. We use the same weights for our registration procedure as described in the methods section for all scan pairs (both tomato and maize datasets) at different growth stages. We also perform a quantitative evaluation of the accuracy of the registration pipeline. Fig 7 illustrates the results of the registration procedure for two example scan pairs from the maize (top) and the tomato (bottom) datasets. For both examples, we show the input point clouds ($$ {\mathcal{P}}_1,{\mathcal{P}}_2$$) along with their corresponding skeletons. The correspondences estimated during the registration procedure are depicted by the dashed lines joining nodes of the skeleton pair. Our approach was able to find the correspondences reliably despite the growth and the change in topology. We visualize the final registered point cloud $$ {\widehat{\mathcal{P}}}_1$$ (in pink) by deforming the point cloud $$ {\mathcal{P}}_1$$ using the deformation parameters estimated by our approach and overlay it on the target point cloud $$ {\mathcal{P}}_2$$ (in gray) and observe that it overlaps well indicating that the registration results are reasonable. Most of the registered point $$ {\widehat{\mathcal{P}}}_1$$ (in pink) overlaps the target point cloud $$ {\mathcal{P}}_2$$, however, we notice that the errors are usually high towards the outer sections of the leaves which are farther away from the skeleton curve. This effect is to be expected as the skeleton curves do not capture this area well.

Fig 7

4D registration of a point cloud pair scanned on consecutive days for maize (top) and tomato (bottom) plant.

The left column shows the two input point clouds ($$ {\mathcal{P}}_1,{\mathcal{P}}_2$$) along with their skeletons, with the estimated correspondences between the skeleton nodes shown by dashed lines, and the right column shows the deformed point cloud $$ {\widehat{\mathcal{P}}}_1$$ (in pink) overlaid on $$ {\mathcal{P}}_2$$.

Further, we quantitatively evaluate the accuracy of our registration pipeline by registering all consecutive scans of the two datasets. First, we compute the accuracy of our skeleton matching procedure by computing the percentage of correspondences estimated correctly. We define the correct correspondences as those which belong to the same organ (i.e., the same leaf or the stem) in the skeleton pair as there is no unique way to define a correct correspondence due to the growth in the plant. We manually label the different organs of the plant with a unique identifier to provide the ground truth to compute this metric. For our tomato datasets, we obtain an average of 95% correct correspondences between consecutive skeleton pairs with most pairs having all the correspondences estimated correctly. For the maize dataset, we obtain 100%of the correspondences between consecutive days correctly. Similarly, we also evaluated the accuracy of the correspondence estimation between skeleton pairs with 2 and 3 days apart from each other. For the tomato dataset, we obtain again an average of 95% correspondences with scans taken 2 days apart, whereas this falls down to 88% with scans taken 3 days apart. Again for the maize dataset, we obtain all the correspondences correctly both with skeletons taken 2 and 3 days apart. The higher accuracy for the maize plants is likely due to the simpler shape of the plant as compared to the tomato plants.

Secondly, we evaluate the accuracy of the estimated registration parameters by computing the error between the deformed source point cloud $$ {\widehat{\mathcal{P}}}_1$$ and the target point cloud $$ {\mathcal{P}}_2$$. We define this registration error e_reg as:

As this registration measure may be susceptible to some degenerate cases, an extreme case being all points $$ p_i\in {\mathcal{P}}_1$$ deforming to one point in $$ p_j\in {\mathcal{P}}_2$$ giving a zero registration error. Therefore, we use the ground truth instance information to determine if such degenerated situations occur. We compute the percentage of points in pc₁ that are deformed to the corresponding ground truth instances in pc₂. This gives a measure of how well the instances are mapped after applying the deformation parameters. We obtain an average accuracy of 97% for the maize dataset and 89% for the tomato dataset with most of the errors occurring at the joints where leaves merge with the stem. This results suggest that the registration parameters deform the instances properly and do not result in degenerate estimates.

As a baseline comparison, we assume a rigid transformation for the whole point cloud between consecutive scans and computed the transformation with a standard ICP approach. The average error e_reg for this experiment is 35 mm and the maximum error is 166 mm. The large errors using a rigid transformation assumption are both due to the plant growth and in some cases, the ICP procedure diverging completely. This indicates that a rigid transformation assumption is inadequate and a non-rigid registration procedure is required to capture the growth and movement of the plant.

We visualize the registration error as a heat map for the two example point cloud pairs in Fig 8. The heat map is projected on $$ {\widehat{\mathcal{P}}}_1$$ to show how well different portions of the plant are registered. The blue regions in the heat map represent a smaller registration error whereas the yellow regions indicate large errors. Most of the regions are blue indicating a successful registration, however, we notice that the errors are usually high towards the outer sections of the leaves which are farther away from the skeleton curve. This effect is to be expected as the skeleton curves do not capture this area well.

Fig 8

Visualizing registration error.

We visualize the registration error as a heatmap for two pairs of tomato plant scans, Day 1 vs. Day 2 and Day 6 vs. Day 10. Blue represents low registration error whereas yellow represents a larger error.

Temporal tracking of phenotypic traits

In this experiment, we show that the spatial-temporal registration results computed by our approach allows us to compute several phenotypic traits and track them temporally. We compute the area l_a and length l_l for leave instances and the diameter s_d and length s_l of the stem for each point cloud in the temporal sequence as describe previously, and associate it over time using the data associations estimated by our approach during the registration process. The tracking results for the three sample plants from the maize and tomato datasets are visualized in Fig 9. The first two columns in Fig 9, track the leaf area and leaf length over time. Different shades of blue and green in these plots represent individual leaf instances. In addition, we can also detect certain events, which mark a topological change in the structure of the plant such as the appearance of a new leaf. These events can be recognized from the leaf area or leaf length plots in Fig 9 whenever a new line rises up from the zero level. For example, for the first plant in the maize dataset, a new leaf emerges on day 2, 3, and 6. In the rightmost column in Fig 9, we see that stem length and diameter for both the datasets increase considerably over the data acquisition period. Such phenotypic information can also be used to compute the BBCH scale [83] of the plant, which is a growth stage scale and provides valuable information to the agronomists.

Fig 9

Tracking phenotypic traits for individual organs of the plant.

Our registration procedures allows us to track the growth of the stem and different leave lengths over time and detect topological events such as the emergence of new leaves. Different shades of blue and green in these plots represent individual leaf instances in the first two columns. The orange and red represent the length and diameter of the stem respectively.

Temporal interpolation of plant point clouds

In the last experiment, we demonstrate that the registration results can be used to interpolate the point clouds at intermediate points in time, i.e., in between the actual acquisition times of the scans. The ability to interpolate is useful for analyzing properties of the plants even when actual measurements are not available. It allows us to predict both, the motion and growth at intermediate time intervals. We visualize the interpolated point cloud at three time instances $$ t_1^i,t_2^i,t_3^i$$ between the two scans in top of Fig 10. This allows us to animate a time-lapse view of the plants. The pink point clouds represent the interpolated scans and overlaps well with point cloud (gray) at time t indicating that the interpolation is reasonable. As the interpolation procedure does not actually model the movement or the growth of the plant, the result of the interpolation may differ from the actual plant at those instances. In order to evaluate the interpolation step, we take the scans on day t − 1 and day t + 1, then interpolate the point cloud at day t and compare against the actual point cloud on day t. We compute the registration error (as described in (14)) and obtain a mean e_reg of 4 mm with a standard deviation of 1.9 mm. We also compute the percentage of points in $$ {\mathcal{P}}_{t-1}$$ that are registered to the corresponding ground truth instances in $$ {\mathcal{P}}_t$$ similar to the measure in the registration experiments. We obtain an average accuracy of 91% for the maize dataset and 83% suggesting that our interpolation is a reasonable approximation of the real plant growth.

Fig 10

Interpolation of point clouds at intermediate time intervals.

Point clouds (gray) at time t − 1 and t come from actual scan measurements whereas the points clouds (pink) at time instants $$ t_1^i,t_2^i,t_3^i$$ visualize the three interpolated scans.