DIAGNOSING CLEFT LIP PATHOLOGY IN 3 D ULTRASOUND : A LANDMARKING-BASED APPROACH

The aim of this work is to automatically diagnose and formalize prenatal cleft lip with representative key points and identify the type of defect (unilateral, bilateral, right, or left) in three-dimensional ultrasonography (3D US). Geometry has been used as a framework for describing facial shapes and curvatures. Then, descriptors coming from this field are employed for identifying the typical key points of the defect and its dimensions. The descriptive accuracy of these descriptors has allowed us to automatically extract reference points, quantitative distances, labial profiles, and to provide information about facial asymmetry. Eighteen foetal faces, ten of healthy foetuses and eight with different types of cleft lips, have been obtained through a Voluson system and used for testing the algorithm. Cleft lip has been diagnosed and correctly characterized in all cases. Transverse and cranio-caudal length of the cleft have been computed and upper lip profile has been automatically extract to have a visual quantification of the overall labial defect. The asymmetry information obtained is consistent with the defect. This algorithm has been designed to support practitioners in identifying and classifying cleft lips. The gained results have shown that geometry might be a proper tool for describing faces and for diagnosis.


INTRODUCTION
Three-dimensional ultrasound (US) has been introduced more than twenty years ago into clinical practice (Riccabona et al., 1997).Its applications on diagnosis of anomalies and diseases were a direct consequence of its use.In particular, cleft lip and palate (CLP) detection, whose incidence is 1/700 in United States (Tonni and Lituania, 2013), was widely addressed, as they could be difficult to diagnose with bidimensional US, especially in earlier gestational ages (Hata et al., 1998).In the effort to quantify the performance of routine ultrasonographic screening on an unselected population, the Eurofetus study (Grandjean et al., 1999) shows that CLP has the lowest rates of detection (18%) and it is diagnosed usually later in pregnancy (only 31.6%before 24 weeks).Furthermore, CLP is identified with a lower occurrence by prenatal US when the anomaly is isolated than in the cases where multiple anomalies coexist, as frequently noticed during autopsies following termination of pregnancy of foetuses with diagnosed multiple diseases (Luck, 1992).Complementarily, 3D, despite some criticisms (Maarse et al., 2010), has been considered more accurate than bi-dimensional data in detecting unaffected lips at less than 24 weeks (Pretorius et al., 1995).
In this work we will focus on cleft lip (CL) alone.CL, "both unilateral and bilateral, includes clefts involving the alveolus and hard palate anterior to the incisive foramen, namely the embryological primary palate" (Demircioglu et al., 2008).The tested rates of antenatal detection of CL range 21-30% (Rotten and Levaillant, 2004).
Cleft lip has been associated with more than onehundred different chromosomal abnormalities and genetic syndromes (Jones, 1993), and may sometimes be the only sign of a chromosomal anomaly, as trisomy 18 (Carlson, 2000), trisomy 13, or syndromes such as Cornelia de Lange or Smith-Lemli-Opitz (Roelfsema et al., 2007).Thus, an accurate scan searching for other foetal anomalies and a genetic counselling are paramount when a cleft lip is diagnosed.Cleft lip does not go with any palatal abnormality in 15-25% cases (Offerdal et al., 2008, Bäumler et al., 2011).More generally, if we consider CLP as a whole, the incidence of structural anomalies and syndromes accompanying cleft lip and palate ranges between 21% and 38% (Campbell et al., 2005).But it is important to note that, although they often occur with each other, cleft lip and palate abnormalities are "developmentally distinct processes" (Lee et al., 2000).In particular, the embryological origins of lips and alveolus clefts appear to be distinct from those of secondary palate cleft (Campbell et al., 2005).Lee et al. (2000) used three-dimensional ultrasonography to support cleft lip and palate detection.CL was identified by an examiner as "a loss of continuity of the orbicularis oris muscle from a coronal or axial view of the lips" (Lee et al., 2000), so the diagnosis was not automatic.Campbell et al. (2005) assessed the clinical value of a three-dimensional US technique, the 'reverse face' view, in the prenatal categorization of orofacial clefts including CL.Then, Platt et al. (2006) proposed the 'flipped face' view to diagnose lip and palate cleftings, relying on 3D US.When a static volume is acquired, it is rotated 90° so that the cut plane is directed in a chinto-nose plane and scrolled to examine in sequential order different zones, including lips.Mailáth-Pokorny et al. (2010) investigated the role of foetal MRI in the antenatal diagnosis of facial clefts, including cleft lip, although no particular detection technique has been employed.Martinez-Ten et al. (2012) investigated whether systematic examination of primary and secondary palate supported the detection of face cleftings during first trimester.Gindes et al. (2013) studied the potential of three-dimensional US for palate view in foetuses at high risk for CLP.An in-depth palate assessment was made adopting both 2D and 3D US on the axial plane.Then, the outcoming prenatal diagnosis was compared to after-birth findings.Some authors used landmarks as reference points.Johnson et al. (2000) assessed the advantages of three-dimensional US in diagnosing cleft lip.The volume data were displayed in two formats: three orthogonal planar images and a three-dimensional rendered image of the foetal facial surface.The planar images were "rotated with the interactive display into a standard anatomic orientation, so that the three planar images corresponded to the frontal, sagittal, and transverse facial planes" (Johnson et al., 2000).The rendered image provided landmarks for the planar images.Roelfsema et al. (2007) used 3D US to perform foetal orofacial clefts examination and quantified the craniofacial variability index (CVI) in distinguishing between isolated cleft lip/palate and cleft lip/palate in chromosomal abnormalities or syndromes.Facial landmarks such as tragus, nasion, gnathion, glabella, subnasion, and others were employed to extract sixteen craniofacial measurements for the evaluation of after-birth abnormal/regular orofacial development.Although none of the foetuses evaluated in their study was affected by cleft lip, Sepulveda et al. (2010) proposed a novel sonographic landmark typical of the first trimester, the 'retronasal triangle', to be adopted for the early screening of CP.This landmark has been termed this way because coronal plane displays three easily identifiable echogenic lines: the two maxilla frontal processes and the primary palate.Manganaro et al. (2011) studied CLP via MRI and ultrasound, although not 3D.Facial landmarks in the zones of forehead, occiput, orbits, nose, lips, chin, mandible were identified and analyzed for each foetus.Tonni and Lituania (2012) proposed a new three-dimensional sonographic software, the OmiView algorithm, and applied it to unilateral CL, bilateral CLP, and isolated CP.They showed that 3D imaging of the foetal hard and soft palates by OmniView was technically easier than with previously reported 3D techniques.OmniView allowed visualization of all anatomical landmarks of the specific targeted zone, i.e., labia, primary palate, alveolar ridge, posterior palate, uvula, velum, and tongue.This work introduces a methodology for automatically diagnosing cleft lip and assessing specific information about the detected cleft, such as transverse and cranio-caudal lengths, upper lip outline, and a quantification of facial asymmetry.

MATERIALS AND METHODS
During 2013, 38 3D volumes of 38 foetuses at 22 nd -32 nd weeks' gestation were acquired.This gestational period was chosen, as it is considered the most suitable for 3D facial scanning; also, 3D reconstruction is possible in most cases (Kurjak et al., 2007).Eight of them were foetuses affected by cleft lip.Written informed consent was obtained from the parents for publication of clinical details, clinical images, and videos.Principles outlined in the Declaration of Helsinki have been followed.Among these acquisitions, 18 were selected and processed for the purposes of the study, keeping all the eight faces with cleft lip.The leftover ones were excluded due to damages, acquisition inaccuracy, noise, and wrong or unusable foetus's position, such as hands on face or similar.
The US equipment was a Voluson system (GE Healthcare, Wauwatosa, WI, USA), with a RAB 4-8 (real time 4D convex transducer probe).The GE RAB 4-8 has a frequency range of 4 to 8 MHz and is used for OB applications (Footprint 63.6 x 37.8 mm, FOV 70°, V 85°x70°).Table 1 shows data details and respective scan settings.4D VIEW software allows to see the acquired images on three orthogonal planes: axial, sagittal, and coronal (Fig. 1, above).The plane chosen for the facial shell modelling is the midsagittal (Fig. 1, below).

Fig. 1. Multiplanar image (above) and midsagittal plane (below).
The distance between two successive slices is 0.4 mm.For each slice composing the whole facial volume, the relative DICOM format file is generated and imported into Simpleware ScanIP software for the three-dimensional model reconstruction.Facial data were collected in point clouds (shells), imported in Matlab®, triangulated, and converted into a squared-grid-based depth map with the function "gritrimesh".The squared grid has been chosen for its advantages in terms of code writing and for its comparability to a Cartesian space, in which every point has a x-, y-, and z-coordinate.
The algorithm we developed for foetal diagnosis of cleft lip was elaborated, implemented, and run on these shells.It relies on the geometrical features of foetus's face in order to detect the deformation.Moreover, the algorithm identifies whether the cleft lip is unilateral or bilateral, localizes some key points of the deformation and performs tailored measurements in order to assess cleft quantification.The upper lip outline is also evaluated, in order to provide shape and size description of the defect.The geometrical descriptors used in this work are defined and described in previous works of our research group on 3D faces at different ages (Calignano and Vezzetti, 2010;Vezzetti et al., 2011;2012;2013;2014).In particular, geometrical descriptors are introduced and mapped point-by-point on adults' 3D facial surfaces to investigate their behaviours and study their local properties (Calignano and Vezzetti, 2010;Vezzetti et al., 2010;2011;Vezzetti and Marcolin, 2012a;2012b).Then, these descriptors have been applied to automatic landmarking, i.e. extraction of typical facial points, in the medical context (Vezzetti and Marcolin, 2012c;Vezzetti et al., 2012;2013;Vezzetti and Marcolin, 2014), for recognition (Vezzetti et al., 2014a) and foetal diagnosis (Vezzetti et al., 2014b) purposes.
The whole set of descriptors, taken from differential geometry background, includes: the six coefficients of the fundamental form; mean and Gaussian curvatures, principal curvatures, shape and curvedness indexes introduced by Koenderink and van Doorn (1992).After the application, i.e. mapping, of all the geometrical descriptors onto cleft-lip-affected foetal faces, we have identified descriptors e (first coefficient of the second fundamental form), k 2 (second principal curvature), and C (curvedness indexes) as the most quantitatively and qualitatively descriptive of the labial deformation.This evidence is noticeable in Fig. 2, which shows the behaviour of the selected descriptors on a facial shell with cleft lip.The choice of these descriptors among others relies on their focused capability of describing the defect and of circumscribing cleft lip facial area.In particular, e shows a maximum in correspondence to the cleft, while k 2 and C underline the "cut" with an accurate curvature line.

DETECTION OF THE DEFORMITY
Cleft lip is a gap/indentation in the upper lip.The proposed algorithm demonstrates that this defect could be detected via point-by-point mapping geometrical descriptors on facial depth map.This indentation is characterized by high numerical values of coefficient e and of curvedness index C in correspondence to the zone of interest, as shown in Fig. 3.Moreover, the two parts of the lip that are located beside the indentation are characterized by two maximums of the principal curvature k 2 , as can be seen in Fig. 4. For the definition of descriptors e, C, and principal curvatures we refer to that provided by Vezzetti and Marcolin (2012a).
The designed algorithm adopts the previous geometrical features to detect whether the cleft lip is present or not in a foetus's face.Moreover, this algorithm is able to automatically distinguish between unilateral and bilateral cleft lip.It is composed by the following steps.
1.The algorithm searches if there are points whose coefficient e and curvedness index C are greater than a threshold value (C > 3 and e > 3.5).These are the geometrical features of the gap in the lip.Thresholds are set via experimentation.
2. If the search of the step 1 gives no results, the cleft lip is not present.Otherwise, another check is performed in order to verify that a cleft lip is really present.For each point that satisfies the conditions of step 1, the algorithm searches if in its neighbourhood there are points with a high value of the principal curvature k 2 (k 2 >0.3).These are the geometrical features of the two parts of the lip beside the gap.This further condition is needed, as in some cases the points close to the alae of the nose could have the same geometrical features searched in step 1.
3. If the search of the previous step gives no results, the cleft lip is not present; otherwise it is.
4. In order to verify if it is an unilateral or bilateral cleft lip, the algorithm checks if the points that satisfy the condition of step 1 and 2 are all in the same neighbourhood or not.If they do not belong to the same neighbourhood, the cleft lip is bilateral.
The steps of the process are explained in the scheme of Fig. 5.In Fig. 6, the points found in step 1 and 2 are shown in a shell with an unilateral cleft lip.
In Fig. 7, the points found in step 1 and 2 are shown in a shell with a bilateral cleft lip.

LOCALIZATION OF KEY POINTS
In order to quantify the deformation, four key points are automatically localized: the two points of the lip that are beside the cleft and the ending points of the cleft, shown in Fig. 8.This method has sound common points and features with the techniques within the wider area of automatic facial landmarking, deeply investigated in the last decade (Ibragimov et al., 2015;Wang et al., 2015).
As mentioned above, the principal curvature k 2 has two maximums in the first two points.The automatic localization algorithm: 1. after deformity detection, identifies two regions in the neighbourhood of the cleft, one on the right side and one on the left side; 2. in each region, selects the points with Dy < 0; 3. maximizes the principal curvature k2.
The other two points are located in the gap of the lip.As said above, this area is characterized by high values of the curvedness index C and of coefficient e.To extract these points, firstly the algorithm localizes the centre of the cleft maximizing the coefficient e.Then, it analyzes the neighbourhood of this point, moving upwards from the centre to the high ending point and downwards till the low ending point.The algorithm, for each y value, maximizes the coefficient e in a neighbourhood of the cleft lip.The ending points are the first two maximums that are lower than a proper threshold value, established via experimentation.

MEASUREMENT OF THE DEFORMATION AND EXTRACTION OF THE UPPER LIP OUTLINE
Transverse diameter of the cleft was evaluated by computing the Euclidean distance between the first two points extracted; its cranio-caudal length by computing the Euclidean distance between the last two points extracted.These two distances, represented in Fig. 9, are the most adopted in the estimation of cleft size.
The outline of the upper lip was also extracted, in order to provide an extra information about its shape.The curvedness index was adopted, as it is one of the geometrical descriptors that more accurately highlights upper lip surface behaviour.As shown in Fig. 10, the curvedness index has a maximum behaviour in correspondence to the upper lip.To extract the outline, for each x value in the upper lip area, the curvedness index is maximized.This way, we obtain a sequence of points, namely a line, in the 3D space that describes the upper lip area.

EVALUATION OF FACE ASYMMETRY
Facial asymmetry was also evaluated in the cases of unilateral cleft lip.Only unilateral cases have been investigated due to corrective surgery reasons.Surgery can fix cleft lip by compensating the defect, which is linked to the asymmetry.In the bilateral CL cases, the asymmetry could result as negligible, so less significant for surgery purposes.
To perform this evaluation, the mean of the absolute value of the coefficient e was computed in both left and right parts of the mouth.By comparing these two mean values, we obtain an useful information about the asymmetry of the face.Coefficient e is chosen and its absolute value is taken, as in correspondence of the cleft lip two minimums and a maximum are present (as it was previously shown in Fig. 8); the absolute value will avoid that the minimum and maximum behaviour will annul each other when the mean area is computed.

RESULTS
The algorithm for the diagnosis of cleft lip was tested on the eighteen ultrasounds (eight with cleft lip and ten without).In the eight ultrasounds with cleft lip, the four key points were localized, shown in Fig. 11.Also, upper lip outline was extracted on all faces with cleft lip.Results are shown in Fig. 12.After the localization of the four key points, transverse and the cranio-caudal length are computed.In Table 2, the values of these distances for each shell are shown.The unit of measurement is the millimetre.
Facial asymmetry was quantified in the six shells with unilateral cleft lip and on the ten shells without cleft lip; the evaluation was not performed on the two shells with bilateral cleft lip.The results are shown in Table 3 (shells with unilateral cleft lip) and 4 (shells without cleft lip).

DISCUSSION
In all these cases, the algorithm correctly detected the presence/absence of the cleft and, in the two shells with bilateral cleft lip, the algorithm detected both the clefts.
To verify that the localization of the key points was accurate, two maxillo-facial surgeons, experts in facial landmarking, were asked to manually locate the "ground truth" key points on separate sessions.A shady 3D model of the foetal face imported in a specific software for 3D point clouds management (Fig. 13) was given to the observers on a laptop, who directly clicked on the points and created the 'manually landmarked face'.The manual landmarked face model was superimposed on the automatic land-marked one and errors were calculated.The global mean error was 0.53 mm and no key points had more than 1.06 mm error.This was considered acceptable by the surgeons, given the low quality of ultrasound facial models.2 showed that the shells that presented the lowest cranio-caudal length were the ones with an incomplete cleft, thus confirming the accurateness of the evaluation of the distances describing the defect.
Concerning asymmetry, as can be seen in Tables 3  and 4, the difference between the two sides of the face is usually higher in the shells with cleft lip.The mean difference value is 0.58 mm in shells with cleft lip instead it is 0.09 mm in shells without cleft lip, i.e. about six times smaller.The shell with cleft lip with the lowest difference value is Hcl, where an incomplete cleft lip is present.Probably in this case the asymmetry was not high because the cleft lip was not accentuated, as can be verified in Table 2, where the shell Hcl has the lowest cranio-caudal length.Also, the other shell with a low difference value, namely shell Ecl, presents an incomplete cleft lip.Instead, in the shells without cleft lip the difference between the two face sides is usually low, with values ranging between 0.05 and 0.18 millimetres.The highest values (B and L) are probably due to the quality of the ultrasound.
The algorithm is written to be flexible and customized depending on the week of gestation.Generally speaking, given that the facial structure becomes visible at the 11 th -12 th week and in the 13 th-14 th weeks face has reached an adequate degree of development for diagnostic purposes (Kurjak et al., 2007), this is the period from which the method here proposed could work properly, at least for detecting the defect.Differential geometry provided us with a set of descriptors leading this research activity.The result is that these descriptors are suitable to describe facial shape and curvedness, allowing an accurate extraction of the facial features under consideration.

Fig. 2 .
Fig. 2. Above.Mapping of the used geometrical descriptors onto a 3D face of a foetus affected by cleft lip.From left to right: the second coefficient of the second fundamental form e, the second principal curvature k 2 , the curvedness index C. Below.The same descriptors applied to a 3D face model of a healthy foetus.

Fig. 3 .
Fig. 3.The behaviour of the coefficient e (left) and of the Curvedness Index C (right) in a foetus with cleft lip.The circle highlights the area of the gap.

Fig. 4 .
Fig. 4. The behaviour of the principal curvature k 2 in a foetus with cleft lip.The two circles highlight the areas of the lip beside the gap.

Fig. 5 .
Fig. 5. Scheme of the process for cleft lip diagnosis.

Fig. 6 .
Fig. 6.On the left: the points found in the first step of the algorithm.On the right: the points found in the second step of the algorithm.

Fig. 7 .
Fig. 7. On the left: the points found in the first step of the algorithm.On the right: the points found in the second step of the algorithm.

Fig. 8 .
Fig. 8.The four key points of the cleft lip.Fig. 9.The transverse (horizontal) and cranio-caudal length (vertical) of the cleft.

Fig. 10 .
Fig. 10.The curvedness index.The two circles highlight the upper lip area.

Fig. 11 .
Fig. 11.The resulting four key points for each shell with cleft lip.

Fig. 12 .
Fig. 12.The resulting upper lip outline for each shell with cleft lip.

Fig. 13 .
Fig. 13.3D face model of a foetus with cleft lip, imported in a software for three-dimensional point clouds management.This is the type of model provided to the surgeons for identifying the "ground truth" key points.

Table 1 .
Weeks' gestation, scan settings, and eventual cleft lip features for each baby.

Table 2 .
Computed distances for each shell.

Table 3 .
Results of asymmetry evaluation for shells with unilateral cleft lip.

Table 4 .
Results of asymmetry evaluation for shells without cleft lip.