1 3 D non-invasive inspection of the skin lesions by close-range and low-cost photogrammetric techniques

In dermatology, one of the most common causes of skin abnormality is an unusual change in skin lesion structure which may exhibit very subtle physical deformation of its 3D shape. However the geometrical sensitivity of current cost-effective inspection and measurement methods may not be sufficient to detect such small progressive changes in skin lesion structure at micro-scale. Our proposed method could provide a low-cost, noninvasive solution to overcome these shortcomings by using very close-range photogrammetric (stereoscopic) imaging techniques to build a 3D surface model for a continuous observation of subtle changes in skin lesions and other features.


Introduction
For dermatological diagnosis, the detection of very early signs of disease, whether physical or structural, is important to ensure early intervention.The measurement sensitivity of current methods may not be sufficient to detect these early signs of disease (melanoma, etc.), which present as changes on the skin features.Small physical changes can be efficiently detected in the 3D domain by using photogrammetric measurement techniques at the required accuracy.Our proposed method is a low-cost non-invasive solution that overcomes the accuracy problems.This is achieved by using a microscale photogrammetric bundle adjustment method (Granshaw,1980) for all 3D skin features, including skin lesions or wounds.The photogrammetric bundle adjustment technique has been widely used in industrial applications (Orun and Alkis,2003) and aerial photography (Orun andNatarajan, 1994) since 1950, but now its very-close-range version may also be utilized at micro scale for medical diagnostic purposes by observing the subtle progressive changes in a lesion structure, or any other physical skin feature (volume, shape, etc.).
The techniques proposed within this research can also be applicable to subcutaneous layers by utilizing a high resolution IR camera to detect micro-level progressive changes of skin features over a specific time period.Nowadays most skin imaging techniques are limited to the 2D domain, and look for the skin features such as color change, lesion symmetry, lesion border irregularity check, etc. (Claridge and Orun, 2002;2003) and are unable to make accurate progressive 3D observations of skin abnormalities such as malignant lesion growth, displacement of micro blood vessels or suspicious changes in mole at micro scale.Other studies which focus on medical 3D feature reconstruction also have some disadvantages.Alvarez et al. (2006) investigated the use of digital photography to measure wounds.They developed the Photo-Digital Planimetry Software (PDPS) utility for this purpose but limited it to 2D planimetric measurements.One other work similar to ours was introduced by Gorpas et al. (2007).They use high-cost twin CCD cameras with telecentric lenses for three-dimensional lesion surface measurements.The cameras they utilized have to be fixed and carefully located; which results in many restrictions to the image data collection process, hence this method is not practical for a flexible clinical environment.In contrast our system uses a low-cost single camera (40-50 times cheaper than the telecentric lenses used by Gorpas et al. ,2007) with a completely free-hand image acquisition style without the need for a fixed camera position or any geometric setup restriction.
For medical applications some researchers have used 3D object reconstruction techniques.Choi et al. (2013) used a comprehensive photogrammetrical system with 3 digital cameras for orthognatic surgery.Goellner et al. (2010) also introduced a similar system used in dentistry which utilises stereo camera pairs with photogrammetric technique.Such systems can only be used with a highly conditioned setup in a restricted environment and are not cost effective.With regards to low-cost solutions, one of studies introduces a close range photogrammetric system using a single camera which was conducted by Khalil (2011).In his experiments he measures the displacement of a moving object only in X,Y directions by keeping the camera at fixed position and the object Z coordinates are disregarded.The theory of photogrammetrical measurement precision by using single camera was studied by Luhman (2009) comprehensively.He emphasizes the importance of avoiding photogrammetrical system restrictions (e.g.synchronization of cameras, spatial observation conditions, system costs, etc.)

Photogrammetric Image acquisition
Photogrammetry is a fundamental technique which specifies the geometrical relationship between image points (any point of an object whose image is taken) and the three dimensional Cartesian coordinates of an object (e.g.skin lesion).Each point in the scene is represented by a unique feature such as an edge, dot or tip of a line, etc. and must also be identified in the corresponding stereo images of the scene.Nowadays many photogrammetric methods are used in several fields of medical imaging but in this study we focus on very-close-range photogrammetry, and particularly the bundle adjustment method, which is sub-set of photogrammetry (Moffitt and Mikhail, 1980).A simple stereoscopic image acquisition process in association with the bundle adjustment technique can be achieved (Figure 1) by using a low-cost single digital camera, and by taking a pair of images of a skin lesion from two different positions (S1 and S2).
In the experiments a reference ring is used whose markings were already measured accurately, to provide Cartesian coordinates of Control and Check Points, as shown in Figure 2. The rings rear side is coated with an adhesive material so that it can be vertically positioned on the surface surrounding a skin lesion.Left and right stereoscopic images of the skin lesion are used to create 3D lesion modelling.The Accuracy of the control points (CP) marked by tiny holes on the reference ring, directly effect the accuracy of the results yielded by the bundle adjustment algorithm.The check points located at the tip of test bars (Figure 2) whose X,Y,Z object coordinates are also known enable corrections to be made to the final lesion model coordinates.

Bundle adjustment technique
The photogrammetric bundle adjustment technique used here aims to generate 3D skin lesion models at micro scale.A similar technique was previously used by Jaspers et al. (1999) by a micro-mirror device instead of a single digital camera.There are also various applications in which close-range photogrametric principles used.Unlike conventional close-range photogrammetry, our suggested application has very-close-range characteristics used with a bundle adjustment utility.The bundle adjustment technique is a basic tool in photogrammetry and has been studied by several authors (Orun and Natarajan, 1990;Granshaw, 1980;Moffitt and Mikhail,1980).Comprehensive studies on the bundle adjustment technique were presented earlier with details by Granshaw (1980).A brief description is as follows.In figure 1, consider ith point on the skin with spatial co-ordinates of (X,Y,Z)i which corresponds to image points a1 and a2 ,and whose image coordinates are (xi,yi)1,2 on (stereoscopic) image sequences 1 and 2. In Figure 1 and Formula 1, f is the principal distance (or focal length if objective = ), which can be calculated by a basic camera calibration in a lab environment.The principal distance (focal length) is a camera interior parameter and has to be re-calculated after the modification of the camera lens system, which makes the camera suitable for veryclose-range measurements.(X, Y, Z)S1,S2 denotes the co-ordinates of the cameras' perspective centres (camera locations) and (w,,1,2 are the rotation angles of the camera at locations 1 and 2 .The collinearity equations which relate the model and image co-ordinates may be defined as: Where rij are the elements of the image rotation matrix (Equation 2) for the each image.For both images, the collinearity equations are solved by iterations to calculate the X,Y,Z object coordinates.This can only be done by an inverse solution, because most parameters are unknown and only the set of image co-ordinates (xi,yi)1,2 can be measured on both images.In the case of a very dense surface model, where a large number of surface points are included, the measurements can be done automatically by an image matching technique.The rotation matrix R is an orthogonal matrix which can be written separately for both images as: (2) (3) The bundle adjustment technique is based on the inverse solution of the collinearity equations (1) by iteration.To achieve this, the collinearity equations are linearised by Taylor's Theorem, and then the normal equations can be configured from these linearised observation equations.Let the partitioned normal equations (Granshaw,1980) be given by :

Basic camera Calibration process (calculation of principle distance)
The principal distance (or focal length if objective = ) can be calculated by a basic level of camera calibration process in a lab environment (Figure 5).This process has to be done after the modification of the camera lens system to make the camera suitable for very-close-range measurements.The focal length () is a camera interior and can be calculated

Results and discussion
)) ./( ( )) ./( ( Experimental tests have been accomplished to investigate how the bundle adjustment method might be exploited by the selection of optimum system parameters to build accurate 3D skin lesion surface models (Table1).In the tests, two different types of lesions are examined (elevated and flat).A sample of elevated skin lesion and its 3D model reconstruction is shown in Figure 6.For the tests each set of seven surface points on two types of lesion (elevated and flat), three check points and four control points located on the reference ring are used in association with the bundle adjustment algorithm.The lesion points can be selected from among other physical details (e.g.pigment dot, natural skin details, etc.) identified on the lesion, and which are also visible on both stereo image pairs.The image pixel coordinates of the object points in both sets of two different camera positions (S1,S2) are measured by the image screening utility manually, and then used as inputs to the bundle adjustment utility.In this experiment the bundle adjustment algorithm is used in a very-close-range domain, as compared to industrial applications, and hence any measurement error on control points coordinates be more effective .In our experiments all image measurements on a skin lesion is made by using a low-cost digital camera (Premier, KSI Trade ltd, UK) with 2032x1520 pixel resolution by taking two sequential images (Figure 1) targeting the same measurement area (including control and test points) from different point of views that called perspective centers Xs,Ys,Zs (Table 1).The focal length of the system remained constant during the image data collections, hence the camera should not have any "auto focus/zoom" facility.
On the reference ring (Figure 2) the Cartesian (X,Y,Z) coordinates of the control points and check points have been measured by using a digital gauge at approximately 10 micron accuracy.The accuracy of the calculated values for the check points yielded by bundle adjustment results corresponds to the accuracy of a reconstructed surface model of a skin lesion.In further stages of progressive 3D analysis of a lesion in a specific time period (e.g. to observe lesion growth, displacement, etc.), each set of model coordinates may compared to previous ones to identify the differences.It would be easier to compare model features (e.g.distances, volumes) rather than the coordinates because it will be difficult to position the reference ring in each time of measurement on its exact original location.The other issue is pin-point identification of control points' or test points' centroids to sub-pixel accuracy due to insufficient camera resolution (Figure 3) and fixed focal length (non-auto focus) characteristic of the camera.
Table 1.The display of the Bundle adjustment algorithm results by which 3D lesion model coordinates (in mm) are calculated after 3 iterations, which includes 7 lesion points (1-7) and 3 check points (8-10).Using this method, the lesion growth progress may also be observed by comparison between the set of coordinates sequentially obtained in a specific time period.Exterior orientation parameters include positions (Xs1, Xs2) and tilts (ω,φ,κ) of a single digital Camera at two stationary locations.

Figure 3 (right)
. By using relatively low resolution images of low-cost (nonmetrological) cameras, it is difficult to identify pinpoint location of central pixel (centroid) of each control point marked as "+" (left image).The image blurring is also originated from non-focus characteristics of the camera due to fixed focal length requirement in bundle adjustment algorithm By using relatively low resolution images taken by a low-cost (non-metrological) and non-focus style cameras, in some cases it may be difficult to pinpoint the location of the central pixel (centroid) on each control point.Incorrect location of a centroid pixel with ±1 pixel displacement may cause up to a few mm errors in 3D lesion coordinates.Fortunately these types of coordinate error can be automatically compensated for by the robust characteristics of the bundle adjustment method, but for some additional errors it may be more effectively applied to the final results, such as inaccurate focal length (less than 4 digit behind the decimal point) or CCD sensor matrix errors in a non-metrological camera.In our experiments, the results of the bundle adjustment calculations are matched with 3 check points (Figure 2) forming a triangular area which help to estimate the errors of the lesion model coordinates (X,Y,Z) for points 1-7.Then by using 2D interpolation method over the accurate triangular area, it is possible to correct lesion model coordinates with up to 10 micron accuracy (Table 1).The iteration procedure is followed by which the reverse solution of collinearity equation is made until a maximum accuracy is reached.By this procedure the software algorithm tries to converge to the optimum values by iteration where the number of iterations depend on the initial values selected arbitrarily.The closer the initial values are to the results, the smaller the number of iterations.

System configuration and adjustment
Before obtaining the stereo image acquisition the major camera interior parameters have to be calculated (focal length, CCD size, etc.).The focal length (principal distance) should be calculated (f = 8.883 mm) due to modifications of the camera lens system to enable 200 pixels the camera to take close-range images.This procedure was carried out by using an optic lab utility and geometric principles (Orun, 1996) which is described by Formula 7. The previous experiments have shown that (Orun and Natarajan,1994) the best geometric configuration for accurate results of Z coordinates are established by ratio of base/height = 1, where "base" refers to distance between two cameras' perspective centers and "height" is distance between the camera and skin surface.It has been proven that (Orun,1990) the optimum B/H ratio for Z coordinates may degrade the planimetric accuracy of X,Y coordinates.Hence in our work the height accuracy (Z coordinates) are given a priority over the planimetric accuracy.This is because a single camera image (positioned vertically to skin surface) would be sufficient to calculate X,Y coordinates of a lesion accurately by using basic perspective geometry principles such as ; Where ; X or Y are Cartesian coordinates in planimetric object domain, and x or y are their corresponding image coordinates, H is the distance between camera and skin surface, f is the principal distance (or focal length if camera objective focuses on ), CCDpixelsize is the size of single CCD unit of camera (in mm) and x or y image coordinates in pixel unit.According to the results shown in Table 1, the errors on the X,Y,Z coordinates of two lesion types can be easily calculated by using check points located on the reference ring (Figure 2).Each check point was precisely measured by a digital gauge at 10 micron accuracy.The horizontal planimetric errors on X,Y coordinates vary between 0.24mm and 2.29mm for the average distance (15mm) between the check points whose locations surround the target lesion.If the maximum horizontal planimetric errors between the check points are distributed over the lesion points coordinates, the max horizontal error for the lesion points corresponds to 0.6mm.This error may be neglected since bundle adjustment method results have distance-preserving characteristics being effective on the planimetric object domain (distances between the lesion points).Normally even a single pixel measurement error on a single image may result a few millimetres Z coordinate errors on lesion points (Figure 3) and this may increase if both image have the same symmetric pixel errors.But fortunately bundle adjustment method also has a unique self-compensation characteristics which applies corrections automatically to the lesion points coordinates.

Conclusion
The techniques introduced can be used with any set of suitable equipment and may be easily adapted to any sensor system which has a basic perspective geometrical characteristic (e.g., microscope, etc.).This would widen its application areas in a broad range and its low-cost characteristics may also strengthen its market potential.The system has compact and portable characteristics, hence it may be used for the legal purposes for measuring or tracking any medical condition of a scar or surface blemish after an injury (e.g. for insurance companies, Medicare reimbursement, etc.).Table 2 -The results of 3D model coordinates (X,Y,Z) for four types of lesions at different elevations are yielded by bundle adjustment algorithm iterations.The further corrections to Z coordinates (Zi) are calculated by interpolation over the Check points (X,Y,Z)a,b,c coordinate values (here the effect of point C can be neglected).In the table, lesion model coordinates (points 1-7) and Check point coordinates ( a, b and c ) are shown for both lesions.To bring planimetric corrections to lesion points (Xi ,Yi), simply the average of all 3 check points (X,Y) a,b,c are taken into account for an approximate results which may fall into a tolerance level.
Here the planimetric (Xi,Yi) errors are less important than Zi elevation errors, since (Xi, Yi) can also be simply calculated by an interpolation on a single image by using check points (X,Y)a,b,c that are already known, disregarding the tilts of image.As only comparative lesion points (Xi,Yi) displacements would be counted.We have to note that if all points' calculated X,Y coordinates have almost equal (or radial) displacements, then local lesion coordinates are not too much affected by these relatively large X, Y planimetric displacements (in lesion C and D cases where radial and rotational displacements of all points exist respectively ).

Figure 1 .Figure 2 .
Figure 1.Stereoscopic image acquisition by very-close-range photogrammetric technique using a single camera whose centre of locations are depicted as S1 and S2.Calculation of a single point (i) coordinates on a skin lesion is made by bundle adjustment iteration algorithm to build 3D model.In the Figure, base (B) refers to distance between the S1 and S2, height (H) is the distance between the camera position (Si ) and skin surface.
p denotes surface point co-ordinates and s indicates the camera parameters, Nps is the normal equations of them.Δxp and Δxs represent corrections to surface point co-ordinates (X,Y,Z)i and camera parameters (X s ,Y s , s ,ω,,respectively.tp and ts are the corresponding right hand sides of normal equations.The inverse of sub-matrix Np can be easily calculated, allowing us to

Figure 4 .
Figure 4 .Flow diagram of the bundle adjustment algorithm (t is the threshold of accuracy to stop the iteration) measured distance between two dots (in the dot-grid calibration plate ) Px, : Single CCD detector size of the camera (e.g.3.1x3.1 micron) L: Distance between the two positions of calibration plate S1, S2 : length of "s" measured on the image (in pixel unit)for positions 1 and 2.

Figure 5 .
Figure 5 .The calibration utility configuration to define camera focal length (principal distance).The images are taken at two different positions (1 and 2) of dot-grid calibration plate.

Figure 6 .
Figure 6.An elevated lesion whose height (Z coordinates) are calculated by using photogrammetric bundle adjustment technique with the low base-to-height ratio: B/H = 0.2 ( top ).Its 3D surface representation (bottom ).

Figure 7 .
Figure 7.The lesion points (pointed by arrows) are selected provided that they are clearly visible on both stereo image pairs and each selected point image coordinates with its corresponding ones are to be measured on both images.