SPATIALLY ADAPTIVE MORPHOLOGICAL IMAGE FILTERING USING INTRINSIC STRUCTURING ELEMENTS

This paper deals with spatially adaptive morphological filtering, extending the theory of mathematical morphology to the paradigm of adaptive neighborhood. The basic idea in this approach is to substitute the extrinsically-defined, fixed-shape, fixed-size structuring elements generally used by morphological operators, by intrinsically-defined, variable-shape, variable-size structuring elements. These last so-called intrinsic structuring elements fit to the local features of the image, with respect to a selected analyzing criterion such as luminance, contrast, thickness, curvature or orientation. The resulting spatially-variant morphological operators perform efficient image processing, without any a priori knowledge of the studied image and some of which satisfy multiscale properties. Moreover, in a lot of practical cases, the elementary adaptive morphological operators are connected, which is topologically relevant. The proposed approach is practically illustrated in several application examples, such as morphological multiscale decomposition, morphological hierarchical segmentation and boundary detection.


INTRODUCTION
Firstly, a lot of image processing techniques use spatially-invariant transformations, with fixed operational windows.This kind of operators, such as morphological operators or convolution filters, give efficient and compact computing structures, in the sense where data and operators are independent.However, due to their fixed operational windows, they consequently have several strong drawbacks such as creating artificial patterns, changing the detailed parts of large objects, damaging transitions or removing significant details (Arce and Foster, 1989).
Alternative approaches towards spatially-variant image processing have been proposed (Gordon and Rangayyan, 1984;Perona and Malik, 1990;Salembier, 1992;Alvarez et al., 1993;Charif-Chefchaouni and Schonfeld, 1994;Vogt, 1994;Braga Neto, 1996;Cuisenaire, 2005;Lerallut et al., 2005) with the introduction of adaptive operators, where the adaptive concept results from the spatial adjustment of the operational window.A spatially adaptive operator will no longer be spatially-invariant, but must vary over the whole image with adaptive windows, taking locally into account the image context.Such transforms perform efficient image processing.
Secondly, usual image processing operators have some limitations concerning their operational windows (adaptive or not).In fact, these last ones are usually extrinsically defined with regard to the local features of the image.A priori constraints are imposed upon the size and/or the shape of the operational windows, which is not the most appropriate.For instance, spatially-invariant approaches such as wavelets (Mallat, 1989), morphological pyramids (Sun and Maragos, 1989;Laporterie et al., 2002), and isotropic scale-spaces (Lindeberg, 1994;Heijmans and Boomgaard, 2000) use sliding windows extrinsically defined with regard to the analyzing scales.Indeed, their size and shape are fixed on the whole image for each scale, i.e., a priori determined, independently of the image context.In an other example (Vogt, 1994), spatially-variant morphological operators are used, where the shape of morphological structuring elements that automatically adjust the gray tones in a range image is rectangular or ellipsoidal, involving a priori knowledge about the image context.Therefore, intrinsic approaches, using self-defined operational windows that fit to the local content of the image, without any a priori spatial constraints, are more appropriate.Following this idea, image processing based on the Adaptive Neighborhood (AN) paradigm (Paranjape et al., 1994) has been proposed.A set of adaptive neighborhoods (ANs set) is defined around each point within the image, whose extent depends on the local features of the image in which the given point is situated.Thus, for each point to be processed, its associated ANs set is used as (intrinsic) operational windows of the considered transformation.The resulting operators perform meaningful image processing as shown in various image filtering processes (Rabie et al., 1994;Rangayyan et al., 1998;Rangayyan and Das, 1998;Ciuc et al., 2000;Buzuloiu et al., 2001;Ciuc, 2002).
In this paper, an intrinsic spatially-variant approach in the context of Mathematical Morphology (MM) is then proposed using autoreflected Structuring Elements (traditionally called symmetric (Serra, 1988a), i.e., structuring elements which are equal to their reflected set), based on the AN paradigm.
While autoreflectedness is not necessary in the general framework of spatially-variant mathematical morphology, as formally proposed by Charif-Chefchaouni and Schonfeld (1994) and practically used by Cuisenaire (2005) and Lerallut et al. (2005), it is relevant for three main reasons: 1. it is more adapted to image analysis for topological and visual reasons, 2. both dualities by adjunction and by involution for dilation and erosion are satisfied, 3. it allows to simplify mathematical expressions of morphological operators, without increasing computational complexity of algorithms.
Thereafter, the fixed-size, fixed-shape SEs generally used for morphological operators are substituted by (intrinsic) Adaptive Structuring Elements (ASEs) adjusted to a specified set of adaptive neighborhoods based on an analyzing criterion.It leads to Adaptive Neighborhood Mathematical Morphology (ANMM), which provides convincing spatially adaptive morphological filters (Debayle and Pinoli, 2005a).Some of which satisfy multiscale properties (Debayle and Pinoli, 2005b).Moreover, in a lot of practical cases, the elementary adaptive morphological operators are connected, contrary to the usual ones which fail to this property.The proposed approach is practically illustrated through several application examples, such as multiscale decomposition, hierarchical segmentation and boundary detection on the 'cameraman' image, the 'tools' image and a metallurgic grains real image, respectively.

ADAPTIVE NEIGHBORHOOD SETS
The first step in this ANMM approach consists in defining a set of adaptive neighborhoods determined on the spatial support of the studied image.
Let D ⊆ R 2 (or more generally in R n (Debayle, 2005)), the (usually rectangular) domain of definition of the images and I the set of image mappings from D into R.For each point x ∈ D of an image f ∈ I, the adaptive neighborhood (AN) sets, denoted V h m (x), are computed in relation with an homogeneity tolerance m ∈ R + on a criterion mapping h (based on a local measurement such as luminance, contrast, curvature, thickness or orientation related to the image f ) belonging to the set C of mappings from D into R.More precisely, for all point x ∈ D its associated AN set V h m (x) ⊆ D: • depends on two parameters: -h: analyzing criterion -m: homogeneity tolerance • fulfills two conditions: -its points have a measurement value close to that of the point x: where C X (x) denotes the path-connected component (with the usual Euclidean topology on D ⊆ R 2 ) of X ⊆ D containing x ∈ D.
Fig. 1 gives an impression, on a 1-D example, of the computation of an AN set.
Fig. 1.One-dimensional computation of an adaptive neighborhood set V h m (x).For a point x, a tube of tolerance m is first computed around h(x).Secondly, the inverse image of this interval gives a subset of the 1-D spatial support.Finally, the path-connected component holding x provides its AN set V h m (x).
Fig. 2 illustrates the AN set of a point x computed with the luminance criterion or the contrast (defined in the sense of (Jourlin et al., 1988;Pinoli, 1991)) Image Anal Stereol 2005;24:145-158 criterion, on an electrophoresis gel image provided by the software Micromorph R .In practice, the choice of the appropriate criterion results from kind of the considered application.In the following, the notion of path is defined so as to get a practical equivalent definition of these AN sets, involving computing interests.

Definition 2 (Path)
A path of extremities x ∈ D and y ∈ D respectively, denoted P y x , is a continuous mapping (with the usual Euclidean topologies on [0, 1] and D) (Choquet, 2000): So, the AN sets V h m (x) are defined with the help of a region growing process, which is of great computing importance, where the aggregating condition is given by: |h(.) − h(x)| ≤ m. where −→ denotes the path-connectivity relationship: Remark 1 This definition of the AN sets V h m (x) is similar to those of connected regions R f m (x) defined by Braga Neto (1996) but are more general in the sense that they take into account a criterion mapping h.
These AN sets satisfy several properties as stated in the following: 2. increasing with respect to m: 3. equality between iso-valued points: 4. addition invariance with respect to h: 5. multiplication compatibility with respect to h: Proof: 2.
3. Let z be a point in V h m (x).So, there exists a path P z x such that: , there exists a path P x y such that: then there exists a path P z y such that: and h(y) = h(x), then y belongs to V h m (x) (seen with the inverse path P y x (.) = P x y (.) = P x y (1 − .)).So, there exists a path P z x such that P z To illustrate the nesting property (Eq.5) with respect to m, the AN sets of four initial points are computed on the 'Lena' image (Fig. 3) with the luminance as analyzing criterion.c).An AN set defined with a certain homogeneity tolerance could be represented by several tinges of the color associated to its seed point.
Fig. 3 shows that the AN sets are, through the analyzing criterion and the homogeneity tolerance, intrinsically-defined with respect to the local structures of the studied image, performing a real spatially adaptive analysis.
So, morphological operations have to be adjusted to these AN sets so as to develop Adaptive Neighborhood Mathematical Morphology (ANMM).

ADAPTIVE NEIGHBORHOOD MATHEMATICAL MORPHOLOGY
The origin of mathematical morphology stems from the study of the geometry of porous media by Matheron (1967).The mathematical analysis is based on set theory, integral geometry and lattice algebra.Its development is characterized by a crossfertilization between applications, methodologies, theories, and algorithms.It leads to several processing Inspired by Braga Neto (1996), the proposed spatially-variant mathematical morphology approach is based on the adaptive neighborhood paradigm proposed by Paranjape et al. (1994).In this paper, only the flat MM (ie with structuring elements as subsets in R 2 ) is considered, though the approach is not restricted and can also address the general case of functional MM (ie with functional structuring elements from a subset of D into R) (Debayle, 2005).
The space of images I is provided with the partial ordering relation ≤ defined in terms of the usual ordering relation ≤ of real numbers: Thus, the partially-ordered set (I, ≤), still named I, is a complete lattice (Serra, 1988a).

ADAPTIVE STRUCTURING ELEMENTS
To get the morphological duality (adjunction) between erosion and dilation, reflected (or transposed) structuring elements (SEs) (Serra, 1988a), whose definition is mentioned below, should be used.

Definition 4 (Reflected subset)
The reflected subset of A(x) ⊆ D, element of a collection {A(z)} z∈D , is defined as: The notion of autoreflectedness is then defined as following (Serra, 1988a): that is to say: Remark 2 The term autoreflectedness is employed in place of symmetry which is generally used in literature (Serra, 1988a), so as to avoid the confusion with the geometrical symmetry.Indeed, an autoreflected subset A(x) ⊆ D belonging to {A(z)} z∈D is generally not symmetric with respect to the point x.
Spatially-variant mathematical morphology using adaptive SEs which do not satisfy the autoreflectedness condition (Eq.11) has been formally proposed by Charif-Chefchaouni and Schonfeld (1994) and practically used in image processing (Lerallut et al., 2005;Cuisenaire, 2005).Nevertheless, while autoreflectedness is restrictive from a strict mathematical point of view, it is relevant for three main reasons: 1. it is more adapted to image analysis for topological and visual reasons (Rem.3), 2. both dualities by adjunction and by involution for dilation and erosion are satisfied, 3. it allows to simplify mathematical expressions of morphological operators, without increasing computational complexity of algorithms.
From this point, autoreflected adaptive structuring elements are considered in this paper.Therefore, the AN sets V h m (x) are not autoreflected (Fig. 4).
point line measurement value Fig. 4. The AN sets {V h m (z)} z∈D are not autoreflected: x ∈ V h 3 (y) and y / ∈ V h 3 (x).Consequently, (V h 3 ) = V h 3 .So, the adaptive SEs, denoted {R h m (x)} x∈D , are defined while satisfying the AN paradigm and the autoreflectedness condition.

Definition 6 (Adaptive structuring elements)
These adaptive SEs (Eq.12) are anisotropic and selfdefined with respect to the criterion image mapping h.They satisfy the following properties: Properties 2 (Adaptive structuring elements) 2. symmetry: DEBAYLE J ET AL: Adaptive neighborhood mathematical morphology 3. reflexivity: 4. increasing with respect to m: 5. addition invariance with respect to h: 6. multiplication compatibility with respect to h: Proof: So, there exists z in D such that y belongs to V h m (z) (with the path P y z ) and x belongs to V h m (z) (with the path P x z ).Thus, the path P The ASEs satisfy the AN paradigm through the geometric nesting (Eq.13).Fig. 5 compares the shape of usual SEs B r (x) as disks of radius r ∈ R + and adaptive SEs R h m (x) as sets self-defined with respect to the criterion mapping h and the homogeneity tolerance m ∈ E + .
Remark 3 Autorefletedness is argued to be more adapted to image analysis from both topological and visual reasons.In fact, it allows a symmetric neighborhood system R h m (x) to be defined at each point x belonging to D. Topologically, it means that if x is in the neighborhood of y at level m (x ∈ R h m (y)), then y is as close to x as x is close to y (y ∈ R h m (x)) (Prop 2.2).In terms of metric, this is a required condition to define a distance function d, starting from all the R h m (.), satisfying the symmetry axiom: d(x, y) = d(y, x) (Cech, 1966).Indeed, symmetry is needed to introduce a non-degenerate topological metric space (the authors are currently working on topological approaches with respect to the AN paradigm).From a visual point of view, the symmetry property appears closely linked to the human visual perception (Gestalt theory, . . . ) (Wertheimer, 1938;Dakin and Hess, 1997).
r, m Fig. 5. Example of adaptive R h m and non-adaptive B r structuring elements with three values both for the homogeneity tolerance parameter m, and for the disks radius r.The shape of B r (x 1 ) and B r (x 2 ) are identical and {B r (x)} r is a family of homothetic sets for each point x ∈ D. On the contrary, the shape of R h m (x 3 ) and R h m (x 4 ) are dissimilar and {R m (x)} m is not a family of homothetic sets.
The next step consists in defining adaptive basic operators of MM in the aim of building adaptive morphological filters.

ADAPTIVE ELEMENTARY MORPHOLOGICAL OPERATORS
The elementary dual operators of adaptive dilation and erosion are defined accordingly to the flat ASEs R h m (x).The formal definitions are given as following: where D h m ( f ) : where E h m ( f ) : Next, the lattice theory (Serra, 1988a) allows to define the most elementary (adaptive) morphological filters (Serra, 1988b).More precisely, the adaptive closing and opening are respectively defined as: O h m : These adaptive elementary morphological operators satisfy several properties stated in the following.Properties 1 through 6 are standard, that is to say analogous to those of the usual morphological operators: 1. increasing: 2. adjunction (morphological duality): 3. extensiveness, anti-extensiveness: 4. distributivity with , : where K is an index set (finite or not).
5. duality with respect to the involution .: 6. idempotence: 7. increasing, decreasing with respect to m: 8. addition invariance with respect to h: 9. multiplication compatibility with respect to h: Proof: 1-6.These properties are inferred from the lattice theory of increasing mappings (Serra, 1988a;b).7-9.It is directly inferred from the properties 4-6 of the adaptive structuring elements (Prop.2).
The usual and adaptive morphological operators of erosion, dilation, opening and closing are illustrated in Fig. 6.Note that Er, Dr, Or and Cr denote the classical erosion, dilation, opening, closing with the disk B r of radius r as isotropic structuring element.
Moreover, with the 'luminance' criterion (h = f ), the adaptive dilation  and erosion (Eqs.21-22) satisfy the connectedness (Serra and Salembier, 1993) condition (Prop.4) which is of great morphological importance.On the contrary, the usual dilation and erosion fail to this property.
Definition 9 (Connectedness) An operator φ : I → I is connected if and only if: Properties 4 (Connectedness of adaptive operators)

Proof:
Let g be in I.For all (x, y) neighboring points (with the usual Euclidean topology on x and y are neighbors with the same gray tone.So, Thereafter, the closing and the opening are connected operators by composition of connected operators (Serra and Salembier, 1993).This property is an overwhelming advantage in comparison to the usual ones which fail to this connectedness condition.Besides, it allows to define several connected operators built by composition or combination with the supremum and the infimum (Serra and Salembier, 1993) of these adaptive elementary morphological operators, as adaptive closings (Eq.23) and openings (Eq.24).Thus, the operators OC , called adaptive opening-closing and adaptive closingopening respectively, are (adaptive) morphological filters (Matheron, 1988), and in addition, connected operators with the luminance criterion.

ADAPTIVE SEQUENTIAL MORPHOLOGICAL OPERATORS
The families of adaptive morphological filters {O h m } m≥0 (Eq.24) and {C h m } m≥0 (Eq.23) are generally not a size distribution and anti-size distribution respectively, due to the notion of semi-group which is generally not satisfied (Serra, 1988a).Nevertheless, such families are built by naturally reiterate adaptive dilation or erosion.Explicitly, adaptive sequential dilation, erosion, closing and opening, are respectively defined as: C h m,p : O h m,p : The morphological duality of D h m,p (Eq.33) and E h m,p (Eq.34) provides, so among other things, the two sequential morphological filters C h m,p (Eq.35) and O h m,p (Eq.36).Moreover, these last ones generate size and antisize distributions: Thus, the extension of the well-known alternating sequential filters (ASFs) (Serra, 1988c) can be defined: ASFCO h m,n : These adaptive ASFs are similar to those defined by Braga Neto (1996) using the AN paradigm.

RESULTS
Adaptive morphological processes are now applied and illustrated in the field of image filtering and segmentation.The results are all achieved with the luminance criterion (mapping h).

MULTISCALE DECOMPOSITION
In this subsection, a multiscale representation of the 'cameraman' image (Fig. 7) is constituted following a specific kind of morphological operators: the alternating sequential filters (ASFs) which satisfy relevant multiscale properties (Serra and Salembier, 1993).Usual ASFs, usual ASFs by reconstruction (Crespo et al., 1995) and adaptive ASFs, are applied, supplying results which are compared and discussed.
Note that ASFCO m,p and ASFRCO m,p stand for the usual ASF and the ASF by reconstruction of order p, respectively, with a disk of radius m as uniform SE.As regards the filter ASFCO f m,p , it denotes the adaptive ASF of order p using the adaptive SEs with the luminance criterion mapping f and the homogeneity tolerance m.
These results show that the connectedness of adaptive ASFs and usual ASFs by reconstruction is an overwhelming advantage.Indeed, the edges are quickly damaged by the usual ASFs, while they are preserved with the connected ASFs.Moreover, the filters by reconstruction remove fine details so far, as revealed in the scene upon the camera (Fig. 7e) and the eye of the human face (Fig. 7e), although they are connected.On the contrary, the decomposition of the original image with ANMM-based filters, does not decimate relevant structures from fine-to-coarse scales (Fig. 7h-j).

BOUNDARY DETECTION
A real example in the field of image segmentation is now illustrated on a metallurgic grain boundaries image (Fig. 8).Several methods have ever been introduced (e.g., Chazallon and Pinoli, 1997) requiring most of the time complex processes.Elementary ANMM-based processing is then suggested and compared with the corresponding usual MM approach.Seeing that the crest lines of the original image fit with the narrow grain boundaries, the watershed transform, denoted W , is directly applied on smoothed images (processed with closing-opening filters) in order to avoid an over-segmentation.Note that CO r (resp.CO h m ) stands for the usual closing-opening using a disk of radius r as uniform SE (resp.using the adaptive SEs with the homogeneity tolerance m, and the criterion mapping h).The adaptive approach overcomes the usual one, achieving an efficient segmentation of the original image, with the expected result for m = 20 .Indeed, the adaptive filters, contrary to the usual ones, do not damage the boundaries and well smooth the image inside the grains.This event is a direct consequence of the connectedness of the adaptive morphological operators.Besides, the minima of the (adaptively) filtered images provide markers, for the watershed transformation, all the more significant so the parameter m increases.This argument leads to the well-segmented image W (CO f 20 ( f )).In their current research works, the authors take an interest in the difficult task of automatically picking the fair parameter m.

ADAPTIVE MORPHOLOGICAL HIERARCHICAL SEGMENTATION
In this last application example, the 'tools' test image (Fig. 9) is hierarchically segmented.The process is achieved in three steps: 1. firstly, the original image is smoothed with adaptive sequential closing filters (Eq.35) which satisfy relevant properties (anti-size distribution and connectedness), supplying a multiscale representation, 2. secondly, the morphological gradient is computed on decomposed images, 3. finally, the watershed transform is applied to the previous images.
In this way, it leads to a hierarchical segmentation (Vachier, 2001) without any operators by reconstruction.
The resulting hierarchical segmentation supplies nested partitions of the spatial support of the original image, which could induce a graph representation (Serra and Salembier, 1993;Vachier, 2001).Thereafter, the process C f 4,n offers the expected result for n = 15 or n > 15.Indeed, this operator is saturated from this value: This characteristic has been studied and promises relevant topological properties (Debayle, 2005).Furthermore, the resulting hierarchical segmentation, where flat zones are nested, is achieved without any operators by reconstruction (Crespo et al., 1995).Filters by reconstruction require geodesic transformations, so as to define connected operators, which are traditionally used for this kind of multiscale segmentation (Serra and Salembier, 1993;Salembier and Serra, 1995;Vachier, 2001).
So, the connectedness of the elementary ANMMbased operators is an overwhelming advantage: all operators built by composition or combination with the supremum and the infimum of the adaptive dilation and erosion, define connected operators.

IMPLEMENTATION ISSUE
The algorithms of the proposed morphological operators are built in two steps.Firstly, the AN sets are computed and stored in random access memory (RAM).The equality property (Prop.1.3) between iso-valued points is used so as to save memory and reduce computation time.Secondly, the operators are run with the stored AN sets.In this way, AN sets are computed once time even for advanced operators, such as adaptive ASFs.Nevertheless, this methodology requires a large random access memory (RAM) so as to store all the AN sets.Moreover, the computational time of AN sets is rather long, about 2-3 minutes for of a 256 × 256 image, with a Pentium IV (3GHz / 2Go RAM) using the software Aphelion TM and C++ language.Conversely, the running of morphological operators is faster (for example 4-5 seconds for a dilation).

CONCLUSION
The proposed spatially-variant morphological approach provides adaptive operators without any a priori knowledge of the studied image.Theoretically, such transforms possess several strong advantages such as the connectedness of the operators, contrary to the usual ones which fail to this property.In practice, it yields good results in the field of image filtering and segmentation.The idea of building spatially adaptive operators using intrinsic operational windows which locally fit to the features of an image is very promising and can suit different image processing tools, more particularly in multiscale image analysis as studied in (Debayle and Pinoli, 2005b).
Currently, the authors are interested in AN sets taking into account physical and psychophysical settings allowing to be consistent with several image formation models and with the human visual perception.In addition, other analyzing criteria are actually being studied.

Fig. 2 .
Fig. 2. Original electrophoresis gel image (a).The adaptive neighborhood set for the seed point highlighted in (d) is respectively homogeneous (e,f), with respect to the tolerance m, in relation to the luminance criterion (b) or to the contrast criterion (c).

Fig. 3 .
Fig. 3. Nesting of AN sets of four seed points (b) using the luminance criterion (a) and different homogeneity tolerances: m = 5, 10, 15, 20 and 25 encoded by the color table(c).An AN set defined with a certain homogeneity tolerance could be represented by several tinges of the color associated to its seed point.
ImageAnal Stereol 2005;24:145-158    tools in the aim of image filtering, image segmentation and classification, image measurements, pattern recognition, or texture analysis.

Fig. 6 .
Fig. 6.Usual vs adaptive morphological operators on a blood vessels (a) image f : usual erosion (b) / dilation (c) / opening (d) / closing (e) with a disk of radius 1 as isotropic SE -adaptive erosion (f) / dilation (g) /opening (h) / closing (i) with adaptive SE computed with the luminance criterion f and the homogeneity tolerance m = 20.

Fig. 8 .
Fig. 8. Usual (b-d) and adaptive (i-k) closing-opening filters of the original image (a) and their corresponding segmentation (resp.(f-h) and (l-n)) using the watershed transformation, denoted W .The original image is primarily filtered in order to avoid an over-segmentation (e).

Fig. 9 .
Fig. 9. Hierarchical pyramidal segmentation of the 'Tools' (a) image.First, the original image is decomposed using adaptive sequential closing filters (b-d).Secondly, the application of the morphological gradient followed by the watershed transformation, denoted S, achieves the images (f-h).The original image is decomposed so as to avoid an over-segmentation (e).The process S(C f 4,n ( f )) provides a well-accepted segmentation for n ≥ 15 (g,h).