Density Based clustering evaluation with DCV algorithm

 Density-Based Cluster Validity (Dcv):

  • It evaluates clusters produced by density based clustering algorithms such as DBSCAN and HDBSCAN.
  • The fundamental idea of Dcv is to compare the density inside clusters and the density outside the clusters (overall dataset).
  • Cluster Density:   
    • For each cluster Ci, the intra-cluster density D(Ci) is computed by averaging the pairwise distances between all points within the cluster. This gives a measure of how tightly packed the points in the cluster are.
  • Overall Dataset Density:
    • The overall dataset density D(X) is computed by looking at the pairwise distances between all points in the dataset, both within and outside of clusters. This acts as a baseline density for comparison.
  • Validity Measure:
    • The Dcv index V is computed by comparing the intra-cluster densities to the overall dataset density:

 

  • ∣Ci∣ is the number of points in cluster Ci.

  • D(x) is the density for each point in cluster Ci.

  • D(X) is the overall dataset density.



  • The Dcv score can be interpreted as the average intra-cluster density divided by the overall dataset density.

  • A high Dcv score indicates that the points within clusters are much denser than the overall dataset, which suggests good clustering.

  • A low Dcv score means the clusters are not much denser than the overall dataset, indicating poor clustering or that the clusters are not well-separated.



  • Pros of Dcv:

  • Works well with non-convex clusters.

  • Can handle varying densities within the dataset.

  • Evaluates noise and outliers effectively.


  • Cons of Dcv:

  • Computationally expensive, especially when calculating pairwise distances for large datasets.

  • Not as widely supported in standard clustering libraries.

  • Sensitive to parameters like ϵ\epsilonϵ and min_samples in DBSCAN/HDBSCAN, meaning careful tuning is needed.

Comments

Post a Comment

Popular posts from this blog

Understanding of Attribute types

Image
  Know your Data What is data It is set of records, having one or more attributes or features. It is also called collection of data objects. Example of Data Name Income Profession Mother Tongue Native Place Ram 70000 Doctor Bengali Village Shyam 50000 Carpenter Hindi Small Town Mohan 60000 Engineer Hindi Suburban   Kabir 90000 Doctor   Bengali Metropolitan   Fig1. As shown in Fig1, There are four rows and five columns. Record: Each row is called record. There are four records in the example dataset. Features or Attribute: Each column is called feature or attribute of a record in the dataset. There are five attributes for each record in example dataset. Understanding of Data Knowing of data is very important in data science, whether you are doing Data Mining or applying Machine Learning Techniques on data. To understand the data you have to understand each attribute of a record. The following properties of numbers are typically used to describe attributes. Distinctne...
Read more

Basic Statistical Description of Data

Measure of Central Tendency and Dispersion of Data: We have learnt about data and different attributes of data in  previous  section. As a next step, we have to understand the different values of same attribute with he help of statistical description of data. That means, we have to carefully observe the values of each attributes of different records in data set.  This will help us to find the variance between records, removing the noise and identifying the outliers. Removing of noise and identify the outliers are used during cleaning of data. This process is used in pre-processing of data. In order to do the cleaning of data, there are certain statistical operations. These operations can be applied on different types of attribute.  I will provide the briefs of each operations in this section. I will explain in details of each statistical operations in subsequent sections. In broader level, there are two major statistical Operations used widely to identify properties ...
Read more

Mean value in Data Science

Image
  Mean value in Data Science Mean of a given attribute from data set calculate the centre or average value of that attribute. This is a popular measure of central tendency in data science. It is mostly used in finding average, clustering and preprocessing of data. Mean is the statistical operation is performed on Numerical attribute type. Below is the formula to calculate Mean value:   Example:   For a given data set calculate the mean value of Age of employee in the organisation. Employee ID Age Year of Experience Department 1001 37 15 Research 1002 25 3 Research 1003 35 10 Payroll 1004 27 9 Research 1005 31 9 HR 1006 40 20 Research 1007 24 2 Research Mean of Age = ( 37+ 25+35+27+31+40+24)/7 = 31.28. Weighted Mean: Sometime, each value of attribute in a set may be associated with weight. The weight reflect the significance, importance or occurrence frequency attached to their respective values. Below is the formula to calculate weighted mean:   Trimmed Mean: A mea...
Read more