Distance and Similarity Measures Overview

Notes on Distance and Similarity Measures

1. Minkowski Distance

• Concept: A generalized form of both Euclidean and Manhattan distances.
• Formula: Varies with the parameter $p$:
  • $p = 1$: Equivalent to Manhattan Distance.
  • $p = 2$: Equivalent to Euclidean Distance.
• Insight: Used to measure the "straight line" distance between two points. Different values of $p$ allow flexibility in how distance is computed.
• Application: Useful in clustering and nearest neighbor algorithms.

2. Euclidean Distance

• Concept: Measures the "straight line" distance between two points in space.
• Application: Extensively used in clustering (e.g., k-nearest neighbors or KNN) and classification tasks.
• Additional Info: European mathematicians formulated this metric; it's fundamental in geometry.

3. Levenshtein Distance (Edit Distance)

• Concept: Measures the similarity between two strings by counting the minimum number of operations (insertions, deletions, or substitutions) required to transform one string into the other.
• Application: Commonly used in spell-checking, DNA sequence alignment, and natural language processing.
• Additional Info: Named after Vladimir Levenshtein, useful in error correction.

Similarity and Correlation Algorithms

1. Jaccard Index

• Concept: Measures the similarity between two sets by dividing the size of their intersection by the size of their union.
• Application: Text analysis for calculating the similarity of characters or words.
• Additional Info: Useful in clustering and recommendation systems.

2. Cosine Similarity

• Concept: Computes the cosine of the angle between two vectors; smaller angles indicate more similarity.
• Application: Widely used in text classification and clustering via vector space models.
• Additional Info: Particularly useful in high-dimensional spaces.

3. Pearson Correlation Coefficient

• Concept: Evaluates the linear correlation between two variables.
• Application: Used in statistics for linear regression analysis to determine the strength of a linear relationship.
• Additional Info: Values range from -1 to 1, indicating the degree of linear relationship.

4. Spearman Correlation

• Concept: Measures the monotonic relationship between variables based on ranked values.
• Application: Serves as an alternative to Pearson in nonlinear analyses.
• Additional Info: Useful for ordinal data.

5. Kullback-Leibler Divergence

• Concept: Measures the difference between two probability distributions.
• Application: Utilized in information theory to evaluate distribution similarity.
• Additional Info: Asymmetric measure, meaning $KL(P || Q) \neq KL(Q || P)$.

6. Hellinger Distance

• Concept: Computes the similarity between two probability distributions.
• Application: Used in statistics and machine learning for assessing distributional differences.
• Additional Info: Related to Euclidean distance but in probability space.

Extended readings: