JNTUH B.Tech IV INFORMATION RETRIEVAL SYSTEMS Previous Question Paper (February/March - 2022 - R15)

The Vector Space Model (VSM) is a fundamental model in Information Retrieval (IR) that represents documents and queries as vectors in a multi-dimensional space. Each dimension in this space corresponds to a unique term (word) present in the collection of documents. The value of a document's vector in a particular dimension represents the weight of that term in the document. This model allows for the calculation of similarity between documents and between documents and queries based on the geometric relationships between their corresponding vectors.

The working principle of the VSM can be explained in the following steps:

1. Document Representation: Each document is converted into a vector. The dimensions of this vector correspond to the terms in the document collection.
2. Term Weighting: The value of each dimension in a document's vector is determined by a term weight. A common and effective weighting scheme is TF-IDF (Term Frequency-Inverse Document Frequency).
• Term Frequency (TF): Measures how frequently a term appears in a document. Higher frequency suggests greater importance of the term to the document's content.
• Inverse Document Frequency (IDF): Measures the rarity of a term across the entire document collection. Terms that appear in many documents have a lower IDF, indicating they are less discriminative. The formula is generally: IDF(t) = log(N/df(t)), where N is the total number of documents and df(t) is the number of documents containing term t.
• TF-IDF Weight: The final weight for a term 't' in document 'd' is calculated as TF-IDF(t, d) = TF(t, d) * IDF(t).
3. Query Representation: A query is also represented as a vector in the same multi-dimensional space, using the same term weighting scheme (e.g., TF-IDF) based on the terms present in the query.
4. Similarity Calculation: The similarity between a query vector (Q) and a document vector (D) is calculated using a similarity measure, most commonly the cosine similarity.
• Cosine Similarity: Measures the cosine of the angle between two vectors. A value closer to 1 indicates higher similarity, while a value closer to 0 indicates lower similarity. The formula is: CosineSimilarity(Q, D) = (Q . D) / (||Q|| * ||D||), where '.' denotes the dot product and '|| ||' denotes the magnitude (Euclidean norm) of the vectors.
5. Ranking: Documents are ranked in descending order of their similarity scores with respect to the query. The documents with the highest similarity scores are considered the most relevant.

Similarity measures are crucial in Information Retrieval to quantify the degree of relevance between a query and a document, or between two documents. They are used to rank documents in response to a query. Several similarity measures exist, each with its own characteristics and applicability. The most common ones are:

1. Cosine Similarity

• Description: This is the most widely used similarity measure in VSM-based IR systems. It measures the cosine of the angle between two non-zero vectors in a multi-dimensional space. It is particularly effective for high-dimensional data, such as text documents represented as vectors.
• Formula: For two vectors, A and B, the cosine similarity is given by: CosineSimilarity(A, B) = (A . B) / (||A|| * ||B||), where A . B is the dot product of A and B, and ||A|| and ||B|| are the Euclidean norms (magnitudes) of A and B, respectively.
• Range: The value ranges from -1 to 1. In text retrieval, where term weights are typically non-negative, the range is usually from 0 to 1. A value of 1 indicates that the vectors are identical in direction (perfect similarity), while 0 indicates they are orthogonal (no similarity).
• Advantages: It is independent of document length, making it suitable for comparing documents of varying sizes.

2. Jaccard Similarity

• Description: This measure is often used for comparing the similarity and diversity of sample sets. In the context of text, it can be applied to sets of terms present in documents. It is defined as the size of the intersection of two sets divided by the size of their union.
• Formula: For two sets, A and B, the Jaccard similarity is: JaccardSimilarity(A, B) = |A ∩ B| / |A ∪ B|
• Range: The value ranges from 0 to 1. A value of 1 indicates that the sets are identical, while 0 indicates they have no elements in common.
• Application: Useful when the presence or absence of terms is more important than their frequency.

3. Euclidean Distance

• Description: While primarily a distance measure, it can be converted into a similarity measure. It calculates the straight-line distance between two points (vectors) in Euclidean space. Shorter distances imply higher similarity.
• Formula: For two vectors, A and B, the Euclidean distance is: EuclideanDistance(A, B) = sqrt(∑(Ai - Bi)²)
• Conversion to Similarity: Similarity can be derived by taking the inverse of the distance, e.g., Similarity = 1 / (1 + Distance), to ensure the similarity is bounded between 0 and 1.
• Disadvantage: It is sensitive to document length, as longer documents tend to have larger Euclidean distances.

4. Overlap Coefficient

• Description: This measure calculates the ratio of the size of the intersection of two sets to the size of the smaller of the two sets. It is useful when one set is a subset of another.
• Formula: For two sets, A and B, the overlap coefficient is: OverlapCoefficient(A, B) = |A ∩ B| / min(|A|, |B|)
• Range: The value ranges from 0 to 1. A value of 1 indicates that one set is a subset of the other (or they are identical).
• Use Case: Can be useful in scenarios where partial matches are of interest, especially if one document is significantly shorter than another.

The choice of similarity measure depends on the specific characteristics of the data and the retrieval task. Cosine similarity is generally preferred for text retrieval due to its robustness against document length variations.