Information retrieval

• Currently reading an article

• Currently reading an article
• Goal: Want to find an article similar to that one

• How do we measure similarity?
• How do we search over articles?

• Organize all articles by similarity of text

• Organize all articles by similarity of text

• Organize all articles by similarity of text

• Organize all articles by similarity of text

• Item (e.g., doc) representation \(x_q\)
• Measure of distance between items : \[ \xi = \textit{distance}(x_i, x_q) \]

Bag of words model

• Igonore order of words
• Count # of instances of each word in vocabulary

Sample sentence with matrix:

• Common words in doc: "the", "player", "field",...
• Dominate rare words like: "cricket", "Shane Watson"

• Emphasizes important words

  • Appears frequently in document (common locally)
  • Term frequency = Word Count

• Appears rarely in corpus (rare globally)

\[ \textit{Inverse doc frequency} = \log\frac{\textit{#docs}}{1 + \textit{#docs using the word}} \]

• tf*idf

Notion of closeness?

• Euclidean distance:

\[ \textit{distance}(x_i,x_q) = |x_i - x_q| \]

• Some features might be more important than others
• Weighing features in distance metrics

• Weight on each feature
• Setting weights as zero or one is same as feature selection

\[ \textit{distance}(x_i,x_q) = \sqrt{(a_1(x_i[1]-x_q[1])^2+\ldots+a_d(x_i[d]-x_q[d])^2))} \]

\[ \frac{\mathbf{x}_i^{T}\mathbf{x}_q}{\|x_i\|\|x_q\|} \]