Assignment 4

Learning Objectives

Learning Objectives

• Evaluate an Information Retrieval System
• Combine Qualitative and Quantitative Methods for System Evaluation

Model Evaluation for Retrieval Systems

External Resources

Read this article about evaluating information retrieval systems. You can stop reading at the section called NDCG@K since the data we will use for evaluation doesn’t match the requirements needed to apply this metric.

Evaluating Machine Learning Systems

Exercise 1

Come up with a plan for evaluating the EchoMinds app. You may consider evaluating the user experience of the system (e.g., do people find the app helpful, usable, etc.) itself, testing the underlying machine learning system (e.g., by evaluating it on a dataset), or some combination. You may also find it useful to consider qualitative versus quantitative methods of assessment or aspects of the user experience.

You can use bullet points or paragraphs to describe your plan, but you should put some real thought into this. Try to spend at least 45 minutes thinking through this problem.

Solution

Model Analysis

Exercise 2

Go through the Colab notebook on analyzing the machine learning model used in the EchoMinds app.

Solution

(Optional) Download the Echo Minds App

If you didn’t do this in class and you want to play around with the app yourself, here are the instructions to get the EchoMinds app on your iOS device. If you do not have an iOS device, please consider coming to my office hours this Wednesday from 1-2pm (MAC212). You can test it on one of the devices in my lab.

1. Download the TestFlight app from the App Store
2. Click the following link (on your iPhone or iPad) to join the EchoMinds beta.
3. Load some notes about accessible tech (find this in the settings menu)