TU Wien:Experiment Design for Data Science VU (Knees)/Exam 2022-01-27

Exam 2022-01-27

The exam consisted of 4 open questions and you had 60 minutes of time. The exam was conducted in TUWEL because of the corona pandemic.

To attend the exam, you had to join a Zoom meeting with two devices, one filming from the front and one from the side/back.

Apparently, there were different groups.

Questions

Question 1

a) Consider the following Confusion Matrix obtained from comparing the outputs of a machine learning classifier with the ground truth:

	Classified positive	Classified negative
Actual positive	611	89
Actual negative	194	106

Calculate Accuracy, Sensitivity, Specificity, Precision, and Recall. Write the equation for each and fill in the numbers.

Notes:

• Presenting a final, real number is not necessary, a fraction is sufficient.
• you can use LaTeX Math notation in your answers, but it should not even be necessary.

b) Now consider the following Cost Matrix, showing the costs associated with making certain errors.

Error Cost	Classified positive	Classified negative
Actual positive	0	12
Actual negative	2	0

Given these costs, would you optimize the machine learning classifier towards Precision or Recall? Explain your answer!

Question 2

For comparison of two machine learning based email spam classifiers A and B, a ground truth annotated dataset is randomly split into 90% training data and 10% test data. Training and test data are used in the same manner for both A and B. This procedure is carried out 20 times. 

1. Explain the concept of repeated test-train spilts and how it differs from other evaluation setups (briefly compare to at least two other strategies). What are advantages and disadvantages?
2. In order to find out whether one of the classifiers is statistically significantly better than the other with regard to the chosen evaluation measure, a statistical test should be carried out. Which test is applicable here and why? (If multiple tests are applicable, pick the one with highest power.)
3. Which types of errors can you make in statistical hypothesis testing? Give a brief definition of each. How can you minimize the chances of making these errors?

Question 3

Below is one statement from the Modelers' Hippocratic Oath (Derman, 2012) and two rules for responsible big data research (Zook et al., 2017). For EACH of the three statements/rules, answer the following questions:

• (i) Explain the statement/rule and state what aspect of Data Science it is meant to warn about.

• (ii) Give one concrete example of a situation in Data Science in which this statement/rule is applicable.

• (iii) Explain which measures should be taken by Data Scientists to satisfy the statement/rule.



Make sure that you number your answers clearly as A(i),(ii),(iii), B(i),(ii),(iii), C(i),(ii),(iii).

• A. Though I will use models boldly to estimate value, I will not be overly impressed by mathematics.

• B. Acknowledge that data are people and can do harm.

• C. Consider the strengths and limitations of your data; big does not automatically mean better

Question 4

Data Management Plans

• 
a) Explain the five common themes / key aspects that need to be addressed in a data management plan and provide examples for each
• b) Describe the motivation and key concepts underlying machine-actionable DMPs

Question 5

Reproducibility

a) List and describe common sources of irreproducibility from a computing perspective

b) Describe the PRIMAD model of reproducibility types and explain the insights gained by priming (modifying) the respective elements

Question 6

Below is one statement from the Modelers' Hippocratic Oath (Derman, 2012) and two rules for responsible big data research (Zook et al., 2017). For EACH of the three

statements/rules, answer the following questions:

(i) Explain the statement/rule and state what aspect of Data Science it is meant to warn about.

(ii) Give one concrete example of a situation in Data Science in which this statement/rule is applicable.

(ili) Explain which measures should be taken by Data Scientists to satisfy the statement/rule.

Make sure that you number your answers clearly as A(i), (li), (ili), B(i), (li), (ili), C(i), (il), (ili).

A. I will remember that I didn't make the world, and it doesn't satisfy my equations.

B. Recognize that privacy is more than a binary value.

C. Design your data and systems for auditability.

Question 7

Consider an experimental setup in which you want to compare two machine learning algorithms A (SVM) and B (CNN). In total, the data set consists of 15.000 instances, and

you are using cross validation with k=50 folds and F-measure (with macro averaging) as performance criteria.

1. Explain the concept of cross validation and how it differs from other evaluation setups (briefly compare to at least two other strategies). What are advantages and

disadvantages?

2.

Outline at least two different strategies to perform a significance test on the generated measurements (F-measure values; or other depending on the strategy). For each

strategy, which significance test is applicable and how many sampled values are underlying your test, i.e. which value does N have?

3. Which types of errors can you make in statistical hypothesis testing? Give a brief definition of each. How can you minimize the chances of making these errors?