Seminar für Diplomand_innen Methods and Paper analysis
Formal vs Applied Research Methods
Benchmarking
Method | Where found | Description |
Survey / Questionnaires | PANAS | “State 3 positive and 3 negative aspects on” |
RESEARCH QUESTIONS | Page 2 | |
Wizard of Oz | Page 3 | Wizard of Oz Prototypes. The course was predefined but people believed it were the actions of their human drivers |
Quantitative online Measurement (Stress): Physiological sensing (ECG) | Page 3 | “How stressed are passengers with different types of drivers?” |
Kruskal-Wallis test[a] | Page 4 | |
Pattern Recognition | Page 4 | From the paper: “In order to reduce data processing load, we extracted a single frame for every second of recorded video and used a C# program to classify the image according to the specification in the API documentation.” |
Classification | Page 4 | Classified the found results |
Survey with Scale | Page 4 | Standardized Survey with a 5 point scale from “Very slightly or not at all” to “Extremely” |
Post/Pre Test Measurements | Page 5 | Circumplex Model |
Interviews | Page 6 | |
Method | Where found | Description |
Research Questions | Page 2 | “RQ1: How to implement exploratory search on EKGs?”, “RQ2: How to identify suitable relatedness metrics?”, “RQ3: Are explanations derived from the EKG helpful?” |
Use Case Analysis | Page 3-4 | |
Prototype | Page 3 | “we designed, implemented and deployed the Semantic Search for Architectural ^Knowledge (STAR) prototype” |
Ontology Engineering | Page 5 | “To construct the STAR ontology, in an initial scoping workshop system purpose,” |
Requirement Engineering | Page 5 | “in an initial scoping workshop system purpose, use cases, and competency questions were clarified” [b] |
Comparative Evaluation | Page 7 | 6.1 Comparative Evaluation of Relatedness Metrics We use statistical methods to compare the overall behaviour of the two metrics over the entire EKG |
User Based Evaluation (including Survey with Scale) | Page 7-9 | 6.2 User Based Evaluation half of the dataset) an explanation of how their relatedness was derived. They were asked to rank the pairs relatedness on a 5-point scale (1 completely unrelated; 5 very related) |
Experimental Evaluation Method | Page 8-9 | Evaluation with experiment-data-set and check quality / usefulness of results |
Method | Where found | Description |
Research Questions | Page 2 | “RQ1: Is IoT malware different than traditional malware? • RQ2: Are current anti-malware techniques effective against IoT malware?” |
Empirical studies | Page 2 | “This work seeks to investigate this matter by systematically and empirically studying the lifecycle of IoT malware and comparing it with traditional malware that target desktop and mobile platforms. We present a large-scale measurement of more than 166K Linux-based IoT malware samples collected over a year.” |
Page 2, 3 | Second, we use our framework to systematize 25 papers that study traditional malware. We study 25 papers from prior works to qualitatively derive subcategories under each component, which are in Appendix A. For example, we cite the work of Holz et al. [42] to support the use of drive-by downloads in desktop malware and their distribution networks. Moreover, we use the MITRE ATT&CK taxonomy to derive additional subcategories that are not found in prior work but are documented by security companies. | |
Qualitative Measures: Analyse of Malware/Code | Page 2, 6 | For RQ2, we qualitatively evaluate how traditional anti-malware techniques work and judge their efficacy based on empirical observations from the IoT malware ecosystem. |
Software Engineering: Code/Framework | Page 2 | First, we propose a novel analysis framework that captures the threat lifecycle of IoT malware, which considers the infection vectors, payload properties, persistence methods, capabilities, and C&C infrastructure. |
(not sure if legit method) Data Science: Publishing Data | Page 2 | Third, we characterize IoT malware by examining more than 166K samples spanning 6 different system architectures collected over a year. |
Literature Research | Page 2 | Second, we use our framework to systematize 25 papers that study traditional malware. |
Baselining | Page 6 | “For network artifacts, we collected network traffic from the VM for 72 hours without executing any malware. We then filter out any traffic that matches the baseline or bogon networks” |
Quantitative Measures: Statistics (to Malware) | Page 7 | However, we were not able to build a VM for M68K architecture due to legacy code incompatibility, therefore, we only considered the M68K samples for static analysis. |
Qualitative Measures: | Page 12 | Approximately 50 minutes later, based on pDNS first seen resolution, the domain is detected and reported to URLHaus. |
These questions must be answered with TRUE/FALSE:
All italics are suggestions and I am not 100% sure
Does the paper "Automated Driving System, Male, or Female Driver: Who’d You Prefer? Comparative Analysis of Passengers’ Mental Conditions, Emotional States & Qualitative Feedback" contain "User study" as one of the methodologies?
TRUE
Does the paper "Automated Driving System, Male, or Female Driver: Who’d You Prefer? Comparative Analysis of Passengers’ Mental Conditions, Emotional States & Qualitative Feedback" contain "High-fidelity Prototype" as one of the methodologies?
Does the paper "Automated Driving System, Male, or Female Driver: Who’d You Prefer? Comparative Analysis of Passengers’ Mental Conditions, Emotional States & Qualitative Feedback" contain "Quantitative study" as one of the methodologies?
TRUE
Does the paper "Automated Driving System, Male, or Female Driver: Who’d You Prefer? Comparative Analysis of Passengers’ Mental Conditions, Emotional States & Qualitative Feedback" contain "Physiological sensing" as one of the methodologies?
TRUE
Does the paper "Automated Driving System, Male, or Female Driver: Who’d You Prefer? Comparative Analysis of Passengers’ Mental Conditions, Emotional States & Qualitative Feedback" contain "Cognitive walkthrough" as one of the methodologies?
FALSE
Does the paper "Automated Driving System, Male, or Female Driver: Who’d You Prefer? Comparative Analysis of Passengers’ Mental Conditions, Emotional States & Qualitative Feedback" contain "Grounded theory" as one of the methodologies?
TRUE[h]
Does the paper "The Circle Of Life: A Large-Scale Study of The IoT Malware Lifecycle" contain "Differential Analysis" as one of the methodologies?
TRUE
Does the paper "The Circle Of Life: A Large-Scale Study of The IoT Malware Lifecycle" contain "Passive Measurements" as one of the methodologies?
TRUE
Does the paper "The Circle Of Life: A Large-Scale Study of The IoT Malware Lifecycle" contain "Dynamic Program Analysis" as one of the methodologies?
TRUE
Does the paper "The Circle Of Life: A Large-Scale Study of The IoT Malware Lifecycle" contain "Active Measurements" as one of the methodologies?
Does the paper "The Circle Of Life: A Large-Scale Study of The IoT Malware Lifecycle" contain "Model Checking" as one of the methodologies?
FALSE(?)
Does the paper "The Circle Of Life: A Large-Scale Study of The IoT Malware Lifecycle" contain "Benchmarking" as one of the methodologies?
FALSE
Does the paper "Exploring Enterprise Knowledge Graphs: A Use Case in Software Engineering" contain "Ontology Engineering" as one of the methodologies?
TRUE
Does the paper "Exploring Enterprise Knowledge Graphs: A Use Case in Software Engineering" contain "Systematic Literature Review" as one of the methodologies?
Does the paper "Exploring Enterprise Knowledge Graphs: A Use Case in Software Engineering" contain "design science/relevance" as one of the methodologies?
Does the paper "Exploring Enterprise Knowledge Graphs: A Use Case in Software Engineering" contain "design science/rigour" as one of the methodologies?
Does the paper "Exploring Enterprise Knowledge Graphs: A Use Case in Software Engineering" contain "theorem proving" as one of the methodologies?
FALSE
Does the paper "Exploring Enterprise Knowledge Graphs: A Use Case in Software Engineering" contain "dynamic bayesian networks" as one of the methodologies?
FALSE
[a]ANOVA as well?
[b]Unsure if Requirement Engineering is valid here or it is just Ontology Engineering
[c]More likely a taxonomy because there are no other relations and constraints?
[d]I agree, i think this should be taxonomy or classification
[e]Found this paper explaining the differences of Ontology/Classification/Taxonomy:
https://www.cs.auckland.ac.nz/research/conferences/w78/papers/W78-37.pdf
[f]Is this false because the prototype used does not look/feel like the final product and is therefor a low-fi prototype?
[g]See section LIMITATIONS AND FUTURE WORK:
"Comparedto today’s standards in computer graphics (e. g., computergames), the rendering of the environment shown in thedriver simulator could have been more realistic."-> low-fidelity graphics -> low fidelity prototype
[h]I would opt for False, as I fail to see the authors creating any form of abstract theory that explains what would or would not lead to a preference of female/male/automated.
[i]I would agree with this since data was only collected passively or taken from datasetsand no network devices were scanned or tested for vulnerabilities.
But don't quote me on that.
[j]Assuming the question is about active and passive _network_ measurements as the slides hint at, I would agree with FALSE, as they only record network traffic and don't scan the C&C IPs or similar active reconnaissance
[k]Don't they collect Data with network traces etc. in the Dynamic analysis part of the Paper?
[l]It should have been true, according to their evaluation
[m]I am really unsure about that. From my point of view, executing malware to cause syscalls and network traffic is active.
[n]I would argue that active measurements only concerns real network traffic analysis. In this case this seems to be traffic of a test VM as part of dynamic binary analysis.
[o]Ofc there was literature research done in this paper, but i cannot see "Systematic Literature Review"
[p]Ah yes, I see your point. If this questions would be asked for the paper "The Circle Of Life: A Large-Scale Study of The IoT Malware Lifecycle" it would be true, imho.
but yes, here FALSE makes more sense
[q]I would answer TRUE since Siemens is presented as relevant business case
[r]I see it as you: "To refine the ontology, several iterations were performed with Siemens partners to validate the relevance and usefulness of these concepts for the use case"
[s]Could someone explain to me why this is true?
[t]I think because the solution is build up onto something existing and using existing knowledge of architecture
