Date of Award
3-26-2015
Document Type
Thesis
Degree Name
Master of Science in Cyber Operations
Department
Department of Electrical and Computer Engineering
First Advisor
Robert F. Mills, PhD.
Abstract
Existing classifier evaluation methods do not fully capture the intended use of classifiers in hybrid intrusion detection systems (IDS), systems that employ machine learning alongside a signature-based IDS. This research challenges traditional classifier evaluation methods in favor of a value-focused evaluation method that incorporates evaluator-specific weights for classifier and prediction threshold selection. By allowing the evaluator to weight known and unknown threat detection by alert classification, classifier selection is optimized to evaluator values for this application. The proposed evaluation methods are applied to a Cyber Defense Exercise (CDX) dataset. Network data is processed to produce connection-level features, then labeled using packet-level alerts from a signature-based IDS. Seven machine learning algorithms are evaluated using traditional methods and the value-focused method. Comparing results demonstrates fallacies with traditional methods that do not consider evaluator values. Classifier selection fallacies are revealed in 2 of 5 notional weighting schemes and prediction threshold selection fallacies are revealed in 5 of 5 weighting schemes.
AFIT Designator
AFIT-ENG-MS-15-M-046
DTIC Accession Number
ADA622990
Recommended Citation
Rich, Michael D., "Evaluating Machine Learning Classifiers for Hybrid Network Intrusion Detection Systems" (2015). Theses and Dissertations. 52.
https://scholar.afit.edu/etd/52