AI for Security: Concepts and Technical Principles
Cybersecurity and Artificial Intelligence
Why Do We Need AI in Cybersecurity?
Security spans a multitude of dimensions: from threat detection, incident response, user authentication to privacy, leak detection, and encryption. Delving into threat detection reveals significant insights.
Traditional Detection Based on Rules – Unarmed Outside the Rulebook
Traditional threat detection mechanisms lean heavily on predefined rules. With a set of 100 rules, you're equipped to identify 100 specific threats. Any threat beyond this set remains undetected.
The Constraints of Human-defined Rules
Human professionals craft these rules. Consider the scalability: how many rules can a single security expert construct in a year after rigorous data assessments and testing? The reliability of these rules also remains questionable.
Shortcomings of Manual Rules: Overlooking the Novel and the Altered
Manual rule creation struggles to catch new, novel threats or slightly modified threats. Even if an event matches a rule, unless that rule is flawlessly designed, the detected event might be a false alarm. Historically, failing to detect real threats ("false negatives") and raising false alarms ("false positives") have both been significant vulnerabilities.
Navigating the Challenges of Detection
Given the potential ramifications of false negatives, one might assume that minimizing them would be paramount. Yet, there's been a resignation to their inevitability, with more emphasis paradoxically placed on reducing false positives.
The Emergence of AI-driven Solutions
In this burgeoning AI era, the transition from human-crafted rules to AI-powered detection mechanisms feels not only inevitable but essential.
AI Security Initiatives and Setbacks in Korea
Consistent Failures in Korea's AI Security – Over-reliance on Supervised Learning
South Korea initiated its journey into AI-enhanced security around 2017. Though an instinctive next step for many, the early endeavors witnessed significant hiccups. The primary reason? An exclusive reliance on supervised learning.
The Inherent Challenges of Supervised Learning in Security
For supervised learning to work, AI requires both a test (data) and the corresponding answers. In the realm of security, this would be event data and its associated threat identification. However, procuring reliable data on threats, especially new or variant ones, is challenging. If you're only capturing known threats, then the value addition of an AI model becomes questionable.
The Catch-22 of Supervised Learning and Unknown Threats
Using supervised learning to identify unfamiliar threats presents a paradox: how can one teach the AI about something unknown? It's evident that to identify and combat new threats, unsupervised learning is a more apt approach.So, the question remains: Why did South Korea persist with supervised learning in its security for such an extended period?
Global Progress in AI Security
Emphasis on Unsupervised Learning Internationally
Internationally, the pivot to unsupervised learning in security came sooner than in Korea. Investment and research in this domain quickly ramped up.By 2018, companies from the US and UK began introducing products centered on unsupervised learning, showcasing their effectiveness. This spurred others to innovate and compete in the same realm.
International Entrants Gaining Ground in Korea
These pioneering companies ventured into the Korean market, gaining traction. Meanwhile, Korea persisted with its emphasis on supervised learning. With tech advancements in the fast lane, there's growing concern among Korean industry stakeholders about international players dominating their local security market.
Why AI Security Stumbled in Korea
The primary roadblock: Absence of foundational technology.
Ready Access to Supervised Learning (Kudos to Tech Giants)
Google and Facebook have made supervised learning widely accessible by offering the tech behind it for free. This freely available technology encompasses deep learning, the linchpin of supervised learning, ready for research and commercial applications.
Unsupervised Learning Lacks Robust Engines
Unsupervised learning tools, especially for security, fall short of expectations. They need to process real-time data influx in security, and currently available public tools aren't up to the mark. While deep learning can manage unsupervised tasks, it lags in speed compared to other unsupervised models.
To Thrive in Unsupervised Learning, Forge Your Own Path...
A high-performance engine, tailored for unsupervised learning, is pivotal. Yet, the absence of such a public tool means companies must craft their own. Can firms accustomed to ready-made AI engines innovate one from scratch?
Diverse Core Technologies Needed for In-House Development
Engine creation is distinct from its usage. It demands profound understanding of machine learning algorithms, advanced mathematics, computer engineering, and a broad spectrum of knowledge beyond AI. Acquiring this foundational tech is time-consuming and cannot be expedited.
Artificial Intelligence in Security Framework
High-Performance Unsupervised Learning Engines - Clustering Engines
Emphasizing High-Performance
The effectiveness of unsupervised learning, especially in detecting new threats, hinges on its high performance. It's imperative for these engines to continually process new user data efficiently.
Diverse Algorithm Usage
The nature of unsupervised learning requires flexibility in choosing algorithms. The specific type of data at hand and the objectives dictate the most suitable algorithm selection.
Incorporating Supervised Learning
Strategic Utilization
Even though supervised learning can't be wholly relied upon due to data limitations, it's crucial when focusing on specific threat detection. It should be employed optimally, but not exclusively.
Supplementary Techniques
Since a 100% supervised learning approach isn't feasible, it's essential to bolster its outcomes with various methods external to machine learning.
Large Data Processing Platforms
Scalable Unsupervised Learning
Security handles an overwhelming amount of data, different from static models like photo recognition. There's a continuous need to feed, train, and analyze data due to emerging threats and user-generated data.
Optimized Data Stream Handling
A potent machine learning engine isn't the sole requirement. A comprehensive platform must manage vast real-time data without any performance drops. This undeniably demands profound computer engineering skills.
Automatic Labeling
Navigating Ambiguous Results
Unlike supervised learning, unsupervised outcomes aren't tied to specific labels. Even supervised models with inadequate resources might produce questionable label suggestions.
Techniques for Enhanced Labeling
For unsupervised learning, there's a need for labeling, and for supervised learning, label enrichment is beneficial. Achieving this requires proficiency that goes beyond the domain of machine learning.