Leveraging the Public Domain: Open Source Threat Intelligence (OSTI) for Proactive Cybersecurity
Introduction: The Need for Proactive Defense
The cyber world keeps changing, and cyber threats are becoming more common and clever. With more people working remotely, using cloud services, and systems being linked together, companies are facing more risks than ever. Security plans need to move from just reacting to problems to actively preventing them. Open Source Threat Intelligence (OSTI) is becoming important, helping organizations gather public information, like news, social media, and databases, to spot and address threats before they cause real damage.
OSTI involves collecting and analyzing public data to create useful insights about cyber threats. Because it’s open to everyone, both big companies and smaller businesses, as well as individuals, can improve their security. But that same openness also means attackers can access this information, which is why being proactive is so important.
Core Concepts: OSINT, Threat Intelligence, and OSTI
- OSINT (Open Source Intelligence): Intelligence derived from public data, including media, internet sources, government records, and technical data.
- Threat Intelligence (TI): Evidence-based knowledge about cyber threats, focused on actionable insights.
- OSTI: The intersection of OSINT and TI, using public data specifically to inform cyber defense.
A key principle is the transformation of raw data into intelligence through rigorous analysis and contextualization, making it relevant and actionable for specific security needs.
Key Techniques and Tools
OSTI relies on a variety of OSINT collection techniques:
- Passive Collection: Monitoring public sources (websites, social media, threat feeds) without direct interaction.
- Semi-Passive/Active Collection: Interacting with systems (DNS lookups, network scanning) within legal and ethical boundaries.
Common techniques include digital footprinting, infrastructure analysis (WHOIS, IP analysis), social media monitoring, web scraping, metadata analysis, and Google Dorking. Tools like Maltego, Shodan, VirusTotal, and SpiderFoot are widely used to automate and visualize intelligence gathering.
The OSTI Ecosystem: Information Sources
OSTI draws from both structured and unstructured sources:
- Structured: Threat intelligence feeds (AlienVault OTX, Abuse.ch), vulnerability databases (CVE, NVD).
- Unstructured: Government advisories (CISA, FBI), security blogs, research papers, vendor reports, hacker forums, code repositories (GitHub), paste sites, and social media.
Each source varies in reliability and timeliness. Analysts must validate and cross-reference information, especially from less trustworthy sources like forums or social media.
The OSTI Lifecycle
OSTI follows a cyclical process:
There’s a lot of open-source info out there, and it can be a bit overwhelming. To make sense of it and turn it into useful threat intelligence, we need a solid plan. The Threat Intelligence Lifecycle gives us that structure, helping to keep our efforts focused and effective. It’s not just a straightforward process; it’s more like a cycle where what we learn in the later stages can help improve our understanding from the earlier ones.
Phase 1: Planning & Direction
This initial stage is really important for shaping the whole intelligence process. It starts with figuring out clear goals based on what the different people in the organization need. This could be top executives looking for a big-picture view, analysts in the Security Operations Center needing specific alerts, or incident responders wanting detailed information. There are a few key questions to ask: What assets or processes need protection? What threats are likely to come after the organization or its industry? What information will help us strengthen our defenses?
These big goals are broken down into specific Priority Intelligence Requirements (PIRs), which are key questions the intelligence team needs to answer to tackle threats. PIRs help focus efforts. For instance, a PIR could be, How are financially motivated hackers using initial access brokers against businesses in the financial sector? You can further refine PIRs into smaller questions called Essential Elements of Information (EEIs), like What tactics are these hackers using for lateral movement?
Once we have the PIRs and EEIs, a collection plan is created. This step involves figuring out which open-source intelligence (OSTI) sources will likely provide the answers we need. The plan evaluates current sources, spots any gaps, and suggests new ones like feeds, blogs, forums, or databases to check out. The scope and deadlines for the intelligence work might also be set.
For OSTI, this planning phase is really important. With so much open-source data out there, having clear PIRs and a focused collection plan helps analysts avoid getting lost in unnecessary information and ensures that they’re collecting data that truly addresses the organization’s main security issues.
Phase 2: Collection:
In this phase, we gather the needed information from various open sources, following our collection plan. We’ll use OSINT techniques, like web scrapers and search engine searches, including Google Dorking. We can also tap into APIs from services like VirusTotal or Shodan, keep an eye on threat feeds, and monitor social media and forums. It’s smart to pull data from a mix of both public sources and, when it makes sense, our own internal logs that might relate to what we find publicly. Automation helps a lot, especially when we’re dealing with a lot of data from feeds or websites.
Esteban Borges
Phase 3: Processing:
Raw data from different open sources is usually messy and not organized. It needs to be changed into a format that can be analyzed. Here are the main steps in processing it:
- Formatting/Structuring: Organizing data, perhaps parsing text, extracting key entities (IPs, domains, actor names), and loading it into databases or spreadsheets.
- Filtering/Sorting: Removing duplicate entries, irrelevant information (noise), and potential false positives.
- Normalization: Standardizing data formats (e.g., date formats, naming conventions).
- Translation: Converting information from foreign languages.
- Decryption/Decoding: Handling encoded or lightly encrypted data found in some sources.
- Initial Evaluation: Assessing the apparent reliability and relevance of the data before deeper analysis.
This stage is really important for OSTI because open-source data can be tricky with its huge amounts and mixed quality. To handle all this data better, we usually use scripts or specific tools (like TIPs) to make it easier to work with and get ready for real analysis.
Phase 4: Analysis:
This is the main stage where we turn data into useful insights. Human analysts, using various tools, look at the information to respond to the key questions we set in Phase 1. This includes:
- Identifying Patterns and Trends: Recognizing connections between seemingly disparate data points, identifying recurring TTPs, and tracking threat actor campaigns.
- Correlation: Linking indicators (IoCs) to specific malware families, threat actors, or vulnerabilities.
- Contextualization: Evaluating the findings in the context of the organization’s specific environment, assets, and risk profile. Assessing the potential impact of identified threats.
- Attribution (where possible): Attempting to identify the threat actors responsible for observed activity.
- Developing Insights: Forming conclusions and actionable recommendations based on the evidence.
Using structured analysis methods like hypothesis testing or comparing different ideas can make the process more reliable. Data visualization tools can also show us connections. The real benefit of OSTI comes out during the analysis phase, turning public info into useful intelligence.
Phase 5: Dissemination:
After the analysis is done, the findings should be shared with the people who need to know, in a way that’s easy for them to understand and use. The format and detail should suit the audience. For instance:
- Executives: May receive concise briefings or high-level reports focusing on strategic risks, business impact, and recommended investments, avoiding deep technical jargon.
- SOC Analysts/IR Teams: May receive detailed technical reports with specific IoCs, TTP analysis, and tactical mitigation steps.
- Security Tools (SIEM, SOAR, Firewalls): May receive intelligence via automated feeds or API integrations for direct operational use (e.g., blocking rules).
Getting the information from OSTI out there helps make better decisions and actually improves security.
Phase 6: Feedback:
In this last but important step, we gather and use feedback from the people who rely on the information. Stakeholders share their thoughts on how timely, relevant, accurate, and useful the information is. This feedback helps shape the next round of the process.
- Were the PIRs adequately addressed?
- Should priorities be adjusted?
- Were the sources effective? Are new sources needed?
- Was the analysis sound?
- Was the dissemination format useful?
This feedback loop is essential for continuous improvement and ensuring the OSTI program remains aligned with the organization’s evolving needs and the dynamic threat landscape.
This process has a cycle to it, especially the feedback loop, and it’s really important for OSTI, maybe even more than info from fancy, private sources. Dealing with the messy and often unreliable public information can be tough, so getting regular input from people involved is key. Without that input, OSTI could lose touch with what really matters for the organization, which can lead to wasting time and effort on data that doesn’t help. A good feedback system helps keep the Planning & Direction phase focused, making sure that the whole OSTI process makes the best use of public resources to tackle the biggest cyber threats.
Benefits and Challenges
Benefits:
- Cost-Effective: Many sources are free, lowering barriers for smaller organizations.
- Broad Visibility: Access to global threat information.
- Timeliness: Real-time updates from the open-source community.
- Proactivity: Enables anticipation of threats.
- Community Collaboration: Leverages global expertise.
Challenges:
- Data Overload: High volume and noise require skilled filtering and analysis.
- Variable Quality: Open sources can be unreliable without validation.
- Resource Demands: Effective OSTI needs trained analysts and appropriate tools.
- Integration Complexity: Merging OSTI with existing security systems can be challenging.
- Ethical and Legal Risks: Practitioners must respect privacy laws, terms of service, and ethical standards.
Integration and Use Cases
OSTI is most effective when integrated into security workflows:
OSTI Integration Points with Core Security Technologies
OSTI for Individuals
OSTI often comes up in talks about organizational security, but its basic ideas about using publicly available information are also super useful for anyone looking to boost their personal cybersecurity. Sure, people don’t have the same resources as big companies for constant, large-scale monitoring, but using OSINT techniques and checking OSTI sources can really help you be more aware and resilient when it comes to your own security.
Assessing Your Personal Risk / Digital Footprint:
You can apply simple OSINT techniques to see how much of your information is out there. Looking up your name, email addresses, usernames, and phone numbers on search engines like Google or DuckDuckGo, as well as on social media, can show you what others can find about you. Checking your email on services like Have I Been Pwned can tell you if your credentials have been exposed in any data breaches. Knowing your digital footprint helps you understand how easy it would be for someone to gather info for social engineering, phishing, or identity theft. For instance, sharing details about your job, location, family, or hobbies could be exploited by others.
Staying Informed about Threats:
You can use many of the same OSTI sources as professionals to keep up with common cyber threats targeting everyday people. Following trusted cybersecurity news websites, security blogs, and consumer advice columns can give you timely info about phishing scams, malware hitting personal devices, and weaknesses in popular software. Government sites also offer alerts and guidance on widespread threats that impact the public, helping you avoid common attacks.
Research and Learning:
If you want to dive deeper into cybersecurity, there are tons of resources available. Public databases like CVE and NVD let you research software flaws. You can read academic papers, blogs, and even well-managed public security forums to get a better understanding of security issues, malware analysis, and specific cyber incidents.
Limitations:
It’s worth noting that individual efforts in OSTI aren’t as extensive as those of organizations. Most people don’t have access to advanced tools or the resources to monitor open sources all the time. For personal use, it’s usually about occasional checks and staying updated, rather than running a full-scale intelligence operation.
The main benefit for individuals who engage with OSTI ideas and resources is an increase in awareness. Understanding OSINT helps you be more mindful of what you share publicly and how it could be misused. Knowing about common threats like phishing and ransomware through easy-to-access OSTI channels equips you to spot and avoid them more effectively. If you find out that your credentials have been leaked, you’ll be prompted to take important steps like changing passwords and setting up multi-factor authentication. So, even a little bit of interaction with OSTI concepts and sources can really reinforce your personal security, making you a tougher target against attacks that rely on public information and oversights.
Ethical and Legal Considerations
Legal Considerations
Open Source Threat Intelligence (OSTI) is all about using a ton of public information to gather insights. But with that comes a lot of responsibility. It’s important to handle OSTI carefully and be aware of the legal and ethical rules to make sure we’re doing things the right way and not causing any problems.
OSTI practitioners must operate within legal boundaries (e.g., GDPR, CCPA), respect privacy, and avoid illegal activities. Ethical sourcing, data minimization, and transparency are essential for trust and compliance.
Ethical Considerations
Besides following the law and privacy rules, there are ethical issues to think about with the sources and how the information is used:
Checking Sources: Analysts need to think about the ethics of the data they’re using. For instance, is it okay to use data from a breach even if it’s out there for the public? What are the risks of looking at forums known for illegal stuff? We need clear rules here.
Avoiding Harm: Professionals need to be careful about the harm that might come from collecting or sharing information, especially if that info is wrong or misused. In cybersecurity, the main goal should be to protect the organization and its assets.
Being Open: While getting consent for using public data isn’t always easy, companies should be open about their practices, at least among their teams. It’s crucial to stick to consent rules when they apply, like with permissions on specific platforms.
Future Trends and Conclusion
Open Source Threat Intelligence (OSTI) has become a key part of today’s cybersecurity approach. By making use of a wealth of publicly available information, OSTI helps organizations and individuals shift from simply reacting to threats to taking steps to prevent them. Its main benefit is improving awareness about potential attacks, giving insights into how adversaries work, spotting weaknesses, and enhancing response efforts and threat detection.
This report covers a few important points. First, turning public information into useful intelligence is crucial. This involves sticking to a structured process that includes planning, collecting, analyzing, sharing, and getting feedback. Second, navigating the wide range of OSTI sources means regularly checking their reliability to deal with issues like data overload and varying quality. Third, effective OSTI isn’t just about gathering data; it requires skilled analysts, the right tools, and clear procedures. Lastly, it’s essential for all OSTI work to be based on solid ethical and legal standards to maintain trust and ensure it can continue operating properly.
Some Future Trends we see:
- AI/ML: Automation will help manage data volume and complexity.
- Data Growth: The expanding digital landscape offers both opportunities and challenges.
- Adversary Use: Defenders must continually adapt as attackers also leverage open sources.
- Ethics and Regulation: Expect growing scrutiny and the need for robust ethical frameworks.
I hope this is useful for some of you when trying to figure out how to best make use of what is out there, for your personal protection, but also within your organisations.
Next time, we will take a look at one of the tools I like in the OSTI space, MISP:
MISP
As usual:
For any inquiries and questions, you can reach me on Session: 059db238ab37c3d92615c5cc24b694da29c598cc13e27886053722404118e14271