ThreatStream

What is ThreatStream?

A comprehensive Python-based CVE intelligence platform that automates the complete lifecycle from vulnerability discovery to executive reporting

About This Project

ThreatStream is a production-ready Python application that fetches Common Vulnerabilities and Exposures (CVE) data from NIST's National Vulnerability Database, processes and enriches it, then generates professional PDF reports and Power BI-ready analytics. The backend requires Python 3.12+ and cannot run on static hosting like GitHub Pages — it must be run locally or on a server with Python runtime.

Automated CVE Ingestion

Fetches the latest vulnerability data from NVD API with incremental updates, pagination handling, rate limiting, and state management to prevent duplicate processing.

Executive Reporting

Generates professional PDF reports with charts, trend analysis, critical vulnerability hotlists, and strategic security recommendations ready for executive distribution.

Power BI Integration

Exports pre-structured CSV files optimized for Power BI dashboards, enabling instant visualization of CVE trends, vendor risks, and CVSS severity distributions.

Threat Intelligence

Enriches CVE data with CVSS scoring, attack vector analysis, vendor risk assessment, CWE (Common Weakness Enumeration) categorization, and exploit reference tracking.

Setup & Installation

Get ThreatStream running in minutes on your local machine or server

System Requirements

ThreatStream requires Python 3.12+ and cannot run on GitHub Pages or static hosting platforms. It needs a Python runtime environment to fetch data from NVD API, process parquet files, generate charts, and create PDF reports. The application can be run locally on Windows, macOS, or Linux, or deployed to a cloud server.

Installation Instructions

Clone the Repository

Download ThreatStream from GitHub

$ git clone https://github.com/DakodaStemen/threat-stream.git

$ cd threat-stream

Create Virtual Environment

Set up isolated Python environment

$ python -m venv .venv

$ .venv\Scripts\activate  # Windows

$ source .venv/bin/activate  # Mac/Linux

Install Dependencies

Install required Python packages

$ pip install -r requirements.txt

Configure API Key (Optional)

Add your NVD API key for better rate limits

$ cp .env.example .env

$ echo "NVD_API_KEY=your_key_here" >> .env

Run the Pipeline

Execute the complete CVE processing workflow

PS C:\threat-stream> python scripts/run_all.py
2025-10-12 15:20:15 - INFO - Starting CVE data pipeline...
✓ Fetched 247 new CVEs
✓ Normalized 247 records to parquet
✓ Generated 4 KPI export files
✓ Created PDF report: CVE_Daily_Report_20251012_1520.pdf
Pipeline completed in 59.8s

Generate Custom Reports

Create reports for specific time periods with interactive menu

PS C:\threat-stream> python scripts/interactive_report.py

╔═══════════════════════════════════════════════╗
║ CVE Report Generator - Interactive ║
╚═══════════════════════════════════════════════╝

Select report period:
1. Daily (last 24 hours)
2. Weekly (last 7 days)
3. Monthly (last 30 days)
4. Yearly (last 365 days)
5. Custom days

Enter choice (1-5): 3
Organization name (or press Enter to skip): Acme Corp

Generating monthly report for Acme Corp...
✓ Report saved: reports/monthly/CVE_Monthly_Report_20251012_1520.pdf

Complete Documentation

For detailed documentation, API reference, and advanced configuration options:

View on GitHub Read Full README

Example: Full Pipeline Execution

See exactly what happens when you run ThreatStream

Complete Output from run_all.py

PS C:\threat-stream> python scripts/run_all.py
2025-10-12 15:20:15 - INFO - Starting CVE data pipeline...
2025-10-12 15:20:15 - INFO - Setting up directories...
✓ Directories created successfully

2025-10-12 15:20:16 - INFO - Fetching CVE data...
Fetching from NVD API (last modified: 2025-10-11)...
✓ Fetched 247 new CVEs
Saved to: data/raw/2025-10-12/cve_data_222015.json

2025-10-12 15:20:45 - INFO - Normalizing CVE data...
Processing 247 CVE records...
✓ Created CVE_Data_October_12_2025.parquet (247 records)
✓ Created Affected_Products_October_12_2025.parquet (428 records)
✓ Created References_October_12_2025.parquet (1,234 references)

2025-10-12 15:21:02 - INFO - Generating KPI tables...
✓ Daily Overview: data/exports/Daily_Overview_October_12_2025.csv
✓ Vendor Weekly: data/exports/Vendor_Weekly_October_12_2025.csv
✓ Hotlist: data/exports/Hotlist_October_12_2025.csv
✓ CWE Trends: data/exports/Cwe_Trends_October_12_2025.csv

2025-10-12 15:21:15 - INFO - Generating professional PDF report...
Creating charts and visualizations...
Building executive summary...
✓ Report saved: reports/daily/CVE_Daily_Report_20251012_1520.pdf

✓ Pipeline completed successfully!
Total execution time: 59.8 seconds

The pipeline automatically fetches, processes, and reports on CVE data in under 60 seconds!

Data Processing Pipeline

Six automated stages transform raw NVD data into actionable intelligence

1

Fetch CVE Data

Incremental fetch from NVD API with state management

2

Normalize Data

Convert JSON to structured parquet datasets

3

Enrich & Classify

Add vendor mappings and CWE categories

4

Build KPIs

Generate analytics and trend metrics

5

Generate Reports

Create professional PDF documents

6

Export Dashboards

Power BI-ready CSV exports

Technology Stack

Python 3.12+

Pandas

NumPy

Parquet

Matplotlib

Seaborn

FPDF2

Requests

PyYAML

Power BI

Key Capabilities

Enterprise-grade features designed for security operations teams

Incremental Updates

Smart state management tracks processed CVEs, fetching only new vulnerabilities since last run. Prevents duplicate processing and optimizes API usage.

Vendor Normalization

Intelligent mapping of vendor aliases to canonical names (e.g., "Microsoft" vs "MS" vs "MSFT") ensures accurate risk assessment across product lines.

CVSS Scoring

Automatic severity classification using CVSS v3 base scores. Identifies critical vulnerabilities (≥9.0) requiring immediate patching priority.

Trend Analysis

Daily, weekly, monthly, and yearly reporting with time-series analysis. Tracks vulnerability disclosure patterns and emerging threat vectors.

CWE Categorization

Maps vulnerabilities to Common Weakness Enumeration categories, enabling root cause analysis and targeted security improvements in development practices.

Attack Vector Analysis

Classifies vulnerabilities by attack vector (Network, Local, Physical, Adjacent) to prioritize infrastructure hardening efforts and exposure reduction.

Professional Reports

Executive-ready PDF reports with comprehensive threat intelligence

Report Features

Executive Summary: High-level overview of threat landscape and key metrics
Trend Visualization: Professional charts showing CVE disclosure patterns
Critical Hotlist: Structured table of high-severity vulnerabilities requiring immediate action
Vendor Risk Assessment: Top affected vendors with CVE counts and severity distribution
CWE Distribution: Breakdown of vulnerability types to guide security focus areas
Strategic Recommendations: Actionable guidance based on data analysis
Custom Branding: Add your organization's logo and name to reports

View Sample Daily Report View Sample Monthly Report

Power BI Integration

ThreatStream exports optimized CSV files that can be directly imported into Power BI for interactive dashboards:

daily_overview_*.csv - Daily CVE counts and trends
vendor_weekly_*.csv - Vendor risk analysis by week
hotlist_*.csv - Critical vulnerabilities for immediate action
cwe_trends_*.csv - Vulnerability type distribution over time

Setup & Installation

Get ThreatStream running in minutes on your local machine or server

System Requirements

ThreatStream requires Python 3.12+ and cannot run on GitHub Pages or static hosting platforms. It needs a Python runtime environment to fetch data from NVD API, process parquet files, generate charts, and create PDF reports. The application can be run locally on Windows, macOS, or Linux, or deployed to a cloud server.

Installation Instructions

Clone the Repository

Download ThreatStream from GitHub

$ git clone https://github.com/DakodaStemen/threat-stream.git

$ cd threat-stream

Create Virtual Environment

Set up isolated Python environment

$ python -m venv .venv

$ .venv\Scripts\activate  # Windows

$ source .venv/bin/activate  # Mac/Linux

Install Dependencies

Install required Python packages

$ pip install -r requirements.txt

Configure API Key (Optional)

Add your NVD API key for better rate limits

$ cp .env.example .env

$ echo "NVD_API_KEY=your_key_here" >> .env

Run the Pipeline

Execute the complete CVE processing workflow

PS C:\threat-stream> python scripts/run_all.py
2025-10-12 15:20:15 - INFO - Starting CVE data pipeline...
✓ Fetched 247 new CVEs
✓ Normalized 247 records to parquet
✓ Generated 4 KPI export files
✓ Created PDF report: CVE_Daily_Report_20251012_1520.pdf
Pipeline completed in 59.8s

Generate Custom Reports

Create reports for specific time periods with interactive menu

PS C:\threat-stream> python scripts/interactive_report.py

╔═══════════════════════════════════════════════╗
║ CVE Report Generator - Interactive ║
╚═══════════════════════════════════════════════╝

Select report period:
1. Daily (last 24 hours)
2. Weekly (last 7 days)
3. Monthly (last 30 days)
4. Yearly (last 365 days)
5. Custom days

Enter choice (1-5): 3
Organization name (or press Enter to skip): Acme Corp

Generating monthly report for Acme Corp...
✓ Report saved: reports/monthly/CVE_Monthly_Report_20251012_1520.pdf

Complete Documentation

For detailed documentation, API reference, and advanced configuration options:

View on GitHub Read Full README

Platform Metrics

Key statistics and capabilities of the ThreatStream platform

6

Pipeline Stages

100%

Automated

NVD

Data Source

PDF

Reports

Parquet

Data Format

CSV

Power BI Export

<60s

Processing Time

Daily

Updates

Transform Vulnerability Data Into Intelligence

Clone ThreatStream, run the pipeline, and start generating professional CVE intelligence reports for your organization. Full source code and documentation available on GitHub.

View on GitHub Connect on LinkedIn Back to Portfolio