Security engineering / AI systems
AegisGraph
A security investigation engineering platform for synthetic fintech telemetry: scenario replay, evidence-grounded AI, hypothesis tracking, and detection regression.
Inspect the system
V2 is live as a read-only, deterministic synthetic fintech investigation environment. Inspect the workflow and its boundaries; this is not a production SOC deployment.
Guided inspection
- System
- Next.js / TypeScript, FastAPI, PostgreSQL, SQLAlchemy, and Alembic.
- Investigation
- Eight synthetic scenarios, causal replay, deterministic detections and correlation receipts, evidence timeline, and hypothesis ledger.
- Validation
- Full-corpus rule comparison, regression gates, deterministic analyst benchmark, and SHA-256 export verification.
- AI boundary
- Bounded evidence, typed claims, exact-context citations, support predicates, and explicit insufficient evidence.
- Public demo
- Read-only replay and fixed comparisons; two deterministic analyst questions. No live OpenAI calls, spend, or saved reviews.
- Local/interview mode
- Saved rule and hypothesis reviews, case workflow, and optional OpenAI analysis. Persistent decisions remain human-controlled.
- Hosting
- Railway frontend, API and PostgreSQL; TLS, restricted database role and tested backup recovery.
Overview
I built AegisGraph to make the path from a security observation to an analyst decision inspectable. V1 established one deep Atlas investigation; V2 tests the reasoning against a wider synthetic corpus and connects it to replay, hypothesis review, detection tuning, and evidence export. It remains a synthetic/demo environment, not a production SOC deployment.
Problem
A suspicious login, role change, or resource request can mean little in isolation. A plausible model answer can misuse a real citation, and a threshold change can suppress a useful signal along with a benign one. I wanted to expose both the evidence behind an interpretation and the consequences of changing the rules.
Investigation loop
- Scenario
- Telemetry
- Normalization
- Detection
- Correlation
- Incident
- Evidence
- Hypothesis
- Bounded AI analysis
- Analyst decision
- Detection tuning
- Replay
- Regression validation
This is an investigation and review loop, not an autonomous pipeline. Schema, citation, and claim-support checks sit between AI proposals and rendered findings. Human review remains separate from automated evidence assessment; saving decisions and approving rule revisions belong to local mode.
Scenario corpus
Eight versioned scenarios contain 4,098 synthetic events. The original Atlas case retains 4,026 events, ten alerts, and 26 evidence items. Additional scenarios include authentication pressure, service-principal access, approved administration, bulk automation, an isolated anomaly, and conflicting context. They provide known conditions for testing both useful signals and uncertainty.
Atlas follows unfamiliar authentication, MFA acceptance, privilege assignment, API enumeration, sensitive-resource access, abnormal volume, related anomalous activity, and privilege reversion. The sequence does not establish who operated the account, a malware family, exfiltration, or financial loss. Synthetic ground truth and evaluation-only expectations stay separate from analyst-visible observations; an answer key never becomes model evidence.
Selected interfaces
Technical highlights
- The model is not the system of record. Typed claims require validated evidence; persistent review decisions remain human-controlled.
Architecture
One Next.js/TypeScript frontend connects to FastAPI, with Pydantic validation, SQLAlchemy, Alembic, and PostgreSQL. The hosted path is Railway Next.js → Railway FastAPI → private Railway PostgreSQL over TLS. The API has SELECT-only database authority; pgBackRest supplies off-volume backups and WAL archiving, with production-backup recovery tested. SQLite supports lightweight local runs and tests. Only the canonical Atlas case is persisted. Other scenario projections remain ephemeral, including those that form an incident during replay.
Four adapters normalize identity, API gateway, endpoint, and Atlas application telemetry into a canonical event model with timestamps, actors, targets, outcomes, network/device/session context, bounded attributes, and synthetic source references. The workspace connects source records, a timeline, and relational entity links. Local annotations, findings, workflow, and report approvals remain separate from source observations.
Replay, detection & correlation
The API computes bounded causal frames from arrived observations; the browser cursor controls which frame is visible. Atlas initializes 4,000 background observations before playing the 26 investigation observations. Start, pause, step, and replay speed expose when signals emerge. Source-time gaps are compressed and disclosed. This is finite telemetry replay, not production streaming ingestion.
Ten rules emit source-cited alerts. Correlation separately requires three distinct rules across two families for one principal within 30 minutes of the first alert. A scenario below that threshold stays an observation scope. Replay creates no stored case, workflow record, or per-visitor server session; opening the canonical Atlas case returns to its preserved investigation.
Each reached principal and alert window has a decision receipt: matched rules and families, configured minimums, time bounds, and the incident or unmet conditions. The same function produces the receipt and decides whether an incident forms. Stable rule and signal ordering makes equal-time input permutations produce the same decision and incident identity.
Atlas finishes with nine distinct rules across five families over 22 minutes and one incident. Approved administration produces two rules from one family over nine minutes, below both diversity minimums. Its approval evidence is separate counterevidence, not a correlation veto; no incident does not prove there was no compromise.
Hypothesis ledger
The ledger separates observation, supported claim, and working hypothesis. Four fixed hypothesis types expose supporting citations, narrow counterevidence, typed missing evidence, and evidence-derived status. Conditional guidance identifies what would help confirm or reject an interpretation without inventing a future observation.
Human acceptance is a separate review: accepting a partially supported hypothesis does not make its evidence conclusive. Changed context makes a prior local review stale. The ledger can inspect richer approval/job counterevidence than the narrower analyst projection; it does not give the model authority to accept a hypothesis or mutate a case.
Evidence-grounded AI
A provider interface supports credential-free deterministic testing and optional OpenAI analysis in local/interview mode. Both use the same validation boundary. The public demo forces deterministic analysis for two curated questions without persisted answers or OpenAI spend. The application selects bounded, allowlisted evidence; evaluator expectations and untrusted free text are excluded.
The provider selects typed claim candidates and exact evidence references, rather than writing unrestricted factual prose. Server-side checks enforce the output schema, evidence existence, incident membership, inclusion in the actual bounded context, and support for the proposed claim. Fixed templates render accepted findings and their cited summary. One invalid claim rejects the entire answer, including any otherwise valid subset.
In historical V1 local live OpenAI validation, “What most likely happened?” returned ten supported findings with 13 citation references from 26 supplied evidence rows. “What malware family was used?” produced insufficient evidence and no invented malware attribution. These are recorded provider observations, not new V2 live-model trials. The provider has no database connection, retrieval tools, or incident-mutation authority.
Detection regression
Eight of the ten rule definitions expose bounded typed parameters. The local workbench saves append-only proposals, full-corpus comparisons, and explicit reviews. Approval independently recomputes the comparison and rejects a stale baseline. Approved revisions affect future replay projections without rewriting historical Atlas alerts or evidence.
For APP-002, raising the threshold from 1,000 to 1,500 records removes the benign reconciliation signal while retaining the required Atlas and service-access signals: PASS. A 1,100-record comparison leaves fixture outcomes unchanged: WARN. At 2,200, a required service signal and its correlated incident disappear: BLOCK. A blocking change cannot be approved; a warning requires deliberate review. Public visitors inspect these three fixed comparisons without saving or approving changes.
These are synthetic fixture measurements, not production precision or recall. TP/FN count two required scenarios; FP/TN count two explicitly benign scenarios. Four other scenarios still participate in full-rule and correlation gates. The workbench exposes those populations and before/after outcomes instead of reducing them to one flattering accuracy score.
Evidence export
A hash-verifiable investigation export captures a bounded, consistent committed snapshot as exact UTF-8 logical files in a JSON container with a SHA-256 manifest. Browser and offline verification check membership, lengths, and hashes without extracting paths. Public exports omit human workflow material.
VALID means agreement with the supplied manifest; changing a file without updating it is detected. The manifest is unsigned and replaceable, so replacing both content and hashes defeats any authenticity inference. Verification does not establish source telemetry truth, cryptographic source provenance, privileged database integrity, or legal/forensic chain-of-custody compliance.
Engineering decisions
- Retain one relational store for evidence joins and transactional reviews. Neither Elasticsearch nor a graph database is justified by the measured fixture workload.
- Use bounded JSON replay frames and a browser cursor; a finite demonstration does not require Kafka, WebSockets, or a distributed replay service.
- Separate detection, incident correlation, hypothesis support, and human approval so each decision can be inspected and tested independently.
- Constrain rule proposals and model claims to explicit types. Arbitrary code editing and unrestricted factual prose would need different validation boundaries.
Security / privacy considerations
Telemetry is untrusted data. In the historical tested telemetry-injection fixture, free text was excluded before the live model call; accepted output remained evidence-grounded and case state stayed unchanged. This validates that fixture's projection and output boundaries, not universal prompt-injection resistance. V2's deterministic benchmark adds named containment and counterevidence obligations without claiming general model safety.
Boundary checks reject foreign-case input, nonexistent citations, cross-incident citations, and current-case citations omitted from bounded context. Public mode denies persistent writes. Local actor labels are not authenticated identities; case scoping is not tenant authorization. Database triggers reject ordinary event and audit edits, but a privileged owner can remove those protections. Replay, comparison, and export work is bounded; caches and abuse controls remain process-local.
Validation
The September 27, 2026 refinement passed 695 backend tests, 129 frontend tests, and 27 browser tests: 20 local plus seven public-mode HTTPS checks. The original 28 deterministic evaluations and 145 V2 benchmark obligations are separate suites. The benchmark's eight metric populations overlap; these are application-boundary checks, not live-model accuracy samples. This is validation coverage, not proof of production security.
The September 27 CI run passed backend, frontend, and browser jobs for source revision f616185, the documentation follow-up to application revision 72647c7. It includes PostgreSQL clean/populated migration checks, dependency audits, and a production build. Both hosted services were verified at f616185; fresh public reads confirmed the correlation receipts and fixed APP-002 BLOCK comparison. That point-in-time check is separate from the earlier complete V2 hosted smoke.
Historical V1 evidence remains dated: the September 17 release review at c95afa6 records 146 backend tests, 25 frontend tests, 13 browser tests, and 28 deterministic evaluation cases. Its live OpenAI record retains three earlier HTTP 429 failures as well as successful scenarios. Neither V2's deterministic benchmark nor its public hosted checks extend that small live-provider sample.
Lessons / constraints
Synthetic ground truth is useful for repeatability, not evidence of real-world detection effectiveness. Bounded context and finite claim predicates can omit useful observations or interpretations. Supported findings can still be incomplete, and matching export hashes do not establish source truth. Human review and honest insufficiency remain necessary.
Production work includes authenticated identities, case and tenant authorization, multi-tenancy, authenticated streaming-scale ingestion, multi-instance rule rollout, broader repeated live-model evaluation, and stronger independently retained evidence and audit protections. The hosted V2 demo and local review workflow are implemented; production ingestion, access control, retention, and operating-scale measurements are not.