How We Used AI to Increase the Security Level of Our Code

Summary: We replaced a once-a-year external security audit with an AI-driven one we run ourselves, across 100% of our services, as frequently as we want. The core is a Claude Code skill we built in-house — within about a day it was already surfacing issues that our previous external audits had missed. We estimate it saves around $150k a year. Here’s how it works, what it found, and the areas where it does not replace a human.

Joko is a shopping companion used by more than 6 million people in France and the US. Our users connect their bank account to the app, and we operate a payment system on top of it. That makes security a first-order concern: we handle sensitive personal and financial data, so a gap in our code is not an abstract risk but a direct one to our users.

Why bring this in-house

The default for a company our size is to pay an external firm for a security audit. Since it’s often fairly costly and requires onboarding an external security analyst, this is something that is generally only done once a year. It works, but it has three structural problems.

It’s expensive. It’s a point-in-time snapshot: by the time the report lands, the code has already moved. And it tends to be strong on generic issues (known CVEs, missing headers) but shallow on the things specific to your product, because the auditors don’t know your domain well.

We wanted the opposite: continuous coverage, across every service, that actually understands our business logic. That only became realistic with AI in the loop.

What a security audit actually is

It helps to split an audit into two halves.

The first half is mechanical scanning: known-CVE lookups, leaked-secret detection, OWASP static patterns. Tools have done this well for years. It isn’t where the money or the expertise goes.

The second half is contextual reasoning: can we forge money in the user’s wallet by going through this endpoint, does this admin handler actually check the requester’s role, is this money-movement webhook safe to replay, does this Lambda log sensitive user payment info to CloudWatch. Answering those means understanding the system, not pattern-matching. That is the part that used to require an expensive human, and it’s exactly the part a capable model now does well.

The whole design follows from that split.

The skill: five passes

We wrote a security-audit skill in our internal Claude Code plugin. Point it at one repo or all of them, and it runs five passes in order: three deterministic, two reasoning-based.

Pass 1 — Dependency CVEs. Most package managers ship their own tool for auditing dependency CVEs. For Node and npm, that’s npm audit --json, for example.

npm audit --json

Pass 2 — Secrets and credentials. This spans a spectrum, from basic grep commands to more advanced, dedicated tooling.

# from a basic grep...
git grep -nE "(api_key|secret|password|token|private_key|connection_string)\s*=\s*['\"][^'\"]{8,}"
# ...to more advanced tools like trufflehog

Pass 3 — OWASP static patterns. Static analysis catches a broad class of known-insecure patterns without running the code. We use semgrep with the OWASP Top Ten ruleset.

semgrep --config=p/owasp-top-ten --json .

Those three are just orchestration around existing tools. The value starts at pass four.

Pass 4 — Infrastructure and config (reasoning). The model reads your infrastructure and deployment configuration directly — IaC, CI/CD, runtime manifests, and environment configuration (e.g. Terraform/CloudFormation, Kubernetes/Helm, Docker, GitHub Actions, and app manifests like package.json, pom.xml, requirements.txt) — looking for over-permissive rights (cloud IAM or K8s RBAC), wildcard actions, missing encryption, open CORS, exposed config, and risky install hooks.

Pass 5 — Business logic (reasoning). The model reads the source for product-specific risks: forging money in the user’s wallet through an unsigned or unauthenticated payload, admin privilege escalation from missing role checks, sensitive user payment info logged to CloudWatch, and sensitive routes accepting unauthenticated calls.

The output is a severity-ranked report (Critical / High / Medium / Low), each finding with a file:line reference and a one-line “why this is dangerous”. It then deduplicates against existing tickets and can propose Linear tickets from the findings — with the option to filter to Critical and High entries only, so you stay focused on the most important points and avoid losing focus. The report turns into triaged work rather than a PDF nobody actions.

Security Audit — payment-service — 2026-06-20

CRITICAL
• (none)

HIGH
• Wildcard IAM action on deploy role (serverless.yml:212) — role can act on any resource

MEDIUM
• PII (email) written to CloudWatch (src/handlers/notify.ts:88)

Tools run: npm audit ✅ | semgrep ✅ | trufflehog ✅

What it found

Naming the classes of issue it surfaced, all since triaged and closed:

• Over-broad rights: wildcard actions on roles, scoped down to least privilege.
• CI credentials: long-lived cloud keys in CI, migrated to short-lived OIDC.
• Security headers: HSTS max-age too short (raised to a year) and missing anti-framing / clickjacking headers (added).
• Observability leakage: Sentry breadcrumbs capturing full request URLs — which can embed tokens or personal data in query parameters — with no scrubbing configured.
• Logging hygiene: PII (personally identifiable information — emails, payment details) reaching logs from a handful of handlers.

None of these are exotic. That’s the point. They are the quiet, boring gaps that accumulate between annual audits, and they’re exactly what a continuous in-house process catches early.

From a one-shot project to a recurring habit

The work split into two phases, the same shape as a cost-cleanup.

One-shot: a full sweep across 100% of backend services and the cross-cutting concerns (secrets, IAM, CI auth, headers, logging). Confirmed findings became tickets; the P1s were pulled into a finite “close the urgent gaps” project with a hard deadline.

Recurring: the run is a skill, so repeating it is negligible for a business of our size. Findings flow into a continuous triage stream reviewed weekly, and the full sweep is now scheduled quarterly, one run per repo. Security coverage stops being an annual event and becomes routine.

The economics

A multi-week external engagement, once a year, versus a skill built in about a day and a sweep that takes a few engineer-days and runs every quarter. Adding up the audits we no longer outsource and the gaps caught before they become incidents, we estimate the in-house process saves on the order of $150k a year, while increasing coverage from once-a-year to continuous.

What we want to improve

The real limitation today is context. All our new business logic now lives in a monorepo, but we still have some legacy business logic spread across old separate repos — so for that legacy code the audit runs one repo at a time and sees each service in isolation. That’s fine for issues contained inside a service, but the most dangerous gaps live at the seams between them: a trust assumption where one service calls another, an auth check that exists upstream but is missing where it actually matters.

We know those gaps are real because we found one. The most serious issue the audit surfaced spanned several services, invisible to any single-repo run; it only came together by manually stitching separate audits into one picture. That was the strongest argument for changing how we run it.

That’s why we want to migrate all the legacy logic into the monorepo, which we already use for everything else today. Consolidating there changes what the audit can see: the model can load the whole system at once and follow a request across service boundaries instead of stopping at the edge of a repo. The skill lives in the monorepo too, versioned alongside the code it audits. More context in one place means the reasoning passes get sharper, which is where the value already is.

Takeaway

AI collapses the expensive middle of a security audit — the contextual reasoning about your own system — and turns a yearly external snapshot into a continuous in-house process. The mechanical scanning was never the hard part. The judgment was, and that’s now cheap enough to run every quarter instead of every year.