Ship AI into banks in weeks rather than 12 months by engineering for compliance velocity: a CVE remediation pipeline with evidence of exploit removal, verified backup-and-restore with data-correctness checks, and redaction guarantees covering LLM judge reasoning — plus organizational readiness like image pull approvals and GPU sourcing.
Instead of just bumping package versions and hoping scanner tools accept it, run agents inside a harness that write exploits against the old package version and tests to verify the exploit is actually gone from the application code, then package that evidence into a report for the customer's security team.
UI redaction hides sensitive data from human users, but LLM judge system prompts (e.g. an instruction to quote the agent's answer for human verification) can cause the judge's persisted reasoning to un-redact PII or MNPI. Audit judge traces for leaks, then persist scores always but persist reasoning only after a sensitivity check, or run a second judge on redacted input.
A service that audits an AI vendor's deployment against the pillars banks actually check — build pipeline, data resiliency, leakage — and identifies the organizational signals (image-registry approval process, GPU procurement, security reviews, custom LM gateway) that stretch time-to-production from weeks to months, then fixes them with engineering, workflow and organizational work streams.
Cisco is a technology company providing networking, security, observability, collaboration, and AI infrastructure and software solutions. Its offerings include networking hardware and software, security enforcement, cloud control, agentic operations, and observability for agents, models, and supporting infrastructure. The videos identify Cisco as the employer and provider of AI observability software deployed inside major banks, where deployments undergo security and compliance reviews covering vulnerability remediation, backup and restore verification, PII leakage, and data sovereignty.
ClickHouse is an open-source, column-oriented database management system for online analytical processing (OLAP) and real-time analytics. It stores data by column and uses optimized compression and vectorized query execution to process analytical SQL queries while using available CPU resources efficiently. It supports large-scale workloads such as real-time analytics, observability, data warehousing, and machine learning or generative AI applications, including vector search and aggregations. ClickHouse can be self-managed on a machine or deployed through ClickHouse Cloud on AWS, Google Cloud, and Microsoft Azure. ClickHouse Local runs queries on local files such as CSV, TSV, and Parquet without a server. The project provides installers for macOS, Linux, FreeBSD, and Windows, as well as Docker and playground options; the self-managed database is open source.
An open-source backup and restore tool for ClickHouse, maintained by Altinity. It provides job logs for monitoring backup and restore operations.
Daytona is cloud sandbox infrastructure for running AI-generated code. Its website describes secure, stateful environments with fast environment creation and enterprise-grade security. It was used as the sandbox provider in early WolfBench evaluations.
Harbor is a framework from the creators of Terminal-Bench for evaluating and optimizing AI agents and language models in container environments. It can run arbitrary agents, including Claude Code, OpenHands, and Codex CLI, against shared or custom benchmarks and environments such as Terminal-Bench, SWE-Bench, and Aider Polyglot. Harbor launches benchmark runs locally with Docker or in parallel through cloud and sandbox providers, and can generate rollouts for reinforcement-learning optimization. It is distributed as a Python package installable with uv or pip and provides command-line tools for running datasets, selecting agents and models, and listing supported benchmarks.
MITRE is identified in the supplied video as a CVE research database used in software security and compliance work, including tracking vulnerabilities and supporting CVE remediation.
NVD is a CVE research database available through the National Institute of Standards and Technology's NVD website.
Postgres is an open-source relational database created by Mike Stonebraker and subsequently maintained and enhanced by a volunteer community and companies. It is presented as a widely used database whose open ownership contributed to its adoption and continued development.
Prisma ORM is an object-relational mapper and query builder for Node.js and TypeScript applications, developed by Prisma. It supports PostgreSQL, MySQL, MariaDB, SQL Server, SQLite, MongoDB, and CockroachDB. The repository's Prisma Next development line is a TypeScript rewrite with a composable core, database support built through a public extension interface, and agent skills that can be registered in a project so AI coding agents can work from a Prisma contract and workflow-specific references. Prisma Next is described as early access and not recommended for production workloads yet.
Redis is an open-source in-memory data structure server, cache, key-value and document database, and query engine for real-time applications. It supports strings, lists, sets, hashes, sorted sets, JSON, time-series data, transactions, scripting, queues, streams, publish/subscribe messaging, full-text and geospatial queries, aggregations, and vector search. The project is used for caching, distributed session storage, event brokering, real-time analytics, and as a supporting service for the support desk shown in the video.
Sysdig is a cloud-security platform that provides real-time visibility from runtime through development. It is described in the video context as a customer-side CVE-scanning tool that produces security reports; its website also presents capabilities for assessing cloud risk, prioritizing critical risks, detecting threats, and responding to them.
Wiz is an AI cybersecurity platform for cloud and AI applications. It connects code, cloud, identities, network, and runtime in a unified security graph, using that context to scan attack surfaces, model lateral movement, privilege escalation, and data-access paths, detect exploitation, and generate code or infrastructure fixes. The videos describe its customer-side CVE scanning as producing security reports and discuss using agents to remediate vulnerabilities.
Searchable transcript of Shipping AI Into 5 Major Banks Without a 12-Month Compliance Wait — Al Chen, Cisco — AI Engineer (17:42). Search for a phrase, then click its timestamp to jump straight to that moment in the video.
Captions sourced from the original video on YouTube, published by AI Engineer. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.
00:01 [music] >> Uh when I was in the 10th grade, I had a math teacher, Mr. Huff, who was uh notoriously brutal on our math tests. He would uh make us write our answers down to the uh fourth decimal place in case you happen to get to the answer by luck or by chance. And most of the credit for the question was actually from the proof that you wrote to show how you got to the answer.
00:37 And when I first started playing AI observability software, uh I realized that most of these compliance and risk teams that I work with every day, they're kind of like Mr. Huff, but with a a multi-million dollar compliance budget. And I would let you know that playing agents AI within banks takes a long long time. And the reason is because that keyword production.
01:04 I'm not talking about prototypes in dev or staging or even like internal rag agents to increase, you know, internal employee productivity. I'm talking about the applications that these banks millions of customers use every single day. The same ones you use when you log into your bank's website or mobile app. And those are the use cases that matter. In my team, we're on the front lines of deploying not only our software, but also engineering processes into the largest banks in the world.
01:32 And I want to share some of the issues that our customers bring to us and the strategies we use to help reduce the time to production from a year down to a couple weeks. So, let's take first take a look to see how standard SaaS AI gets deployed at a bank. Uh spoiler alert, it doesn't work. There's outbound egress to the internet and sensitive information can potentially be leaked out.
02:04 Within when you deploy on prem AI at an air-gapped bank, the flow is a little more complicated. I'm going to walk through three pillars today. Number one is the actual build pipeline. How our banks pull images from our software into their cluster. If a region fails mid-request, what is the data resiliency plan? And then finally, one of the most important parts is how do you ensure no sensitive information leaks out to internal bank employees and outside of the the bank's VPC.
02:42 Compliance velocity uh and CVEs go hand in hand. And no platform worth their salt does not have a plan for how they address this first compliance hurdle, which is common vulnerabilities and exposures. I'm going to walk you through why this process is complicated. So, number one, you can take one CVE and you can research on the NVD, CISA, or MITRE websites.
03:03 But, in reality, our platform, most platforms, you have multiple images and each image has multiple CVEs. And I'm not trying to remediate CVEs for one bank, but rather for multiple banks. So, we have to create a system and process around this. And the complication there is that our banks are on different versions of our images, each with their own unique CVEs.
03:29 On top of that, banks utilize different scanning tools, which output a report on what CVEs are present. Furthermore, they have specific policies around what are the actual CVEs in the images, what the severity level is, and then what the SLA is for time to remediation. Our images rely on many open-source projects and those projects contain many transient dependencies which have CVEs that our engineering team cannot remediate.
03:59 So, our team is we're constantly looking into the release notes for a lot of these open-source projects cuz we have to provide a detailed explanation back to the security teams at these banks on why we cannot remediate and why we're dependent on these open-source project maintainers. So, once you've identified the CVEs that your team your engineering team can actually remediate there's multiple ways of remediation.
04:23 You can bump the package version up to a higher version or rebuild your image against the base Docker image. But, then we also have this complication of certain banks saying we want to be on version 3.8 of the package. But, another bank might say hold wait, hold on. That version is too too high. There's too many CVEs. I want to be on an older version of that package.
04:46 So, given all of this your software still has to work and function. And so, one solution that you can use is creating a scale. So, we have a scale that does a lot of the heavy heavy lifting and reporting and the internal research as well. And the output is a nice kind of table format that has tells our engineering team what CVEs we have to fix and also provides explanations back to our security teams from the bank on what can we cannot fix due to these transient dependencies.
05:23 Most of these scanning tools Sysdig, Wiz, Prisma they just check to see if the package has been bumped to a specific version. But, it says absolutely nothing about whether or not that exploit is actually still in your application code. And so, how do you actually test for that? So, one solution is creating a harbor task and this is kind of like creating a harness or a world for your for an agent to try things out.
05:53 And so within this harbor task you have instructions for the agent, you have an environment which contains the actual docker images for your application, and then tests that you can run against the agent to see if they passed. The instruction here is a CVE for uh this service, this package. And the instruction I tell the agent is 2.6.2 of this package supposedly remediates the CVE.
06:17 I want you to go and write an exploit based on the older version to see if this new version still has that exploit. Then write a test to see if that exploit actually passed or not. And then with harbor you can run a bunch of docker containers or um harbor uh modal or Daytona sandboxes, run these all in parallel, and you have a bunch of trials to see how the agent performed over various trials.
06:44 And the What I'm trying to say here is that it's not enough just to say we bumped the package version up to a specific number. We have to provide the evidence to show that the exploit is no longer actually in the application code. Uh the second pillar, disaster recovery, is not like optional infra hygiene for a bank. The complication here is that the main database platform most AI observability tools use, including some of the companies that are sponsoring this conference, there's no managed service for failover across
07:21 regions and data replication. Anyone happen to know what database platform I'm talking about? It isn't this So, with Redis, Postgres, there's managed failover across regions, but this one database does not have that story. ClickHouse is an amazing database platform, but even in their data resiliency docs they call out that there's no managed service managed failover across regions.
07:46 So this means the banks are kind of up to themselves to to figure out that disaster recovery situation. So a simple pattern that you can employ is you have ClickHouse in a primary region, a job that backs up to a bucket, and then another job that restores that bucket to a DR region. Within that region, you might have multiple pods that have replicas of that data.
08:10 This whole procedure relies on ClickHouse backup, which is an open-source project maintained by Altinity, who they themselves are a competitor to ClickHouse in terms of managing this data failover across regions. It's uh so with this procedure, they ClickHouse backup shares logs on whether or not the backup and restore happens, so you can get this nice table.
08:37 But this status says nothing about your data at this point in time. A bad disk, a bad node, um bad migration, replication lag, these are all issues that can cause your data to become stale, that is different from when this job actually ran. So you can go the extra step in checking the row counts between tables, uh running a check to see if the replicas within a region have converged to each other.
09:05 And the takeaway here is that it's about data correctness, and not necessarily just checking a box and seeing if the backup and restore job ran. Similar to the CVs, you can't just bump the version and assume the exploit's gone. You have to do these extra checks. And uh one call out for any of the uh architects who are creating this kind of backup and restore within your cluster is that you're also making a decision around data residency, whether you like it or not.
09:32 When you're replicating to a bucket, the bucket doesn't care if it's in Oregon or Virginia. Data residency, however, cares about where those individual records are stored. So, one solution is you can replicate just the IDs and the metadata around your traces and your data into different regions. But, then the actual core trace data, that's stored within one region and doesn't leave that region because of certain jurisdiction laws.
10:07 Uh the final pillar is most banks think that they are not leaking any sensitive information to their internal employees, their users outside of their VPC, but there may be side doors that are open. And this comes in the form of PII MNPI being redacted. And there's many different strategies for this. You can redact information at the SDK level, at the gateway level, using various middleware.
10:34 And what this looks like in practice is on the UI, you might have some customer information, sensitive information is redacted. But, what about the judges that you're using? I challenge all of you to look at the system prompts for your LLM's judges and check the traces to see if those traces with the reasoning from those judges contain any PII or MNPI.
11:01 Here's a system prompt for a simple judge. It outputs a score and explanation on how well the agent performed for a given task. But, there's one line here in the prompt, which is seemingly innocuous because you want the human reviewer to be able to verify if the agent did a good job and was accurate. This just potentially may have allowed the judge to leak and unredact PII or MNPI.
11:29 So, your UI still looks good, but then on the back-end your LLM as LLM as judges reasoning may un-redact customer information and sensitive information. So, one strategy for this is you can still have your judge read the raw trace data, but the reasoning for that judge is not persisted. The score, however, you want that score it's for good good for evals, that always persists through the UI, but then on this other branch the reasoning, as we saw, can still leak PII.
12:01 But in your application code, you can have a check that tells the agent this record does not contain sensitive information and therefore can persist to the UI. So, now you have a score plus the reasoning for the for this record. And then if that check comes back negative, it does have sensitive information, then you just suppress that reasoning from the UI.
12:22 Most of the banks we work with, they want to see the LLM as judges reasoning on this sensitive information. And the extra upgrade you can add to this flow is you have a second LLM as a judge where the input has is redacted of sensitive information and therefore you can have confidence that the output from the judge is clean and then you can have that show up in the UI.
12:50 The uh takeaway here is that having zero leakage and data sovereignty is not free. In this case, it's the cost of running another LLM call on the same request. So, one way we're able to build trust with these security and compliance teams is that we're not just saying trust us, here's the answer. We have to show the evidence and the work on how we arrived at the at the answer.
13:17 And many vendors will tell you, you know, we're our deployment time to production takes a few weeks, but what they're not saying is they didn't check that time against your infrastructure, your security posture, and the nuances about your specific cluster. And so certain signals that we look for in terms of shortening that time to production is really just having a lot of processes set up in terms of, you know, how you pull images down from an from a registry, having GPUs sourced and ready to go for running inference.
13:53 That timeline gets pushed out a couple months when you have formal security reviews, or you're building a custom LM gateway or RBA configuration. And then that timeline gets pushed even further out when you don't even have an approval process for pulling images down to your your registry, and not having a not having a story around sourcing and procuring GPUs.
14:15 Now, I found that most of these signals that push the timeline out are less technical and more about internal human and organizational challenges and alignments at the banks. Uh but still important factors to figure out. So, the theme for today is showing your work. That is the real lever that you have on compliance velocity when working with a bank.
14:45 And when our banks, when one of our customers come to us and tell us, "Hey, our customers are having problems." Those problems now become my team's problems. And we break those problems down into uh engineering, workflow, and human and organizational work streams. The goal is to automate as much as possible, but those problems are the ones that keep me up at night.
15:08 And I want to ensure that our competitors don't outwork us and don't outsmart us on any of those vectors. And I would say that you don't want to just be another vendor, you want to actually be solving your customers' problems. And I would say the biggest lesson I've learned from working with these banks and security teams is that there's so much at stake.
15:38 Checking accounts, mortgages, credit cards. Getting your answers down to the fourth decimal place matters in these situations. And there's a reason why a lot of the software within these banks is old and outdated. But the the line between legacy software and old banking processes and innovation is moving a little bit every day and it's because of all of you.
16:07 So I think all of you're, you know, exploring different agent frameworks and building with, you know, new harnesses, playing with Jev. And I would say for the few of you who care about the the slow, the boring, the bureaucratic, the methodical, the messy human and organizational challenges within a bank, there's nothing like deploying AI within the financial services sector because there's so much so much at stake.
16:36 And yeah, if you all want to score your own deployments against um these pillars, just just reach out. I only discussed three pillars today, but um you know, there's a whole production checklist that we use to uh that underpins these pillars. Going back to my 10th grade math test, one of the best parts about those tests were afterwards we me and my friends we, you know, go to the lunch table and we'd try to scribble down all the problems we remembered from the test.
17:08 And then we use that time to talk about, "Hey, here's how we approached solving this problem and that problem." And it allowed me to kind of think about different approaches and different ways to solving these problems. And conferences like this are kind of like that uh lunch table back from when I was in 10th grade because I want to learn from all of you on how you're solving your your problems for for your customers. Thank you very much. >> [applause]