Putting AI agents into production on a 15TB live payments system across 70+ countries

  • Category: Enterprise
  • Industry: Consumer Finance / Payments
  • Service: Agentic AI / ML & LLMOps
  • Client: Global consumer finance / payments platform

A global pay-as-you-go consumer finance platform, serving millions of customers across more than 70 countries, partnered with Symphony to close the context gap between its engineers and a vast, complex system. Symphony built a production multi-agent platform with a four-layer architecture and a semantic layer over a 15TB, 1,500-table database. Specialized agents now handle debugging, analytics, and proactive monitoring in production, with intelligent payment rerouting running at over 99.99% accuracy and payment support calls all but eliminated.

Problem

Operating a payments system at this scale meant that debugging, analytics, and incident response all depended on deep, specialized knowledge of a vast and complex data architecture. A small engineering team maintained a live system processing hundreds of millions of payments at peak rates of 200 per second, integrating with more than a hundred banks across more than 70 countries. As senior engineers rotated out, tribal knowledge left with them, onboarding was slow, and the service desk filled with tickets, many of which were not bugs but misconfigurations or unclear expectations. The product was stuck in maintenance mode while competitors moved.

Solution

Symphony built a four-layer agentic platform on top of the live system, designed around a simple principle: an agent only performs as well as the context it is given. Rather than relying on naive retrieval that guesses which data it needs, Symphony transformed the client's code, databases, and infrastructure into a unified semantic layer the agents can query and reason about. Specialized agents handle distinct functions, debugging, analytics, and proactive monitoring, reasoning from documentation and established patterns rather than guesswork, with engineers connecting their own tools to the same foundation.

  • Four-layer architecture with a semantic layer over a 15TB, 1,500-table database
  • Specialized agents for debugging, analytics, and proactive monitoring running in production
  • Hybrid retrieval combining semantic search, BM25, and cross-encoder reranking
  • Custom MCP servers connecting the agents to the live system at production scale
  • Intelligent payment rerouting at over 99.99% accuracy
  • A proactive monitoring agent that catches issues before users notice them
  • Shared context layer that engineers connect their own development tools to

Technology: Symphony built a nightly pipeline that ingests code, database schema, and infrastructure, gives them meaning in plain language, and embeds that meaning into a semantic vector store, making the entire system searchable. A layer of custom-built MCP servers connects the agents to the indexed system: semantic search with hybrid retrieval, secure read-only production database querying with query-complexity scoring, production log navigation, and ticketing, email, and cloud integrations. Specialized agents run on top, reasoning from this context to diagnose issues, run analytics, and monitor the system proactively.

Under the hood

Technical architecture

  • Semantic pipeline that chunks code and database schema and embeds them into a vector store, refreshed nightly
  • Hybrid retrieval: cosine similarity for candidate selection, cross-encoder reranking for precision, BM25 for exact-term search, fused via Reciprocal Rank Fusion
  • Custom MCP servers for semantic search, live production DB querying, log navigation, ticketing, email, and cloud
  • Query analyzer scoring every database query for complexity to protect billion-row tables
  • Containerized agent runtime deployed across managed Kubernetes
  • Specialized agents for debugging, analytics, and proactive monitoring
  • Continuous learning loop capturing conversation ratings into a reusable knowledge store

Business Impact

  • Closed the context gap between engineers and a 15TB, 1,500-table production system
  • Freed a small engineering team from constant support load to move the product forward again
  • Made advanced analytics accessible beyond engineering, so non-technical teams can self-serve reports and trends
  • Shifted operations from reactive firefighting to proactive monitoring that catches issues before customers do
  • 99.99%+ - Payment-rerouting accuracy in production
  • Eliminated - Payment support calls all but removed
  • 15TB - Production database covered by the semantic layer
  • 70+ - Countries served by the live payments system

https://symphony.is/our-impact/case-studies/global-payments-ai-agents-platform