Why NYSE Chose Snowflake to Handle Trillion-Message Days

Wall Street has always been about information. Two centuries ago, traders gathered under a buttonwood tree on lower Manhattan because proximity meant faster access to shipping news and commodity prices. Today, the New York Stock Exchange processes 1.2 trillion incoming order messages on peak trading days, processing more data in a single trading session than the entire internet generated in a day during the early 2000s.

Each of those trillion messages represents someone making a financial decision that ripples through the global economy: pension funds managing retirement savings for millions of teachers and firefighters, sovereign wealth funds deploying national reserves, insurance companies investing policyholder premiums and retail investors using smartphone apps to buy fractional shares of Apple or Tesla.

Lynn Martin, President of NYSE Group, describes this evolution as both an operational challenge and a fundamental shift in how markets function. “We have two incredibly important jobs: one is around market integrity focused on transparency, and the other is around efficient risk management – so ensuring that those messages get processed incredibly efficiently so that people can manage their portfolios,” Lynn says.

The numbers behind this transformation reflect several trends that have reshaped Wall Street over the past decade. Retail trading volumes surged during the pandemic as commission-free brokers attracted millions of new investors, whilst algorithmic trading strategies have become more sophisticated and generate far more order messages even for relatively small position changes. Exchange-traded funds have grown to US$7tn in assets, requiring continuous creation and redemption transactions that add layers of operational complexity to what was already an intricate system.

What makes these volumes particularly challenging is their irregular distribution throughout the trading day. High-frequency trading firms can generate 40,000 orders per second during volatile periods, with algorithms cancelling and replacing orders faster than traditional systems can process them. Market makers must simultaneously quote prices across thousands of securities whilst managing inventory risk that can shift by hundreds of millions during periods of market stress, all while regulators expect the same level of oversight that applied when humans handled far simpler transaction flows.

Regulatory complexity drives NYSE towards AI-powered compliance systems

The regulatory environment has become another significant driver of technological change at NYSE, though the irony is hard to miss. Wall Street, which built its reputation on personal relationships and handshake deals, now operates under some of the most prescriptive rules in American business. Rule sets have increased by 40% over the past decade, but the rules themselves have also become more detailed rather than principles-based, which makes compliance increasingly difficult to manage through traditional approaches that once relied on experienced traders knowing what looked suspicious.

The Securities and Exchange Commission requires detailed audit trails for every transaction, with data retention periods extending seven years. The Consolidated Audit Trail, which was mandated after the 2010 Flash Crash exposed gaps in market oversight, requires exchanges to capture and report granular transaction data across all US markets. Meanwhile, the exchange must monitor every order for potential manipulation patterns including spoofing, where traders submit large orders they intend to cancel to move prices artificially.

This surveillance work has traditionally relied on systems that flag suspicious patterns for human review, but the exponential growth in message volumes means compliance teams cannot manually investigate every alert. That bottleneck has pushed NYSE towards AI solutions that can filter genuine anomalies from routine trading activity, allowing human analysts to focus on cases that actually require investigation.

NYSE has worked with Snowflake for six years, initially for data storage before expanding into AI applications through the platform’s Cortex capabilities. The partnership reflects the exchange's focus on maintaining data quality whilst scaling operations to handle exponential growth in both trading volumes and regulatory requirements.

“What really powers AI is that good source of truth. If you don’t have a source of truth, you're going to have unfortunate outcomes,” Lynn says, speaking on stage at Snowflake Summit 25.

The exchange has actually used machine learning models for over a decade to enhance pattern recognition, but newer generative AI capabilities enable natural language queries of surveillance databases and automated preliminary investigations. This approach allows the compliance team to maintain the same level of oversight whilst handling far larger data volumes, though Lynn emphasises the deliberate nature of their AI deployment strategy.

“We always come back to our principles of market integrity and transparency. When you stick to your core principles, the use cases will follow,” Lynn says.

Financial services firms demonstrate measurable returns from targeted AI implementations

NYSE's approach reflects a broader trend across financial services, where firms are moving beyond experimental AI projects towards implementations that deliver measurable business results. Chicago Trading Company achieved 54% cost savings worth millions of dollars annually by consolidating transformation workloads to Snowflake using Snowpark, which demonstrates the kind of efficiency gains that justify infrastructure investments.

Sridhar Ramaswamy, Snowflake’s CEO, says these implementations validate his view that enterprise AI requires robust data foundations rather than standalone applications. He emphasises that the complexity of modern business operations demands simplicity in technology solutions, particularly when organisations are dealing with mission-critical applications like market surveillance.

“Complexity creates cost. Complexity creates friction and makes it harder to get the job done. Whereas simplicity drives results,” Sridhar says.

His company’s platform now supports thousands of customers sharing data through its marketplace, which contains over 3,000 listings from more than 750 providers. This connectivity enables what Sridhar describes as the fundamental principle underlying effective AI implementation.

“There is no AI strategy without a data strategy. Data is the fuel, and Snowflake's AI Data Cloud is powered by a connected ecosystem of data,” Sridhar says.

Financial services firms including Stripe and State Street use the platform for secure data sharing with partner ecosystems, with many maintaining hundreds of live connections. CME Group uses it for real-time market intelligence, whilst other companies deploy AI for insurance underwriting and investor report analysis.

The NYSE's parent company, Intercontinental Exchange, has achieved particular success with Snowflake's Snowpark platform for data engineering workloads. The exchange migrated its regulatory reporting operations to Snowpark after finding that maintaining separate compute environments was both costly and operationally burdensome. Anand Pradhan, Senior Director of Regulatory and NMS Tech at NYSE, describes the challenge: “Our regulatory reporting workload looks at hundreds of billions of records in the data set and constantly merges them together to find different patterns — all while joining various reference data sources. It requires a massive amount of compute.”

The migration delivered immediate financial benefits. Related data costs fell by more than 50% within one month of deploying Snowpark for regulatory reporting, despite consistent data volumes. According to Pradhan: “Cost reduction with Snowpark was so impactful that I had to adjust my budget for this year.”

The pattern that emerges from these implementations is that successful AI projects tend to focus on specific operational challenges rather than broad technology deployments. Banks deploying AI for anti-money laundering, for example, need systems that can identify suspicious transactions whilst minimising false positives that overwhelm compliance staff. Insurance companies face similar requirements when using AI for claims processing, where they must balance automation efficiency with regulatory oversight requirements.

Sridhar argues that this focus on solving specific problems reflects a broader principle about how technology should integrate with business operations. “You should be able to ask a question with a voice memo and get an answer from your enterprise data. You should be able to launch a customer app without having to write a line of code. You should be able to harness the world's best AI models to create agents that can learn about your business,” he says.

Market surveillance systems require balance between automation and human oversight

The Department of Defense IL5 authorisation that Snowflake recently received demonstrates the security standards required for financial market infrastructure that processes sensitive trading data. 

NYSE’s surveillance operations show how this security framework applies in practice. The exchange monitors 1.2 trillion daily messages for market manipulation patterns whilst ensuring that automated systems don't replace human judgment in critical decisions. Traditional surveillance systems generate thousands of alerts daily, but AI helps filter these down to cases that genuinely require human investigation.

The platform achieves 90% accuracy for AI applications, though Sridhar says the company continues working to improve these levels. The accuracy requirements reflect the nature of financial market applications, where data quality directly impacts market confidence and participant protection.

“We know you [customers] need to move ahead with confidence that the right people are using the right data for the right purpose,” Sridhar says.

This focus on data governance becomes particularly important when financial institutions deploy AI for customer-facing applications or regulatory reporting, where errors can have significant compliance implications.

"When it comes to AI, we have developed a very deliberate approach,” Lynn explains. “We always come back to our principles of market integrity and transparency. So we’ve been using a version of AI – LLMs, not generative – for more than a decade to add transparency to our markets. We continue to be very deliberate and not stray from our two core principles around transparency and market integrity.”

To read the full article in the magazine, click HERE.