Skip to main content

[@ServeTheHomeVideo] Enterprise AI Using RAG - Supermicro Open Storage Summit Session 8

· 8 min read
[@ServeTheHomeVideo] Summarizer
Video Bot

@ServeTheHomeVideo - "Enterprise AI Using RAG - Supermicro Open Storage Summit Session 8"

Link: https://youtu.be/MHmX-QM7WYQ

Short Summary​

Okay, here's a breakdown of the key takeaway and executive summary based on the transcript:

Number One Most Important Action Item/Takeaway:

Enterprises should start with a specific customer-focused use case in mind and work backward to build their AI stack, prioritizing evaluation and guardrails for accurate retrieval and augmentation in RAG architectures. Rather than simply deploying AI, solve a customer problem.

Concise Executive Summary (2-3 Sentences):

The Super Micro Open Storage Summit panel emphasized the shift towards enterprises wanting to own their AI stacks for differentiation. They highlighted the importance of modern AI-native infrastructure with scalable storage, secure data protection, and flexible environments to support Retrieval Augmented Generation (RAG) architectures. The key is to use the current hardware and plan for the future where AI is used to solve customer problems by focusing on utilization, sustainability, and planning for future agentic workflows and their potential impact.

Key Quotes​

Here are five quotes from the YouTube video transcript that represent valuable insights:

  1. Ben Lee, Super Micro: "So what's a paradigm shift? Uh I think the true answer is like now you need to build anti AI factories." This quote highlights the transition from simply experimenting with AI to building dedicated infrastructure for AI development and deployment within enterprises.
  2. Sab Giri, Voltage Park: "Customers want an end-to-end AI stack that's very tightly coupled across the hardware and software layers because then they get simplicity, predictability and cost performance and that is what Voltagebug does." This points to a need for integrated solutions that eliminate the complexities of deploying AI.
  3. Phil Mennez, Vast Data: "So we really set out to make AI secure, scalable, and easy for every enterprise is and we've just announced kind of the final pieces of our broad vision which is the vast AI operating system." This quote emphasizes the importance of simplifying AI deployment through a comprehensive, integrated platform, addressing scalability and security concerns.
  4. NAV Algarissi, Nvidia: "The data is needs to be secure, needs to be maintained continuously. So in fact bringing AI to your data becomes a more and more evolving and available solution especially when working with VAST and other platforms." This quote underscores that data gravity, data security, and continuous data maintenance are critical, making "bringing AI to your data" a more viable and evolving solution.
  5. Kelly Osborne, Grade Technology: "The goal of this is to eliminate latencies in these data paths between these workflows. Um, so that especially in rag where you're incorporating your own data, there are bottlenecks all over the place in moving data from your own environment to an LLM..." This identifies a critical bottleneck in RAG architectures: latency in data paths when integrating enterprise data with LLMs.

Detailed Summary​

Here's a detailed summary of the YouTube video transcript, focusing on key topics and arguments, excluding advertisements:

  • Introduction & Focus:

    • Super Micro's Open Storage Summit episode focuses on how Retrieval Augmented Generation (RAG) influences enterprise architectures in the context of enterprise AI.
    • The discussion emphasizes the shift from experimenting with generative AI to deploying it in production environments.
    • Key challenge: How to connect foundation models with proprietary data securely, efficiently, and with low latency.
    • Enterprises need modern, scalable AI infrastructure to handle structured and unstructured data across edge, core, and cloud.

  • Panel Introduction:

    • Host Rob Streche introduces the panel:
      • Ben Lee (Super Micro)
      • Sab Giri (Voltage Park)
      • Phil Mennez (Vast Data)
      • Nav Algarissi (Nvidia)
      • Kelly Osborne (Grade Technology)
      • Tamid Ramen (Solidigm)

  • Super Micro's Perspective (Ben Lee):

    • Super Micro is seeing rapid adoption of AI by enterprises beyond just inference workflows; they're also scaling RAG and building their own models.
    • This puts pressure on enterprise infrastructure, requiring more AI flops, memory bandwidth, fast networking, and decisions about on-premise vs. cloud deployment.
    • Paradigm shift: The need to build "AI factories."
    • Super Micro's five-layer approach:
      1. GPU-accelerated compute
      2. High-performance storage
      3. Low-latency networking
      4. AI software stack (e.g., Nvidia's enterprise suite)
      5. LLM strategy (pre-trained, local, or cloud-based APIs)
    • The biggest challenge: Connecting enterprise data to LLMs, which RAG aims to solve.
    • Super Micro offers turnkey, scalable, and optimized solutions, working with customers from initial design to deployment, including validation and testing.
    • Goal: Simplify AI infrastructure and accelerate AI factory deployments.

  • Voltage Park's Perspective (Sab Giri):

    • Voltage Park is a cloud service provider specializing in AI workloads.
    • They provide an end-to-end AI stack (hardware and software) focused on simplicity, predictability, and cost-performance.
    • Voltage Park partnered with Super Micro and Vast Data due to the foundational role of storage in scaling AI workloads.
    • Storage throughput impacts RAG latency, training cycle times, and system costs.
    • Checkpointing is critical for large training runs to minimize downtime.
    • Real-world inference use cases like RAG require large amounts of data to be available in real time.

  • Vast Data's Perspective (Phil Mennez):

    • Vast partners with Super Micro to build AI supercomputers.
    • They serve model builders, service providers (like Voltage Park), and enterprises.
    • The goal is to make AI secure, scalable, and easy for every enterprise.
    • Vast AI Operating System brings together:
      • Vast Data Store: Hyperscale data environment for fast access to block, file, and object data.
      • Database: Captures structured data and embeddings in real time.
      • Data Space: Unifies data across edge, service providers, and cloud.
      • Data Engine: Runtime for embedding services and data-driven workflows.
      • Agent Engine: Easy interface to build and deploy agents at scale.
    • Underlying architecture is a disaggregated, shared-everything architecture ("DAYS") for linear scalability, real-time performance, extreme resilience, and affordable data access.

  • Nvidia's Perspective (Nav Algarissi):

    • Enterprises want to connect their data to their AI.
    • Data gravity is becoming more apparent, requiring secure and continuously maintained data.
    • Nvidia aims to accelerate AI adoption and make production easier.
    • Nemo Retriever: Microservices to enable enterprises to extract, embed, and retrieve data from various sources (text, visual, PDF) in real time for RAG.
    • Key components are delivered as modular pieces and blueprints.
    • Nvidia Inference Microservice (NIM): Containerized inference software deployable anywhere, with enterprise-ready security and optimizations.
    • Focus on multimodal retrieval (visual content retrieval).
    • Emphasizes the portability and modularity of the system.

  • Grade Technology's Perspective (Kelly Osborne):

    • Moving large amounts of data between workflows can introduce latency.
    • Traditional software RAID and hardware RAID haven't kept pace with performance improvements.
    • Grade Technology's Supreme Raid is a new way of doing data protection.
    • Supreme Raid uses Nvidia GPUs to handle the mathematical calculations for parity, offloading the CPU.
    • They use a patented peer-to-peer DMA technology and GPU Direct Storage.
    • Goal: Eliminate latencies in data paths, especially important in edge environments where real-time data collection and inference occur.

  • Solidigm's Perspective (Tamid Ramen):

    • Solidigm focuses on data center storage for AI, where storage needs to be responsive and expansive.
    • They are a vertically integrated company with media, controller, and system optimization.
    • They address different pipeline stages of AI, from data ingest to training.
    • High-density SSDs are a good fit for data ingest, prep, and archive.
    • Gen 5 SSDs (high performance, high bandwidth) are used in the training phase.
    • SSDs offer throughput improvements compared to HDDs, improving checkpointing and IOPS.
    • Future needs: Fast access, increasing data set sizes, and efficient rack designs.
    • SSD-based solutions can offer significant power and footprint reductions compared to HDD-based solutions.

  • Q&A and Recommendations:

    • Voltage Park (Sab Giri): AI factories are about integrating AI to augment existing products or build new ones. Enterprises want to own their AI stack end-to-end for data privacy. Focus on use cases to deliver AI stack.
    • Super Micro (Ben Lee): Flexibility and complete solution are important. Support multiple deployments and different customer preferences.
    • Solidigm (Tamid Ramen): Fast access to data sets is increasing. Techniques can come in place where disk specific ANN models can be leveraged, also softload some of the rack database on SSD.
    • Vast Data (Phil Mennez): Focus on creating a complete, reliable, manageable stack deployed as a turnkey service. As you shift to real time you need a more scaleable approach to vector databases.
    • Nvidia (Nav Algarissi): Make it as easy, accessible and cost efficient as possible to connect enterprise data to AI. Leverage customer obsession and focus on specific use cases.
    • Grade Technology (Kelly Osborne): Customers aren't alone, and they don't have to do this alone. Partnership ecosystem exists to help overcome roadblocks.