Open source extensions from Timescale

Timescale, the PostgreSQL cloud database company, has unveiled two open-source extensions, pgvectorscale and pgai, which significantly enhance PostgreSQL’s scalability and ease of use for AI applications. Licensed under the open-source PostgreSQL license, these extensions empower developers to build retrieval augmented generation, search, and AI agent applications with PostgreSQL, rivaling the performance of specialized vector databases like Pinecone at a fraction of the cost. Complete benchmark information with details around performance and cost is available here.

New open source extensions. Innovations driving AI application performance and developer productivity

pgvectorscale enables developers to build more scalable AI applications, with higher performance embedding search and cost-efficient storage. pgvectorscale complements pgvector - the popular open-source extension for vector data in PostgreSQL - and introduces two key innovations: a StreamingDiskANN index (adapted from research from Microsoft) and Statistical Binary Quantization (developed by Timescale researchers and which improves on standard Binary Quantization techniques). Timescale’s benchmarks reveal that with pgvectorscale, PostgreSQL achieves 28x lower p95 latency and 16x higher query throughput compared to Pinecone for approximate nearest neighbor queries at 99% recall. In contrast to pgvector, which is written in C, pgvectorscale is developed in the Rust programming language, offering the PostgreSQL community a new avenue for contributing to vector support.

pgai brings more AI workflows to PostgreSQL, making it easier for developers to build search and retrieval augmented generation (RAG) applications. The initial release supports creating OpenAI embeddings and obtaining OpenAI chat completions from models like GPT4o directly within PostgreSQL. This integration allows for classification, summarization, and data enrichment tasks on existing relational data, streamlining the development process from proof of concept to production.

“Pgvectorscale and pgai are incredibly exciting for building AI applications with PostgreSQL. Having embedding functions directly within the database is a huge bonus,” states Web Begole, CTO of Market Reader, a company using Timescale’s cloud PostgreSQL offering to build an AI-enabled financial information platform. “Previously, updating our saved embeddings was a tedious task, but now, with everything integrated, it promises to be much simpler and more efficient. This will save us a significant amount of time and effort.”

“Pgvectorscale and pgai are great additions to the PostgreSQL AI ecosystem. The introduction of Statistical Binary Quantization promises lightning performance for vector search and will be valuable as we scale our vector workload,” said John McBride, Head of Infrastructure at OpenSauced, a company building an AI-enabled analytics platform for open-source projects using Timescale’s cloud PostgreSQL offering. “Pgai removes the need for developers to re-implement common functionality themselves, and I’m excited for the use cases it enables.”

Challenging the need for standalone specialized vector databases

The primary advantage of dedicated vector databases like Pinecone and many others has been their performance, coming from purpose-built architectures and algorithms for storing and searching large volumes of vector data. However, Timescale’s pgvectorscale challenges this notion by bringing such specialized architectures and algorithms to PostgreSQL in the form of an extension, helping the popular general purpose database deliver comparable and often superior performance than specialized vector databases. According to Timescale’s benchmarks, which involved querying a dataset of 50 million Cohere embeddings (768 dimension), PostgreSQL outperforms Pinecone’s storage optimized index (s1) with 28x lower p95 latency and 16x higher query throughput for approximate nearest neighbor queries at 99% recall. Furthermore, PostgreSQL with pgvectorscale achieves 1.4x lower p95 latency and 1.5x higher query throughput compared to Pinecone’s performance optimized index (p2) at 90% recall on the same dataset.

The cost benefits are equally compelling. Self-hosting PostgreSQL with pgvector and pgvectorscale is 4-5 times cheaper than using Pinecone, with PostgreSQL costing approximately $835 per month on AWS EC2, compared to Pinecone’s $3,241 per month for the storage optimized index and $3,889 per month for the performance optimized index.

PostgreSQL: The all-in-one database solution for AI applications

Timescale’s new extensions reinforce the “PostgreSQL for Everything” movement, where developers seek to simplify complex data architectures by leveraging PostgreSQL’s robust, versatile, and reliable ecosystem. With its rich array of extensions and proven reliability, PostgreSQL stands as the ideal foundation for the future of data and AI-driven applications.

“Timescale’s goal is to lower the barriers for developers adopting and scaling PostgreSQL for AI applications,” says Ajay Kulkarni, CEO of Timescale. “By open-sourcing pgvectorscale and pgai, Timescale aims to establish PostgreSQL as the default database for AI applications. This eliminates the need for separate vector databases and simplifies the data architecture for developers as they scale.”

The Timescale pgvectorscale and pgai extensions products are available now. Go to this blog post to learn more or www.timescale.com/contact for additional details to schedule a demo.