Module 5: Setting Up Pinecone

Your First Managed Vector Database

---

Introduction

Pinecone is the most popular managed vector database. It handles all infrastructure, scaling, and maintenance, letting you focus on building your application.

By the end of this module, you'll have:

• A Pinecone account and index
• Working code to insert and query vectors
• Understanding of Pinecone's concepts and terminology

---

5.1 Getting Started with Pinecone

Create an Account

1. Go to pinecone.io
2. Sign up for a free account
3. You get a free tier with 1 index and limited storage

Get Your API Key

1. Navigate to the Pinecone console
2. Go to "API Keys" in the sidebar
3. Copy your API key

# Add to .env
PINECONE_API_KEY=your-api-key-here

Create Your First Index

In the Pinecone console:

1. Click "Create Index"
2. Name: my-first-index
3. Dimensions: 1536 (for OpenAI embeddings)
4. Metric: cosine
5. Cloud/Region: Choose closest to your users

Or via API:

import { Pinecone } from '@pinecone-database/pinecone'

const pinecone = new Pinecone({
  apiKey: process.env.PINECONE_API_KEY!
})

// Create index (one-time operation)
await pinecone.createIndex({
  name: 'my-first-index',
  dimension: 1536,
  metric: 'cosine',
  spec: {
    serverless: {
      cloud: 'aws',
      region: 'us-east-1'
    }
  }
})

---

5.2 Pinecone Concepts

Indexes

An index is a collection of vectors. Think of it as a database.

• Each index has a fixed dimension (e.g., 1536)
• All vectors in an index must have the same dimension
• You query within an index

Namespaces

Namespaces partition an index. Use them for:

• Multi-tenancy (one namespace per customer)
• Separating environments (dev, staging, prod)
• Organizing different data types

// Query specific namespace
const index = pinecone.index('my-first-index')
const namespace = index.namespace('customer-123')

await namespace.query({
  vector: queryEmbedding,
  topK: 10
})

Records (Vectors)

A record contains:

• id: Unique identifier (string)
• values: The vector (number[])
• metadata: Optional JSON object for filtering

const record = {
  id: 'doc-123',
  values: [0.1, -0.2, 0.3, ...],  // 1536 numbers
  metadata: {
    title: 'How to use Pinecone',
    category: 'tutorial',
    createdAt: '2024-01-15'
  }
}

---

5.3 Basic Operations

Install the SDK

npm install @pinecone-database/pinecone openai dotenv

Initialize the Client

import { Pinecone } from '@pinecone-database/pinecone'
import OpenAI from 'openai'
import dotenv from 'dotenv'

dotenv.config()

const pinecone = new Pinecone({
  apiKey: process.env.PINECONE_API_KEY!
})

const openai = new OpenAI()

const index = pinecone.index('my-first-index')

Inserting Vectors (Upsert)

async function upsertDocuments(documents: { id: string; text: string; metadata: object }[]) {
  // Generate embeddings
  const response = await openai.embeddings.create({
    model: 'text-embedding-3-small',
    input: documents.map(d => d.text)
  })

  // Prepare records
  const records = documents.map((doc, i) => ({
    id: doc.id,
    values: response.data[i].embedding,
    metadata: doc.metadata
  }))

  // Upsert in batches of 100
  const batchSize = 100
  for (let i = 0; i < records.length; i += batchSize) {
    const batch = records.slice(i, i + batchSize)
    await index.upsert(batch)
  }
}

// Usage
await upsertDocuments([
  {
    id: 'doc-1',
    text: 'Vector databases store embeddings for similarity search',
    metadata: { category: 'databases', source: 'tutorial' }
  },
  {
    id: 'doc-2',
    text: 'Pinecone is a managed vector database service',
    metadata: { category: 'databases', source: 'product' }
  }
])

Querying Vectors

async function searchSimilar(query: string, topK = 10, filter?: object) {
  // Generate query embedding
  const response = await openai.embeddings.create({
    model: 'text-embedding-3-small',
    input: query
  })
  const queryEmbedding = response.data[0].embedding

  // Query Pinecone
  const results = await index.query({
    vector: queryEmbedding,
    topK,
    filter,
    includeMetadata: true
  })

  return results.matches
}

// Usage
const results = await searchSimilar('How do I store embeddings?')
console.log(results)
// [
//   { id: 'doc-1', score: 0.89, metadata: { category: 'databases', ... } },
//   { id: 'doc-2', score: 0.76, metadata: { category: 'databases', ... } },
// ]

Filtering Results

Pinecone supports metadata filtering:

// Single condition
const results = await index.query({
  vector: queryEmbedding,
  topK: 10,
  filter: { category: 'databases' }
})

// Multiple conditions (AND)
const results = await index.query({
  vector: queryEmbedding,
  topK: 10,
  filter: {
    category: 'databases',
    source: 'tutorial'
  }
})

// Complex filters
const results = await index.query({
  vector: queryEmbedding,
  topK: 10,
  filter: {
    $and: [
      { category: { $eq: 'databases' } },
      { createdAt: { $gte: '2024-01-01' } }
    ]
  }
})

Supported operators:

• $eq, $ne: Equal, not equal
• $gt, $gte, $lt, $lte: Comparisons
• $in, $nin: In array, not in array
• $and, $or: Logical operators

Fetching by ID

const record = await index.fetch(['doc-1', 'doc-2'])
console.log(record.records['doc-1'])

Deleting Records

// Delete by ID
await index.deleteOne('doc-1')

// Delete multiple
await index.deleteMany(['doc-1', 'doc-2', 'doc-3'])

// Delete by filter
await index.deleteMany({
  filter: { category: 'deprecated' }
})

// Delete all in namespace
await index.namespace('old-data').deleteAll()

Updating Records

// Update metadata only
await index.update({
  id: 'doc-1',
  metadata: { category: 'updated-category' }
})

// Update vector and metadata
await index.update({
  id: 'doc-1',
  values: newEmbedding,
  metadata: { category: 'updated' }
})

---

5.4 Complete Example: Document Search

Here's a complete example of a document search system:

import { Pinecone } from '@pinecone-database/pinecone'
import OpenAI from 'openai'
import dotenv from 'dotenv'

dotenv.config()

const pinecone = new Pinecone()
const openai = new OpenAI()

interface Document {
  id: string
  title: string
  content: string
  category: string
}

interface SearchResult {
  id: string
  title: string
  score: number
}

class DocumentSearch {
  private index

  constructor(indexName: string) {
    this.index = pinecone.index(indexName)
  }

  async addDocuments(documents: Document[]): Promise<void> {
    // Generate embeddings
    const contents = documents.map(d => `${d.title}\n\n${d.content}`)
    const embeddingsResponse = await openai.embeddings.create({
      model: 'text-embedding-3-small',
      input: contents
    })

    // Prepare records
    const records = documents.map((doc, i) => ({
      id: doc.id,
      values: embeddingsResponse.data[i].embedding,
      metadata: {
        title: doc.title,
        category: doc.category
      }
    }))

    // Upsert
    await this.index.upsert(records)
    console.log(`Added ${documents.length} documents`)
  }

  async search(
    query: string,
    options: { topK?: number; category?: string } = {}
  ): Promise<SearchResult[]> {
    const { topK = 10, category } = options

    // Generate query embedding
    const response = await openai.embeddings.create({
      model: 'text-embedding-3-small',
      input: query
    })

    // Build filter
    const filter = category ? { category } : undefined

    // Query
    const results = await this.index.query({
      vector: response.data[0].embedding,
      topK,
      filter,
      includeMetadata: true
    })

    return results.matches?.map(m => ({
      id: m.id,
      title: m.metadata?.title as string,
      score: m.score ?? 0
    })) ?? []
  }
}

// Usage
async function main() {
  const search = new DocumentSearch('my-first-index')

  // Add documents
  await search.addDocuments([
    {
      id: '1',
      title: 'Introduction to Vector Databases',
      content: 'Vector databases are specialized systems for storing and querying embeddings...',
      category: 'tutorial'
    },
    {
      id: '2',
      title: 'Pinecone Quick Start',
      content: 'Pinecone is a managed vector database that makes it easy to build...',
      category: 'guide'
    }
  ])

  // Search
  const results = await search.search('How do I get started with vectors?')
  console.log('Results:', results)

  // Search with filter
  const tutorialResults = await search.search('vector basics', { category: 'tutorial' })
  console.log('Tutorial results:', tutorialResults)
}

main()

---

5.5 Best Practices

Batching

Always batch operations:

// Bad: One at a time
for (const doc of documents) {
  await index.upsert([doc])  // Slow!
}

// Good: Batch of 100
const batchSize = 100
for (let i = 0; i < documents.length; i += batchSize) {
  const batch = documents.slice(i, i + batchSize)
  await index.upsert(batch)
}

Metadata Design

Keep metadata lean:

// Bad: Large metadata
{
  id: 'doc-1',
  values: [...],
  metadata: {
    fullContent: '10,000 words of text...',  // Too large!
    imageData: 'base64...'  // Don't do this
  }
}

// Good: Reference only
{
  id: 'doc-1',
  values: [...],
  metadata: {
    title: 'Document Title',
    category: 'tutorial',
    documentId: 'ref-to-full-content'  // Fetch full content separately
  }
}

Index Names

Use descriptive, consistent naming:

// Good patterns
'production-documents'
'staging-customer-support'
'dev-test-index'

---

Key Takeaways

1. Pinecone is managed—no infrastructure to worry about
2. Indexes hold vectors, namespaces partition them
3. Upsert inserts or updates, query finds similar vectors
4. Metadata filtering lets you combine similarity with exact matches
5. Batch operations for efficiency

---

Exercise: Build a FAQ Search

1. Create a Pinecone index for FAQ entries
2. Add 10 sample FAQ questions and answers
3. Implement search that finds relevant FAQs
4. Add filtering by category

// Starter template
const faqs = [
  {
    id: 'faq-1',
    question: 'How do I reset my password?',
    answer: 'Go to settings and click reset password...',
    category: 'account'
  },
  // Add more...
]

---

Next up: Module 6 - Setting Up pgvector (PostgreSQL)