image_management_api/README.md

SEREACT - Secure Image Management API

SEREACT is a secure API for storing, organizing, and retrieving images with advanced search capabilities powered by AI-generated embeddings.

Features

• Secure image storage in Google Cloud Storage
• Team-based organization and access control
• Hybrid authentication model: Public management endpoints + API key protected data endpoints
• Asynchronous image processing with Pub/Sub and Cloud Functions
• AI-powered image embeddings using Google Vertex AI Multimodal Embedding API
• Semantic search using vector similarity with Qdrant Vector Database
• Self-hosted vector database on Google Compute Engine VM
• Automatic retry mechanism for failed processing (up to 3 attempts)
• Metadata extraction and storage
• Image processing capabilities
• Multi-team support
• Public user and team management APIs for easy integration
• Comprehensive E2E testing with real database support

Architecture

sereact/
  ├── images/                    # Sample images for testing
  ├── deployment/                # Deployment configurations
  │   ├── cloud-function/        # **Cloud Function for image processing**
  │   ├── cloud-run/             # Google Cloud Run configuration
  │   └── terraform/             # Infrastructure as code
  │       ├── vm.tf              # **Vector database VM configuration**
  │       └── scripts/           # **VM installation scripts**
  ├── docs/                      # Documentation
  │   ├── api/                   # API documentation
  │   └── TESTING.md             # Comprehensive testing guide
  ├── scripts/                   # Utility scripts
  ├── src/                       # Source code
  │   ├── api/                   # API endpoints and routers
  │   │   └── v1/                # API version 1 routes
  │   ├── auth/                  # Authentication and authorization
  │   ├── config/                # Configuration management
  │   ├── core/                  # Core application logic
  │   ├── db/                    # Database layer
  │   │   ├── providers/         # Database providers (Firestore)
  │   │   └── repositories/      # Data access repositories
  │   ├── models/                # Database models
  │   ├── schemas/               # API request/response schemas
  │   ├── services/              # Business logic services
  │   │   ├── pubsub_service.py  # **Pub/Sub message publishing**
  │   │   └── vector_db.py       # **Qdrant vector database service**
  │   └── utils/                 # Utility functions
  ├── tests/                     # Test code
  │   ├── api/                   # API tests
  │   ├── auth/                  # Authentication tests
  │   ├── models/                # Model tests
  │   ├── services/              # Service tests
  │   ├── integration/           # Integration tests
  │   │   └── test_cloud_function.py  # **Cloud Function tests**
  │   └── test_e2e.py           # **Comprehensive E2E workflow tests**
  ├── main.py                    # Application entry point
  ├── requirements.txt           # Python dependencies
  └── README.md                  # This file

System Architecture

┌─────────────┐         ┌─────────────┐         ┌─────────────┐
│             │         │             │         │             │
│  FastAPI    │ ───────▶│  Firestore  │◀────────│  Cloud      │
│  Backend    │         │  Database   │         │  Functions  │
│             │         │             │         │             │
└─────┬───────┘         └─────────────┘         └──────┬──────┘
      │                                                │
      │                                                │
      ▼                                                │
┌─────────────┐         ┌─────────────┐                │
│             │         │             │                │
│  Cloud      │         │  Pub/Sub    │                │
│  Storage    │────────▶│  Queue      │────────────────┘
│             │         │             │
└─────────────┘         └─────────────┘
                               │
                               │
                               ▼
                        ┌─────────────┐         ┌─────────────┐
                        │             │         │             │
                        │  Cloud      │         │  Qdrant     │
                        │  Vision API │────────▶│  Vector DB  │
                        │             │         │  (VM)       │
                        └─────────────┘         └─────────────┘

Image Processing Workflow

1. Image Upload Flow:

• User uploads image through FastAPI backend
• Image is stored in Google Cloud Storage
• Image metadata is saved to Firestore with embedding_status: "pending"
• Pub/Sub message is published to trigger async processing

2. Embedding Generation Flow (Asynchronous):

• Cloud Function is triggered by Pub/Sub message
• Function updates image status to "processing"
• Function downloads image from Cloud Storage
• Function calls Google Vertex AI Multimodal Embedding API to generate 1408-dimensional embeddings
• Embeddings are stored in Qdrant Vector Database on dedicated VM
• Firestore is updated with embedding info and status: "success"

3. Error Handling & Retry:

• Failed processing updates status to "failed" with error message
• Automatic retry up to 3 times using Pub/Sub retry policy
• Dead letter queue for permanently failed messages

4. Search Flow:

• Search queries processed by FastAPI backend
• Vector similarity search performed against Qdrant VM
• Results combined with metadata from Firestore

Technology Stack

• FastAPI - Web framework
• Firestore - Database
• Google Cloud Storage - Image storage
• Google Pub/Sub - Message queue for async processing
• Google Cloud Functions - Serverless image processing
• Google Vertex AI Multimodal Embedding API - AI-powered image analysis and embedding generation
• Qdrant - Self-hosted vector database for semantic search (on Google Compute Engine VM)
• Google Compute Engine - VM hosting for vector database
• Pydantic - Data validation

Vector Database Infrastructure

Qdrant Vector Database VM

The system includes a dedicated Google Compute Engine VM running Qdrant vector database:

• VM Specifications: 2 vCPUs, 8GB RAM, 50GB disk (e2-standard-2)
• Operating System: Ubuntu 22.04 LTS
• Vector Database: Qdrant (latest version via Docker)
• Ports: 6333 (HTTP API), 6334 (gRPC API)
• Features:
  • Automatic installation and configuration via startup script
  • Daily automated backups
  • Health monitoring
  • Firewall configuration
  • Optional static IP assignment
  • API key authentication support

Vector Database Features

• High Performance: Optimized for image vector similarity search
• Scalable: Can handle millions of image vectors
• Persistent Storage: Data persisted on VM disk with automated backups
• RESTful API: Easy integration with Python client
• Cosine Similarity: Optimized for image embedding comparisons
• Metadata Filtering: Support for complex search filters

AI Embedding Model

SEREACT uses Google's Vertex AI multimodal embedding model for generating high-quality image embeddings:

• Model: multimodalembedding@001
• Provider: Google Vertex AI
• Type: Multimodal embedding model (supports both images and text)
• Output Dimensions: 1408-dimensional vectors
• Data Type: Float32, normalized vectors (norm ≈ 1.0)
• Similarity Metric: Cosine similarity
• Use Case: Semantic image search and similarity matching

⚠️ Important Configuration: Ensure your Qdrant collection is configured with 1408 dimensions to match the Vertex AI model output. Dimension mismatches will cause embedding storage failures.

Setup and Installation

Prerequisites

• Python 3.8+
• Google Cloud account with Firestore, Storage, Pub/Sub, Cloud Functions, Compute Engine, and Vision API enabled
• Terraform (for infrastructure deployment)

Installation

1. Clone the repository:

   git clone https://github.com/yourusername/sereact.git
   cd sereact
   

2. Create and activate a virtual environment:

   python -m venv venv
   source venv/bin/activate  # Linux/macOS
   venv\Scripts\activate     # Windows
   

3. Install dependencies:

   pip install -r requirements.txt
   

4. Create a .env file with the following environment variables:

   # Firestore
   FIRESTORE_PROJECT_ID=your-gcp-project-id
   FIRESTORE_CREDENTIALS_FILE=path/to/firestore-credentials.json
   
   # Google Cloud Storage
   GCS_BUCKET_NAME=your-bucket-name
   GCS_CREDENTIALS_FILE=path/to/credentials.json
   
   # Google Pub/Sub
   PUBSUB_TOPIC=image-processing-topic
   PUBSUB_SUBSCRIPTION=image-processing-subscription
   
   # Google Vertex AI (for image embeddings)
   VERTEX_AI_LOCATION=us-central1
   GOOGLE_CLOUD_PROJECT=your-gcp-project-id
   
   # Security
   API_KEY_SECRET=your-secret-key
   
   # Vector database (Qdrant)
   QDRANT_HOST=your-vm-external-ip
   QDRANT_API_KEY=your-qdrant-api-key  # Optional
   QDRANT_COLLECTION=image_vectors
   QDRANT_HTTPS=false
   

5. Deploy Infrastructure (Required for vector database):

   # Configure Terraform variables
   cd deployment/terraform
   cp terraform.tfvars.example terraform.tfvars
   # Edit terraform.tfvars with your values
   
   # Deploy infrastructure including vector database VM
   terraform init
   terraform plan
   terraform apply
   
   # Note the output values for VM IP addresses
   

6. Deploy Cloud Function (Optional - for production):

   cd ../cloud-function
   ./deploy.sh
   

7. Run the application:

   uvicorn main:app --reload
   

8. Visit http://localhost:8000/docs in your browser to access the API documentation.

Deployment

Complete Infrastructure Deployment

Deploy the entire infrastructure including the vector database VM:

cd deployment/terraform

# Configure your variables
cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars with your specific values:
# - project_id: Your GCP project ID
# - storage_bucket_name: Unique bucket name
# - qdrant_api_key: Secure API key for Qdrant (optional)
# - allowed_cidr_blocks: Your IP address/range for security
# - use_static_ip: Set to true for production

# Deploy infrastructure
terraform init
terraform plan
terraform apply

This will create:

• Google Compute Engine VM with Qdrant vector database
• Firewall rules for vector database access
• Service accounts and IAM bindings
• Pub/Sub topic and subscription with retry policy
• Cloud Storage bucket
• Firestore database
• Cloud Run service

Vector Database VM Outputs

After deployment, Terraform will output:

• vector_db_vm_external_ip: External IP address of the VM
• qdrant_http_endpoint: HTTP API endpoint for Qdrant
• qdrant_grpc_endpoint: gRPC API endpoint for Qdrant

Cloud Function Deployment

The image processing Cloud Function can be deployed using the provided script:

cd deployment/cloud-function

# Set environment variables
export GOOGLE_CLOUD_PROJECT=your-project-id
export QDRANT_HOST=your-vm-external-ip
export QDRANT_API_KEY=your-qdrant-api-key

# Deploy the function
./deploy.sh

Vector Database Management

Accessing the Vector Database

# SSH into the VM
gcloud compute ssh sereact-vector-db --zone=us-central1-a

# Check Qdrant status
sudo systemctl status qdrant

# View logs
sudo journalctl -u qdrant -f

# Run health check
sudo /opt/qdrant/health_check.sh

# Manual backup
sudo /opt/qdrant/backup.sh

Vector Database API Usage

from src.services.vector_db import VectorDatabaseService

# Initialize service
vector_db = VectorDatabaseService(
    host="your-vm-external-ip",
    api_key="your-qdrant-api-key"  # Optional
)

# Add image vector
point_id = vector_db.add_image_vector(
    image_id="img_123",
    vector=[0.1, 0.2, ...],  # 1408-dimensional vector
    metadata={"filename": "image.jpg", "size": 1024}
)

# Search similar images
results = vector_db.search_similar_images(
    query_vector=[0.1, 0.2, ...],
    limit=10,
    score_threshold=0.7
)

API Endpoints

The API provides the following main endpoints with their authentication and pagination support:

🔓 Public Endpoints (No Authentication Required)

Authentication & API Key Management

• /api/v1/auth/bootstrap - Initial system setup (creates first team, admin user, and API key)
• /api/v1/auth/api-keys (POST) - Create new API key (requires user_id and team_id parameters)

Team Management

• /api/v1/teams/* - Complete team management (no authentication required)
  • POST /api/v1/teams - Create new team
  • GET /api/v1/teams - List all teams (no pagination - returns all teams)
  • GET /api/v1/teams/{team_id} - Get team by ID
  • PUT /api/v1/teams/{team_id} - Update team
  • DELETE /api/v1/teams/{team_id} - Delete team

User Management

• /api/v1/users/* - Complete user management (no authentication required)
  • POST /api/v1/users - Create new user (requires team_id)
  • GET /api/v1/users - List users (no pagination - returns all users, optionally filtered by team)
  • GET /api/v1/users/{user_id} - Get user by ID
  • PUT /api/v1/users/{user_id} - Update user
  • DELETE /api/v1/users/{user_id} - Delete user
  • GET /api/v1/users/me?user_id={id} - Get user info (requires user_id parameter)
  • PUT /api/v1/users/me?user_id={id} - Update user info (requires user_id parameter)

🔐 Protected Endpoints (API Key Authentication Required)

API Key Management (Authenticated)

• /api/v1/auth/api-keys (GET) - List API keys for current user
• /api/v1/auth/api-keys/{key_id} (DELETE) - Revoke API key
• /api/v1/auth/admin/api-keys/{user_id} (POST) - Create API key for another user (admin only)
• /api/v1/auth/verify - Verify current authentication

Image Management ✅ Fully Paginated & Protected

• /api/v1/images/* - Image upload, download, and management (with async processing)
  • GET /api/v1/images - List images with full pagination support
    • Query Parameters:
      • skip (default: 0, min: 0) - Number of items to skip
      • limit (default: 50, min: 1, max: 100) - Number of items per page
      • collection_id (optional) - Filter by collection
    • Response includes: images, total, skip, limit

Search Functionality ✅ Fully Paginated & Protected

• /api/v1/search/* - Image search functionality (semantic search via Qdrant)
  • GET /api/v1/search - Search images with pagination support
    • Query Parameters:
      • q (required) - Search query
      • limit (default: 10, min: 1, max: 50) - Number of results
      • similarity_threshold (default: 0.7, min: 0.0, max: 1.0) - Similarity threshold
      • collection_id (optional) - Filter by collection
    • Response includes: results, total, limit, similarity_threshold, query
  • POST /api/v1/search - Advanced search with same pagination

🔑 Authentication Model

SEREACT uses a hybrid authentication model:

1. Public Management Endpoints: Users, teams, and API key creation are publicly accessible for easy integration and setup
2. Protected Data Endpoints: Image storage, search, and API key management require API key authentication
3. Bootstrap Process: Initial setup via /auth/bootstrap creates the first team, admin user, and API key

Typical Workflow:

# 1. Bootstrap initial setup (public)
POST /api/v1/auth/bootstrap
{
  "team_name": "My Team",
  "admin_email": "admin@example.com", 
  "admin_name": "Admin User"
}
# Returns: API key for subsequent authenticated requests

# 2. Create additional users (public)
POST /api/v1/users
{
  "name": "John Doe",
  "email": "john@example.com",
  "team_id": "team_id_from_bootstrap"
}

# 3. Create API keys for users (public)
POST /api/v1/auth/api-keys?user_id={user_id}&team_id={team_id}
{
  "name": "John's API Key",
  "description": "For image uploads"
}

# 4. Use API key for protected operations
GET /api/v1/images
Headers: X-API-Key: your_api_key_here

Authentication & Pagination Status

Endpoint Category	Authentication	Pagination Status	Notes
Users Management	🔓 Public	❌ Not Implemented	Complete CRUD operations, no auth required
Teams Management	🔓 Public	❌ Not Implemented	Complete CRUD operations, no auth required
API Key Creation	🔓 Public	N/A	Requires user_id and team_id parameters
Images API	🔐 Protected	✅ Fully Implemented	skip, limit, total with proper validation
Search API	🔐 Protected	✅ Fully Implemented	limit, total with similarity scoring
API Key Management	🔐 Protected	❌ Not Implemented	List/revoke existing keys (small datasets)

Note: Public endpoints (users, teams) don't implement pagination as they typically return small datasets and are designed for management use cases where full data visibility is preferred.

Image Processing Status

Images now include embedding processing status:

{
  "id": "image-id",
  "filename": "example.jpg",
  "embedding_status": "success",  // "pending", "processing", "success", "failed"
  "embedding_error": null,
  "embedding_retry_count": 0,
  "has_embedding": true
}

Refer to the Swagger UI documentation at /docs for detailed endpoint information.

Development

Running Tests

# Run all tests
pytest

# Run specific test categories
pytest tests/services/test_pubsub_service.py  # Pub/Sub service tests
pytest tests/services/test_vector_db.py       # Vector database tests
pytest tests/integration/test_cloud_function.py  # Cloud Function tests
pytest tests/api/test_images_pubsub.py        # API integration tests

Comprehensive End-to-End Testing

SEREACT includes a comprehensive E2E testing suite that covers complete user workflows with completely self-contained artificial test data:

# Run all E2E tests (completely self-contained - no setup required!)
python scripts/run_tests.py e2e

# Run unit tests only (fast)
python scripts/run_tests.py unit

# Run integration tests (requires real database)
python scripts/run_tests.py integration

Infrastructure Costs

Estimated Monthly Costs (USD)

• Compute Engine VM (e2-standard-2): ~$50-70/month
• Cloud Storage: $0.02/GB/month
• Firestore: $0.18/100K reads, $0.18/100K writes
• Pub/Sub: $0.40/million messages
• Cloud Functions: $0.40/million invocations
• Cloud Vision API: $1.50/1000 images

Total estimated cost for moderate usage: ~$60-100/month

Cost Optimization Tips

• Use preemptible VM instances for development (50-91% cost reduction)
• Set up automatic VM shutdown during off-hours
• Use regional persistent disks instead of SSD for cost savings
• Monitor and set up billing alerts

License

This project is licensed under the MIT License - see the LICENSE file for details.

API Modules Architecture

The SEREACT API is organized into the following key modules to ensure separation of concerns and maintainable code:

src/
  ├── api/             # API endpoints and routers
  │   └── v1/          # API version 1 routes
  ├── auth/            # Authentication and authorization
  ├── config/          # Configuration management
  ├── models/          # Database models
  ├── services/        # Business logic services
  │   └── vector_db.py # **Qdrant vector database service**
  └── utils/           # Utility functions

Module Responsibilities

Router Module

• Defines API endpoints and routes
• Handles HTTP requests and responses
• Validates incoming request data
• Directs requests to appropriate services
• Implements API versioning

Auth Module

• Manages user authentication
• Handles API key validation and verification
• Implements role-based access control
• Provides security middleware
• Manages user sessions and tokens

Services Module

• Contains core business logic
• Orchestrates operations across multiple resources
• Implements domain-specific rules and workflows
• Integrates with external services (Cloud Vision, Storage, Qdrant)
• Handles image processing and embedding generation

Models Module

• Defines data structures and schemas
• Provides database entity representations
• Handles data validation and serialization
• Implements data relationships and constraints
• Manages database migrations

Utils Module

• Provides helper functions and utilities
• Implements common functionality used across modules
• Handles error processing and logging
• Provides formatting and conversion utilities
• Implements reusable middleware components

Config Module

• Manages application configuration
• Handles environment variable loading
• Provides centralized settings management
• Configures service connections and credentials
• Defines application constants and defaults

Module Interactions

┌─────────────┐         ┌─────────────┐         ┌─────────────┐
│             │         │             │         │             │
│  Router     │ ───────▶│  Services   │ ◀───────│  Config     │
│  Module     │         │  Module     │         │  Module     │
│             │         │             │         │             │
└──────┬──────┘         └──────┬──────┘         └─────────────┘
       │                       │
       │                       │
       ▼                       ▼
┌─────────────┐         ┌─────────────┐
│             │         │             │
│  Auth       │         │  Models     │
│  Module     │         │  Module     │
│             │         │             │
└──────┬──────┘         └──────┬──────┘
       │                       │
       │                       │
       └───────────────────────┘
                 │
                 ▼
          ┌─────────────┐
          │             │
          │  Utils      │
          │  Module     │
          │             │
          └─────────────┘

The modules interact in the following ways:

• Request Flow:

  • Client request arrives at the Router Module
  • Auth Module validates the request authentication
  • Router delegates to appropriate Service functions
  • Service uses Models to interact with the database
  • Service integrates with Qdrant Vector Database for similarity search
  • Service returns data to Router which formats the response

• Cross-Cutting Concerns:

  • Config Module provides settings to all other modules
  • Utils Module provides helper functions across the application
  • Auth Module secures access to routes and services

• Dependency Direction:

  • Router depends on Services and Auth
  • Services depend on Models and Config
  • Models depend on Utils for helper functions
  • Auth depends on Models for user information
  • All modules may use Utils and Config

This modular architecture provides several benefits:

• Maintainability: Changes in one module have minimal impact on others
• Testability: Modules can be tested in isolation with mocked dependencies
• Scalability: New features can be added by extending existing modules
• Reusability: Common functionality is centralized for consistent implementation
• Security: Authentication and authorization are handled consistently

TODO

High Priority

• Remove Pinecone integration and complete Qdrant migration
• Test and validate vector search functionality with Qdrant
• Implement proper pagination for search results and all endpoints COMPLETED - Images and Search APIs have full pagination
• Test Cloud Function image processing pipeline
• Validate VM setup for self-hosted Qdrant instance

Medium Priority

• Add comprehensive logging for vector search operations
• Implement caching layer for frequently accessed embeddings
• Implement caching for frequently accessed data
• Add monitoring and alerting for vector database performance
• Document vector search API endpoints
• Set up Qdrant cluster with multiple nodes
• Consider adding pagination to admin endpoints (users, teams, API keys) if datasets grow large

Low Priority

• Terraform dependencies
• Move all auth logic to auth module
• Remove bootstrap endpoint
• Move cloud function code to src folder and reuse code with embedding service
• Thumbnail generation

Pagination Status ✅

• ✅ Images API: Fully implemented with skip, limit, total parameters
• ✅ Search API: Fully implemented with limit, total, similarity scoring
• ℹ️ Users/Teams/API Keys: No pagination (small datasets, admin use cases)

Recent Changes

• Implemented hybrid authentication model: Users, teams, and API key creation are now publicly accessible
• Removed authentication requirements from user and team management endpoints for easier integration
• Migrated from Pinecone to self-hosted Qdrant
• Added Cloud Function for async image processing
• Implemented vector similarity search
• Added E2E testing infrastructure
• Set up VM for vector database hosting