SEREACT - Secure Image Management API

SEREACT is a secure API for storing, organizing, and retrieving images with advanced search capabilities powered by AI-generated embeddings.

Features

• Secure image storage in Google Cloud Storage
• Team-based organization and access control
• API key authentication
• Asynchronous image processing with Pub/Sub and Cloud Functions
• AI-powered image embeddings using Google Cloud Vision API
• Semantic search using vector similarity with Qdrant Vector Database
• Self-hosted vector database on Google Compute Engine VM
• Automatic retry mechanism for failed processing (up to 3 attempts)
• Metadata extraction and storage
• Image processing capabilities
• Multi-team support
• Comprehensive E2E testing with real database support

Architecture

sereact/
  ├── images/                    # Sample images for testing
  ├── deployment/                # Deployment configurations
  │   ├── cloud-function/        # **Cloud Function for image processing**
  │   ├── cloud-run/             # Google Cloud Run configuration
  │   └── terraform/             # Infrastructure as code
  │       ├── vm.tf              # **Vector database VM configuration**
  │       └── scripts/           # **VM installation scripts**
  ├── docs/                      # Documentation
  │   ├── api/                   # API documentation
  │   └── TESTING.md             # Comprehensive testing guide
  ├── scripts/                   # Utility scripts
  ├── src/                       # Source code
  │   ├── api/                   # API endpoints and routers
  │   │   └── v1/                # API version 1 routes
  │   ├── auth/                  # Authentication and authorization
  │   ├── config/                # Configuration management
  │   ├── core/                  # Core application logic
  │   ├── db/                    # Database layer
  │   │   ├── providers/         # Database providers (Firestore)
  │   │   └── repositories/      # Data access repositories
  │   ├── models/                # Database models
  │   ├── schemas/               # API request/response schemas
  │   ├── services/              # Business logic services
  │   │   ├── pubsub_service.py  # **Pub/Sub message publishing**
  │   │   └── vector_db.py       # **Qdrant vector database service**
  │   └── utils/                 # Utility functions
  ├── tests/                     # Test code
  │   ├── api/                   # API tests
  │   ├── auth/                  # Authentication tests
  │   ├── models/                # Model tests
  │   ├── services/              # Service tests
  │   ├── integration/           # Integration tests
  │   │   └── test_cloud_function.py  # **Cloud Function tests**
  │   └── test_e2e.py           # **Comprehensive E2E workflow tests**
  ├── main.py                    # Application entry point
  ├── requirements.txt           # Python dependencies
  └── README.md                  # This file

System Architecture

┌─────────────┐         ┌─────────────┐         ┌─────────────┐
│             │         │             │         │             │
│  FastAPI    │ ───────▶│  Firestore  │◀────────│  Cloud      │
│  Backend    │         │  Database   │         │  Functions  │
│             │         │             │         │             │
└─────┬───────┘         └─────────────┘         └──────┬──────┘
      │                                                │
      │                                                │
      ▼                                                │
┌─────────────┐         ┌─────────────┐                │
│             │         │             │                │
│  Cloud      │         │  Pub/Sub    │                │
│  Storage    │────────▶│  Queue      │────────────────┘
│             │         │             │
└─────────────┘         └─────────────┘
                               │
                               │
                               ▼
                        ┌─────────────┐         ┌─────────────┐
                        │             │         │             │
                        │  Cloud      │         │  Qdrant     │
                        │  Vision API │────────▶│  Vector DB  │
                        │             │         │  (VM)       │
                        └─────────────┘         └─────────────┘

Image Processing Workflow

1. Image Upload Flow:

• User uploads image through FastAPI backend
• Image is stored in Google Cloud Storage
• Image metadata is saved to Firestore with embedding_status: "pending"
• Pub/Sub message is published to trigger async processing

2. Embedding Generation Flow (Asynchronous):

• Cloud Function is triggered by Pub/Sub message
• Function updates image status to "processing"
• Function downloads image from Cloud Storage
• Function calls Google Cloud Vision API to generate embeddings
• Embeddings are stored in Qdrant Vector Database on dedicated VM
• Firestore is updated with embedding info and status: "success"

3. Error Handling & Retry:

• Failed processing updates status to "failed" with error message
• Automatic retry up to 3 times using Pub/Sub retry policy
• Dead letter queue for permanently failed messages

4. Search Flow:

• Search queries processed by FastAPI backend
• Vector similarity search performed against Qdrant VM
• Results combined with metadata from Firestore

Technology Stack

• FastAPI - Web framework
• Firestore - Database
• Google Cloud Storage - Image storage
• Google Pub/Sub - Message queue for async processing
• Google Cloud Functions - Serverless image processing
• Google Cloud Vision API - AI-powered image analysis and embedding generation
• Qdrant - Self-hosted vector database for semantic search (on Google Compute Engine VM)
• Google Compute Engine - VM hosting for vector database
• Pydantic - Data validation

Vector Database Infrastructure

Qdrant Vector Database VM

The system includes a dedicated Google Compute Engine VM running Qdrant vector database:

• VM Specifications: 2 vCPUs, 8GB RAM, 50GB disk (e2-standard-2)
• Operating System: Ubuntu 22.04 LTS
• Vector Database: Qdrant (latest version via Docker)
• Ports: 6333 (HTTP API), 6334 (gRPC API)
• Features:
  • Automatic installation and configuration via startup script
  • Daily automated backups
  • Health monitoring
  • Firewall configuration
  • Optional static IP assignment
  • API key authentication support

Vector Database Features

• High Performance: Optimized for image vector similarity search
• Scalable: Can handle millions of image vectors
• Persistent Storage: Data persisted on VM disk with automated backups
• RESTful API: Easy integration with Python client
• Cosine Similarity: Optimized for image embedding comparisons
• Metadata Filtering: Support for complex search filters

Setup and Installation

Prerequisites

• Python 3.8+
• Google Cloud account with Firestore, Storage, Pub/Sub, Cloud Functions, Compute Engine, and Vision API enabled
• Terraform (for infrastructure deployment)

Installation

1. Clone the repository:

   git clone https://github.com/yourusername/sereact.git
   cd sereact
   

2. Create and activate a virtual environment:

   python -m venv venv
   source venv/bin/activate  # Linux/macOS
   venv\Scripts\activate     # Windows
   

3. Install dependencies:

   pip install -r requirements.txt
   

4. Create a .env file with the following environment variables:

   # Firestore
   FIRESTORE_PROJECT_ID=your-gcp-project-id
   FIRESTORE_CREDENTIALS_FILE=path/to/firestore-credentials.json
   
   # Google Cloud Storage
   GCS_BUCKET_NAME=your-bucket-name
   GCS_CREDENTIALS_FILE=path/to/credentials.json
   
   # Google Pub/Sub
   PUBSUB_TOPIC=image-processing-topic
   PUBSUB_SUBSCRIPTION=image-processing-subscription
   
   # Google Cloud Vision
   VISION_API_ENABLED=true
   
   # Security
   API_KEY_SECRET=your-secret-key
   
   # Vector database (Qdrant)
   QDRANT_HOST=your-vm-external-ip
   QDRANT_API_KEY=your-qdrant-api-key  # Optional
   

5. Deploy Infrastructure (Required for vector database):

   # Configure Terraform variables
   cd deployment/terraform
   cp terraform.tfvars.example terraform.tfvars
   # Edit terraform.tfvars with your values
   
   # Deploy infrastructure including vector database VM
   terraform init
   terraform plan
   terraform apply
   
   # Note the output values for VM IP addresses
   

6. Deploy Cloud Function (Optional - for production):

   cd ../cloud-function
   ./deploy.sh
   

7. Run the application:

   uvicorn main:app --reload
   

8. Visit http://localhost:8000/docs in your browser to access the API documentation.

Deployment

Complete Infrastructure Deployment

Deploy the entire infrastructure including the vector database VM:

cd deployment/terraform

# Configure your variables
cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars with your specific values:
# - project_id: Your GCP project ID
# - storage_bucket_name: Unique bucket name
# - qdrant_api_key: Secure API key for Qdrant (optional)
# - allowed_cidr_blocks: Your IP address/range for security
# - use_static_ip: Set to true for production

# Deploy infrastructure
terraform init
terraform plan
terraform apply

This will create:

• Google Compute Engine VM with Qdrant vector database
• Firewall rules for vector database access
• Service accounts and IAM bindings
• Pub/Sub topic and subscription with retry policy
• Cloud Storage bucket
• Firestore database
• Cloud Run service

Vector Database VM Outputs

After deployment, Terraform will output:

• vector_db_vm_external_ip: External IP address of the VM
• qdrant_http_endpoint: HTTP API endpoint for Qdrant
• qdrant_grpc_endpoint: gRPC API endpoint for Qdrant

Cloud Function Deployment

The image processing Cloud Function can be deployed using the provided script:

cd deployment/cloud-function

# Set environment variables
export GOOGLE_CLOUD_PROJECT=your-project-id
export QDRANT_HOST=your-vm-external-ip
export QDRANT_API_KEY=your-qdrant-api-key

# Deploy the function
./deploy.sh

Vector Database Management

Accessing the Vector Database

# SSH into the VM
gcloud compute ssh sereact-vector-db --zone=us-central1-a

# Check Qdrant status
sudo systemctl status qdrant

# View logs
sudo journalctl -u qdrant -f

# Run health check
sudo /opt/qdrant/health_check.sh

# Manual backup
sudo /opt/qdrant/backup.sh

Vector Database API Usage

from src.services.vector_db import VectorDatabaseService

# Initialize service
vector_db = VectorDatabaseService(
    host="your-vm-external-ip",
    api_key="your-qdrant-api-key"  # Optional
)

# Add image vector
point_id = vector_db.add_image_vector(
    image_id="img_123",
    vector=[0.1, 0.2, ...],  # 512-dimensional vector
    metadata={"filename": "image.jpg", "size": 1024}
)

# Search similar images
results = vector_db.search_similar_images(
    query_vector=[0.1, 0.2, ...],
    limit=10,
    score_threshold=0.7
)

API Endpoints

The API provides the following main endpoints:

• /api/v1/auth/* - Authentication and API key management
• /api/v1/teams/* - Team management
• /api/v1/users/* - User management
• /api/v1/images/* - Image upload, download, and management (with async processing)
• /api/v1/search/* - Image search functionality (semantic search via Qdrant)

Image Processing Status

Images now include embedding processing status:

{
  "id": "image-id",
  "filename": "example.jpg",
  "embedding_status": "success",  // "pending", "processing", "success", "failed"
  "embedding_error": null,
  "embedding_retry_count": 0,
  "has_embedding": true
}

Refer to the Swagger UI documentation at /docs for detailed endpoint information.

Development

Running Tests

# Run all tests
pytest

# Run specific test categories
pytest tests/services/test_pubsub_service.py  # Pub/Sub service tests
pytest tests/services/test_vector_db.py       # Vector database tests
pytest tests/integration/test_cloud_function.py  # Cloud Function tests
pytest tests/api/test_images_pubsub.py        # API integration tests

Comprehensive End-to-End Testing

SEREACT includes a comprehensive E2E testing suite that covers complete user workflows with completely self-contained artificial test data:

# Run all E2E tests (completely self-contained - no setup required!)
python scripts/run_tests.py e2e

# Run unit tests only (fast)
python scripts/run_tests.py unit

# Run integration tests (requires real database)
python scripts/run_tests.py integration

Infrastructure Costs

Estimated Monthly Costs (USD)

• Compute Engine VM (e2-standard-2): ~$50-70/month
• Cloud Storage: $0.02/GB/month
• Firestore: $0.18/100K reads, $0.18/100K writes
• Pub/Sub: $0.40/million messages
• Cloud Functions: $0.40/million invocations
• Cloud Vision API: $1.50/1000 images

Total estimated cost for moderate usage: ~$60-100/month

Cost Optimization Tips

• Use preemptible VM instances for development (50-91% cost reduction)
• Set up automatic VM shutdown during off-hours
• Use regional persistent disks instead of SSD for cost savings
• Monitor and set up billing alerts

License

This project is licensed under the MIT License - see the LICENSE file for details.

API Modules Architecture

The SEREACT API is organized into the following key modules to ensure separation of concerns and maintainable code:

src/
  ├── api/             # API endpoints and routers
  │   └── v1/          # API version 1 routes
  ├── auth/            # Authentication and authorization
  ├── config/          # Configuration management
  ├── models/          # Database models
  ├── services/        # Business logic services
  │   └── vector_db.py # **Qdrant vector database service**
  └── utils/           # Utility functions

Module Responsibilities

Router Module

• Defines API endpoints and routes
• Handles HTTP requests and responses
• Validates incoming request data
• Directs requests to appropriate services
• Implements API versioning

Auth Module

• Manages user authentication
• Handles API key validation and verification
• Implements role-based access control
• Provides security middleware
• Manages user sessions and tokens

Services Module

• Contains core business logic
• Orchestrates operations across multiple resources
• Implements domain-specific rules and workflows
• Integrates with external services (Cloud Vision, Storage, Qdrant)
• Handles image processing and embedding generation

Models Module

• Defines data structures and schemas
• Provides database entity representations
• Handles data validation and serialization
• Implements data relationships and constraints
• Manages database migrations

Utils Module

• Provides helper functions and utilities
• Implements common functionality used across modules
• Handles error processing and logging
• Provides formatting and conversion utilities
• Implements reusable middleware components

Config Module

• Manages application configuration
• Handles environment variable loading
• Provides centralized settings management
• Configures service connections and credentials
• Defines application constants and defaults

Module Interactions

┌─────────────┐         ┌─────────────┐         ┌─────────────┐
│             │         │             │         │             │
│  Router     │ ───────▶│  Services   │ ◀───────│  Config     │
│  Module     │         │  Module     │         │  Module     │
│             │         │             │         │             │
└──────┬──────┘         └──────┬──────┘         └─────────────┘
       │                       │
       │                       │
       ▼                       ▼
┌─────────────┐         ┌─────────────┐
│             │         │             │
│  Auth       │         │  Models     │
│  Module     │         │  Module     │
│             │         │             │
└──────┬──────┘         └──────┬──────┘
       │                       │
       │                       │
       └───────────────────────┘
                 │
                 ▼
          ┌─────────────┐
          │             │
          │  Utils      │
          │  Module     │
          │             │
          └─────────────┘

The modules interact in the following ways:

• Request Flow:

  • Client request arrives at the Router Module
  • Auth Module validates the request authentication
  • Router delegates to appropriate Service functions
  • Service uses Models to interact with the database
  • Service integrates with Qdrant Vector Database for similarity search
  • Service returns data to Router which formats the response

• Cross-Cutting Concerns:

  • Config Module provides settings to all other modules
  • Utils Module provides helper functions across the application
  • Auth Module secures access to routes and services

• Dependency Direction:

  • Router depends on Services and Auth
  • Services depend on Models and Config
  • Models depend on Utils for helper functions
  • Auth depends on Models for user information
  • All modules may use Utils and Config

This modular architecture provides several benefits:

• Maintainability: Changes in one module have minimal impact on others
• Testability: Modules can be tested in isolation with mocked dependencies
• Scalability: New features can be added by extending existing modules
• Reusability: Common functionality is centralized for consistent implementation
• Security: Authentication and authorization are handled consistently

TODO

High Priority

• Remove Pinecone integration and complete Qdrant migration
• Test and validate vector search functionality with Qdrant
• Implement proper pagination for search results
• Test Cloud Function image processing pipeline
• Validate VM setup for self-hosted Qdrant instance

Medium Priority

• Add comprehensive logging for vector search operations
• Implement caching layer for frequently accessed embeddings
• Add monitoring and alerting for vector database performance
• Create backup strategy for vector database
• Document vector search API endpoints

Low Priority

• Add support for batch embedding generation
• Implement embedding versioning
• Add support for custom embedding models
• Create visualization tools for embedding space
• Add support for embedding fine-tuning

Recent Changes

• Migrated from Pinecone to self-hosted Qdrant
• Added Cloud Function for async image processing
• Implemented vector similarity search
• Added E2E testing infrastructure
• Set up VM for vector database hosting