johnpccd/image_management_api
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
image_management_api/README.md
johnpccd 4723dec85a cp
2025-05-24 17:31:54 +02:00

19 KiB
Raw Blame History

SEREACT - Secure Image Management API

SEREACT is a secure API for storing, organizing, and retrieving images with advanced search capabilities powered by AI-generated embeddings.

Features

  • Secure image storage in Google Cloud Storage
  • Team-based organization and access control
  • API key authentication
  • Asynchronous image processing with Pub/Sub and Cloud Functions
  • AI-powered image embeddings using Google Cloud Vision API
  • Semantic search using vector similarity with Qdrant Vector Database
  • Self-hosted vector database on Google Compute Engine VM
  • Automatic retry mechanism for failed processing (up to 3 attempts)
  • Metadata extraction and storage
  • Image processing capabilities
  • Multi-team support
  • Comprehensive E2E testing with real database support

Architecture

sereact/
  ├── images/                    # Sample images for testing
  ├── deployment/                # Deployment configurations
  │   ├── cloud-function/        # **Cloud Function for image processing**
  │   ├── cloud-run/             # Google Cloud Run configuration
  │   └── terraform/             # Infrastructure as code
  │       ├── vm.tf              # **Vector database VM configuration**
  │       └── scripts/           # **VM installation scripts**
  ├── docs/                      # Documentation
  │   ├── api/                   # API documentation
  │   └── TESTING.md             # Comprehensive testing guide
  ├── scripts/                   # Utility scripts
  ├── src/                       # Source code
  │   ├── api/                   # API endpoints and routers
  │   │   └── v1/                # API version 1 routes
  │   ├── auth/                  # Authentication and authorization
  │   ├── config/                # Configuration management
  │   ├── core/                  # Core application logic
  │   ├── db/                    # Database layer
  │   │   ├── providers/         # Database providers (Firestore)
  │   │   └── repositories/      # Data access repositories
  │   ├── models/                # Database models
  │   ├── schemas/               # API request/response schemas
  │   ├── services/              # Business logic services
  │   │   ├── pubsub_service.py  # **Pub/Sub message publishing**
  │   │   └── vector_db.py       # **Qdrant vector database service**
  │   └── utils/                 # Utility functions
  ├── tests/                     # Test code
  │   ├── api/                   # API tests
  │   ├── auth/                  # Authentication tests
  │   ├── models/                # Model tests
  │   ├── services/              # Service tests
  │   ├── integration/           # Integration tests
  │   │   └── test_cloud_function.py  # **Cloud Function tests**
  │   └── test_e2e.py           # **Comprehensive E2E workflow tests**
  ├── main.py                    # Application entry point
  ├── requirements.txt           # Python dependencies
  └── README.md                  # This file

System Architecture

┌─────────────┐         ┌─────────────┐         ┌─────────────┐
│             │         │             │         │             │
│  FastAPI    │ ───────▶│  Firestore  │◀────────│  Cloud      │
│  Backend    │         │  Database   │         │  Functions  │
│             │         │             │         │             │
└─────┬───────┘         └─────────────┘         └──────┬──────┘
      │                                                │
      │                                                │
      ▼                                                │
┌─────────────┐         ┌─────────────┐                │
│             │         │             │                │
│  Cloud      │         │  Pub/Sub    │                │
│  Storage    │────────▶│  Queue      │────────────────┘
│             │         │             │
└─────────────┘         └─────────────┘
                               │
                               │
                               ▼
                        ┌─────────────┐         ┌─────────────┐
                        │             │         │             │
                        │  Cloud      │         │  Qdrant     │
                        │  Vision API │────────▶│  Vector DB  │
                        │             │         │  (VM)       │
                        └─────────────┘         └─────────────┘

Image Processing Workflow

1. Image Upload Flow:

  • User uploads image through FastAPI backend
  • Image is stored in Google Cloud Storage
  • Image metadata is saved to Firestore with embedding_status: "pending"
  • Pub/Sub message is published to trigger async processing

2. Embedding Generation Flow (Asynchronous):

  • Cloud Function is triggered by Pub/Sub message
  • Function updates image status to "processing"
  • Function downloads image from Cloud Storage
  • Function calls Google Cloud Vision API to generate embeddings
  • Embeddings are stored in Qdrant Vector Database on dedicated VM
  • Firestore is updated with embedding info and status: "success"

3. Error Handling & Retry:

  • Failed processing updates status to "failed" with error message
  • Automatic retry up to 3 times using Pub/Sub retry policy
  • Dead letter queue for permanently failed messages

4. Search Flow:

  • Search queries processed by FastAPI backend
  • Vector similarity search performed against Qdrant VM
  • Results combined with metadata from Firestore

Technology Stack

  • FastAPI - Web framework
  • Firestore - Database
  • Google Cloud Storage - Image storage
  • Google Pub/Sub - Message queue for async processing
  • Google Cloud Functions - Serverless image processing
  • Google Cloud Vision API - AI-powered image analysis and embedding generation
  • Qdrant - Self-hosted vector database for semantic search (on Google Compute Engine VM)
  • Google Compute Engine - VM hosting for vector database
  • Pydantic - Data validation

Vector Database Infrastructure

Qdrant Vector Database VM

The system includes a dedicated Google Compute Engine VM running Qdrant vector database:

  • VM Specifications: 2 vCPUs, 8GB RAM, 50GB disk (e2-standard-2)
  • Operating System: Ubuntu 22.04 LTS
  • Vector Database: Qdrant (latest version via Docker)
  • Ports: 6333 (HTTP API), 6334 (gRPC API)
  • Features:
    • Automatic installation and configuration via startup script
    • Daily automated backups
    • Health monitoring
    • Firewall configuration
    • Optional static IP assignment
    • API key authentication support

Vector Database Features

  • High Performance: Optimized for image vector similarity search
  • Scalable: Can handle millions of image vectors
  • Persistent Storage: Data persisted on VM disk with automated backups
  • RESTful API: Easy integration with Python client
  • Cosine Similarity: Optimized for image embedding comparisons
  • Metadata Filtering: Support for complex search filters

Setup and Installation

Prerequisites

  • Python 3.8+
  • Google Cloud account with Firestore, Storage, Pub/Sub, Cloud Functions, Compute Engine, and Vision API enabled
  • Terraform (for infrastructure deployment)

Installation

  1. Clone the repository:

    git clone https://github.com/yourusername/sereact.git
cd sereact

  2. Create and activate a virtual environment:

    python -m venv venv
source venv/bin/activate  # Linux/macOS
venv\Scripts\activate     # Windows

  3. Install dependencies:

    pip install -r requirements.txt

  4. Create a .env file with the following environment variables:

    # Firestore
FIRESTORE_PROJECT_ID=your-gcp-project-id
FIRESTORE_CREDENTIALS_FILE=path/to/firestore-credentials.json

# Google Cloud Storage
GCS_BUCKET_NAME=your-bucket-name
GCS_CREDENTIALS_FILE=path/to/credentials.json

# Google Pub/Sub
PUBSUB_TOPIC=image-processing-topic
PUBSUB_SUBSCRIPTION=image-processing-subscription

# Google Cloud Vision
VISION_API_ENABLED=true

# Security
API_KEY_SECRET=your-secret-key

# Vector database (Qdrant)
QDRANT_HOST=your-vm-external-ip
QDRANT_API_KEY=your-qdrant-api-key  # Optional

  5. Deploy Infrastructure (Required for vector database):

    # Configure Terraform variables
cd deployment/terraform
cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars with your values

# Deploy infrastructure including vector database VM
terraform init
terraform plan
terraform apply

# Note the output values for VM IP addresses

  6. Deploy Cloud Function (Optional - for production):

    cd ../cloud-function
./deploy.sh

  7. Run the application:

    uvicorn main:app --reload

  8. Visit http://localhost:8000/docs in your browser to access the API documentation.

Deployment

Complete Infrastructure Deployment

Deploy the entire infrastructure including the vector database VM:

cd deployment/terraform

# Configure your variables
cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars with your specific values:
# - project_id: Your GCP project ID
# - storage_bucket_name: Unique bucket name
# - qdrant_api_key: Secure API key for Qdrant (optional)
# - allowed_cidr_blocks: Your IP address/range for security
# - use_static_ip: Set to true for production

# Deploy infrastructure
terraform init
terraform plan
terraform apply

This will create:

  • Google Compute Engine VM with Qdrant vector database
  • Firewall rules for vector database access
  • Service accounts and IAM bindings
  • Pub/Sub topic and subscription with retry policy
  • Cloud Storage bucket
  • Firestore database
  • Cloud Run service

Vector Database VM Outputs

After deployment, Terraform will output:

  • vector_db_vm_external_ip: External IP address of the VM
  • qdrant_http_endpoint: HTTP API endpoint for Qdrant
  • qdrant_grpc_endpoint: gRPC API endpoint for Qdrant

Cloud Function Deployment

The image processing Cloud Function can be deployed using the provided script:

cd deployment/cloud-function

# Set environment variables
export GOOGLE_CLOUD_PROJECT=your-project-id
export QDRANT_HOST=your-vm-external-ip
export QDRANT_API_KEY=your-qdrant-api-key

# Deploy the function
./deploy.sh

Vector Database Management

Accessing the Vector Database

# SSH into the VM
gcloud compute ssh sereact-vector-db --zone=us-central1-a

# Check Qdrant status
sudo systemctl status qdrant

# View logs
sudo journalctl -u qdrant -f

# Run health check
sudo /opt/qdrant/health_check.sh

# Manual backup
sudo /opt/qdrant/backup.sh

Vector Database API Usage

from src.services.vector_db import VectorDatabaseService

# Initialize service
vector_db = VectorDatabaseService(
    host="your-vm-external-ip",
    api_key="your-qdrant-api-key"  # Optional
)

# Add image vector
point_id = vector_db.add_image_vector(
    image_id="img_123",
    vector=[0.1, 0.2, ...],  # 512-dimensional vector
    metadata={"filename": "image.jpg", "size": 1024}
)

# Search similar images
results = vector_db.search_similar_images(
    query_vector=[0.1, 0.2, ...],
    limit=10,
    score_threshold=0.7
)

API Endpoints

The API provides the following main endpoints:

  • /api/v1/auth/* - Authentication and API key management
  • /api/v1/teams/* - Team management
  • /api/v1/users/* - User management
  • /api/v1/images/* - Image upload, download, and management (with async processing)
  • /api/v1/search/* - Image search functionality (semantic search via Qdrant)

Image Processing Status

Images now include embedding processing status:

{
  "id": "image-id",
  "filename": "example.jpg",
  "embedding_status": "success",  // "pending", "processing", "success", "failed"
  "embedding_error": null,
  "embedding_retry_count": 0,
  "has_embedding": true
}

Refer to the Swagger UI documentation at /docs for detailed endpoint information.

Development

Running Tests

# Run all tests
pytest

# Run specific test categories
pytest tests/services/test_pubsub_service.py  # Pub/Sub service tests
pytest tests/services/test_vector_db.py       # Vector database tests
pytest tests/integration/test_cloud_function.py  # Cloud Function tests
pytest tests/api/test_images_pubsub.py        # API integration tests

Comprehensive End-to-End Testing

SEREACT includes a comprehensive E2E testing suite that covers complete user workflows with completely self-contained artificial test data:

# Run all E2E tests (completely self-contained - no setup required!)
python scripts/run_tests.py e2e

# Run unit tests only (fast)
python scripts/run_tests.py unit

# Run integration tests (requires real database)
python scripts/run_tests.py integration

Infrastructure Costs

Estimated Monthly Costs (USD)

  • Compute Engine VM (e2-standard-2): ~$50-70/month
  • Cloud Storage: $0.02/GB/month
  • Firestore: $0.18/100K reads, $0.18/100K writes
  • Pub/Sub: $0.40/million messages
  • Cloud Functions: $0.40/million invocations
  • Cloud Vision API: $1.50/1000 images

Total estimated cost for moderate usage: ~$60-100/month

Cost Optimization Tips

  • Use preemptible VM instances for development (50-91% cost reduction)
  • Set up automatic VM shutdown during off-hours
  • Use regional persistent disks instead of SSD for cost savings
  • Monitor and set up billing alerts

License

This project is licensed under the MIT License - see the LICENSE file for details.

API Modules Architecture

The SEREACT API is organized into the following key modules to ensure separation of concerns and maintainable code:

src/
  ├── api/             # API endpoints and routers
  │   └── v1/          # API version 1 routes
  ├── auth/            # Authentication and authorization
  ├── config/          # Configuration management
  ├── models/          # Database models
  ├── services/        # Business logic services
  │   └── vector_db.py # **Qdrant vector database service**
  └── utils/           # Utility functions

Module Responsibilities

Router Module

  • Defines API endpoints and routes
  • Handles HTTP requests and responses
  • Validates incoming request data
  • Directs requests to appropriate services
  • Implements API versioning

Auth Module

  • Manages user authentication
  • Handles API key validation and verification
  • Implements role-based access control
  • Provides security middleware
  • Manages user sessions and tokens

Services Module

  • Contains core business logic
  • Orchestrates operations across multiple resources
  • Implements domain-specific rules and workflows
  • Integrates with external services (Cloud Vision, Storage, Qdrant)
  • Handles image processing and embedding generation

Models Module

  • Defines data structures and schemas
  • Provides database entity representations
  • Handles data validation and serialization
  • Implements data relationships and constraints
  • Manages database migrations

Utils Module

  • Provides helper functions and utilities
  • Implements common functionality used across modules
  • Handles error processing and logging
  • Provides formatting and conversion utilities
  • Implements reusable middleware components

Config Module

  • Manages application configuration
  • Handles environment variable loading
  • Provides centralized settings management
  • Configures service connections and credentials
  • Defines application constants and defaults

Module Interactions

┌─────────────┐         ┌─────────────┐         ┌─────────────┐
│             │         │             │         │             │
│  Router     │ ───────▶│  Services   │ ◀───────│  Config     │
│  Module     │         │  Module     │         │  Module     │
│             │         │             │         │             │
└──────┬──────┘         └──────┬──────┘         └─────────────┘
       │                       │
       │                       │
       ▼                       ▼
┌─────────────┐         ┌─────────────┐
│             │         │             │
│  Auth       │         │  Models     │
│  Module     │         │  Module     │
│             │         │             │
└──────┬──────┘         └──────┬──────┘
       │                       │
       │                       │
       └───────────────────────┘
                 │
                 ▼
          ┌─────────────┐
          │             │
          │  Utils      │
          │  Module     │
          │             │
          └─────────────┘

The modules interact in the following ways:

  • Request Flow:

    • Client request arrives at the Router Module
    • Auth Module validates the request authentication
    • Router delegates to appropriate Service functions
    • Service uses Models to interact with the database
    • Service integrates with Qdrant Vector Database for similarity search
    • Service returns data to Router which formats the response

  • Cross-Cutting Concerns:

    • Config Module provides settings to all other modules
    • Utils Module provides helper functions across the application
    • Auth Module secures access to routes and services

  • Dependency Direction:

    • Router depends on Services and Auth
    • Services depend on Models and Config
    • Models depend on Utils for helper functions
    • Auth depends on Models for user information
    • All modules may use Utils and Config

This modular architecture provides several benefits:

  • Maintainability: Changes in one module have minimal impact on others
  • Testability: Modules can be tested in isolation with mocked dependencies
  • Scalability: New features can be added by extending existing modules
  • Reusability: Common functionality is centralized for consistent implementation
  • Security: Authentication and authorization are handled consistently