# SEREACT - Secure Image Management API SEREACT is a secure API for storing, organizing, and retrieving images with advanced search capabilities powered by AI-generated embeddings. ## Features - Secure image storage in Google Cloud Storage - Team-based organization and access control - API key authentication - **Asynchronous image processing with Pub/Sub and Cloud Functions** - **AI-powered image embeddings using Google Cloud Vision API** - **Semantic search using vector similarity with Qdrant Vector Database** - **Self-hosted vector database on Google Compute Engine VM** - **Automatic retry mechanism for failed processing (up to 3 attempts)** - Metadata extraction and storage - Image processing capabilities - Multi-team support - **Comprehensive E2E testing with real database support** ## Architecture ``` sereact/ ├── images/ # Sample images for testing ├── deployment/ # Deployment configurations │ ├── cloud-function/ # **Cloud Function for image processing** │ ├── cloud-run/ # Google Cloud Run configuration │ └── terraform/ # Infrastructure as code │ ├── vm.tf # **Vector database VM configuration** │ └── scripts/ # **VM installation scripts** ├── docs/ # Documentation │ ├── api/ # API documentation │ └── TESTING.md # Comprehensive testing guide ├── scripts/ # Utility scripts ├── src/ # Source code │ ├── api/ # API endpoints and routers │ │ └── v1/ # API version 1 routes │ ├── auth/ # Authentication and authorization │ ├── config/ # Configuration management │ ├── core/ # Core application logic │ ├── db/ # Database layer │ │ ├── providers/ # Database providers (Firestore) │ │ └── repositories/ # Data access repositories │ ├── models/ # Database models │ ├── schemas/ # API request/response schemas │ ├── services/ # Business logic services │ │ ├── pubsub_service.py # **Pub/Sub message publishing** │ │ └── vector_db.py # **Qdrant vector database service** │ └── utils/ # Utility functions ├── tests/ # Test code │ ├── api/ # API tests │ ├── auth/ # Authentication tests │ ├── models/ # Model tests │ ├── services/ # Service tests │ ├── integration/ # Integration tests │ │ └── test_cloud_function.py # **Cloud Function tests** │ └── test_e2e.py # **Comprehensive E2E workflow tests** ├── main.py # Application entry point ├── requirements.txt # Python dependencies └── README.md # This file ``` ## System Architecture ``` ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ │ │ │ │ FastAPI │ ───────▶│ Firestore │◀────────│ Cloud │ │ Backend │ │ Database │ │ Functions │ │ │ │ │ │ │ └─────┬───────┘ └─────────────┘ └──────┬──────┘ │ │ │ │ ▼ │ ┌─────────────┐ ┌─────────────┐ │ │ │ │ │ │ │ Cloud │ │ Pub/Sub │ │ │ Storage │────────▶│ Queue │────────────────┘ │ │ │ │ └─────────────┘ └─────────────┘ │ │ ▼ ┌─────────────┐ ┌─────────────┐ │ │ │ │ │ Cloud │ │ Qdrant │ │ Vision API │────────▶│ Vector DB │ │ │ │ (VM) │ └─────────────┘ └─────────────┘ ``` ## **Image Processing Workflow** ### 1. **Image Upload Flow**: - User uploads image through FastAPI backend - Image is stored in Google Cloud Storage - Image metadata is saved to Firestore with `embedding_status: "pending"` - **Pub/Sub message is published to trigger async processing** ### 2. **Embedding Generation Flow** (Asynchronous): - **Cloud Function is triggered by Pub/Sub message** - Function updates image status to `"processing"` - **Function downloads image from Cloud Storage** - **Function calls Google Cloud Vision API to generate embeddings** - **Embeddings are stored in Qdrant Vector Database on dedicated VM** - **Firestore is updated with embedding info and status: "success"** ### 3. **Error Handling & Retry**: - **Failed processing updates status to "failed" with error message** - **Automatic retry up to 3 times using Pub/Sub retry policy** - **Dead letter queue for permanently failed messages** ### 4. **Search Flow**: - Search queries processed by FastAPI backend - Vector similarity search performed against Qdrant VM - Results combined with metadata from Firestore ## Technology Stack - **FastAPI** - Web framework - **Firestore** - Database - **Google Cloud Storage** - Image storage - **Google Pub/Sub** - Message queue for async processing - **Google Cloud Functions** - Serverless image processing - **Google Cloud Vision API** - AI-powered image analysis and embedding generation - **Qdrant** - Self-hosted vector database for semantic search (on Google Compute Engine VM) - **Google Compute Engine** - VM hosting for vector database - **Pydantic** - Data validation ## **Vector Database Infrastructure** ### **Qdrant Vector Database VM** The system includes a dedicated Google Compute Engine VM running Qdrant vector database: - **VM Specifications**: 2 vCPUs, 8GB RAM, 50GB disk (e2-standard-2) - **Operating System**: Ubuntu 22.04 LTS - **Vector Database**: Qdrant (latest version via Docker) - **Ports**: 6333 (HTTP API), 6334 (gRPC API) - **Features**: - Automatic installation and configuration via startup script - Daily automated backups - Health monitoring - Firewall configuration - Optional static IP assignment - API key authentication support ### **Vector Database Features** - **High Performance**: Optimized for image vector similarity search - **Scalable**: Can handle millions of image vectors - **Persistent Storage**: Data persisted on VM disk with automated backups - **RESTful API**: Easy integration with Python client - **Cosine Similarity**: Optimized for image embedding comparisons - **Metadata Filtering**: Support for complex search filters ## Setup and Installation ### Prerequisites - Python 3.8+ - Google Cloud account with Firestore, Storage, Pub/Sub, Cloud Functions, Compute Engine, and Vision API enabled - Terraform (for infrastructure deployment) ### Installation 1. Clone the repository: ```bash git clone https://github.com/yourusername/sereact.git cd sereact ``` 2. Create and activate a virtual environment: ```bash python -m venv venv source venv/bin/activate # Linux/macOS venv\Scripts\activate # Windows ``` 3. Install dependencies: ```bash pip install -r requirements.txt ``` 4. Create a `.env` file with the following environment variables: ``` # Firestore FIRESTORE_PROJECT_ID=your-gcp-project-id FIRESTORE_CREDENTIALS_FILE=path/to/firestore-credentials.json # Google Cloud Storage GCS_BUCKET_NAME=your-bucket-name GCS_CREDENTIALS_FILE=path/to/credentials.json # Google Pub/Sub PUBSUB_TOPIC=image-processing-topic PUBSUB_SUBSCRIPTION=image-processing-subscription # Google Cloud Vision VISION_API_ENABLED=true # Security API_KEY_SECRET=your-secret-key # Vector database (Qdrant) QDRANT_HOST=your-vm-external-ip QDRANT_API_KEY=your-qdrant-api-key # Optional ``` 5. **Deploy Infrastructure** (Required for vector database): ```bash # Configure Terraform variables cd deployment/terraform cp terraform.tfvars.example terraform.tfvars # Edit terraform.tfvars with your values # Deploy infrastructure including vector database VM terraform init terraform plan terraform apply # Note the output values for VM IP addresses ``` 6. **Deploy Cloud Function** (Optional - for production): ```bash cd ../cloud-function ./deploy.sh ``` 7. Run the application: ```bash uvicorn main:app --reload ``` 8. Visit `http://localhost:8000/docs` in your browser to access the API documentation. ## **Deployment** ### **Complete Infrastructure Deployment** Deploy the entire infrastructure including the vector database VM: ```bash cd deployment/terraform # Configure your variables cp terraform.tfvars.example terraform.tfvars # Edit terraform.tfvars with your specific values: # - project_id: Your GCP project ID # - storage_bucket_name: Unique bucket name # - qdrant_api_key: Secure API key for Qdrant (optional) # - allowed_cidr_blocks: Your IP address/range for security # - use_static_ip: Set to true for production # Deploy infrastructure terraform init terraform plan terraform apply ``` This will create: - **Google Compute Engine VM with Qdrant vector database** - **Firewall rules for vector database access** - **Service accounts and IAM bindings** - **Pub/Sub topic and subscription with retry policy** - **Cloud Storage bucket** - **Firestore database** - **Cloud Run service** ### **Vector Database VM Outputs** After deployment, Terraform will output: - `vector_db_vm_external_ip`: External IP address of the VM - `qdrant_http_endpoint`: HTTP API endpoint for Qdrant - `qdrant_grpc_endpoint`: gRPC API endpoint for Qdrant ### **Cloud Function Deployment** The image processing Cloud Function can be deployed using the provided script: ```bash cd deployment/cloud-function # Set environment variables export GOOGLE_CLOUD_PROJECT=your-project-id export QDRANT_HOST=your-vm-external-ip export QDRANT_API_KEY=your-qdrant-api-key # Deploy the function ./deploy.sh ``` ### **Vector Database Management** #### **Accessing the Vector Database** ```bash # SSH into the VM gcloud compute ssh sereact-vector-db --zone=us-central1-a # Check Qdrant status sudo systemctl status qdrant # View logs sudo journalctl -u qdrant -f # Run health check sudo /opt/qdrant/health_check.sh # Manual backup sudo /opt/qdrant/backup.sh ``` #### **Vector Database API Usage** ```python from src.services.vector_db import VectorDatabaseService # Initialize service vector_db = VectorDatabaseService( host="your-vm-external-ip", api_key="your-qdrant-api-key" # Optional ) # Add image vector point_id = vector_db.add_image_vector( image_id="img_123", vector=[0.1, 0.2, ...], # 512-dimensional vector metadata={"filename": "image.jpg", "size": 1024} ) # Search similar images results = vector_db.search_similar_images( query_vector=[0.1, 0.2, ...], limit=10, score_threshold=0.7 ) ``` ## API Endpoints The API provides the following main endpoints: - `/api/v1/auth/*` - Authentication and API key management - `/api/v1/teams/*` - Team management - `/api/v1/users/*` - User management - `/api/v1/images/*` - **Image upload, download, and management (with async processing)** - `/api/v1/search/*` - **Image search functionality (semantic search via Qdrant)** ### **Image Processing Status** Images now include embedding processing status: ```json { "id": "image-id", "filename": "example.jpg", "embedding_status": "success", // "pending", "processing", "success", "failed" "embedding_error": null, "embedding_retry_count": 0, "has_embedding": true } ``` Refer to the Swagger UI documentation at `/docs` for detailed endpoint information. ## Development ### Running Tests ```bash # Run all tests pytest # Run specific test categories pytest tests/services/test_pubsub_service.py # Pub/Sub service tests pytest tests/services/test_vector_db.py # Vector database tests pytest tests/integration/test_cloud_function.py # Cloud Function tests pytest tests/api/test_images_pubsub.py # API integration tests ``` ### **Comprehensive End-to-End Testing** SEREACT includes a comprehensive E2E testing suite that covers complete user workflows with **completely self-contained artificial test data**: ```bash # Run all E2E tests (completely self-contained - no setup required!) python scripts/run_tests.py e2e # Run unit tests only (fast) python scripts/run_tests.py unit # Run integration tests (requires real database) python scripts/run_tests.py integration ``` ## **Infrastructure Costs** ### **Estimated Monthly Costs (USD)** - **Compute Engine VM (e2-standard-2)**: ~$50-70/month - **Cloud Storage**: $0.02/GB/month - **Firestore**: $0.18/100K reads, $0.18/100K writes - **Pub/Sub**: $0.40/million messages - **Cloud Functions**: $0.40/million invocations - **Cloud Vision API**: $1.50/1000 images **Total estimated cost for moderate usage**: ~$60-100/month ### **Cost Optimization Tips** - Use preemptible VM instances for development (50-91% cost reduction) - Set up automatic VM shutdown during off-hours - Use regional persistent disks instead of SSD for cost savings - Monitor and set up billing alerts ## License This project is licensed under the MIT License - see the LICENSE file for details. ## API Modules Architecture The SEREACT API is organized into the following key modules to ensure separation of concerns and maintainable code: ``` src/ ├── api/ # API endpoints and routers │ └── v1/ # API version 1 routes ├── auth/ # Authentication and authorization ├── config/ # Configuration management ├── models/ # Database models ├── services/ # Business logic services │ └── vector_db.py # **Qdrant vector database service** └── utils/ # Utility functions ``` ### Module Responsibilities #### Router Module - Defines API endpoints and routes - Handles HTTP requests and responses - Validates incoming request data - Directs requests to appropriate services - Implements API versioning #### Auth Module - Manages user authentication - Handles API key validation and verification - Implements role-based access control - Provides security middleware - Manages user sessions and tokens #### Services Module - Contains core business logic - Orchestrates operations across multiple resources - Implements domain-specific rules and workflows - Integrates with external services (Cloud Vision, Storage, **Qdrant**) - Handles image processing and embedding generation #### Models Module - Defines data structures and schemas - Provides database entity representations - Handles data validation and serialization - Implements data relationships and constraints - Manages database migrations #### Utils Module - Provides helper functions and utilities - Implements common functionality used across modules - Handles error processing and logging - Provides formatting and conversion utilities - Implements reusable middleware components #### Config Module - Manages application configuration - Handles environment variable loading - Provides centralized settings management - Configures service connections and credentials - Defines application constants and defaults ### Module Interactions ``` ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ │ │ │ │ Router │ ───────▶│ Services │ ◀───────│ Config │ │ Module │ │ Module │ │ Module │ │ │ │ │ │ │ └──────┬──────┘ └──────┬──────┘ └─────────────┘ │ │ │ │ ▼ ▼ ┌─────────────┐ ┌─────────────┐ │ │ │ │ │ Auth │ │ Models │ │ Module │ │ Module │ │ │ │ │ └──────┬──────┘ └──────┬──────┘ │ │ │ │ └───────────────────────┘ │ ▼ ┌─────────────┐ │ │ │ Utils │ │ Module │ │ │ └─────────────┘ ``` The modules interact in the following ways: - **Request Flow**: - Client request arrives at the Router Module - Auth Module validates the request authentication - Router delegates to appropriate Service functions - Service uses Models to interact with the database - **Service integrates with Qdrant Vector Database for similarity search** - Service returns data to Router which formats the response - **Cross-Cutting Concerns**: - Config Module provides settings to all other modules - Utils Module provides helper functions across the application - Auth Module secures access to routes and services - **Dependency Direction**: - Router depends on Services and Auth - Services depend on Models and Config - Models depend on Utils for helper functions - Auth depends on Models for user information - All modules may use Utils and Config This modular architecture provides several benefits: - **Maintainability**: Changes in one module have minimal impact on others - **Testability**: Modules can be tested in isolation with mocked dependencies - **Scalability**: New features can be added by extending existing modules - **Reusability**: Common functionality is centralized for consistent implementation - **Security**: Authentication and authorization are handled consistently ## TODO ### High Priority - [ ] Remove Pinecone integration and complete Qdrant migration - [ ] Test and validate vector search functionality with Qdrant - [ ] Implement proper pagination for search results - [ ] Test Cloud Function image processing pipeline - [ ] Validate VM setup for self-hosted Qdrant instance ### Medium Priority - [ ] Add comprehensive logging for vector search operations - [ ] Implement caching layer for frequently accessed embeddings - [ ] Add monitoring and alerting for vector database performance - [ ] Create backup strategy for vector database - [ ] Document vector search API endpoints ### Low Priority - [ ] Add support for batch embedding generation - [ ] Implement embedding versioning - [ ] Add support for custom embedding models - [ ] Create visualization tools for embedding space - [ ] Add support for embedding fine-tuning ## Recent Changes - Migrated from Pinecone to self-hosted Qdrant - Added Cloud Function for async image processing - Implemented vector similarity search - Added E2E testing infrastructure - Set up VM for vector database hosting