image_management_api/README.md

# SEREACT - Secure Image Management API

SEREACT is a secure API for storing, organizing, and retrieving images with advanced search capabilities powered by AI-generated embeddings.

## Features

- Secure image storage in Google Cloud Storage
- Team-based organization and access control
- API key authentication
- **Asynchronous image processing with Pub/Sub and Cloud Functions**
- **AI-powered image embeddings using Google Cloud Vision API**
- **Semantic search using vector similarity with Qdrant Vector Database**
- **Self-hosted vector database on Google Compute Engine VM**
- **Automatic retry mechanism for failed processing (up to 3 attempts)**
- Metadata extraction and storage
- Image processing capabilities
- Multi-team support
- **Comprehensive E2E testing with real database support**

## Architecture

```
sereact/
  ├── images/                    # Sample images for testing
  ├── deployment/                # Deployment configurations
  │   ├── cloud-function/        # **Cloud Function for image processing**
  │   ├── cloud-run/             # Google Cloud Run configuration
  │   └── terraform/             # Infrastructure as code
  │       ├── vm.tf              # **Vector database VM configuration**
  │       └── scripts/           # **VM installation scripts**
  ├── docs/                      # Documentation
  │   ├── api/                   # API documentation
  │   └── TESTING.md             # Comprehensive testing guide
  ├── scripts/                   # Utility scripts
  ├── src/                       # Source code
  │   ├── api/                   # API endpoints and routers
  │   │   └── v1/                # API version 1 routes
  │   ├── auth/                  # Authentication and authorization
  │   ├── config/                # Configuration management
  │   ├── core/                  # Core application logic
  │   ├── db/                    # Database layer
  │   │   ├── providers/         # Database providers (Firestore)
  │   │   └── repositories/      # Data access repositories
  │   ├── models/                # Database models
  │   ├── schemas/               # API request/response schemas
  │   ├── services/              # Business logic services
  │   │   ├── pubsub_service.py  # **Pub/Sub message publishing**
  │   │   └── vector_db.py       # **Qdrant vector database service**
  │   └── utils/                 # Utility functions
  ├── tests/                     # Test code
  │   ├── api/                   # API tests
  │   ├── auth/                  # Authentication tests
  │   ├── models/                # Model tests
  │   ├── services/              # Service tests
  │   ├── integration/           # Integration tests
  │   │   └── test_cloud_function.py  # **Cloud Function tests**
  │   └── test_e2e.py           # **Comprehensive E2E workflow tests**
  ├── main.py                    # Application entry point
  ├── requirements.txt           # Python dependencies
  └── README.md                  # This file
```

## System Architecture

```
┌─────────────┐         ┌─────────────┐         ┌─────────────┐
│             │         │             │         │             │
│  FastAPI    │ ───────▶│  Firestore  │◀────────│  Cloud      │
│  Backend    │         │  Database   │         │  Functions  │
│             │         │             │         │             │
└─────┬───────┘         └─────────────┘         └──────┬──────┘
      │                                                │
      │                                                │
      ▼                                                │
┌─────────────┐         ┌─────────────┐                │
│             │         │             │                │
│  Cloud      │         │  Pub/Sub    │                │
│  Storage    │────────▶│  Queue      │────────────────┘
│             │         │             │
└─────────────┘         └─────────────┘
                               │
                               │
                               ▼
                        ┌─────────────┐         ┌─────────────┐
                        │             │         │             │
                        │  Cloud      │         │  Qdrant     │
                        │  Vision API │────────▶│  Vector DB  │
                        │             │         │  (VM)       │
                        └─────────────┘         └─────────────┘
```

## **Image Processing Workflow**

### 1. **Image Upload Flow**:
   - User uploads image through FastAPI backend
   - Image is stored in Google Cloud Storage
   - Image metadata is saved to Firestore with `embedding_status: "pending"`
   - **Pub/Sub message is published to trigger async processing**

### 2. **Embedding Generation Flow** (Asynchronous):
   - **Cloud Function is triggered by Pub/Sub message**
   - Function updates image status to `"processing"`
   - **Function downloads image from Cloud Storage**
   - **Function calls Google Cloud Vision API to generate embeddings**
   - **Embeddings are stored in Qdrant Vector Database on dedicated VM**
   - **Firestore is updated with embedding info and status: "success"**

### 3. **Error Handling & Retry**:
   - **Failed processing updates status to "failed" with error message**
   - **Automatic retry up to 3 times using Pub/Sub retry policy**
   - **Dead letter queue for permanently failed messages**

### 4. **Search Flow**:
   - Search queries processed by FastAPI backend
   - Vector similarity search performed against Qdrant VM
   - Results combined with metadata from Firestore

## Technology Stack

- **FastAPI** - Web framework
- **Firestore** - Database
- **Google Cloud Storage** - Image storage
- **Google Pub/Sub** - Message queue for async processing
- **Google Cloud Functions** - Serverless image processing
- **Google Cloud Vision API** - AI-powered image analysis and embedding generation
- **Qdrant** - Self-hosted vector database for semantic search (on Google Compute Engine VM)
- **Google Compute Engine** - VM hosting for vector database
- **Pydantic** - Data validation

## **Vector Database Infrastructure**

### **Qdrant Vector Database VM**

The system includes a dedicated Google Compute Engine VM running Qdrant vector database:

- **VM Specifications**: 2 vCPUs, 8GB RAM, 50GB disk (e2-standard-2)
- **Operating System**: Ubuntu 22.04 LTS
- **Vector Database**: Qdrant (latest version via Docker)
- **Ports**: 6333 (HTTP API), 6334 (gRPC API)
- **Features**:
  - Automatic installation and configuration via startup script
  - Daily automated backups
  - Health monitoring
  - Firewall configuration
  - Optional static IP assignment
  - API key authentication support

### **Vector Database Features**

- **High Performance**: Optimized for image vector similarity search
- **Scalable**: Can handle millions of image vectors
- **Persistent Storage**: Data persisted on VM disk with automated backups
- **RESTful API**: Easy integration with Python client
- **Cosine Similarity**: Optimized for image embedding comparisons
- **Metadata Filtering**: Support for complex search filters

## Setup and Installation

### Prerequisites

- Python 3.8+
- Google Cloud account with Firestore, Storage, Pub/Sub, Cloud Functions, Compute Engine, and Vision API enabled
- Terraform (for infrastructure deployment)

### Installation

1. Clone the repository:
   ```bash
   git clone https://github.com/yourusername/sereact.git
   cd sereact
   ```

2. Create and activate a virtual environment:
   ```bash
   python -m venv venv
   source venv/bin/activate  # Linux/macOS
   venv\Scripts\activate     # Windows
   ```

3. Install dependencies:
   ```bash
   pip install -r requirements.txt
   ```

4. Create a `.env` file with the following environment variables:
   ```
   # Firestore
   FIRESTORE_PROJECT_ID=your-gcp-project-id
   FIRESTORE_CREDENTIALS_FILE=path/to/firestore-credentials.json
   
   # Google Cloud Storage
   GCS_BUCKET_NAME=your-bucket-name
   GCS_CREDENTIALS_FILE=path/to/credentials.json
   
   # Google Pub/Sub
   PUBSUB_TOPIC=image-processing-topic
   PUBSUB_SUBSCRIPTION=image-processing-subscription
   
   # Google Cloud Vision
   VISION_API_ENABLED=true
   
   # Security
   API_KEY_SECRET=your-secret-key
   
   # Vector database (Qdrant)
   QDRANT_HOST=your-vm-external-ip
   QDRANT_API_KEY=your-qdrant-api-key  # Optional
   ```

5. **Deploy Infrastructure** (Required for vector database):
   ```bash
   # Configure Terraform variables
   cd deployment/terraform
   cp terraform.tfvars.example terraform.tfvars
   # Edit terraform.tfvars with your values
   
   # Deploy infrastructure including vector database VM
   terraform init
   terraform plan
   terraform apply
   
   # Note the output values for VM IP addresses
   ```

6. **Deploy Cloud Function** (Optional - for production):
   ```bash
   cd ../cloud-function
   ./deploy.sh
   ```

7. Run the application:
   ```bash
   uvicorn main:app --reload
   ```

8. Visit `http://localhost:8000/docs` in your browser to access the API documentation.

## **Deployment**

### **Complete Infrastructure Deployment**

Deploy the entire infrastructure including the vector database VM:

```bash
cd deployment/terraform

# Configure your variables
cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars with your specific values:
# - project_id: Your GCP project ID
# - storage_bucket_name: Unique bucket name
# - qdrant_api_key: Secure API key for Qdrant (optional)
# - allowed_cidr_blocks: Your IP address/range for security
# - use_static_ip: Set to true for production

# Deploy infrastructure
terraform init
terraform plan
terraform apply
```

This will create:
- **Google Compute Engine VM with Qdrant vector database**
- **Firewall rules for vector database access**
- **Service accounts and IAM bindings**
- **Pub/Sub topic and subscription with retry policy**
- **Cloud Storage bucket**
- **Firestore database**
- **Cloud Run service**

### **Vector Database VM Outputs**

After deployment, Terraform will output:
- `vector_db_vm_external_ip`: External IP address of the VM
- `qdrant_http_endpoint`: HTTP API endpoint for Qdrant
- `qdrant_grpc_endpoint`: gRPC API endpoint for Qdrant

### **Cloud Function Deployment**

The image processing Cloud Function can be deployed using the provided script:

```bash
cd deployment/cloud-function

# Set environment variables
export GOOGLE_CLOUD_PROJECT=your-project-id
export QDRANT_HOST=your-vm-external-ip
export QDRANT_API_KEY=your-qdrant-api-key

# Deploy the function
./deploy.sh
```

### **Vector Database Management**

#### **Accessing the Vector Database**

```bash
# SSH into the VM
gcloud compute ssh sereact-vector-db --zone=us-central1-a

# Check Qdrant status
sudo systemctl status qdrant

# View logs
sudo journalctl -u qdrant -f

# Run health check
sudo /opt/qdrant/health_check.sh

# Manual backup
sudo /opt/qdrant/backup.sh
```

#### **Vector Database API Usage**

```python
from src.services.vector_db import VectorDatabaseService

# Initialize service
vector_db = VectorDatabaseService(
    host="your-vm-external-ip",
    api_key="your-qdrant-api-key"  # Optional
)

# Add image vector
point_id = vector_db.add_image_vector(
    image_id="img_123",
    vector=[0.1, 0.2, ...],  # 512-dimensional vector
    metadata={"filename": "image.jpg", "size": 1024}
)

# Search similar images
results = vector_db.search_similar_images(
    query_vector=[0.1, 0.2, ...],
    limit=10,
    score_threshold=0.7
)
```

## API Endpoints

The API provides the following main endpoints with their pagination support:

### Authentication & Authorization
- `/api/v1/auth/*` - Authentication and API key management
  - `GET /api/v1/auth/api-keys` - List API keys (no pagination - returns all keys for user)

### Team Management  
- `/api/v1/teams/*` - Team management
  - `GET /api/v1/teams` - List teams (no pagination - admin only, returns all teams)

### User Management
- `/api/v1/users/*` - User management  
  - `GET /api/v1/users` - List users (no pagination - returns all users in team/organization)

### Image Management ✅ **Fully Paginated**
- `/api/v1/images/*` - **Image upload, download, and management (with async processing)**
  - `GET /api/v1/images` - List images with **full pagination support**
    - **Query Parameters:**
      - `skip` (default: 0, min: 0) - Number of items to skip
      - `limit` (default: 50, min: 1, max: 100) - Number of items per page
      - `collection_id` (optional) - Filter by collection
      - `tags` (optional) - Filter by comma-separated tags
    - **Response includes:** `images`, `total`, `skip`, `limit`

### Search Functionality ✅ **Fully Paginated**
- `/api/v1/search/*` - **Image search functionality (semantic search via Qdrant)**
  - `GET /api/v1/search` - Search images with **pagination support**
    - **Query Parameters:**
      - `q` (required) - Search query
      - `limit` (default: 10, min: 1, max: 50) - Number of results
      - `threshold` (default: 0.7, min: 0.0, max: 1.0) - Similarity threshold
      - `collection_id` (optional) - Filter by collection
      - `tags` (optional) - Filter by comma-separated tags
    - **Response includes:** `results`, `total`, `limit`, `threshold`, `query`
  - `POST /api/v1/search` - Advanced search with same pagination
  - `GET /api/v1/search/similar/{image_id}` - Find similar images with pagination

### Pagination Implementation Status

| Endpoint | Pagination Status | Notes |
|----------|------------------|-------|
| `GET /api/v1/images` | ✅ **Fully Implemented** | `skip`, `limit`, `total` with proper validation |
| `GET /api/v1/search` | ✅ **Fully Implemented** | `limit`, `total` with similarity scoring |
| `GET /api/v1/users` | ❌ **Not Implemented** | Returns all users (typically small datasets) |
| `GET /api/v1/teams` | ❌ **Not Implemented** | Returns all teams (typically small datasets) |
| `GET /api/v1/auth/api-keys` | ❌ **Not Implemented** | Returns all keys for user (typically small datasets) |

**Note:** The endpoints without pagination (users, teams, API keys) typically return small datasets and are designed for admin/management use cases where full data visibility is preferred.

### **Image Processing Status**

Images now include embedding processing status:

```json
{
  "id": "image-id",
  "filename": "example.jpg",
  "embedding_status": "success",  // "pending", "processing", "success", "failed"
  "embedding_error": null,
  "embedding_retry_count": 0,
  "has_embedding": true
}
```

Refer to the Swagger UI documentation at `/docs` for detailed endpoint information.

## Development

### Running Tests

```bash
# Run all tests
pytest

# Run specific test categories
pytest tests/services/test_pubsub_service.py  # Pub/Sub service tests
pytest tests/services/test_vector_db.py       # Vector database tests
pytest tests/integration/test_cloud_function.py  # Cloud Function tests
pytest tests/api/test_images_pubsub.py        # API integration tests
```

### **Comprehensive End-to-End Testing**

SEREACT includes a comprehensive E2E testing suite that covers complete user workflows with **completely self-contained artificial test data**:

```bash
# Run all E2E tests (completely self-contained - no setup required!)
python scripts/run_tests.py e2e

# Run unit tests only (fast)
python scripts/run_tests.py unit

# Run integration tests (requires real database)
python scripts/run_tests.py integration
```

## **Infrastructure Costs**

### **Estimated Monthly Costs (USD)**

- **Compute Engine VM (e2-standard-2)**: ~$50-70/month
- **Cloud Storage**: $0.02/GB/month
- **Firestore**: $0.18/100K reads, $0.18/100K writes
- **Pub/Sub**: $0.40/million messages
- **Cloud Functions**: $0.40/million invocations
- **Cloud Vision API**: $1.50/1000 images

**Total estimated cost for moderate usage**: ~$60-100/month

### **Cost Optimization Tips**

- Use preemptible VM instances for development (50-91% cost reduction)
- Set up automatic VM shutdown during off-hours
- Use regional persistent disks instead of SSD for cost savings
- Monitor and set up billing alerts

## License

This project is licensed under the MIT License - see the LICENSE file for details.

## API Modules Architecture

The SEREACT API is organized into the following key modules to ensure separation of concerns and maintainable code:

```
src/
  ├── api/             # API endpoints and routers
  │   └── v1/          # API version 1 routes
  ├── auth/            # Authentication and authorization
  ├── config/          # Configuration management
  ├── models/          # Database models
  ├── services/        # Business logic services
  │   └── vector_db.py # **Qdrant vector database service**
  └── utils/           # Utility functions
```

### Module Responsibilities

#### Router Module
- Defines API endpoints and routes
- Handles HTTP requests and responses
- Validates incoming request data
- Directs requests to appropriate services
- Implements API versioning

#### Auth Module
- Manages user authentication
- Handles API key validation and verification
- Implements role-based access control
- Provides security middleware
- Manages user sessions and tokens

#### Services Module
- Contains core business logic
- Orchestrates operations across multiple resources
- Implements domain-specific rules and workflows
- Integrates with external services (Cloud Vision, Storage, **Qdrant**)
- Handles image processing and embedding generation

#### Models Module
- Defines data structures and schemas
- Provides database entity representations
- Handles data validation and serialization
- Implements data relationships and constraints
- Manages database migrations

#### Utils Module
- Provides helper functions and utilities
- Implements common functionality used across modules
- Handles error processing and logging
- Provides formatting and conversion utilities
- Implements reusable middleware components

#### Config Module
- Manages application configuration
- Handles environment variable loading
- Provides centralized settings management
- Configures service connections and credentials
- Defines application constants and defaults

### Module Interactions

```
┌─────────────┐         ┌─────────────┐         ┌─────────────┐
│             │         │             │         │             │
│  Router     │ ───────▶│  Services   │ ◀───────│  Config     │
│  Module     │         │  Module     │         │  Module     │
│             │         │             │         │             │
└──────┬──────┘         └──────┬──────┘         └─────────────┘
       │                       │
       │                       │
       ▼                       ▼
┌─────────────┐         ┌─────────────┐
│             │         │             │
│  Auth       │         │  Models     │
│  Module     │         │  Module     │
│             │         │             │
└──────┬──────┘         └──────┬──────┘
       │                       │
       │                       │
       └───────────────────────┘
                 │
                 ▼
          ┌─────────────┐
          │             │
          │  Utils      │
          │  Module     │
          │             │
          └─────────────┘
```

The modules interact in the following ways:
- **Request Flow**:
  - Client request arrives at the Router Module
  - Auth Module validates the request authentication
  - Router delegates to appropriate Service functions
  - Service uses Models to interact with the database
  - **Service integrates with Qdrant Vector Database for similarity search**
  - Service returns data to Router which formats the response

- **Cross-Cutting Concerns**:
  - Config Module provides settings to all other modules
  - Utils Module provides helper functions across the application
  - Auth Module secures access to routes and services

- **Dependency Direction**:
  - Router depends on Services and Auth
  - Services depend on Models and Config
  - Models depend on Utils for helper functions
  - Auth depends on Models for user information
  - All modules may use Utils and Config

This modular architecture provides several benefits:
- **Maintainability**: Changes in one module have minimal impact on others
- **Testability**: Modules can be tested in isolation with mocked dependencies
- **Scalability**: New features can be added by extending existing modules
- **Reusability**: Common functionality is centralized for consistent implementation
- **Security**: Authentication and authorization are handled consistently

## TODO

### High Priority
- [ ] Remove Pinecone integration and complete Qdrant migration
- [ ] Test and validate vector search functionality with Qdrant
- [x] ~~Implement proper pagination for search results and all endpoints~~ **COMPLETED** - Images and Search APIs have full pagination
- [ ] Test Cloud Function image processing pipeline
- [ ] Validate VM setup for self-hosted Qdrant instance

### Medium Priority
- [ ] Add comprehensive logging for vector search operations
- [ ] Implement caching layer for frequently accessed embeddings
- [ ] Implement caching for frequently accessed data
- [ ] Add monitoring and alerting for vector database performance
- [ ] Document vector search API endpoints
- [ ] Set up Qdrant cluster with multiple nodes
- [ ] Consider adding pagination to admin endpoints (users, teams, API keys) if datasets grow large

### Low Priority
- [ ] Terraform dependencies

### Pagination Status ✅
- **✅ Images API**: Fully implemented with `skip`, `limit`, `total` parameters
- **✅ Search API**: Fully implemented with `limit`, `total`, similarity scoring
- **ℹ️ Users/Teams/API Keys**: No pagination (small datasets, admin use cases)

## Recent Changes
- Migrated from Pinecone to self-hosted Qdrant
- Added Cloud Function for async image processing
- Implemented vector similarity search
- Added E2E testing infrastructure
- Set up VM for vector database hosting