johnpccd/image_management_api
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
image_management_api/README.md
johnpccd 11923229f6 cp
2025-05-24 20:13:11 +02:00

568 lines
20 KiB
Markdown
Raw Blame History

# SEREACT - Secure Image Management API
SEREACT is a secure API for storing, organizing, and retrieving images with advanced search capabilities powered by AI-generated embeddings.
## Features
- Secure image storage in Google Cloud Storage
- Team-based organization and access control
- API key authentication
- **Asynchronous image processing with Pub/Sub and Cloud Functions**
- **AI-powered image embeddings using Google Cloud Vision API**
- **Semantic search using vector similarity with Qdrant Vector Database**
- **Self-hosted vector database on Google Compute Engine VM**
- **Automatic retry mechanism for failed processing (up to 3 attempts)**
- Metadata extraction and storage
- Image processing capabilities
- Multi-team support
- **Comprehensive E2E testing with real database support**
## Architecture
```
sereact/
├── images/ # Sample images for testing
├── deployment/ # Deployment configurations
│ ├── cloud-function/ # **Cloud Function for image processing**
│ ├── cloud-run/ # Google Cloud Run configuration
│ └── terraform/ # Infrastructure as code
│ ├── vm.tf # **Vector database VM configuration**
│ └── scripts/ # **VM installation scripts**
├── docs/ # Documentation
│ ├── api/ # API documentation
│ └── TESTING.md # Comprehensive testing guide
├── scripts/ # Utility scripts
├── src/ # Source code
│ ├── api/ # API endpoints and routers
│ │ └── v1/ # API version 1 routes
│ ├── auth/ # Authentication and authorization
│ ├── config/ # Configuration management
│ ├── core/ # Core application logic
│ ├── db/ # Database layer
│ │ ├── providers/ # Database providers (Firestore)
│ │ └── repositories/ # Data access repositories
│ ├── models/ # Database models
│ ├── schemas/ # API request/response schemas
│ ├── services/ # Business logic services
│ │ ├── pubsub_service.py # **Pub/Sub message publishing**
│ │ └── vector_db.py # **Qdrant vector database service**
│ └── utils/ # Utility functions
├── tests/ # Test code
│ ├── api/ # API tests
│ ├── auth/ # Authentication tests
│ ├── models/ # Model tests
│ ├── services/ # Service tests
│ ├── integration/ # Integration tests
│ │ └── test_cloud_function.py # **Cloud Function tests**
│ └── test_e2e.py # **Comprehensive E2E workflow tests**
├── main.py # Application entry point
├── requirements.txt # Python dependencies
└── README.md # This file
```
## System Architecture
```
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ │ │ │ │ │
│ FastAPI │ ───────▶│ Firestore │◀────────│ Cloud │
│ Backend │ │ Database │ │ Functions │
│ │ │ │ │ │
└─────┬───────┘ └─────────────┘ └──────┬──────┘
│ │
│ │
▼ │
┌─────────────┐ ┌─────────────┐ │
│ │ │ │ │
│ Cloud │ │ Pub/Sub │ │
│ Storage │────────▶│ Queue │────────────────┘
│ │ │ │
└─────────────┘ └─────────────┘
│
│
▼
┌─────────────┐ ┌─────────────┐
│ │ │ │
│ Cloud │ │ Qdrant │
│ Vision API │────────▶│ Vector DB │
│ │ │ (VM) │
└─────────────┘ └─────────────┘
```
## **Image Processing Workflow**
### 1. **Image Upload Flow**:
- User uploads image through FastAPI backend
- Image is stored in Google Cloud Storage
- Image metadata is saved to Firestore with `embedding_status: "pending"`
- **Pub/Sub message is published to trigger async processing**
### 2. **Embedding Generation Flow** (Asynchronous):
- **Cloud Function is triggered by Pub/Sub message**
- Function updates image status to `"processing"`
- **Function downloads image from Cloud Storage**
- **Function calls Google Cloud Vision API to generate embeddings**
- **Embeddings are stored in Qdrant Vector Database on dedicated VM**
- **Firestore is updated with embedding info and status: "success"**
### 3. **Error Handling & Retry**:
- **Failed processing updates status to "failed" with error message**
- **Automatic retry up to 3 times using Pub/Sub retry policy**
- **Dead letter queue for permanently failed messages**
### 4. **Search Flow**:
- Search queries processed by FastAPI backend
- Vector similarity search performed against Qdrant VM
- Results combined with metadata from Firestore
## Technology Stack
- **FastAPI** - Web framework
- **Firestore** - Database
- **Google Cloud Storage** - Image storage
- **Google Pub/Sub** - Message queue for async processing
- **Google Cloud Functions** - Serverless image processing
- **Google Cloud Vision API** - AI-powered image analysis and embedding generation
- **Qdrant** - Self-hosted vector database for semantic search (on Google Compute Engine VM)
- **Google Compute Engine** - VM hosting for vector database
- **Pydantic** - Data validation
## **Vector Database Infrastructure**
### **Qdrant Vector Database VM**
The system includes a dedicated Google Compute Engine VM running Qdrant vector database:
- **VM Specifications**: 2 vCPUs, 8GB RAM, 50GB disk (e2-standard-2)
- **Operating System**: Ubuntu 22.04 LTS
- **Vector Database**: Qdrant (latest version via Docker)
- **Ports**: 6333 (HTTP API), 6334 (gRPC API)
- **Features**:
- Automatic installation and configuration via startup script
- Daily automated backups
- Health monitoring
- Firewall configuration
- Optional static IP assignment
- API key authentication support
### **Vector Database Features**
- **High Performance**: Optimized for image vector similarity search
- **Scalable**: Can handle millions of image vectors
- **Persistent Storage**: Data persisted on VM disk with automated backups
- **RESTful API**: Easy integration with Python client
- **Cosine Similarity**: Optimized for image embedding comparisons
- **Metadata Filtering**: Support for complex search filters
## Setup and Installation
### Prerequisites
- Python 3.8+
- Google Cloud account with Firestore, Storage, Pub/Sub, Cloud Functions, Compute Engine, and Vision API enabled
- Terraform (for infrastructure deployment)
### Installation
1. Clone the repository:
```bash
git clone https://github.com/yourusername/sereact.git
cd sereact
```
2. Create and activate a virtual environment:
```bash
python -m venv venv
source venv/bin/activate # Linux/macOS
venv\Scripts\activate # Windows
```
3. Install dependencies:
```bash
pip install -r requirements.txt
```
4. Create a `.env` file with the following environment variables:
```
# Firestore
FIRESTORE_PROJECT_ID=your-gcp-project-id
FIRESTORE_CREDENTIALS_FILE=path/to/firestore-credentials.json
# Google Cloud Storage
GCS_BUCKET_NAME=your-bucket-name
GCS_CREDENTIALS_FILE=path/to/credentials.json
# Google Pub/Sub
PUBSUB_TOPIC=image-processing-topic
PUBSUB_SUBSCRIPTION=image-processing-subscription
# Google Cloud Vision
VISION_API_ENABLED=true
# Security
API_KEY_SECRET=your-secret-key
# Vector database (Qdrant)
QDRANT_HOST=your-vm-external-ip
QDRANT_API_KEY=your-qdrant-api-key # Optional
```
5. **Deploy Infrastructure** (Required for vector database):
```bash
# Configure Terraform variables
cd deployment/terraform
cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars with your values
# Deploy infrastructure including vector database VM
terraform init
terraform plan
terraform apply
# Note the output values for VM IP addresses
```
6. **Deploy Cloud Function** (Optional - for production):
```bash
cd ../cloud-function
./deploy.sh
```
7. Run the application:
```bash
uvicorn main:app --reload
```
8. Visit `http://localhost:8000/docs` in your browser to access the API documentation.
## **Deployment**
### **Complete Infrastructure Deployment**
Deploy the entire infrastructure including the vector database VM:
```bash
cd deployment/terraform
# Configure your variables
cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars with your specific values:
# - project_id: Your GCP project ID
# - storage_bucket_name: Unique bucket name
# - qdrant_api_key: Secure API key for Qdrant (optional)
# - allowed_cidr_blocks: Your IP address/range for security
# - use_static_ip: Set to true for production
# Deploy infrastructure
terraform init
terraform plan
terraform apply
```
This will create:
- **Google Compute Engine VM with Qdrant vector database**
- **Firewall rules for vector database access**
- **Service accounts and IAM bindings**
- **Pub/Sub topic and subscription with retry policy**
- **Cloud Storage bucket**
- **Firestore database**
- **Cloud Run service**
### **Vector Database VM Outputs**
After deployment, Terraform will output:
- `vector_db_vm_external_ip`: External IP address of the VM
- `qdrant_http_endpoint`: HTTP API endpoint for Qdrant
- `qdrant_grpc_endpoint`: gRPC API endpoint for Qdrant
### **Cloud Function Deployment**
The image processing Cloud Function can be deployed using the provided script:
```bash
cd deployment/cloud-function
# Set environment variables
export GOOGLE_CLOUD_PROJECT=your-project-id
export QDRANT_HOST=your-vm-external-ip
export QDRANT_API_KEY=your-qdrant-api-key
# Deploy the function
./deploy.sh
```
### **Vector Database Management**
#### **Accessing the Vector Database**
```bash
# SSH into the VM
gcloud compute ssh sereact-vector-db --zone=us-central1-a
# Check Qdrant status
sudo systemctl status qdrant
# View logs
sudo journalctl -u qdrant -f
# Run health check
sudo /opt/qdrant/health_check.sh
# Manual backup
sudo /opt/qdrant/backup.sh
```
#### **Vector Database API Usage**
```python
from src.services.vector_db import VectorDatabaseService
# Initialize service
vector_db = VectorDatabaseService(
host="your-vm-external-ip",
api_key="your-qdrant-api-key" # Optional
)
# Add image vector
point_id = vector_db.add_image_vector(
image_id="img_123",
vector=[0.1, 0.2, ...], # 512-dimensional vector
metadata={"filename": "image.jpg", "size": 1024}
)
# Search similar images
results = vector_db.search_similar_images(
query_vector=[0.1, 0.2, ...],
limit=10,
score_threshold=0.7
)
```
## API Endpoints
The API provides the following main endpoints:
- `/api/v1/auth/*` - Authentication and API key management
- `/api/v1/teams/*` - Team management
- `/api/v1/users/*` - User management
- `/api/v1/images/*` - **Image upload, download, and management (with async processing)**
- `/api/v1/search/*` - **Image search functionality (semantic search via Qdrant)**
### **Image Processing Status**
Images now include embedding processing status:
```json
{
"id": "image-id",
"filename": "example.jpg",
"embedding_status": "success", // "pending", "processing", "success", "failed"
"embedding_error": null,
"embedding_retry_count": 0,
"has_embedding": true
}
```
Refer to the Swagger UI documentation at `/docs` for detailed endpoint information.
## Development
### Running Tests
```bash
# Run all tests
pytest
# Run specific test categories
pytest tests/services/test_pubsub_service.py # Pub/Sub service tests
pytest tests/services/test_vector_db.py # Vector database tests
pytest tests/integration/test_cloud_function.py # Cloud Function tests
pytest tests/api/test_images_pubsub.py # API integration tests
```
### **Comprehensive End-to-End Testing**
SEREACT includes a comprehensive E2E testing suite that covers complete user workflows with **completely self-contained artificial test data**:
```bash
# Run all E2E tests (completely self-contained - no setup required!)
python scripts/run_tests.py e2e
# Run unit tests only (fast)
python scripts/run_tests.py unit
# Run integration tests (requires real database)
python scripts/run_tests.py integration
```
## **Infrastructure Costs**
### **Estimated Monthly Costs (USD)**
- **Compute Engine VM (e2-standard-2)**: ~$50-70/month
- **Cloud Storage**: $0.02/GB/month
- **Firestore**: $0.18/100K reads, $0.18/100K writes
- **Pub/Sub**: $0.40/million messages
- **Cloud Functions**: $0.40/million invocations
- **Cloud Vision API**: $1.50/1000 images
**Total estimated cost for moderate usage**: ~$60-100/month
### **Cost Optimization Tips**
- Use preemptible VM instances for development (50-91% cost reduction)
- Set up automatic VM shutdown during off-hours
- Use regional persistent disks instead of SSD for cost savings
- Monitor and set up billing alerts
## License
This project is licensed under the MIT License - see the LICENSE file for details.
## API Modules Architecture
The SEREACT API is organized into the following key modules to ensure separation of concerns and maintainable code:
```
src/
├── api/ # API endpoints and routers
│ └── v1/ # API version 1 routes
├── auth/ # Authentication and authorization
├── config/ # Configuration management
├── models/ # Database models
├── services/ # Business logic services
│ └── vector_db.py # **Qdrant vector database service**
└── utils/ # Utility functions
```
### Module Responsibilities
#### Router Module
- Defines API endpoints and routes
- Handles HTTP requests and responses
- Validates incoming request data
- Directs requests to appropriate services
- Implements API versioning
#### Auth Module
- Manages user authentication
- Handles API key validation and verification
- Implements role-based access control
- Provides security middleware
- Manages user sessions and tokens
#### Services Module
- Contains core business logic
- Orchestrates operations across multiple resources
- Implements domain-specific rules and workflows
- Integrates with external services (Cloud Vision, Storage, **Qdrant**)
- Handles image processing and embedding generation
#### Models Module
- Defines data structures and schemas
- Provides database entity representations
- Handles data validation and serialization
- Implements data relationships and constraints
- Manages database migrations
#### Utils Module
- Provides helper functions and utilities
- Implements common functionality used across modules
- Handles error processing and logging
- Provides formatting and conversion utilities
- Implements reusable middleware components
#### Config Module
- Manages application configuration
- Handles environment variable loading
- Provides centralized settings management
- Configures service connections and credentials
- Defines application constants and defaults
### Module Interactions
```
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ │ │ │ │ │
│ Router │ ───────▶│ Services │ ◀───────│ Config │
│ Module │ │ Module │ │ Module │
│ │ │ │ │ │
└──────┬──────┘ └──────┬──────┘ └─────────────┘
│ │
│ │
▼ ▼
┌─────────────┐ ┌─────────────┐
│ │ │ │
│ Auth │ │ Models │
│ Module │ │ Module │
│ │ │ │
└──────┬──────┘ └──────┬──────┘
│ │
│ │
└───────────────────────┘
│
▼
┌─────────────┐
│ │
│ Utils │
│ Module │
│ │
└─────────────┘
```
The modules interact in the following ways:
- **Request Flow**:
- Client request arrives at the Router Module
- Auth Module validates the request authentication
- Router delegates to appropriate Service functions
- Service uses Models to interact with the database
- **Service integrates with Qdrant Vector Database for similarity search**
- Service returns data to Router which formats the response
- **Cross-Cutting Concerns**:
- Config Module provides settings to all other modules
- Utils Module provides helper functions across the application
- Auth Module secures access to routes and services
- **Dependency Direction**:
- Router depends on Services and Auth
- Services depend on Models and Config
- Models depend on Utils for helper functions
- Auth depends on Models for user information
- All modules may use Utils and Config
This modular architecture provides several benefits:
- **Maintainability**: Changes in one module have minimal impact on others
- **Testability**: Modules can be tested in isolation with mocked dependencies
- **Scalability**: New features can be added by extending existing modules
- **Reusability**: Common functionality is centralized for consistent implementation
- **Security**: Authentication and authorization are handled consistently
## TODO
### High Priority
- [ ] Remove Pinecone integration and complete Qdrant migration
- [ ] Test and validate vector search functionality with Qdrant
- [ ] Implement proper pagination for search results and all endpoints
- [ ] Test Cloud Function image processing pipeline
- [ ] Validate VM setup for self-hosted Qdrant instance
### Medium Priority
- [ ] Add comprehensive logging for vector search operations
- [ ] Implement caching layer for frequently accessed embeddings
- [ ] Implement caching for frequently accessed data
- [ ] Add monitoring and alerting for vector database performance
- [ ] Document vector search API endpoints
- [ ] Set up Qdrant cluster with multiple nodes
### Low Priority
## Recent Changes
- Migrated from Pinecone to self-hosted Qdrant
- Added Cloud Function for async image processing
- Implemented vector similarity search
- Added E2E testing infrastructure
- Set up VM for vector database hosting
Reference in New Issue View Git Blame Copy Permalink