diff --git a/README.md b/README.md index da2b2d5..6938783 100644 --- a/README.md +++ b/README.md @@ -205,116 +205,11 @@ Uses Google's Vertex AI multimodal embedding model for generating high-quality i QDRANT_PREFER_GRPC=false ``` -5. **Deploy Infrastructure** (Required for vector database): +5. **Deploy Infrastructure** ```bash - # Configure Terraform variables - cd deployment/terraform - cp terraform.tfvars.example terraform.tfvars - # Edit terraform.tfvars with your values - - # Deploy infrastructure including vector database VM - terraform init - terraform plan - terraform apply - - # Note the output values for VM IP addresses + ./deployment/deploy.sh --build --deploy ``` -6. **Deploy Cloud Function** (Optional - for production): - ```bash - cd ../cloud-function - ./deploy.sh - ``` - -7. Run the application: - ```bash - uvicorn main:app --reload - ``` - -8. Visit `http://localhost:8000/docs` in your browser to access the API documentation. - -## **Deployment** - -### **Complete Infrastructure Deployment** - -Deploy the entire infrastructure including the vector database VM: - -```bash -cd deployment/terraform - -# Configure your variables -cp terraform.tfvars.example terraform.tfvars -# Edit terraform.tfvars with your specific values: -# - project_id: Your GCP project ID -# - storage_bucket_name: Unique bucket name -# - qdrant_api_key: Secure API key for Qdrant (optional) -# - allowed_cidr_blocks: Your IP address/range for security -# - use_static_ip: Set to true for production - -# Deploy infrastructure -terraform init -terraform plan -terraform apply -``` - -This will create: -- **Google Compute Engine VM with Qdrant vector database** -- **Firewall rules for vector database access** -- **Service accounts and IAM bindings** -- **Pub/Sub topic and subscription with retry policy** -- **Cloud Storage bucket** -- **Firestore database** -- **Cloud Run service** - -### **Vector Database VM Outputs** - -After deployment, Terraform will output: -- `vector_db_vm_external_ip`: External IP address of the VM -- `qdrant_http_endpoint`: HTTP API endpoint for Qdrant -- `qdrant_grpc_endpoint`: gRPC API endpoint for Qdrant - -### **Cloud Function Deployment** - -The image processing Cloud Function can be deployed using the provided script: - -```bash -cd deployment/cloud-function - -# Set environment variables -export GOOGLE_CLOUD_PROJECT=your-project-id -export QDRANT_HOST=your-vm-external-ip -export QDRANT_API_KEY=your-qdrant-api-key - -# Deploy the function -./deploy.sh -``` - - -#### **Vector Database API Usage** - -```python -from src.services.vector_db import VectorDatabaseService - -# Initialize service -vector_db = VectorDatabaseService( - host="your-vm-external-ip", - api_key="your-qdrant-api-key" # Optional -) - -# Add image vector -point_id = vector_db.add_image_vector( - image_id="img_123", - vector=[0.1, 0.2, ...], # 1408-dimensional vector - metadata={"filename": "image.jpg", "size": 1024} -) - -# Search similar images -results = vector_db.search_similar_images( - query_vector=[0.1, 0.2, ...], - limit=10, - score_threshold=0.7 -) -``` ## API Endpoints @@ -392,21 +287,6 @@ A **hybrid authentication model**: **Note:** Public endpoints (users, teams) don't implement pagination as they typically return small datasets and are designed for management use cases where full data visibility is preferred. -### **Image Processing Status** - -Images now include embedding processing status: - -```json -{ - "id": "image-id", - "filename": "example.jpg", - "embedding_status": "success", // "pending", "processing", "success", "failed" - "embedding_error": null, - "embedding_retry_count": 0, - "has_embedding": true -} -``` - Refer to the Swagger UI documentation at `/docs` for detailed endpoint information. ## Development @@ -414,55 +294,9 @@ Refer to the Swagger UI documentation at `/docs` for detailed endpoint informati ### Running Tests ```bash -# Run all tests -pytest - -# Run specific test categories -pytest tests/services/test_pubsub_service.py # Pub/Sub service tests -pytest tests/services/test_vector_db.py # Vector database tests -pytest tests/integration/test_cloud_function.py # Cloud Function tests -pytest tests/api/test_images_pubsub.py # API integration tests +source venv/Scripts/activate && python scripts/run_tests.py all ``` -### **Comprehensive End-to-End Testing** - -SEREACT includes a comprehensive E2E testing suite that covers complete user workflows with **completely self-contained artificial test data**: - -```bash -# Run all E2E tests (completely self-contained - no setup required!) -python scripts/run_tests.py e2e - -# Run unit tests only (fast) -python scripts/run_tests.py unit - -# Run integration tests (requires real database) -python scripts/run_tests.py integration -``` - -## **Infrastructure Costs** - -### **Estimated Monthly Costs (USD)** - -- **Compute Engine VM (e2-standard-2)**: ~$50-70/month -- **Cloud Storage**: $0.02/GB/month -- **Firestore**: $0.18/100K reads, $0.18/100K writes -- **Pub/Sub**: $0.40/million messages -- **Cloud Functions**: $0.40/million invocations -- **Cloud Vision API**: $1.50/1000 images - -**Total estimated cost for moderate usage**: ~$60-100/month - -### **Cost Optimization Tips** - -- Use preemptible VM instances for development (50-91% cost reduction) -- Set up automatic VM shutdown during off-hours -- Use regional persistent disks instead of SSD for cost savings -- Monitor and set up billing alerts - -## License - -This project is licensed under the MIT License - see the LICENSE file for details. - ## API Modules Architecture The SEREACT API is organized into the following key modules to ensure separation of concerns and maintainable code: @@ -587,7 +421,6 @@ This modular architecture provides several benefits: ### High Priority - [ ] Remove Pinecone integration and complete Qdrant migration - [ ] Test and validate vector search functionality with Qdrant -- [x] ~~Implement proper pagination for search results and all endpoints~~ **COMPLETED** - Images and Search APIs have full pagination - [ ] Test Cloud Function image processing pipeline - [ ] Validate VM setup for self-hosted Qdrant instance @@ -605,17 +438,3 @@ This modular architecture provides several benefits: - [ ] Move all auth logic to auth module - [ ] Move cloud function code to src folder and reuse code with embedding service - [ ] Thumbnail generation - -### Pagination Status ✅ -- **✅ Images API**: Fully implemented with `skip`, `limit`, `total` parameters -- **✅ Search API**: Fully implemented with `limit`, `total`, similarity scoring -- **ℹ️ Users/Teams/API Keys**: No pagination (small datasets, admin use cases) - -## Recent Changes -- **Implemented hybrid authentication model**: Users, teams, and API key creation are now publicly accessible -- **Removed authentication requirements** from user and team management endpoints for easier integration -- Migrated from Pinecone to self-hosted Qdrant -- Added Cloud Function for async image processing -- Implemented vector similarity search -- Added E2E testing infrastructure -- Set up VM for vector database hosting diff --git a/deployment/terraform/variables.tf b/deployment/terraform/variables.tf index 67eda76..c545c0c 100644 --- a/deployment/terraform/variables.tf +++ b/deployment/terraform/variables.tf @@ -58,18 +58,6 @@ variable "qdrant_api_key" { default = "" } -variable "vector_db_environment" { - description = "Vector database environment (e.g., gcp-starter for Pinecone)" - type = string - default = "gcp-starter" -} - -variable "vector_db_index_name" { - description = "Vector database index name for image embeddings" - type = string - default = "image-embeddings" -} - variable "allowed_cidr_blocks" { description = "CIDR blocks allowed to access the vector database" type = string diff --git a/requirements.txt b/requirements.txt index 23a9e1e..135a652 100644 --- a/requirements.txt +++ b/requirements.txt @@ -13,6 +13,7 @@ python-jose==3.3.0 passlib==1.7.4 tenacity==8.2.3 pytest==7.4.3 +pytest-asyncio==0.21.1 httpx==0.25.1 pillow==10.1.0 python-slugify==8.0.1 diff --git a/src/services/embedding_service.py b/src/services/embedding_service.py index 99ba27d..5255a24 100644 --- a/src/services/embedding_service.py +++ b/src/services/embedding_service.py @@ -117,34 +117,6 @@ class EmbeddingService: logger.error(f"Error generating text embedding: {e}") return None - async def process_image_async(self, image_id: str, storage_path: str) -> bool: - """ - Process image asynchronously to generate embeddings - - Args: - image_id: Image ID - storage_path: Path to image in storage - - Returns: - True if processing started successfully - """ - try: - # In a real implementation, this would: - # 1. Publish a message to Pub/Sub queue - # 2. Cloud Function would pick up the message - # 3. Generate embeddings using Cloud Vision API - # 4. Store embeddings in Pinecone - # 5. Update image record with embedding info - - logger.info(f"Starting async processing for image {image_id} at {storage_path}") - - # For now, just log that processing would start - # In production, this would integrate with Google Pub/Sub - return True - except Exception as e: - logger.error(f"Error starting async image processing: {e}") - return False - async def delete_embedding(self, embedding_id: str) -> bool: """ Delete embedding from vector database