cleanup

README.md

@@ -205,116 +205,11 @@ Uses Google's Vertex AI multimodal embedding model for generating high-quality i
     QDRANT_PREFER_GRPC=false
    ```
 
-5. **Deploy Infrastructure** (Required for vector database):
+5. **Deploy Infrastructure**
    ```bash
-   # Configure Terraform variables
+   ./deployment/deploy.sh --build --deploy
-   cd deployment/terraform
-   cp terraform.tfvars.example terraform.tfvars
-   # Edit terraform.tfvars with your values
-   
-   # Deploy infrastructure including vector database VM
-   terraform init
-   terraform plan
-   terraform apply
-   
-   # Note the output values for VM IP addresses
    ```
 
-6. **Deploy Cloud Function** (Optional - for production):
-   ```bash
-   cd ../cloud-function
-   ./deploy.sh
-   ```
-
-7. Run the application:
-   ```bash
-   uvicorn main:app --reload
-   ```
-
-8. Visit `http://localhost:8000/docs` in your browser to access the API documentation.
-
-## **Deployment**
-
-### **Complete Infrastructure Deployment**
-
-Deploy the entire infrastructure including the vector database VM:
-
-```bash
-cd deployment/terraform
-
-# Configure your variables
-cp terraform.tfvars.example terraform.tfvars
-# Edit terraform.tfvars with your specific values:
-# - project_id: Your GCP project ID
-# - storage_bucket_name: Unique bucket name
-# - qdrant_api_key: Secure API key for Qdrant (optional)
-# - allowed_cidr_blocks: Your IP address/range for security
-# - use_static_ip: Set to true for production
-
-# Deploy infrastructure
-terraform init
-terraform plan
-terraform apply
-```
-
-This will create:
-- **Google Compute Engine VM with Qdrant vector database**
-- **Firewall rules for vector database access**
-- **Service accounts and IAM bindings**
-- **Pub/Sub topic and subscription with retry policy**
-- **Cloud Storage bucket**
-- **Firestore database**
-- **Cloud Run service**
-
-### **Vector Database VM Outputs**
-
-After deployment, Terraform will output:
-- `vector_db_vm_external_ip`: External IP address of the VM
-- `qdrant_http_endpoint`: HTTP API endpoint for Qdrant
-- `qdrant_grpc_endpoint`: gRPC API endpoint for Qdrant
-
-### **Cloud Function Deployment**
-
-The image processing Cloud Function can be deployed using the provided script:
-
-```bash
-cd deployment/cloud-function
-
-# Set environment variables
-export GOOGLE_CLOUD_PROJECT=your-project-id
-export QDRANT_HOST=your-vm-external-ip
-export QDRANT_API_KEY=your-qdrant-api-key
-
-# Deploy the function
-./deploy.sh
-```
-
-
-#### **Vector Database API Usage**
-
-```python
-from src.services.vector_db import VectorDatabaseService
-
-# Initialize service
-vector_db = VectorDatabaseService(
-    host="your-vm-external-ip",
-    api_key="your-qdrant-api-key"  # Optional
-)
-
-# Add image vector
-point_id = vector_db.add_image_vector(
-    image_id="img_123",
-    vector=[0.1, 0.2, ...],  # 1408-dimensional vector
-    metadata={"filename": "image.jpg", "size": 1024}
-)
-
-# Search similar images
-results = vector_db.search_similar_images(
-    query_vector=[0.1, 0.2, ...],
-    limit=10,
-    score_threshold=0.7
-)
-```
 
 ## API Endpoints
 
@@ -392,21 +287,6 @@ A **hybrid authentication model**:
 
 **Note:** Public endpoints (users, teams) don't implement pagination as they typically return small datasets and are designed for management use cases where full data visibility is preferred.
 
-### **Image Processing Status**
-
-Images now include embedding processing status:
-
-```json
-{
-  "id": "image-id",
-  "filename": "example.jpg",
-  "embedding_status": "success",  // "pending", "processing", "success", "failed"
-  "embedding_error": null,
-  "embedding_retry_count": 0,
-  "has_embedding": true
-}
-```
-
 Refer to the Swagger UI documentation at `/docs` for detailed endpoint information.
 
 ## Development
@@ -414,55 +294,9 @@ Refer to the Swagger UI documentation at `/docs` for detailed endpoint informati
 ### Running Tests
 
 ```bash
-# Run all tests
+source venv/Scripts/activate && python scripts/run_tests.py all
-pytest
-
-# Run specific test categories
-pytest tests/services/test_pubsub_service.py  # Pub/Sub service tests
-pytest tests/services/test_vector_db.py       # Vector database tests
-pytest tests/integration/test_cloud_function.py  # Cloud Function tests
-pytest tests/api/test_images_pubsub.py        # API integration tests
 ```
 
-### **Comprehensive End-to-End Testing**
-
-SEREACT includes a comprehensive E2E testing suite that covers complete user workflows with **completely self-contained artificial test data**:
-
-```bash
-# Run all E2E tests (completely self-contained - no setup required!)
-python scripts/run_tests.py e2e
-
-# Run unit tests only (fast)
-python scripts/run_tests.py unit
-
-# Run integration tests (requires real database)
-python scripts/run_tests.py integration
-```
-
-## **Infrastructure Costs**
-
-### **Estimated Monthly Costs (USD)**
-
-- **Compute Engine VM (e2-standard-2)**: ~$50-70/month
-- **Cloud Storage**: $0.02/GB/month
-- **Firestore**: $0.18/100K reads, $0.18/100K writes
-- **Pub/Sub**: $0.40/million messages
-- **Cloud Functions**: $0.40/million invocations
-- **Cloud Vision API**: $1.50/1000 images
-
-**Total estimated cost for moderate usage**: ~$60-100/month
-
-### **Cost Optimization Tips**
-
-- Use preemptible VM instances for development (50-91% cost reduction)
-- Set up automatic VM shutdown during off-hours
-- Use regional persistent disks instead of SSD for cost savings
-- Monitor and set up billing alerts
-
-## License
-
-This project is licensed under the MIT License - see the LICENSE file for details.
-
 ## API Modules Architecture
 
 The SEREACT API is organized into the following key modules to ensure separation of concerns and maintainable code:
@@ -587,7 +421,6 @@ This modular architecture provides several benefits:
 ### High Priority
 - [ ] Remove Pinecone integration and complete Qdrant migration
 - [ ] Test and validate vector search functionality with Qdrant
-- [x] ~~Implement proper pagination for search results and all endpoints~~ **COMPLETED** - Images and Search APIs have full pagination
 - [ ] Test Cloud Function image processing pipeline
 - [ ] Validate VM setup for self-hosted Qdrant instance
 
@@ -605,17 +438,3 @@ This modular architecture provides several benefits:
 - [ ] Move all auth logic to auth module
 - [ ] Move cloud function code to src folder and reuse code with embedding service
 - [ ] Thumbnail generation
-
-### Pagination Status ✅
-- **✅ Images API**: Fully implemented with `skip`, `limit`, `total` parameters
-- **✅ Search API**: Fully implemented with `limit`, `total`, similarity scoring
-- **ℹ️ Users/Teams/API Keys**: No pagination (small datasets, admin use cases)
-
-## Recent Changes
-- **Implemented hybrid authentication model**: Users, teams, and API key creation are now publicly accessible
-- **Removed authentication requirements** from user and team management endpoints for easier integration
-- Migrated from Pinecone to self-hosted Qdrant
-- Added Cloud Function for async image processing
-- Implemented vector similarity search
-- Added E2E testing infrastructure
-- Set up VM for vector database hosting

deployment/terraform/variables.tf

@@ -58,18 +58,6 @@ variable "qdrant_api_key" {
   default     = ""
 }
 
-variable "vector_db_environment" {
-  description = "Vector database environment (e.g., gcp-starter for Pinecone)"
-  type        = string
-  default     = "gcp-starter"
-}
-
-variable "vector_db_index_name" {
-  description = "Vector database index name for image embeddings"
-  type        = string
-  default     = "image-embeddings"
-}
-
 variable "allowed_cidr_blocks" {
   description = "CIDR blocks allowed to access the vector database"
   type        = string

requirements.txt

@@ -13,6 +13,7 @@ python-jose==3.3.0
 passlib==1.7.4
 tenacity==8.2.3
 pytest==7.4.3
+pytest-asyncio==0.21.1
 httpx==0.25.1
 pillow==10.1.0
 python-slugify==8.0.1 

src/services/embedding_service.py

@@ -117,34 +117,6 @@ class EmbeddingService:
             logger.error(f"Error generating text embedding: {e}")
             return None
     
-    async def process_image_async(self, image_id: str, storage_path: str) -> bool:
-        """
-        Process image asynchronously to generate embeddings
-        
-        Args:
-            image_id: Image ID
-            storage_path: Path to image in storage
-            
-        Returns:
-            True if processing started successfully
-        """
-        try:
-            # In a real implementation, this would:
-            # 1. Publish a message to Pub/Sub queue
-            # 2. Cloud Function would pick up the message
-            # 3. Generate embeddings using Cloud Vision API
-            # 4. Store embeddings in Pinecone
-            # 5. Update image record with embedding info
-            
-            logger.info(f"Starting async processing for image {image_id} at {storage_path}")
-            
-            # For now, just log that processing would start
-            # In production, this would integrate with Google Pub/Sub
-            return True
-        except Exception as e:
-            logger.error(f"Error starting async image processing: {e}")
-            return False
-    
     async def delete_embedding(self, embedding_id: str) -> bool:
         """
         Delete embedding from vector database