cleanup
This commit is contained in:
parent
8983114868
commit
cd26c397e3
187
README.md
187
README.md
@ -205,116 +205,11 @@ Uses Google's Vertex AI multimodal embedding model for generating high-quality i
|
||||
QDRANT_PREFER_GRPC=false
|
||||
```
|
||||
|
||||
5. **Deploy Infrastructure** (Required for vector database):
|
||||
5. **Deploy Infrastructure**
|
||||
```bash
|
||||
# Configure Terraform variables
|
||||
cd deployment/terraform
|
||||
cp terraform.tfvars.example terraform.tfvars
|
||||
# Edit terraform.tfvars with your values
|
||||
|
||||
# Deploy infrastructure including vector database VM
|
||||
terraform init
|
||||
terraform plan
|
||||
terraform apply
|
||||
|
||||
# Note the output values for VM IP addresses
|
||||
./deployment/deploy.sh --build --deploy
|
||||
```
|
||||
|
||||
6. **Deploy Cloud Function** (Optional - for production):
|
||||
```bash
|
||||
cd ../cloud-function
|
||||
./deploy.sh
|
||||
```
|
||||
|
||||
7. Run the application:
|
||||
```bash
|
||||
uvicorn main:app --reload
|
||||
```
|
||||
|
||||
8. Visit `http://localhost:8000/docs` in your browser to access the API documentation.
|
||||
|
||||
## **Deployment**
|
||||
|
||||
### **Complete Infrastructure Deployment**
|
||||
|
||||
Deploy the entire infrastructure including the vector database VM:
|
||||
|
||||
```bash
|
||||
cd deployment/terraform
|
||||
|
||||
# Configure your variables
|
||||
cp terraform.tfvars.example terraform.tfvars
|
||||
# Edit terraform.tfvars with your specific values:
|
||||
# - project_id: Your GCP project ID
|
||||
# - storage_bucket_name: Unique bucket name
|
||||
# - qdrant_api_key: Secure API key for Qdrant (optional)
|
||||
# - allowed_cidr_blocks: Your IP address/range for security
|
||||
# - use_static_ip: Set to true for production
|
||||
|
||||
# Deploy infrastructure
|
||||
terraform init
|
||||
terraform plan
|
||||
terraform apply
|
||||
```
|
||||
|
||||
This will create:
|
||||
- **Google Compute Engine VM with Qdrant vector database**
|
||||
- **Firewall rules for vector database access**
|
||||
- **Service accounts and IAM bindings**
|
||||
- **Pub/Sub topic and subscription with retry policy**
|
||||
- **Cloud Storage bucket**
|
||||
- **Firestore database**
|
||||
- **Cloud Run service**
|
||||
|
||||
### **Vector Database VM Outputs**
|
||||
|
||||
After deployment, Terraform will output:
|
||||
- `vector_db_vm_external_ip`: External IP address of the VM
|
||||
- `qdrant_http_endpoint`: HTTP API endpoint for Qdrant
|
||||
- `qdrant_grpc_endpoint`: gRPC API endpoint for Qdrant
|
||||
|
||||
### **Cloud Function Deployment**
|
||||
|
||||
The image processing Cloud Function can be deployed using the provided script:
|
||||
|
||||
```bash
|
||||
cd deployment/cloud-function
|
||||
|
||||
# Set environment variables
|
||||
export GOOGLE_CLOUD_PROJECT=your-project-id
|
||||
export QDRANT_HOST=your-vm-external-ip
|
||||
export QDRANT_API_KEY=your-qdrant-api-key
|
||||
|
||||
# Deploy the function
|
||||
./deploy.sh
|
||||
```
|
||||
|
||||
|
||||
#### **Vector Database API Usage**
|
||||
|
||||
```python
|
||||
from src.services.vector_db import VectorDatabaseService
|
||||
|
||||
# Initialize service
|
||||
vector_db = VectorDatabaseService(
|
||||
host="your-vm-external-ip",
|
||||
api_key="your-qdrant-api-key" # Optional
|
||||
)
|
||||
|
||||
# Add image vector
|
||||
point_id = vector_db.add_image_vector(
|
||||
image_id="img_123",
|
||||
vector=[0.1, 0.2, ...], # 1408-dimensional vector
|
||||
metadata={"filename": "image.jpg", "size": 1024}
|
||||
)
|
||||
|
||||
# Search similar images
|
||||
results = vector_db.search_similar_images(
|
||||
query_vector=[0.1, 0.2, ...],
|
||||
limit=10,
|
||||
score_threshold=0.7
|
||||
)
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
@ -392,21 +287,6 @@ A **hybrid authentication model**:
|
||||
|
||||
**Note:** Public endpoints (users, teams) don't implement pagination as they typically return small datasets and are designed for management use cases where full data visibility is preferred.
|
||||
|
||||
### **Image Processing Status**
|
||||
|
||||
Images now include embedding processing status:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "image-id",
|
||||
"filename": "example.jpg",
|
||||
"embedding_status": "success", // "pending", "processing", "success", "failed"
|
||||
"embedding_error": null,
|
||||
"embedding_retry_count": 0,
|
||||
"has_embedding": true
|
||||
}
|
||||
```
|
||||
|
||||
Refer to the Swagger UI documentation at `/docs` for detailed endpoint information.
|
||||
|
||||
## Development
|
||||
@ -414,55 +294,9 @@ Refer to the Swagger UI documentation at `/docs` for detailed endpoint informati
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
pytest
|
||||
|
||||
# Run specific test categories
|
||||
pytest tests/services/test_pubsub_service.py # Pub/Sub service tests
|
||||
pytest tests/services/test_vector_db.py # Vector database tests
|
||||
pytest tests/integration/test_cloud_function.py # Cloud Function tests
|
||||
pytest tests/api/test_images_pubsub.py # API integration tests
|
||||
source venv/Scripts/activate && python scripts/run_tests.py all
|
||||
```
|
||||
|
||||
### **Comprehensive End-to-End Testing**
|
||||
|
||||
SEREACT includes a comprehensive E2E testing suite that covers complete user workflows with **completely self-contained artificial test data**:
|
||||
|
||||
```bash
|
||||
# Run all E2E tests (completely self-contained - no setup required!)
|
||||
python scripts/run_tests.py e2e
|
||||
|
||||
# Run unit tests only (fast)
|
||||
python scripts/run_tests.py unit
|
||||
|
||||
# Run integration tests (requires real database)
|
||||
python scripts/run_tests.py integration
|
||||
```
|
||||
|
||||
## **Infrastructure Costs**
|
||||
|
||||
### **Estimated Monthly Costs (USD)**
|
||||
|
||||
- **Compute Engine VM (e2-standard-2)**: ~$50-70/month
|
||||
- **Cloud Storage**: $0.02/GB/month
|
||||
- **Firestore**: $0.18/100K reads, $0.18/100K writes
|
||||
- **Pub/Sub**: $0.40/million messages
|
||||
- **Cloud Functions**: $0.40/million invocations
|
||||
- **Cloud Vision API**: $1.50/1000 images
|
||||
|
||||
**Total estimated cost for moderate usage**: ~$60-100/month
|
||||
|
||||
### **Cost Optimization Tips**
|
||||
|
||||
- Use preemptible VM instances for development (50-91% cost reduction)
|
||||
- Set up automatic VM shutdown during off-hours
|
||||
- Use regional persistent disks instead of SSD for cost savings
|
||||
- Monitor and set up billing alerts
|
||||
|
||||
## License
|
||||
|
||||
This project is licensed under the MIT License - see the LICENSE file for details.
|
||||
|
||||
## API Modules Architecture
|
||||
|
||||
The SEREACT API is organized into the following key modules to ensure separation of concerns and maintainable code:
|
||||
@ -587,7 +421,6 @@ This modular architecture provides several benefits:
|
||||
### High Priority
|
||||
- [ ] Remove Pinecone integration and complete Qdrant migration
|
||||
- [ ] Test and validate vector search functionality with Qdrant
|
||||
- [x] ~~Implement proper pagination for search results and all endpoints~~ **COMPLETED** - Images and Search APIs have full pagination
|
||||
- [ ] Test Cloud Function image processing pipeline
|
||||
- [ ] Validate VM setup for self-hosted Qdrant instance
|
||||
|
||||
@ -605,17 +438,3 @@ This modular architecture provides several benefits:
|
||||
- [ ] Move all auth logic to auth module
|
||||
- [ ] Move cloud function code to src folder and reuse code with embedding service
|
||||
- [ ] Thumbnail generation
|
||||
|
||||
### Pagination Status ✅
|
||||
- **✅ Images API**: Fully implemented with `skip`, `limit`, `total` parameters
|
||||
- **✅ Search API**: Fully implemented with `limit`, `total`, similarity scoring
|
||||
- **ℹ️ Users/Teams/API Keys**: No pagination (small datasets, admin use cases)
|
||||
|
||||
## Recent Changes
|
||||
- **Implemented hybrid authentication model**: Users, teams, and API key creation are now publicly accessible
|
||||
- **Removed authentication requirements** from user and team management endpoints for easier integration
|
||||
- Migrated from Pinecone to self-hosted Qdrant
|
||||
- Added Cloud Function for async image processing
|
||||
- Implemented vector similarity search
|
||||
- Added E2E testing infrastructure
|
||||
- Set up VM for vector database hosting
|
||||
|
||||
@ -58,18 +58,6 @@ variable "qdrant_api_key" {
|
||||
default = ""
|
||||
}
|
||||
|
||||
variable "vector_db_environment" {
|
||||
description = "Vector database environment (e.g., gcp-starter for Pinecone)"
|
||||
type = string
|
||||
default = "gcp-starter"
|
||||
}
|
||||
|
||||
variable "vector_db_index_name" {
|
||||
description = "Vector database index name for image embeddings"
|
||||
type = string
|
||||
default = "image-embeddings"
|
||||
}
|
||||
|
||||
variable "allowed_cidr_blocks" {
|
||||
description = "CIDR blocks allowed to access the vector database"
|
||||
type = string
|
||||
|
||||
@ -13,6 +13,7 @@ python-jose==3.3.0
|
||||
passlib==1.7.4
|
||||
tenacity==8.2.3
|
||||
pytest==7.4.3
|
||||
pytest-asyncio==0.21.1
|
||||
httpx==0.25.1
|
||||
pillow==10.1.0
|
||||
python-slugify==8.0.1
|
||||
|
||||
@ -117,34 +117,6 @@ class EmbeddingService:
|
||||
logger.error(f"Error generating text embedding: {e}")
|
||||
return None
|
||||
|
||||
async def process_image_async(self, image_id: str, storage_path: str) -> bool:
|
||||
"""
|
||||
Process image asynchronously to generate embeddings
|
||||
|
||||
Args:
|
||||
image_id: Image ID
|
||||
storage_path: Path to image in storage
|
||||
|
||||
Returns:
|
||||
True if processing started successfully
|
||||
"""
|
||||
try:
|
||||
# In a real implementation, this would:
|
||||
# 1. Publish a message to Pub/Sub queue
|
||||
# 2. Cloud Function would pick up the message
|
||||
# 3. Generate embeddings using Cloud Vision API
|
||||
# 4. Store embeddings in Pinecone
|
||||
# 5. Update image record with embedding info
|
||||
|
||||
logger.info(f"Starting async processing for image {image_id} at {storage_path}")
|
||||
|
||||
# For now, just log that processing would start
|
||||
# In production, this would integrate with Google Pub/Sub
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Error starting async image processing: {e}")
|
||||
return False
|
||||
|
||||
async def delete_embedding(self, embedding_id: str) -> bool:
|
||||
"""
|
||||
Delete embedding from vector database
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user