cleanup
This commit is contained in:
parent
0a4038404f
commit
ba2c00db38
@ -19,7 +19,6 @@ API_KEY_SECRET=super-secret-key-for-development-only
|
||||
API_KEY_EXPIRY_DAYS=365
|
||||
|
||||
# Vector Database settings
|
||||
QDRANT_HOST=35.193.174.125
|
||||
QDRANT_PORT=6333
|
||||
QDRANT_HTTPS=false
|
||||
QDRANT_PREFER_GRPC=false
|
||||
@ -199,7 +199,6 @@ Uses Google's Vertex AI multimodal embedding model for generating high-quality i
|
||||
API_KEY_EXPIRY_DAYS=365
|
||||
|
||||
# Vector Database settings
|
||||
QDRANT_HOST=35.193.174.125
|
||||
QDRANT_PORT=6333
|
||||
QDRANT_HTTPS=false
|
||||
QDRANT_PREFER_GRPC=false
|
||||
@ -208,8 +207,13 @@ Uses Google's Vertex AI multimodal embedding model for generating high-quality i
|
||||
5. **Deploy Infrastructure**
|
||||
```bash
|
||||
./deployment/deploy.sh --build --deploy
|
||||
python ./scripts/seed_firestore.py
|
||||
```
|
||||
|
||||
6. **Destroy Infrastructure**
|
||||
```bash
|
||||
./deployment/deploy.sh --destroy
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
|
||||
@ -1,120 +0,0 @@
|
||||
# Cloud Function for Image Embedding Processing
|
||||
|
||||
This Cloud Function processes images to generate embeddings using Google's Vertex AI multimodal embedding model and stores them in a Qdrant vector database.
|
||||
|
||||
## Overview
|
||||
|
||||
The function is triggered by Pub/Sub messages containing image processing tasks. It:
|
||||
|
||||
1. Downloads images from Google Cloud Storage
|
||||
2. Generates embeddings using Vertex AI's `multimodalembedding@001` model
|
||||
3. Stores embeddings in Qdrant vector database
|
||||
4. Updates image metadata in Firestore
|
||||
|
||||
## Key Features
|
||||
|
||||
- **Vertex AI Multimodal Embeddings**: Uses Google's state-of-the-art multimodal embedding model
|
||||
- **1408-dimensional vectors**: High-quality embeddings for semantic image search
|
||||
- **Automatic retry**: Built-in retry logic for failed processing
|
||||
- **Status tracking**: Real-time status updates in Firestore
|
||||
- **Scalable**: Auto-scaling Cloud Function with configurable limits
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `google-cloud-aiplatform`: Vertex AI SDK for multimodal embeddings
|
||||
- `google-cloud-firestore`: Firestore database client
|
||||
- `google-cloud-storage`: Cloud Storage client
|
||||
- `qdrant-client`: Vector database client
|
||||
- `numpy`: Numerical operations
|
||||
- `Pillow`: Image processing
|
||||
|
||||
## Environment Variables
|
||||
|
||||
The function requires these environment variables:
|
||||
|
||||
```bash
|
||||
# Google Cloud Configuration
|
||||
GOOGLE_CLOUD_PROJECT=your-project-id
|
||||
VERTEX_AI_LOCATION=us-central1
|
||||
|
||||
# Firestore Configuration
|
||||
FIRESTORE_PROJECT_ID=your-project-id
|
||||
FIRESTORE_DATABASE_NAME=(default)
|
||||
|
||||
# Cloud Storage Configuration
|
||||
GCS_BUCKET_NAME=your-bucket-name
|
||||
|
||||
# Qdrant Configuration
|
||||
QDRANT_HOST=your-qdrant-host
|
||||
QDRANT_PORT=6333
|
||||
QDRANT_API_KEY=your-api-key
|
||||
QDRANT_COLLECTION=image_vectors
|
||||
QDRANT_HTTPS=false
|
||||
|
||||
# Logging
|
||||
LOG_LEVEL=INFO
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
### Local Testing
|
||||
|
||||
1. Set up your environment:
|
||||
```bash
|
||||
export GOOGLE_CLOUD_PROJECT=your-project-id
|
||||
export VERTEX_AI_LOCATION=us-central1
|
||||
```
|
||||
|
||||
2. Install dependencies:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
3. Run the test script:
|
||||
```bash
|
||||
python test_vertex_ai_embeddings.py
|
||||
```
|
||||
|
||||
This will create a test image and verify that embeddings are generated correctly.
|
||||
|
||||
### Expected Output
|
||||
|
||||
The test should output something like:
|
||||
```
|
||||
INFO:__main__:Testing Vertex AI multimodal embeddings...
|
||||
INFO:__main__:Using project: your-project-id
|
||||
INFO:__main__:Creating test image...
|
||||
INFO:__main__:Created test image with 1234 bytes
|
||||
INFO:__main__:Generating embeddings using Vertex AI...
|
||||
INFO:__main__:Generated embeddings with shape: (1408,)
|
||||
INFO:__main__:Embeddings dtype: float32
|
||||
INFO:__main__:Embeddings range: [-0.1234, 0.5678]
|
||||
INFO:__main__:Embeddings norm: 1.0000
|
||||
INFO:__main__:✅ All tests passed! Vertex AI embeddings are working correctly.
|
||||
INFO:__main__:🎉 Test completed successfully!
|
||||
```
|
||||
|
||||
## Deployment
|
||||
|
||||
The function is deployed using Terraform. See the main deployment documentation for details.
|
||||
|
||||
## Monitoring
|
||||
|
||||
- Check Cloud Function logs in Google Cloud Console
|
||||
- Monitor Firestore for image status updates
|
||||
- Check Qdrant for stored embeddings
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Authentication errors**: Ensure the service account has `roles/aiplatform.user` permission
|
||||
2. **API not enabled**: Ensure `aiplatform.googleapis.com` is enabled
|
||||
3. **Quota limits**: Check Vertex AI quotas in your project
|
||||
4. **Network issues**: Ensure the function can reach Qdrant and other services
|
||||
|
||||
### Error Messages
|
||||
|
||||
- `"Failed to generate embeddings - no image embedding returned"`: Check image format and size
|
||||
- `"PROJECT_ID not found in environment variables"`: Set `GOOGLE_CLOUD_PROJECT`
|
||||
- `"Error generating embeddings"`: Check Vertex AI API access and quotas
|
||||
Loading…
x
Reference in New Issue
Block a user