johnpccd/image_management_api
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

cp fix api start

Browse Source
This commit is contained in:
johnpccd 2025-05-24 23:22:05 +02:00
parent d6cfb05b04
commit 6265ac534a
5 changed files with 1280 additions and 197 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

58
README.md
View File

@ -340,13 +340,54 @@ results = vector_db.search_similar_images(
## API Endpoints
The API provides the following main endpoints:
The API provides the following main endpoints with their pagination support:
### Authentication & Authorization
- `/api/v1/auth/*` - Authentication and API key management
- `GET /api/v1/auth/api-keys` - List API keys (no pagination - returns all keys for user)
### Team Management
- `/api/v1/teams/*` - Team management
- `/api/v1/users/*` - User management
- `GET /api/v1/teams` - List teams (no pagination - admin only, returns all teams)
### User Management
- `/api/v1/users/*` - User management
- `GET /api/v1/users` - List users (no pagination - returns all users in team/organization)
### Image Management ✅ **Fully Paginated**
- `/api/v1/images/*` - **Image upload, download, and management (with async processing)**
- `GET /api/v1/images` - List images with **full pagination support**
- **Query Parameters:**
- `skip` (default: 0, min: 0) - Number of items to skip
- `limit` (default: 50, min: 1, max: 100) - Number of items per page
- `collection_id` (optional) - Filter by collection
- `tags` (optional) - Filter by comma-separated tags
- **Response includes:** `images`, `total`, `skip`, `limit`
### Search Functionality ✅ **Fully Paginated**
- `/api/v1/search/*` - **Image search functionality (semantic search via Qdrant)**
- `GET /api/v1/search` - Search images with **pagination support**
- **Query Parameters:**
- `q` (required) - Search query
- `limit` (default: 10, min: 1, max: 50) - Number of results
- `threshold` (default: 0.7, min: 0.0, max: 1.0) - Similarity threshold
- `collection_id` (optional) - Filter by collection
- `tags` (optional) - Filter by comma-separated tags
- **Response includes:** `results`, `total`, `limit`, `threshold`, `query`
- `POST /api/v1/search` - Advanced search with same pagination
- `GET /api/v1/search/similar/{image_id}` - Find similar images with pagination
### Pagination Implementation Status
| Endpoint | Pagination Status | Notes |
|----------|------------------|-------|
| `GET /api/v1/images` | ✅ **Fully Implemented** | `skip`, `limit`, `total` with proper validation |
| `GET /api/v1/search` | ✅ **Fully Implemented** | `limit`, `total` with similarity scoring |
| `GET /api/v1/users` | ❌ **Not Implemented** | Returns all users (typically small datasets) |
| `GET /api/v1/teams` | ❌ **Not Implemented** | Returns all teams (typically small datasets) |
| `GET /api/v1/auth/api-keys` | ❌ **Not Implemented** | Returns all keys for user (typically small datasets) |
**Note:** The endpoints without pagination (users, teams, API keys) typically return small datasets and are designed for admin/management use cases where full data visibility is preferred.
### **Image Processing Status**
@ -538,26 +579,31 @@ This modular architecture provides several benefits:
- **Reusability**: Common functionality is centralized for consistent implementation
- **Security**: Authentication and authorization are handled consistently
## TODO
### High Priority
- [ ] Remove Pinecone integration and complete Qdrant migration
- [ ] Test and validate vector search functionality with Qdrant
- [ ] Implement proper pagination for search results and all endpoints
- [x] ~~Implement proper pagination for search results and all endpoints~~ **COMPLETED** - Images and Search APIs have full pagination
- [ ] Test Cloud Function image processing pipeline
- [ ] Validate VM setup for self-hosted Qdrant instance
### Medium Priority
- [ ] Add comprehensive logging for vector search operations
- [ ] Implement caching layer for frequently accessed embeddings
- [ ] Implement caching for frequently accessed data
- [ ] Implement caching for frequently accessed data
- [ ] Add monitoring and alerting for vector database performance
- [ ] Document vector search API endpoints
- [ ] Set up Qdrant cluster with multiple nodes
- [ ] Consider adding pagination to admin endpoints (users, teams, API keys) if datasets grow large
### Low Priority
- [ ] Terraform dependencies
### Pagination Status ✅
- **✅ Images API**: Fully implemented with `skip`, `limit`, `total` parameters
- **✅ Search API**: Fully implemented with `limit`, `total`, similarity scoring
- **ℹ️ Users/Teams/API Keys**: No pagination (small datasets, admin use cases)
## Recent Changes
- Migrated from Pinecone to self-hosted Qdrant

42
deployment/terraform/main.tf
View File

@ -130,31 +130,31 @@ resource "google_cloud_run_service" "sereact" {
value = "INFO"
}
# CORS Configuration - These were missing!
env {
name = "CORS_ORIGINS"
value = "*"
}
# # CORS Configuration - These were missing!
# env {
# name = "CORS_ORIGINS"
# value = "[\"*\"]"
# }
env {
name = "CORS_METHODS"
value = "GET,POST,PUT,DELETE,OPTIONS"
}
# env {
# name = "CORS_METHODS"
# value = "GET,POST,PUT,DELETE,OPTIONS"
# }
env {
name = "CORS_HEADERS"
value = "Content-Type,Authorization,X-Requested-With"
}
# env {
# name = "CORS_HEADERS"
# value = "Content-Type,Authorization,X-Requested-With"
# }
env {
name = "CORS_EXPOSE_HEADERS"
value = "Content-Length,Content-Range"
}
# env {
# name = "CORS_EXPOSE_HEADERS"
# value = "Content-Length,Content-Range"
# }
env {
name = "CORS_MAX_AGE"
value = "3600"
}
# env {
# name = "CORS_MAX_AGE"
# value = "3600"
# }
}
}

1158
deployment/terraform/terraform.tfstate
View File

File diff suppressed because one or more lines are too long

216
deployment/terraform/terraform.tfstate.backup
View File

@ -1,7 +1,7 @@
{
"version": 4,
"terraform_version": "1.10.1",
"serial": 273,
"serial": 363,
"lineage": "a183cd95-f987-8698-c6dd-84e933c394a5",
"outputs": {
"cloud_run_qdrant_host": {
@ -9,38 +9,13 @@
"type": "string"
},
"cloud_run_qdrant_host_internal": {
"value": "10.128.0.4",
"type": "string"
},
"cloud_run_url": {
"value": "https://sereact-p64zpdtkta-uc.a.run.app",
"value": "10.128.0.6",
"type": "string"
},
"container_registry_url": {
"value": "gcr.io/gen-lang-client-0424120530/sereact",
"type": "string"
},
"deployment_summary": {
"value": {
"cloud_run_url": "https://sereact-p64zpdtkta-uc.a.run.app",
"firestore_database": "sereact-imagedb",
"qdrant_endpoint": "http://34.71.6.1:6333",
"qdrant_host_ip": "34.71.6.1",
"static_ip_enabled": false,
"storage_bucket": "sereact-images"
},
"type": [
"object",
{
"cloud_run_url": "string",
"firestore_database": "string",
"qdrant_endpoint": "string",
"qdrant_host_ip": "string",
"static_ip_enabled": "bool",
"storage_bucket": "string"
}
]
},
"firestore_database_id": {
"value": "projects/gen-lang-client-0424120530/databases/sereact-imagedb",
"type": "string"
@ -74,7 +49,7 @@
"type": "string"
},
"vector_db_vm_internal_ip": {
"value": "10.128.0.4",
"value": "10.128.0.6",
"type": "string"
},
"vector_db_vm_name": {
@ -123,6 +98,7 @@
"provider": "provider[\"registry.terraform.io/hashicorp/google\"]",
"instances": [
{
"status": "tainted",
"schema_version": 2,
"attributes": {
"autogenerate_revision_name": false,
@ -134,67 +110,25 @@
"run.googleapis.com/ingress": "all"
},
"effective_annotations": {
"run.googleapis.com/ingress": "all",
"run.googleapis.com/ingress-status": "all",
"run.googleapis.com/operation-id": "8e5e22cf-c77d-4cb8-8587-5cf9d409193c",
"run.googleapis.com/urls": "[\"https://sereact-761163285547.us-central1.run.app\",\"https://sereact-p64zpdtkta-uc.a.run.app\"]",
"serving.knative.dev/creator": "johnpccd3@gmail.com",
"serving.knative.dev/lastModifier": "johnpccd3@gmail.com"
"run.googleapis.com/ingress": "all"
},
"effective_labels": {
"cloud.googleapis.com/location": "us-central1",
"goog-terraform-provisioned": "true"
},
"generation": 5,
"labels": {},
"namespace": "gen-lang-client-0424120530",
"resource_version": "AAY15+pAZzs",
"self_link": "/apis/serving.knative.dev/v1/namespaces/761163285547/services/sereact",
"generation": 0,
"labels": null,
"namespace": "",
"resource_version": "",
"self_link": "",
"terraform_labels": {
"goog-terraform-provisioned": "true"
},
"uid": "f67eed78-59c9-4bb6-a118-fb940f206d09"
"uid": ""
}
],
"name": "sereact",
"project": "gen-lang-client-0424120530",
"status": [
{
"conditions": [
{
"message": "Revision 'sereact-00005-5n6' is not ready and cannot serve traffic. The user-provided container failed the configured startup probe checks. Logs for this revision might contain more information.\n\nLogs URL: https://console.cloud.google.com/logs/viewer?project=gen-lang-client-0424120530\u0026resource=cloud_run_revision/service_name/sereact/revision_name/sereact-00005-5n6\u0026advancedFilter=resource.type%3D%22cloud_run_revision%22%0Aresource.labels.service_name%3D%22sereact%22%0Aresource.labels.revision_name%3D%22sereact-00005-5n6%22 \nFor more troubleshooting guidance, see https://cloud.google.com/run/docs/troubleshooting#container-failed-to-start",
"reason": "HealthCheckContainerError",
"status": "False",
"type": "Ready"
},
{
"message": "The user-provided container failed the configured startup probe checks. Logs for this revision might contain more information.\n\nLogs URL: https://console.cloud.google.com/logs/viewer?project=gen-lang-client-0424120530\u0026resource=cloud_run_revision/service_name/sereact/revision_name/sereact-00005-5n6\u0026advancedFilter=resource.type%3D%22cloud_run_revision%22%0Aresource.labels.service_name%3D%22sereact%22%0Aresource.labels.revision_name%3D%22sereact-00005-5n6%22 \nFor more troubleshooting guidance, see https://cloud.google.com/run/docs/troubleshooting#container-failed-to-start",
"reason": "",
"status": "True",
"type": "ConfigurationsReady"
},
{
"message": "Revision 'sereact-00005-5n6' is not ready and cannot serve traffic. The user-provided container failed the configured startup probe checks. Logs for this revision might contain more information.\n\nLogs URL: https://console.cloud.google.com/logs/viewer?project=gen-lang-client-0424120530\u0026resource=cloud_run_revision/service_name/sereact/revision_name/sereact-00005-5n6\u0026advancedFilter=resource.type%3D%22cloud_run_revision%22%0Aresource.labels.service_name%3D%22sereact%22%0Aresource.labels.revision_name%3D%22sereact-00005-5n6%22 \nFor more troubleshooting guidance, see https://cloud.google.com/run/docs/troubleshooting#container-failed-to-start",
"reason": "HealthCheckContainerError",
"status": "False",
"type": "RoutesReady"
}
],
"latest_created_revision_name": "sereact-00005-5n6",
"latest_ready_revision_name": "sereact-00001-rsq",
"observed_generation": 5,
"traffic": [
{
"latest_revision": true,
"percent": 100,
"revision_name": "sereact-00001-rsq",
"tag": "",
"url": ""
}
],
"url": "https://sereact-p64zpdtkta-uc.a.run.app"
}
],
"status": null,
"template": [
{
"metadata": [
@ -203,9 +137,7 @@
"autoscaling.knative.dev/maxScale": "10"
},
"generation": 0,
"labels": {
"run.googleapis.com/startupProbeType": "Custom"
},
"labels": {},
"name": "",
"namespace": "",
"resource_version": "",
@ -215,11 +147,11 @@
],
"spec": [
{
"container_concurrency": 80,
"container_concurrency": 0,
"containers": [
{
"args": [],
"command": [],
"args": null,
"command": null,
"env": [
{
"name": "CORS_EXPOSE_HEADERS",
@ -241,11 +173,6 @@
"value": "GET,POST,PUT,DELETE,OPTIONS",
"value_from": []
},
{
"name": "CORS_ORIGINS",
"value": "*",
"value_from": []
},
{
"name": "FIRESTORE_DATABASE_NAME",
"value": "sereact-imagedb",
@ -309,7 +236,7 @@
"ports": [
{
"container_port": 8000,
"name": "http1",
"name": "",
"protocol": ""
}
],
@ -319,34 +246,18 @@
"cpu": "1",
"memory": "1Gi"
},
"requests": {}
}
],
"startup_probe": [
{
"failure_threshold": 6,
"grpc": [],
"http_get": [
{
"http_headers": [],
"path": "/health",
"port": 8000
}
],
"initial_delay_seconds": 30,
"period_seconds": 10,
"tcp_socket": [],
"timeout_seconds": 5
"requests": null
}
],
"startup_probe": [],
"volume_mounts": [],
"working_dir": ""
}
],
"node_selector": {},
"service_account_name": "761163285547-compute@developer.gserviceaccount.com",
"node_selector": null,
"service_account_name": "",
"serving_state": "",
"timeout_seconds": 300,
"timeout_seconds": 0,
"volumes": []
}
]
@ -416,33 +327,10 @@
},
{
"mode": "managed",
"type": "google_cloud_run_service_iam_member",
"name": "public_access",
"type": "google_compute_address",
"name": "vector_db_static_ip",
"provider": "provider[\"registry.terraform.io/hashicorp/google\"]",
"instances": [
{
"schema_version": 0,
"attributes": {
"condition": [],
"etag": "BwY15bNaQMM=",
"id": "v1/projects/gen-lang-client-0424120530/locations/us-central1/services/sereact/roles/run.invoker/allUsers",
"location": "us-central1",
"member": "allUsers",
"project": "gen-lang-client-0424120530",
"role": "roles/run.invoker",
"service": "v1/projects/gen-lang-client-0424120530/locations/us-central1/services/sereact"
},
"sensitive_attributes": [],
"private": "bnVsbA==",
"dependencies": [
"google_cloud_run_service.sereact",
"google_compute_address.vector_db_static_ip",
"google_compute_instance.vector_db_vm",
"google_project_service.services",
"google_service_account.vector_db_sa"
]
}
]
"instances": []
},
{
"mode": "managed",
@ -462,7 +350,7 @@
"protocol": "tcp"
}
],
"creation_timestamp": "2025-05-24T10:57:27.640-07:00",
"creation_timestamp": "2025-05-24T14:09:17.159-07:00",
"deny": [],
"description": "",
"destination_ranges": [],
@ -552,7 +440,7 @@
"can_ip_forward": false,
"confidential_instance_config": [],
"cpu_platform": "Intel Broadwell",
"creation_timestamp": "2025-05-24T10:57:41.174-07:00",
"creation_timestamp": "2025-05-24T14:09:30.477-07:00",
"current_status": "RUNNING",
"deletion_protection": false,
"description": "",
@ -565,7 +453,7 @@
"hostname": "",
"id": "projects/gen-lang-client-0424120530/zones/us-central1-a/instances/sereact-vector-db",
"instance_encryption_key": [],
"instance_id": "8076617322369889915",
"instance_id": "1246596410843827045",
"key_revocation_action_type": "",
"label_fingerprint": "vezUS-42LLM=",
"labels": {},
@ -592,7 +480,7 @@
"name": "nic0",
"network": "https://www.googleapis.com/compute/v1/projects/gen-lang-client-0424120530/global/networks/default",
"network_attachment": "",
"network_ip": "10.128.0.4",
"network_ip": "10.128.0.6",
"nic_type": "",
"queue_count": 0,
"stack_type": "IPV4_ONLY",
@ -653,7 +541,18 @@
[
{
"type": "get_attr",
"value": "metadata_startup_script"
"value": "boot_disk"
},
{
"type": "index",
"value": {
"value": 0,
"type": "number"
}
},
{
"type": "get_attr",
"value": "disk_encryption_key_rsa"
}
],
[
@ -676,18 +575,7 @@
[
{
"type": "get_attr",
"value": "boot_disk"
},
{
"type": "index",
"value": {
"value": 0,
"type": "number"
}
},
{
"type": "get_attr",
"value": "disk_encryption_key_rsa"
"value": "metadata_startup_script"
}
]
],
@ -716,8 +604,8 @@
"database_edition": "STANDARD",
"delete_protection_state": "DELETE_PROTECTION_DISABLED",
"deletion_policy": "ABANDON",
"earliest_version_time": "2025-05-24T19:55:31.455016Z",
"etag": "IM+J1/v+vI0DMLa2lbPIvI0D",
"earliest_version_time": "2025-05-24T21:09:24.677010Z",
"etag": "IOyxoJSEvY0DMJL7/oiCvY0D",
"id": "projects/gen-lang-client-0424120530/databases/sereact-imagedb",
"key_prefix": "",
"location_id": "us-central1",
@ -726,12 +614,12 @@
"project": "gen-lang-client-0424120530",
"timeouts": null,
"type": "FIRESTORE_NATIVE",
"uid": "30634721-c8c7-4673-b605-c1a91dff3eab",
"uid": "f4aa945b-2205-4f94-b903-57815519f2fc",
"update_time": "",
"version_retention_period": "3600s"
},
"sensitive_attributes": [],
"private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjEyMDAwMDAwMDAwMDAsInVwZGF0ZSI6MTIwMDAwMDAwMDAwMH0sInNjaGVtYV92ZXJzaW9uIjoiMCJ9",
"private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjEyMDAwMDAwMDAwMDAsInVwZGF0ZSI6MTIwMDAwMDAwMDAwMH19",
"dependencies": [
"google_project_service.services"
]
@ -963,7 +851,7 @@
"schema_version": 0,
"attributes": {
"condition": [],
"etag": "BwY15XOdN50=",
"etag": "BwY16CF7/Zw=",
"id": "projects/gen-lang-client-0424120530/subscriptions/image-processing-topic-subscription/roles/pubsub.subscriber",
"members": [
"serviceAccount:761163285547-compute@developer.gserviceaccount.com"
@ -1075,7 +963,7 @@
"schema_version": 0,
"attributes": {
"condition": [],
"etag": "BwY15XMktMU=",
"etag": "BwY16CEHGDA=",
"id": "projects/gen-lang-client-0424120530/topics/image-processing-topic/roles/pubsub.publisher",
"members": [
"serviceAccount:761163285547-compute@developer.gserviceaccount.com"
@ -1113,7 +1001,7 @@
"name": "projects/gen-lang-client-0424120530/serviceAccounts/vector-db-sa@gen-lang-client-0424120530.iam.gserviceaccount.com",
"project": "gen-lang-client-0424120530",
"timeouts": null,
"unique_id": "108486040628486621877"
"unique_id": "115648202672357296665"
},
"sensitive_attributes": [],
"private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjozMDAwMDAwMDAwMDB9fQ=="
@ -1159,7 +1047,7 @@
"self_link": "https://www.googleapis.com/storage/v1/b/sereact-images",
"soft_delete_policy": [
{
"effective_time": "2025-05-24T17:57:31.611Z",
"effective_time": "2025-05-24T21:09:21.315Z",
"retention_duration_seconds": 604800
}
],
@ -1167,10 +1055,10 @@
"terraform_labels": {
"goog-terraform-provisioned": "true"
},
"time_created": "2025-05-24T17:57:31.611Z",
"time_created": "2025-05-24T21:09:21.315Z",
"timeouts": null,
"uniform_bucket_level_access": true,
"updated": "2025-05-24T17:57:31.611Z",
"updated": "2025-05-24T21:09:21.315Z",
"url": "gs://sereact-images",
"versioning": [],
"website": []

3
src/config/config.py
View File

@ -23,9 +23,6 @@ class Settings(BaseSettings):
# Read from environment variable first
env_origins = os.getenv("CORS_ORIGINS")
if env_origins:
# Handle wildcard case
if env_origins.strip() == "*":
return ["*"]
if env_origins.startswith("[") and env_origins.endswith("]"):
# Handle list format like "['*']"
import ast