image_management_api/deployment/terraform

Terraform Configuration for Sereact

This directory contains Terraform configurations to provision the required Google Cloud resources for Sereact:

• Google Cloud Run service
• Google Cloud Function for image processing
• Google Container Registry (GCR)
• Firestore database
• Cloud Storage bucket
• Pub/Sub topics and subscriptions
• Qdrant vector database VM

Prerequisites

1. Install Terraform (v1.0.0+)
2. Install Google Cloud SDK
3. Authenticate with Google Cloud:

   gcloud auth login
   gcloud auth application-default login
   

4. Create or select a Google Cloud project:

   gcloud projects create PROJECT_ID --name="Sereact Project" # optional
   gcloud config set project PROJECT_ID
   

Architecture Overview

The infrastructure includes:

• Cloud Run Service: Main application service
• Cloud Function: Image processing and embedding generation
• Qdrant VM: Vector database running on Compute Engine
• Firestore: Document database
• Cloud Storage: File storage
• Pub/Sub: Message queue for asynchronous image processing
• Automatic IP Configuration: Qdrant VM IP is automatically passed to Cloud Run and Cloud Function

Cloud Function Integration

The Cloud Function is now fully integrated into Terraform and handles:

1. Image Processing: Triggered by Pub/Sub messages
2. Embedding Generation: Uses Google Cloud Vision API
3. Vector Storage: Stores embeddings in Qdrant
4. Status Updates: Updates Firestore with processing results

Automatic Configuration

The Cloud Function is automatically configured with:

• Qdrant Connection: VM IP address passed as environment variable
• IAM Permissions: Firestore, Storage, and Vision API access
• Retry Policy: Automatic retries on failure
• Scaling: 0-10 instances based on demand

Setup and Usage

Option 1: Automated Deployment (Recommended)

Use the provided script that handles API enablement and proper deployment order:

cd deployment/terraform
./deploy-with-apis.sh

This script will:

1. Enable all required APIs
2. Wait for API propagation
3. Deploy infrastructure in the correct order
4. Handle Cloud Functions v2 dependencies

Option 2: Manual Deployment

1. Copy the example variables file and edit it with your values:

   cp terraform.tfvars.example terraform.tfvars
   # Edit terraform.tfvars with your project-specific values
   

2. Initialize Terraform:

   terraform init
   

3. Enable APIs first (required for Cloud Functions v2):

   terraform apply -target=google_project_service.services
   

4. Wait for APIs to propagate (2-3 minutes), then deploy everything:

   terraform apply
   

Required APIs

The following APIs are automatically enabled by Terraform:

• cloudresourcemanager.googleapis.com - Resource management
• containerregistry.googleapis.com - Container Registry
• run.googleapis.com - Cloud Run
• firestore.googleapis.com - Firestore database
• storage.googleapis.com - Cloud Storage
• compute.googleapis.com - Compute Engine (for Qdrant VM)
• cloudfunctions.googleapis.com - Cloud Functions
• cloudbuild.googleapis.com - Cloud Build (for function deployment)
• eventarc.googleapis.com - Eventarc (required for Cloud Functions v2)
• pubsub.googleapis.com - Pub/Sub messaging

Note: If you get an Eventarc API error, use the automated deployment script or wait 2-3 minutes after enabling APIs before deploying the Cloud Function.

5. After provisioning, you'll see outputs including:
   • Cloud Run service URL
   • Storage bucket name
   • Firestore database ID
   • Container Registry URL

Managing Secrets

Secrets for environment variables (API_KEY_SECRET, VECTOR_DB_API_KEY, etc.) should be managed separately using Google Secret Manager:

# Create secrets
gcloud secrets create sereact-api-key-secret --replication-policy="automatic"
gcloud secrets create sereact-vector-db-key --replication-policy="automatic"

# Add secret versions
echo -n "your-secret-value" | gcloud secrets versions add sereact-api-key-secret --data-file=-
echo -n "your-secret-value" | gcloud secrets versions add sereact-vector-db-key --data-file=-

# Update Cloud Run service to use secrets
gcloud run services update sereact \
  --update-secrets=API_KEY_SECRET=sereact-api-key-secret:latest,VECTOR_DB_API_KEY=sereact-vector-db-key:latest

CI/CD Integration

To integrate this with CI/CD, store the terraform.tfvars securely in your CI/CD system and run Terraform as part of your deployment pipeline:

# Example GitHub Actions step
- name: Terraform Apply
  run: |
    cd deployment/terraform
    terraform init
    terraform apply -auto-approve
  env:
    GOOGLE_APPLICATION_CREDENTIALS: ${{ secrets.GCP_SA_KEY }}

Destroying Resources

To destroy all provisioned resources:

terraform destroy

Terraform Infrastructure for Sereact

This directory contains Terraform configurations to deploy the complete Sereact infrastructure on Google Cloud Platform, including automatic configuration of Qdrant VM IP addresses for Cloud Run.

Architecture Overview

The infrastructure includes:

• Cloud Run Service: Main application service
• Qdrant VM: Vector database running on Compute Engine
• Firestore: Document database
• Cloud Storage: File storage
• Automatic IP Configuration: Qdrant VM IP is automatically passed to Cloud Run

Qdrant VM IP Address Integration

The Terraform configuration automatically handles passing the Qdrant VM IP address to your Cloud Run service:

How It Works

1. VM Creation: Terraform creates a Compute Engine VM running Qdrant
2. IP Address Extraction: Terraform extracts the VM's external IP address
3. Environment Variable Injection: The IP is automatically set as QDRANT_HOST in Cloud Run
4. Dependency Management: Cloud Run waits for the VM to be created before deploying

Configuration Options

Static vs Dynamic IP

You can choose between static and dynamic IP addresses:

# In terraform.tfvars
use_static_ip = true   # Use static IP (recommended for production)
use_static_ip = false  # Use ephemeral IP (cheaper for development)

Environment Variables Set Automatically

The following environment variables are automatically configured in Cloud Run:

QDRANT_HOST=<vm-external-ip>      # Automatically set from VM
QDRANT_PORT=6333                  # HTTP port
QDRANT_API_KEY=<your-api-key>     # From terraform.tfvars

Quick Start

1. Prerequisites

# Install required tools
gcloud auth login
gcloud config set project YOUR_PROJECT_ID
terraform --version

2. Configuration

# Copy and edit configuration
cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars with your values

3. Deploy Everything

# From the root directory
./deployment/deploy.sh --deploy --build

4. Get Qdrant Information

# Get Qdrant VM IP and endpoints
./deployment/terraform/scripts/get_qdrant_ip.sh

# Get internal IP (for VPC connections)
./deployment/terraform/scripts/get_qdrant_ip.sh --internal

Terraform Outputs

The configuration provides several outputs for integration:

# Get specific outputs
terraform output cloud_run_url              # Cloud Run service URL
terraform output vector_db_vm_external_ip    # Qdrant VM external IP
terraform output vector_db_vm_internal_ip    # Qdrant VM internal IP
terraform output qdrant_http_endpoint        # Full HTTP endpoint
terraform output cloud_run_qdrant_host       # IP configured for Cloud Run

# Get deployment summary
terraform output deployment_summary

File Structure

terraform/
├── main.tf              # Cloud Run and main resources
├── vm.tf                # Qdrant VM configuration
├── outputs.tf           # Output definitions
├── variables.tf         # Variable definitions
├── terraform.tfvars     # Your configuration values
├── scripts/
│   ├── get_qdrant_ip.sh # Helper script to get VM IP
│   └── install_qdrant.sh # VM startup script
└── README.md            # This file

Environment Variables Reference

Automatically Configured

These are set automatically by Terraform:

Variable	Source	Description
QDRANT_HOST	VM external IP	Qdrant server IP address
QDRANT_PORT	Static value	Qdrant HTTP port (6333)
QDRANT_API_KEY	terraform.tfvars	Qdrant authentication key
FIRESTORE_PROJECT_ID	terraform.tfvars	GCP project ID
GCS_BUCKET_NAME	terraform.tfvars	Storage bucket name

Manual Configuration Required

Set these in terraform.tfvars:

project_id = "your-gcp-project-id"
storage_bucket_name = "your-bucket-name"
qdrant_api_key = "your-qdrant-api-key"
use_static_ip = true  # or false

Networking Configuration

Firewall Rules

The configuration automatically creates firewall rules to allow:

• Qdrant HTTP traffic (port 6333)
• Qdrant gRPC traffic (port 6334)
• Access from Cloud Run to Qdrant VM

IP Address Types

External IP (Default)

• Used for Cloud Run → Qdrant communication
• Accessible from internet (with firewall rules)
• Can be static or ephemeral

Internal IP (Alternative)

• Available for VPC-native connections
• More secure but requires VPC configuration
• Use cloud_run_qdrant_host_internal output

Troubleshooting

Check Qdrant Connectivity

# Get IP and test connection
./deployment/terraform/scripts/get_qdrant_ip.sh

# Manual test
QDRANT_IP=$(terraform output -raw vector_db_vm_external_ip)
curl http://$QDRANT_IP:6333/health

Verify Cloud Run Environment

# Check Cloud Run service
gcloud run services describe sereact --region=us-central1

# Check environment variables
gcloud run services describe sereact --region=us-central1 \
  --format="value(spec.template.spec.containers[0].env[].name,spec.template.spec.containers[0].env[].value)"

Common Issues

1. VM not ready: Qdrant installation takes 2-3 minutes after VM creation
2. Firewall blocking: Check that your IP is in allowed_cidr_blocks
3. Wrong IP type: Ensure you're using external IP for Cloud Run connections

Security Considerations

Production Recommendations

1. Use Static IP: Set use_static_ip = true
2. Restrict Access: Set specific CIDR blocks in allowed_cidr_blocks
3. Use Internal IP: Consider VPC-native networking for internal communication
4. Secure API Key: Store qdrant_api_key in Secret Manager

Development Setup

# terraform.tfvars for development
use_static_ip = false
allowed_cidr_blocks = "0.0.0.0/0"  # Open access (not for production!)

Cost Optimization

Development

• Use use_static_ip = false to avoid static IP charges
• Use smaller VM sizes: machine_type = "e2-micro"

Production

• Use use_static_ip = true for consistent connectivity
• Use appropriate VM sizes based on load

Advanced Configuration

Custom VM Configuration

Edit vm.tf to customize:

• Machine type
• Disk size
• Startup script
• Network configuration

VPC-Native Networking

For internal-only communication:

1. Create VPC connector
2. Configure Cloud Run to use VPC
3. Use internal IP output: cloud_run_qdrant_host_internal

Support

For issues with:

• Terraform: Check terraform plan output
• VM connectivity: Use the helper script get_qdrant_ip.sh
• Cloud Run: Check logs with gcloud run logs tail sereact