# Task 6: SQS & SNS, Async Microservices Communication

[DamirSt]

What was done?

Core Implementation

✅ Task 6.1 - SQS Integration & Batch Processing

• Created catalogBatchProcess lambda function in Product Service stack
• Implemented catalogItemsQueue SQS queue with batch size of 5 messages
• Configured SQS event trigger for catalogBatchProcess lambda
• Lambda processes SQS messages and creates products in DynamoDB tables

✅ Task 6.2 - Import Service Enhancement

• Updated importFileParser lambda function to send CSV records to SQS instead of CloudWatch logging
• Removed verbose logging from CSV stream processing
• Implemented SQS message sending for each CSV record with proper data validation

✅ Task 6.3 - SNS Integration

• Created createProductTopic SNS topic with email subscription
• Updated catalogBatchProcess lambda to send SNS notifications after product creation
• Configured email subscription for damirstanojevic@gmail.com

Additional Enhancements

✅ Unit Tests

• Created comprehensive test suite for catalogBatchProcess lambda function
• Tests cover multiple scenarios: successful processing, validation errors, empty events, malformed JSON

✅ SNS Filter Policies & Multiple Subscriptions

• Created additional email subscriptions:
  • damirstanojevic+expensive@gmail.com for products > $25
  • damirstanojevic+rock@gmail.com for Rock genre products
• Filter policies configured for targeted message distribution

Architecture & Infrastructure

✅ CDK Stack Configuration

• Complete SQS queue configuration with proper permissions
• SNS topic setup with email subscriptions
• Lambda functions configured with proper IAM policies for SQS and SNS access
• Environment variables properly configured for all services

---

Product Schema

interface Product {
  id: string;                    // UUID generated automatically
  title: string;                 // Product name (required)
  description: string;           // Product description (required)
  price: number;                 // Price in dollars (required)
  count: number;                 // Stock quantity (required)
  artist?: string;               // Artist name (optional, default: "Unknown Artist")
  category?: string;             // Product category (optional, default: "Music")
  genre?: string;                // Music genre (optional, default: "Rock")
  year?: number;                 // Release year (optional, default: current year)
  imageUrl?: string;             // Album cover URL (optional, default placeholder)
}

---

API Endpoints

Product Service API

• Base URL: https://uf4ds80g46.execute-api.us-east-1.amazonaws.com/prod
• Products List: GET /products
• Product by ID: GET /products/{productId}
• Create Product: POST /products

Import Service API

• Base URL: https://i6nl249ace.execute-api.us-east-1.amazonaws.com/prod
• Import Products: GET /import?fileName={filename} (returns signed URL for CSV upload)

---

Technical Implementation Details

SQS Flow

CSV Upload → S3 → importFileParser → SQS catalogItemsQueue → catalogBatchProcess → DynamoDB

SNS Notifications

Product Creation → SNS createProductTopic → Email Subscriptions → Filtered Distribution

Data Processing

• CSV parsing with proper error handling
• Data validation and type conversion
• Atomic DynamoDB transactions (products + stock tables)
• Batch processing with configurable size (5 messages)

---

Testing & Validation

Test Coverage

• Unit tests for catalogBatchProcess lambda (5 test cases)
• Integration testing via CSV upload workflow
• SNS notification testing with multiple email recipients

Test Data

• Created comprehensive Foo Fighters albums CSV with 11 records
• Real album cover images from Wikimedia Commons
• Various price points and genres for filter testing

---

Deployment & Configuration

AWS Resources Created

• SQS: catalogItemsQueue
• SNS: createProductTopic with 3 email subscriptions
• Lambda: catalogBatchProcess (updated), importFileParser (updated)
• DynamoDB: products and stock tables (existing)
• API Gateway: Product and Import service endpoints

Environment Variables

• PRODUCTS_TABLE, STOCK_TABLE, CREATE_PRODUCT_TOPIC_ARN
• CATALOG_ITEMS_QUEUE_URL, IMPORT_BUCKET_NAME

---

Additional Scope

• Jest testing framework setup and configuration
• AWS SDK v3 integration
• Error handling and logging improvements
• TypeScript type safety improvements
• IAM permissions and security best practices

---

Status: ✅ COMPLETE - All tasks implemented and deployed successfully
Branch: task-6 from latest master
Ready for: Code review and merge to master branch