Legacy Systems & Migrations

Historical documentation for deprecated systems and completed migrations.

Overview

This document covers systems that have been removed or deprecated from the Eventify platform:

• Kafka - Message streaming (removed)
• Elasticsearch - Search engine (removed)
• Debezium CDC - Change data capture (disabled)
• PostgreSQL Arrays - Migrated to JSON strings for D1 compatibility

---

Kafka Removal

Date: October 2024
Status: ✅ Fully removed

Why Removed

1. Primary consumer (Elasticsearch) was removed
2. No active consumers of change streams
3. Simplified infrastructure
4. Reduced deployment complexity

What Was Removed

Services: - Kafka broker (port 9092) - Zookeeper (ports 2181, 2888, 3888) - Debezium Connect (ports 8083, 5005)

Dependencies: - kafkajs npm package

Files Disabled: - consumer/main.ts - Kafka consumer service - consumer/processors/premise-fulfilled/ - CDC processors

Infrastructure Impact

Before:

# docker-compose.yml
services:
  kafka:
  zookeeper:
  connect:  # Debezium

After:

# All commented out with # KAFKA REMOVED markers

Rollback Instructions

If Kafka needs to be restored:

1. Uncomment services in docker-compose.yml
2. Install dependency: npm install kafkajs@^2.2.4
3. Restore consumer logic in consumer/main.ts
4. Run npm install to update lockfile

---

Elasticsearch Removal

Date: September 2024
Status: ✅ Fully removed

Why Removed

1. Migrating to Cloudflare D1 (SQLite)
2. No complex full-text search requirements
3. Reduced infrastructure costs
4. Simplified deployment

What Was Removed

Services: - Elasticsearch cluster

Dependencies: - @elastic/elasticsearch - @elastic/transport

Files Disabled: - src/esClient.ts - ES client (now stub) - src/lib/actions/premise-filter/listing.ts - Search queries - src/lib/actions/premise-filter/aggregations.ts - Aggregations - consumer/processors/premise-fulfilled/upsert-premise-fulfilled-index.ts - Indexing

Return Value Changes

Function	Before	After
getPremises()	ES search results	{ hits: [], total: 0 }
getPremiseAggregations()	ES aggregations	{}
getPremiseAggregationsByType()	Type aggregations	{ buckets: [] }

Frontend Impact

• Premise search pages show no results
• Map search interface shows empty map
• Filter components render without options
• All non-search pages function normally

Rollback Instructions

1. Install dependencies: npm install @elastic/elasticsearch @elastic/transport
2. Search for // ELASTICSEARCH REMOVED comments
3. Uncomment relevant code blocks
4. Restore environment variables:

   ES_INDEX_PREMISE_FULFILLED=your_index_name
   

5. Uncomment ES service in docker-compose.yml

---

Debezium CDC (Disabled)

Status: ⚠️ Configured but inactive

What is Debezium?

Debezium is a change data capture (CDC) platform that monitors databases and streams row-level changes to Kafka topics.

Previous Use Case

• Monitor PostgreSQL for INSERT/UPDATE/DELETE
• Stream changes to Kafka topics
• Feed real-time data to Elasticsearch
• Enable event-driven architecture

Current State

Database Image: Still uses debezium/postgres:16-alpine

This image includes: - Standard PostgreSQL 16 - WAL configuration for CDC - Logical replication slots - pg_output plugin

Configuration Preserved:

wal_level = logical
max_replication_slots = 4
max_wal_senders = 4

Why Disabled

1. Elasticsearch removal eliminated primary consumer
2. Kafka removal eliminated message bus
3. No downstream systems need change streams
4. Application queries database directly

Re-enabling Debezium

If CDC is needed again:

1. Uncomment services in docker-compose.yml:

   zookeeper:
   kafka:
   connect:
   

2. Configure connector via REST:

   curl -X POST http://localhost:8083/connectors \
     -H "Content-Type: application/json" \
     -d '{
       "name": "postgres-connector",
       "config": {
         "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
         "database.hostname": "db",
         "database.port": "5432",
         "database.user": "postgres",
         "database.password": "postgres",
         "database.dbname": "eventify",
         "database.server.name": "dbserver1",
         "table.include.list": "public.premises,public.events"
       }
     }'
   

3. Monitor change streams:

   docker exec -it kafka kafka-console-consumer \
     --bootstrap-server localhost:9092 \
     --topic dbserver1.public.premises
   

Debezium Benefits

When active: - Real-time sync to downstream systems - Event-driven architecture - Complete audit trail - Exactly-once delivery - Schema evolution support

Alternative Solutions

Without Debezium:

1. Database Triggers - Custom triggers for specific tables
2. Application Events - Emit events from business logic
3. Polling - Periodic batch synchronization
4. Materialized Views - For aggregated data

---

JSON Array Migration

Date: October 2024
Status: ✅ Completed
Reason: SQLite/Cloudflare D1 doesn't support PostgreSQL arrays

Fields Migrated

Premise Model

• amenities: String[] → amenities: String (JSON)

Event Model

• tags: String[] → tags: String (JSON)

User Model

• interests: String[] → interests: String (JSON)

Place Model (Deprecated)

• amenities: String[] → amenities: String (JSON)
• parkingArea: String[] → parkingArea: String (JSON)
• typeOfPremises: String[] → typeOfPremises: String (JSON)

Helper Functions

Use utilities from src/lib/utils/json-arrays.ts:

import { 
  parseJsonArray, 
  stringifyJsonArray, 
  jsonArrayIncludes 
} from '@/lib/utils/json-arrays';

// Parse JSON array
const amenities = parseJsonArray<string>(premise.amenities);
// Returns: ['wifi', 'projector', 'parking']

// Convert to JSON
const json = stringifyJsonArray(['wifi', 'projector']);
// Returns: '["wifi","projector"]'

// Check inclusion
const hasWifi = jsonArrayIncludes(premise.amenities, 'wifi');
// Returns: true

Updated Files

Core Logic: - src/lib/utils/json-arrays.ts - Helper functions - src/lib/actions/premise/prepare-amenity-list.ts - Returns JSON - src/app/[locale]/profile/(company)/venues/[id]/_premise-form/default-form-amenities.ts

Components: - src/components/premise/PremiseAmenities.tsx

Utilities: - src/utils/profileCompleteness.ts

Seeds: - prisma/seed.ts - prisma/seed.prod.ts

SQLite JSON Queries

-- Check if array contains value
SELECT * FROM Premise 
WHERE json_extract(amenities, '$') LIKE '%wifi%';

-- Get array length
SELECT json_array_length(amenities) 
FROM Premise;

-- Extract element
SELECT json_extract(amenities, '$[0]') 
FROM Premise;

Performance Considerations

Indexing: - JSON fields cannot be directly indexed in SQLite - Consider junction tables for frequent queries - Use computed columns for common operations

Querying: - JSON operations slower than native arrays - Cache parsed data in application layer - Denormalize for high-performance needs

Common Issues

Type Errors:

// ❌ Wrong
const amenities: string[] = premise.amenities;

// ✅ Correct
const amenities = parseJsonArray<string>(premise.amenities);

Null/Undefined:

// ✅ Always provide fallback
const items = parseJsonArray(jsonString, []); // Default to empty array

Performance: - Cache parsed arrays in component state - Use React.memo for expensive parsing - Move heavy operations server-side

---

Migration Timeline

Date	Change	Status
Sep 2024	Elasticsearch removal	✅ Complete
Oct 2024	Kafka removal	✅ Complete
Oct 2024	JSON array migration	✅ Complete
Oct 2024	Debezium disabled	⚠️ Inactive
Nov 2024	PostgreSQL → D1 migration	🚧 In progress

---

Related Documentation

• Setup & Local Development
• Database Schema
• Storage
• Deployment