Part of Eyevinn Open Analytics Solution
Eyevinn Open Analytics is an open source solution for tracking events from video players. Based on the open standard Eyevinn Player Analytics (EPAS) it enables a modular framework where you are not locked in with a specific vendor. This is the worker module that process the data from the eventsink and processing queue and stores it in a database.
Available as an open web service in Eyevinn Open Source Cloud. Read this documentation to quickly get started with the hosted solution.
To run it with SQS and Clickhouse hosted in Eyevinn Open Source Cloud:
% npm install
% npm run build
% AWS_ACCESS_KEY_ID=<SQS-ACCESS-KEY-ID> \
AWS_SECRET_ACCESS_KEY=<SQS-SECRET-ACCESS-KEY> \
SQS_ENDPOINT=https://<tenant-id>-<instance-name>.poundifdef-smoothmq.auto.prod.osaas.io \
SQS_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/1/events \
CLICKHOUSE_URL=https://<username>:<password>@<tenant-id>-<instance-name>.clickhouse-clickhouse.auto.prod.osaas.io \
DB_TYPE=CLICKHOUSE \
QUEUE_TYPE=SQS \
AWS_REGION=dummy \
npm start
The workers should start polling the SQS queue for messages and writing them to the Clickhouse database.
| Variable | Description | Default |
|---|---|---|
QUEUE_TYPE |
Queue type to use (SQS, beanstalkd, redis) |
Required |
DB_TYPE |
Database type (DYNAMODB, CLICKHOUSE) |
Required |
NUMBER_OF_WORKERS |
Number of worker instances to run | 2 |
SQS_PULL_INTERVAL |
Interval in ms between SQS polling cycles | 1000 |
SQS_CONCURRENT_RECEIVES |
Number of concurrent SQS receive calls per interval (max 10 msgs each) | 5 |
DB_PROCESS_INTERVAL |
Interval in ms between database write cycles | 2000 |
STARTUP_JITTER_MS |
Maximum random startup delay in ms to spread out worker starts | 5000 |
SKIP_QUEUE_EXISTS_CHECK |
Skip queue existence validation on startup (true/false) |
false |
MAX_AGE |
Maximum age in ms for events before they are discarded | 60000 |
PAUSE_DURATION |
Duration in ms to pause worker after repeated failures | 300000 |
Each worker runs two independent concurrent loops:
- SQS Producer Loop: Polls messages from SQS and adds them to an internal memory queue
- DB Consumer Loop: Processes messages from the internal queue and writes them to the database
This architecture allows SQS polling and database writes to happen concurrently, improving throughput. The internal queue acts as a buffer, decoupling the two operations.
When running multiple workers, the STARTUP_JITTER_MS setting helps avoid the "thundering herd" problem by adding a random delay (between 0 and the configured value) before each worker starts. This spreads out initial SQS and database connections across workers.
Join our community on Slack where you can post any questions regarding any of our open source projects. Eyevinn's consulting business can also offer you:
- Further development of this component
- Customization and integration of this component into your platform
- Support and maintenance agreement
Contact sales@eyevinn.se if you are interested.
Eyevinn Technology is an independent consultant firm specialized in video and streaming. Independent in a way that we are not commercially tied to any platform or technology vendor.
At Eyevinn, every software developer consultant has a dedicated budget reserved for open source development and contribution to the open source community. This give us room for innovation, team building and personal competence development. And also gives us as a company a way to contribute back to the open source community.
Want to know more about Eyevinn and how it is to work here. Contact us at work@eyevinn.se!