Featured engineering case studies / Customer diagnostics
On-premise customer diagnostics for SCAILIUM
I proposed and built a diagnostics stack for SCAILIUM customer installations in under two months. It collected logs, metrics and query traces, and replaced the previous ELK workflow across customer environments.
- Company
- Alar Studios
- Product
- SCAILIUM
- Role
- Senior Software Engineer (Python)
- Contribution
- Proposal, Python service, APIs, tests, Docker Compose, dashboards and documentation
- Stack
- Python, Flask, Grafana, Loki, Prometheus, Tempo, Docker Compose
overview
Overview
SCAILIUM runs on customer infrastructure. Customer engineers need diagnostic data from their own installations, and customers need a package they can deploy without giving an external team access to their systems.
The existing ELK setup exposed logs, but it did not provide one path through node metrics, application logs and query traces. Investigations could take hours and sometimes longer.
previous workflow
Previous workflow
Engineers inspected host metrics and searched logs in the customer environment. When a critical bug, a large query or a performance bottleneck involved several signals, they had to reconstruct the sequence manually.
The diagnostics package also needed to work across different customer installations. SCAILIUM supplied the release, while customers deployed it on their own infrastructure.
constraints
Constraints
- The entire stack had to run on premises.
- Customers needed a repeatable Docker Compose distribution.
- The solution had to collect node metrics and telemetry from the SCAILIUM binary.
- Logs ranged from critical messages to debug output.
- The package could not depend on Alar Studios operating customer infrastructure.
decision
Decision
I proposed a small Flask service around Grafana, Loki, Prometheus and Tempo. The service prepared and exposed the diagnostic data, while the observability components stored and visualised logs, metrics and query traces.
C++ engineers added the required telemetry hooks to the SCAILIUM binary. I wrote the Python service and most of the surrounding solution, including tests, packaging, dashboards and documentation.
Architecture
Public system view
- 01SCAILIUM nodes→
- 02Logs, metrics and query traces→
- 03Python service→
- 04Loki, Prometheus and Tempo→
- 05Grafana dashboards
All components run inside the customer's on-premise environment.
role
My role
I proposed the project and worked through the product, architecture and implementation questions with the relevant teams. I then built the Flask service, backend APIs, Docker Compose distribution, dashboards and technical documentation.
I added automated tests, supported the shared release repository and explained the system to engineers joining the codebase. C++ engineers handled the changes required inside the SCAILIUM binary.
testing
Testing
The project reached approximately 300 automated scenarios and 69% code coverage. The tests covered the backend and automated database operations, including checks that logs and telemetry arrived in the expected form.
outcome
Outcome
All customer installations moved from the previous ELK workflow to the new diagnostics package. Approximately 500 engineers across around 20 companies used it.
Customer developers rated the solution 8.7 out of 10 on average. An investigation that had previously taken hours or longer could usually be completed in about one hour.
lessons
Lessons
Start with the investigation path
The useful unit was the engineer's full investigation, not a single log or metric. That led to a stack that connected node state, application events and query traces.
Package the operating assumptions
Customers deployed the solution themselves, so configuration and documentation were part of the product. Docker Compose gave each installation the same starting point.
Ask for feedback after engineers have used the release
The 8.7/10 score came from customer developers who had worked with the released solution. They rated the complete investigation workflow, not an isolated dashboard.