Featured engineering case studies / Customer diagnostics

On-premise customer diagnostics for SCAILIUM

I proposed and built a diagnostics stack for SCAILIUM customer installations in under two months. It collected logs, metrics and query traces, and replaced the previous ELK workflow across customer environments.

Product
SCAILIUM
Role
Senior Software Engineer (Python)
Contribution
Proposal, Python service, APIs, tests, Docker Compose, dashboards and documentation
Stack
Python, Flask, Grafana, Loki, Prometheus, Tempo, Docker Compose
Built in under two monthsDelivery
Approximately 500 engineersUsers
Around 20 companiesCustomers
8.7/10 average ratingFeedback
About one hourInvestigation

overview

Overview

SCAILIUM runs on customer infrastructure. Customer engineers need diagnostic data from their own installations, and customers need a package they can deploy without giving an external team access to their systems.

The existing ELK setup exposed logs, but it did not provide one path through node metrics, application logs and query traces. Investigations could take hours and sometimes longer.

previous workflow

Previous workflow

Engineers inspected host metrics and searched logs in the customer environment. When a critical bug, a large query or a performance bottleneck involved several signals, they had to reconstruct the sequence manually.

The diagnostics package also needed to work across different customer installations. SCAILIUM supplied the release, while customers deployed it on their own infrastructure.

constraints

Constraints

  • The entire stack had to run on premises.
  • Customers needed a repeatable Docker Compose distribution.
  • The solution had to collect node metrics and telemetry from the SCAILIUM binary.
  • Logs ranged from critical messages to debug output.
  • The package could not depend on Alar Studios operating customer infrastructure.

decision

Decision

I proposed a small Flask service around Grafana, Loki, Prometheus and Tempo. The service prepared and exposed the diagnostic data, while the observability components stored and visualised logs, metrics and query traces.

C++ engineers added the required telemetry hooks to the SCAILIUM binary. I wrote the Python service and most of the surrounding solution, including tests, packaging, dashboards and documentation.

Architecture

Public system view

  1. 01SCAILIUM nodes
  2. 02Logs, metrics and query traces
  3. 03Python service
  4. 04Loki, Prometheus and Tempo
  5. 05Grafana dashboards

All components run inside the customer's on-premise environment.

role

My role

I proposed the project and worked through the product, architecture and implementation questions with the relevant teams. I then built the Flask service, backend APIs, Docker Compose distribution, dashboards and technical documentation.

I added automated tests, supported the shared release repository and explained the system to engineers joining the codebase. C++ engineers handled the changes required inside the SCAILIUM binary.

testing

Testing

The project reached approximately 300 automated scenarios and 69% code coverage. The tests covered the backend and automated database operations, including checks that logs and telemetry arrived in the expected form.

outcome

Outcome

All customer installations moved from the previous ELK workflow to the new diagnostics package. Approximately 500 engineers across around 20 companies used it.

Customer developers rated the solution 8.7 out of 10 on average. An investigation that had previously taken hours or longer could usually be completed in about one hour.

lessons

Lessons

Start with the investigation path

The useful unit was the engineer's full investigation, not a single log or metric. That led to a stack that connected node state, application events and query traces.

Package the operating assumptions

Customers deployed the solution themselves, so configuration and documentation were part of the product. Docker Compose gave each installation the same starting point.

Ask for feedback after engineers have used the release

The 8.7/10 score came from customer developers who had worked with the released solution. They rated the complete investigation workflow, not an isolated dashboard.