Go Back

View Project

DCE Infosec | Hayanalysis

Internal Softwares

We built a scalable, cloud-native log analytics platform for Zamuun that enables real-time log ingestion, dynamic querying across PPL, SQL, and DSL, and custom dashboards—all integrated with AWS services, MongoDB, OpenSearch, and Redis. It empowers teams to manage observability at scale with full control over their data workflows.

May 16, 2025

6 months

Log Analytics, MERN, Opensearch, AWS

Problem Statement

Hayanalysis needed a high-performance log analysis platform to ingest, search, and visualize application logs in real time. Existing solutions were expensive or inflexible. The challenge was to build a system that supported multiple query languages, scaled horizontally, and gave users the ability to create custom dashboards and alerts—all without vendor lock-in or reliance on external processing pipelines.

Problem Statement

Hayanalysis needed a high-performance log analysis platform to ingest, search, and visualize application logs in real time. Existing solutions were expensive or inflexible. The challenge was to build a system that supported multiple query languages, scaled horizontally, and gave users the ability to create custom dashboards and alerts—all without vendor lock-in or reliance on external processing pipelines.

Approach

We developed a full-stack MERN application with OpenSearch as the core query engine, Firebase authentication, and MongoDB for state management. Our architecture integrated Logstash, S3, and Redis for scalable log ingestion and caching. We implemented a Chevrotain-based parser for PPL, DSL, and SQL support, connected it to OpenSearch, and built dynamic dashboards using Apache ECharts, all hosted on AWS.

Solution Overview

Solution Overview

Hayanalysis Logs is a self-hosted, extensible log intelligence platform that supports custom queries, user-level dashboards, alerts, and reports. It features a modular execution engine, multi-format query parsing, AWS-based ETL pipelines, and a frontend built with React, Firebase, and ECharts. The system enables complete observability with no external dependencies.

Solution Overview

Hayanalysis Logs is a self-hosted, extensible log intelligence platform that supports custom queries, user-level dashboards, alerts, and reports. It features a modular execution engine, multi-format query parsing, AWS-based ETL pipelines, and a frontend built with React, Firebase, and ECharts. The system enables complete observability with no external dependencies.

Multi-Format Query Parser

Multi-Format Query Parser

We built a Chevrotain-based parser to support PPL, DSL, and SQL queries, converting them into OpenSearch-compatible formats. The parser interprets clauses like `search`, `where`, and `stats`, and gracefully handles malformed input. This gives users the flexibility to interact with logs using their preferred query style—without switching tools or syntax frameworks.

Multi-Format Query Parser

We built a Chevrotain-based parser to support PPL, DSL, and SQL queries, converting them into OpenSearch-compatible formats. The parser interprets clauses like `search`, `where`, and `stats`, and gracefully handles malformed input. This gives users the flexibility to interact with logs using their preferred query style—without switching tools or syntax frameworks.

Autoscaling Execution Engine

Autoscaling Execution Engine

Our execution engine was designed to autoscale with traffic, ensuring stability under load. It handles parallel query execution, standardized responses across languages, and retry logic. The backend API routes dynamically allocate compute based on the size and complexity of incoming queries, making the platform resilient and responsive in high-throughput scenarios.

Autoscaling Execution Engine

Our execution engine was designed to autoscale with traffic, ensuring stability under load. It handles parallel query execution, standardized responses across languages, and retry logic. The backend API routes dynamically allocate compute based on the size and complexity of incoming queries, making the platform resilient and responsive in high-throughput scenarios.

Custom User Dashboards

Custom User Dashboards

We enabled users to create and save personalized dashboards using Apache ECharts. These dashboards support visualizations like bar, line, pie, and area charts, with real-time updates. Users can map data fields, toggle filters, and generate reports—all without code. This modular dashboarding engine offers enterprise-grade observability tailored to each user’s workflow.

Custom User Dashboards

We enabled users to create and save personalized dashboards using Apache ECharts. These dashboards support visualizations like bar, line, pie, and area charts, with real-time updates. Users can map data fields, toggle filters, and generate reports—all without code. This modular dashboarding engine offers enterprise-grade observability tailored to each user’s workflow.

ETL with Logstash and OpenSearch

ETL with Logstash and OpenSearch

We set up a complete log ingestion pipeline using Logstash on AWS EC2 to pull structured and unstructured logs from S3, transform them, and push to OpenSearch. We added conditional logic to handle multiple formats and tagged logs with user-specific metadata. Redis was used for fast retrieval and caching, enabling efficient ETL and rapid log search at scale.

ETL with Logstash and OpenSearch

We set up a complete log ingestion pipeline using Logstash on AWS EC2 to pull structured and unstructured logs from S3, transform them, and push to OpenSearch. We added conditional logic to handle multiple formats and tagged logs with user-specific metadata. Redis was used for fast retrieval and caching, enabling efficient ETL and rapid log search at scale.

Impact

Hayanalysis Logs delivered a robust observability layer that matched enterprise standards—at a fraction of the complexity and cost. With advanced querying, real-time visualizations, and scalable ingestion, it empowered engineering teams with end-to-end visibility and control over their application infrastructure.

Zero downtime at peak load

100% log indexing via ETL pipeline

Zero downtime at peak load

100% log indexing via ETL pipeline

Throw us a challenge?

Book a call

Book a call

Or

Send inquiery

Send inquiery

Throw us a challenge?

Book a call

Book a call

Or

Send inquiery

Send inquiery

Throw us a challenge?

Book a call

Book a call

Or

Send inquiery

Send inquiery

Throw us a challenge?

Book a call

Book a call

Send inquiery

Send inquiery