Skip to content

Humio Implementation
Best Practices

Based on Vijilan’s proven approach to log analytics and security operations

Abstract:​

Humio provides a solution for effective log management. It enables real time, large-scale log analytics, a necessity for Security Operations (SecOps), and DevOps. Vijilan, a longtime Humio partner, offers a unique Humio implementation process based on best practices and its own proprietary technology developed to make Humio work optimally. This paper explores these best practices and highlights what they mean for a SecOps and DevOps team who want to improve their logs collection, log storage and log analytics. While this paper focuses on SecOps, the methodology for logging and storing log in Humio for DevOps is performed in a similar fashion.
Humio provides effective log management

Introduction​

Effective log management is a critical, though often elusive, element of Security Operations (SecOps). Successful incident response is impossible without it. However, getting on top of logs from multiple devices, users, applications and other log sources can be a significant challenge for a SecOps team. Humio offers a solution, one that delivers real time, large-scale log analytics.

Making Humio work requires a focused, meticulous implementation. Vijilan, a longtime Humio partner, has developed a proven set of best practices for Humio implementation based on numerous client engagements over the last several years. Vijilan augments these practices with its own proprietary technology. This paper explores Vijilan’s Humio implementation best practices and highlights what they mean for a SecOps team that wants to improve its log analytics and incident response capabilities.

Overview

SecOps is responsible for the operational side of defending an organization’s digital assets against malicious actors. In practice, this means detecting and responding to threats quickly enough to prevent negative outcomes. A SecOps team will be doing its job well if it can, for example, detect a ransomware attack and respond by isolating the malware before it encrypts any valuable data. 

Successful SecOps involves the rapid, simultaneous analysis of enormous amounts of data from multiple sources. In particular, the SecOps team must run advanced analytics on logs from multiple systems, including security tools like intrusion protection systems (IPS’s) and firewalls, as well as operational systems like network switches and identity and access management (IAM) systems. With today’s highly sophisticated attackers, the evidence of a threat can easily be hiding in plain sight amongst a deluge of log data. Having the ability to correlate telemetry data from several security products on the fly allows SOC analysts to find therats quicker than performing it manually If SecOps can detect the threat quickly enough, it can prevent a breach.

Log management, analytics and SecOps

Log management challenges in SecOps

Log management presents a number of challenges to SecOps teams. In terms of requirements, log management demand fast ingestion of large amounts of log data from a variety of sources at great velocity. This is easier said than done, and some solutions require the SecOps team to set up server clusters to manage growth in log data sources. As log data volume grows—and it usually grows exponentially—it may also be necessary to deploy indexing farms. Storage requirements balloon in the process, too. Performance can lag, which may negatively affect the security analysis that is the point of the whole endeavor. SecOps may need to assign engineers just to the work of managing and updating the infrastructure that supports log management. 

Humio:

Solving log management challenges

Humio is a modern log management platform designed to process the scale and complexity of today’s log management workloads. Humio is designed with two key differentiators which make real-time analytics at scale possible: Data-streaming in an index-free architecture and high compression storage. These two factors combine to enable Humio users to ask anything and get instant responses from log data.

Best practices for Humio implementation

Vijilan has assessed and validated Humio as a log management solution for applications and security for over four hundred organizations in finance, healthcare, government, manufacturing, legal and education. A growing set of best practices has emerged from these experiences. When Humio is deployed in accordance with best practices, the client enjoys optimal usage of Humio, in terms of SecOps results, utilization of IT assets and team productivity.

Some of the best practices described here are based on a security event management web application that Vijilan has developed for Humio. The application enables users to better deal with real-time monitoring, correlation of events, notifications, visualizations, and presentation. It can correlate sequences of events, traps, logs and metrics across multiple dimensions directly from Humio.

Vijilan uses Humio for long-term storage, analysis, and reporting of log data, leveraging Humio as a Security Information and Event Management (SIEM) solution with advanced analytical capabilities. Through this partnership, Vijilan can provide a complete, end-to-end SIEM with log collection, parsing and normalization, real-time detection, analysis, and reporting.

Humio can run ultra-fast searches and queries against raw log data in seconds and presenting the information visually to users. The query window generates results from inputs ranging from a simple text-based search to a multi-variable structured query of billions of records within seconds.

Core processes

Best practices for Humio integration start with a series of core processes:
Some of these are not directly related to log management and analysis. The broader SecOps context matters for getting log management and analysis right. Humio implementers should understand where log management fits into the SecOps team’s mandate and obligations, e.g., if real time ransomware detection is a priority, then decisions about which logs to integrate into Humio ought to align with that priority. In this way, SecOps’ strategy and goals form a baseline for the Humio implementation project.
Humio:

Determine log integration requirements: current and future

The Vijilan approach to Humio implementation includes the development of an architecture definition that includes project requirements. As part of this architecture definition, the Humio implementation team must determine log integration requirements: which logs will be integrated? What connectors and other technologies will be needed? What are the parameters of the log data parsing and normalization? 

This thought process and development of requirements should apply to current and future log integrations alike. It may be difficult to know future requirements for sure but thinking through likely scenarios for future integration is a best practice, one that will pay off as the Humio instance inevitably evolves over time.

Operationalize log collection

The implementation process continues with the setup of log collection. This will involve the deployment of threat sensors and cloud connectors. At the functional level, what is happening is the creation of a data lake into which a huge, diverse amount of log data will flow. Making this work means ingesting and normalizing all the various log data streams and then “shipping” them into Humio. Vijilan has a pre-built solution for on-premises and cloud log collection. 

A best practice at this stage is to collect historical log data in addition to real time and recent data flows. Forensic analysis of old logs can reveal critical information about past attacks, which may have not been detected. Having access to historical log data in Humio can also facilitate predictive threat detection going forward.

Determine alerts and align with SecOps staffing and workflows

Vijilan collaborates closely with clients on determining alerts for Humio users. It is essential that alerts align with SecOps staffing and workflows. For any alert, there needs to be a SecOps team member to deal with it. or the alert needs to feed into an automated incident response system, such as ITSM solution. This matters because alert fatigue is a major issue in SecOps. Flooding the team with alerts is never optimal. Indeed, the practice can cause important threats to be missed. The best practice is to map the staffing and workflows carefully and design alerts to match those parameters.

Design a log analytics dashboard in alignment with SecOps staffing and workflows

Visualization of Humio log management is a key to SecOps effectiveness and team productivity. Vijilan works with Humio implementation clients to design and deploy customized dashboards. The best practice is to follow a dashboard development process that includes widget design, mapping of log sources, validation, and documentation.

Document Humio log analytics solution for specific SecOps parameters

Documentation tends to be neglected in IT projects, but that does not make it any less important. A Humio implementation must be documented to some extent. Because Humio already offers documentation of its solution, there is no need to draft a big book to go along with the implementation. A succinct piece of documentation, which highlights the parameters of the implementation, will prove itself to be extremely valuable as the Humio instance expands and evolves.

Proactive services

As the Humio implementation comes to life, Vijilan engages in further proactive services. Each of the following could be considered a best practice for getting the most out of Humio:

Run test cases and provide feedback to SecOps team

Identify and train key personnel

Assess the suitability of outsourcing log monitoring

Vijlian’s toolset offers a range of security analytics capabilities integrated with Humio’s functionality, including:

Run test cases and provide feedback to SecOps team

Setting up Humio is not a push button process. The solution needs tuning. The best practice, therefore, i is a continuous and iterative process with well-defined milestones and goals. As Humio processes the test cases, Vijilan provides feedback to the SecOps team. In this way, Vijilan can get Humio properly tuned while simultaneously mentoring the SecOps team on how to improve their log analytics and incident response processes.

Identify and train key personnel

People will need to make Humio work once it’s deployed. If the client does not already have staff trained for Humio, it’s a wise idea to use the implementation project period to identify and train key personnel. Management and admin for Humio does not have to be a full-time job. It can be part of someone’s role in SecOps. But someone will need to take responsibility for monitoring Humio and its supporting infrastructure.

Assess the suitability of outsourcing log monitoring

Outsourcing log monitoring can be a best practice for some organizations. The decision to engage with an external service provider for this workload may arise from a lack of personnel on staff or a desire to focus human resources elsewhere. Given the constraints on SecOps staff and the general difficulty in finding competent employees in this specialized work, letting an outsourced provider handle log monitoring may be a good move.

Conclusion

Conclusion

Getting Humio to work effectively means following best practices. With Vijilan as an implementation partner, an organization can realize its goals for real time, large-scale log analytics. SecOps will improve as a result. Best practices include determining log integration requirements for the present and the future, operationalizing log collection and solving the problem of “log shipping” and determining alerts and aligning them with SecOps staffing and workflows. A well-designed dashboard is essential, as is the process of documentation. Vijilan offers its own unique, proprietary toolset to facilitate the implementation in accordance with these best practices, and others, such as running test cases and training key personnel. As these factors come together, a successful Humio implementation will be the result.

About Humio

Humio is a modern log management platform that is purpose-built for today’s complex systems and scale. Humio is built with two key differentiators which make real-time analytics at scale possible: Data-streaming and index-free architecture and high compression storage. This enables customers to ask anything and get instant responses. The Humio platform also has a robust ecosystem that integrates with technologies (such as Vijilan) that will allow Humio to be used as a security solution among large-scale enterprise and education institutions.

About Humio

Humio is a modern log management platform that is purpose-built for today’s complex systems and scale. Humio is built with two key differentiators which make real-time analytics at scale possible: Data-streaming and index-free architecture and high compression storage. This enables customers to ask anything and get instant responses. The Humio platform also has a robust ecosystem that integrates with technologies (such as Vijilan) that will allow Humio to be used as a security solution among large-scale enterprise and education institutions.

About Vijilan

Vijilan Security has over fifteen years of experience specializing in monitoring, detecting, and responding to information security incidents. Vijilan is a US-based Limited Liability Company found in Aventura, Florida. Vijilan’s primary Security Operations Centers (SOCs) run 24/7, collecting events from private and public networks globally. Vijilan operates and stores all US-based customers’ information in the USA. This company is led by a team of engineers, developers, and highly skilled information security professionals, and services more than 500 small and medium businesses (SMB), focused on Banking, Health Care, Government and Education, in the US, Australia, South Africa, Brazil, and the UK. Vijilan has specialized in creating the security analytics necessary to collect, analyze and filter out relevant information from multiple verticals. Having broad visibility into hundreds of distinct and isolated networks has given Vijilan the ability to develop complex sequential and non-sequential correlations rules to detect malicious insiders and outsiders within minutes. With its unique, comprehensive cloud-based security incident response application, Vijilan can quickly detect, triage, and escalate to proper levels for immediate response. Additionally, Vijilan’s elastic, cloud-based architecture offers high availability, uptime and scalability, and seamless integration with virtually any on-premises and cloud application.

What our clients say

Share this page on:

Contact Information

Experience peace of mind with a reliable top tier security solution.
Need more information?
  • 954-334-9988

  • https://www.linkedin.com/company/vijilan-security-llc/

  • info@vijilan.com

  • 20803 Biscayne Blvd #302 - Aventura, Florida 33180