I’ve got a bone to pick with Forrester’s article “If You’re Not Using Data Pipeline Management for Security And IT, You Need To”.
In today’s cybersecurity landscape, organizations face mounting pressure to reduce costs, particularly in their Security Information and Event Management (SIEM) operations. While Forrester is pushing the idea called Data Pipeline Management, I argue that selective data ingestion to manage expenses may create more problems than it solves. In my experience, restricting SIEM data could significantly impair security operations and ultimately prove more costly in the long run.
The argument for comprehensive data collection is more compelling when we examine the realities of modern security operations. Security analysts require complete visibility to effectively detect and respond to threats. When organizations artificially limit data ingestion, they create dangerous blind spots that sophisticated attackers can exploit. These visibility gaps are particularly problematic during incident investigations, where missing data points cannot be recreated after the fact, potentially hampering forensic analysis and incident response efforts.
Moreover, selective data restriction significantly impacts a SOC analyst’s ability to maintain situational awareness. Modern cyber threats often leverage multiple attack vectors and progress through various stages. Without comprehensive data access, analysts may miss crucial connections between seemingly unrelated events or fail to identify subtle patterns that indicate a broader attack campaign.
The economic argument for data restriction has also weakened considerably in recent years. Overall storage costs have decreased dramatically, making the cost-per-gigabyte argument less compelling than it was a decade ago. Additionally, modern data lakes and tiered storage solutions offer increasingly cost-effective alternatives for long-term data retention, allowing organizations to maintain comprehensive historical data without breaking the budget.
Technological advances in data compression and storage optimization provide another counter to the restriction argument. While some sources suggest that indexed security data can expand to 3-5 times its original size, modern compression algorithms and purpose-built security data compression tools can achieve significantly better ratios. When combined with columnar storage formats and deduplication techniques, organizations can maintain comprehensive data sets while minimizing storage requirements.
Perhaps most importantly, restricting data access creates unnecessary obstacles for SOC analysts who already face significant challenges in their daily operations. The cybersecurity skills gap remains a pressing issue, and limiting analysts’ access to potentially crucial data only makes their jobs more difficult. The cost of missing a security incident due to incomplete data could far outweigh any storage savings realized through data restriction.
Looking toward the future, comprehensive data collection becomes even more critical. Machine learning and advanced analytics require robust datasets for effective pattern recognition and anomaly detection. As new analysis techniques emerge, historical data that might seem unnecessary today could prove invaluable tomorrow. Organizations that restrict their data collection now may find themselves at a significant disadvantage when implementing these advanced capabilities.
Instead of focusing on data restriction, organizations should invest in smarter data management strategies. This includes implementing intelligent data tiering, leveraging modern compression techniques, and developing tools that make data more accessible and actionable. By maintaining comprehensive data collection while optimizing storage and analysis capabilities, organizations can better position themselves to meet both current and future security challenges.
In the end, the true cost of security isn’t measured in storage fees—it’s measured in our ability to detect, prevent, and respond to threats effectively. When we artificially limit our visibility into our security environment, we risk creating false economies that could ultimately prove far more expensive than the storage costs we’re trying to save.
At Impelix, we do not subscribe to the idea of selective data ingestion. Come checkout our IMPACT Security Operations platform where we allow unlimited ingestion of data and help SOC analysts have full situational awareness.