By Guest Blogger, David Humphrey
In this day and age of; “plug-and-play”, the need for instant online gratification, and appliances that encapsulate all of the functionality you need in one convenient shiny wrapper, the idea of installing a SIEM into the corporate environment is not met with that much trepidation. After all, how bad can it be? Everything is made to work with everything else out there, and implementing a SIEM should not be that much of a challenge.
The perception that much of our IT management have is that implementing new IT functionality will not impact the IT service management model significantly, and should always be easier than it once was in the recent past for other IT projects. Yet, while a SIEM installation is not rocket science, it is prudent to recognize just what the true cost of running and maintaining a SIEM will be. And the first step to take along this path is to set the correct level of expectations on just what the project is...
One should consider it to be on the level of replacing the piston rings in the engine of your car.
Now, if you think that clearing the space in your garage for such a project, acquiring the tools to properly address this engineering feat, and mustering the patience to tackle a lot of unknown issues with your very-familiar vehicle might be daunting - you’re on the right plane of where your level of expectations should be for putting a SIEM into the environment. Sure, there are plenty of guys out there that can replace a head gasket, but for most of us, that task is well over our heads.
Most of us are going to pay the dealership cost to have them do it. And that cost will be far more that we want to pay for maintaining our vehicle. But this is because we never put the maintenance cost into the evaluation matrix for when we originally purchased said transportation. When the time came to select that new VW bug over the Honda Accord, well the VW won out on all the cool features. Now comes the cost of maintenance…
I still remember the vendor meetings at my company when I brought the “technology” in to meet the IT staff. Oh, the questions they asked! Here is a wonderful example: since logs are, by default, available on the network, what needs to be done to this new SIEM to suck these down and begin reporting on the environment? What will it cost for the vendor to configure their new toy to work with application ‘X’? Does it work with application ‘Y’?
The answers were succinctly typical in respective order; it comes with a lot of reporting functionality right out of the box; ‘yes’ and ‘yes’. It will work with all of your IT systems.
And that was true.
What no one realized needed to be asked; was not how well the SIEM could be used in the environment, but what needed to be done in the environment to use a SIEM.
All of the Unix systems need a centralized logging system to be developed for the SIEM – the logs were not magically available in the ether to be sucked off the network. The domain controllers needed a GPO update to determine what was going to be logged, and whether or not storage was going to be sufficient, locally, to wait for the SIEM to query them. The Oracle databases had to be configured to use Audit logging, and 30 or more configuration statements would need to be tuned to create pertinent audit logs, including securing access to those logs, and an automatic purging stored procedure to dump logs older than several days so that the database did not wipe out all available disk space. All of those shiny appliances needed to be administered so that they too would log directly to the SIEM, and each web-server would need a new process installed on the server to monitor the web logs so that those lines too could be sent to the SIEM. In short, the work necessary for the environment to log to the SIEM involved three separate management teams, hundreds of change management entries, and a bit of network / storage re-engineering to just begin the process of getting logs.
Don’t get me started on what it takes to log from AWS and O365.
So what is the first thing to note about the true cost of owning and operating a SIEM? It was this: when you determine a SIEM is necessary for your environment, your true cost of integration is directly proportionate to the ITSM management cost of your entire IT infrastructure. It will take a strong IT wizard at the level of your BMW mechanic, to get to all of the silos in your organization and talk technical about the needs and methods to extract the logs that the system needs to work. It will take time and effort to get things set up to log to the SIEM, and this is going to be a manpower initiative proportional to the complexity of your organization.
Now we need to fill in the rest of the story for your TCO on this functionality!
Why? Because you’re growing and changing. You cloned your production database to upgrade your back-end hardware and switched over the instance to a new TNS identifier for the database itself. It all went smoothly. But the SIEM didn’t know this. And now your Oracle logs are gone. When you upgraded the NAS mount for your logging host, and turned off the “RSyslogd” daemon to “RSync” the old filesystem to the new one, you forgot to restart the daemon. The SIEM didn’t know this, and now your Unix logs are gone. You went with the newest cloud service provider to support your AntiVirus implementation, and realize that you don’t have logs from the cloud to the SIEM. You’ll need to provision a logging relay in the DMZ to get those logs into the SIEM, and… yes, now you no longer have endpoint logging anymore.
And we haven’t even gotten to reporting and analysis yet.
There was a reason to put a SIEM into the environment, and whether it was for audit compliance, centralized logging, automatic report generation of events, incident alerting to your SOC monitor, or complex user analytics - once the data is in there, you have to be able to do something with it. And once you start doing things with the data, you will need personnel to act and respond to the heuristics coming out of that tool. Someone is going to review your “dial-in” VPN access logs to see if one UserID is used to get on the network from Bangalore India at the same time that it was used to login in from the Eastern Ukraine; someone is going to figure out if 10 or more password attempts on an account within 1 minute is normal or not. In short, the SIEM will deftly be able to produce this output for you, but someone has to know enough about your global IT footprint to act upon it. And they should also be able to act upon data anomalies in those report – did the logging on the IPS get turned off for a reason, or did some hacker turn it off to cover his tracks. They need to be fairly IT knowledgeable.
The true cost of owning a SIEM? Wisdom. Your management should recognize the high level of IT personnel resources needed to maintain this component, and invest in that personnel. The return is enormous. But directly proportional to this personnel investment. Your environment is dynamic with a lot of moving IT parts. The SIEM has to keep up, it also has to be maintained with patching and hardware refresh. You will need some part of personnel to maintain and expand the data flows as they change and grow. You will need some part of personnel to analyze the data, and update the reporting. And while this may no longer require your BMW mechanic, it is certainly going to require a good garage mechanic. In other words, you need to staff accordingly, provision the manpower, and pro-actively monitor your data sources daily. You can outsource the SIEM to the cloud, and get exactly what you paid for; a logging repository with a helpdesk to turn to when you figure out what changes you need to have made. But you are still going to maintain those data flows, and you cannot outsource your business intelligence – only you have the resources for that.
By PJ Bihuniak
To begin with, we want to thank David for his contribution to our blog. We thought our clients and partners would appreciate a fresh perspective on the challenges and true costs of owning and operating a SIEM and will look to have other guest contributors to this space.
There are many ways to keep the costs of running and operating a SIEM under control. Some of those methods include:
- On premise SIEM with in-house management and event monitoring
- On premise SIEM with outsourced management and event monitoring
- On premise SIEM with outsourced management and in-house event monitoring
- Outsourced SIEM in the cloud where the provider does the management and event monitoring
As David indicated, it is vitally important to understand that regardless of your method, only you will have the business intelligence and context of the events that are in your environment. No event monitoring service will possess the wealth of intelligence that you possess, no matter how much you work with your partner. Your environments change too quickly and it is almost impossible to stay abreast of these changes as a partner.
Our experience is that clients really appreciate the value of Option 3 as they get the financial benefits of outsourced management and monitoring while they get to focus on the content creation and security event monitoring as they possess the business context of the alerts. If you are interested in learning more about this path and how ThetaPoint can deliver this for you, please don’t hesitate to contact us directly.
About ThetaPoint, Inc.
ThetaPoint is a leading provider of strategic consulting and managed security services. We help clients plan, build and run successful SIEM and Log Management platforms and work with the leading technology providers to properly align capabilities to clients needs. Recognized for our unique technical experience, in addition to our ability to quickly and rapidly solve complex customer challenges, ThetaPoint partners with some of the largest and most demanding clients in the commercial and public sector. For more information, visit www.theta-point.com or follow us on Twitter or Linked-In.