The opinions expressed herein are my own and do not represent my past, present and future employer’s views in any way. Nothing posted here should be considered official or sanctioned by my past, present or future employer or any other organization I’m affiliated with.
MSS - Managed Shitty Services
Written by Duarte Castelo Grande de Carvalho (dcgc)
The informatization and technological adaptation by archaic, antiquated and classic industries and businesses such as financing (e.g., banks, trading), manufacturing (e.g., factories, shipment), public service (e.g., government, NGO), among others, as led to instituions within these fields to procure outsourcing services that can help them keep up with the demand to enter the new digital world and to be able to serve and fulfill the demand of the globalization and the ramping rate at which these businesses work nowadays. Of course, also because, outsourcing saves costs for capital expenditures! Several companies such as the Big 5 and other Vendor companies provide services that help institutions such as the aforementioned, to be able to adapt with ease and painlessly to these current norms. One would imagine and expect that by having contract signings with high fee-rates, high buy-in, high investment with these service providers, that customer companies would be betting on having a top-notch managed IT services and qualified personnel. Unfortunately, because of the competitive nature of the contract signing between several of these competitors in the market, it entails that a lot of corners have to be cut, meaning that things like budget, expertise and technology of these managed IT services, in the end, are affected and their overall caliber is diminished, turning it into a sub-par rendered service.
Shortly after my TAC experience at the Vendor company I was at (you can read about it in another blog post), I moved towards one of these so-called Managed Security Services (MSS), in which it serves hundreds of customers, have different teams around the globe, follow strictly IT Service Management and IT management frameworks such as ITIL, and that pride themselves on having premium on-the-clock support in case any incident arises. In theory, MSS are the systematic approach to manage a business’s information security needs. MSS offers a comprehensive set of security options managed by external service providers and these options include 24/7 automated monitoring, security upgrades, assessments, and audits. In reality, MSS are the systematic approach to maintain and not let a hanging-by-the-wire infrastructure (designed by architects who are no longer with the company) to be destroyed, so that the business can continue to operate. MSS offers 24/7 manual and flawed monitoring done by “engineers” in developing countries or in countries and states in the EMEAR and AMER region, respectively, where the labor is cheap, and also provides checkbox assessments and audits, half-assed problem-inducing security changes and upgrades (sometimes done “by-the-book”).
The trend we see happening currently due to the piss-poor MSS and disguised Break/Fix that companies have faced during the last 10 to 15 years, due to the fact that security breaches and targeted hacking have been mainstreamed and popularized (fear-mongering by media), and due to the ridiculously high contract expenditure needed to hire a contractor service, is that it became clear that more and more companies are starting to build these teams in-house and attracting talent with higher salaries and with the promise of building a MSS infrastructure from the ground up (which can be tempting and alluring to several security professionals). It is not a huge shift, but it is clear that it is happening due to the amount of customers not renewing or breaking contract with the MSS’s. The MSS I was in was one of the last that run its course, having its department budget being cut in more than half, with no training and onboarding whatsoever being provided and with shallow hiring that leaned on having contractors working inside the company that probably shouldn’t have ended up there.
The typical rendered service, technology-stack and process/framework of a MSS
There are several MSS that are provided by big companies, as explained before, where the actual offer can differ from one another due to the fact that the companies offering the MSS service have their own strengths and weaknesses regarding security technologies, product/service catalogs that possess, and their market placement within the IT industry. So when it comes to packaging the MSS offer, some companies could be:
- More oriented towards a perimeter management and monitoring of the system/networking infrastructure (Health and Wellfare (H&W)) (possibly, with a security frosting on it) and allowing the hands-on maintenance, improvement and performance trend analysis of the infrastructure.
- Based on product resale where the value-added hardware and software (for a variety of security-related tasks) become available (being deployed, or not, in the customer premises) to the customer and allow archival, normalization and presentation of data that is useful for enrichment of other security services within the MSS offer or outside of it.
- Assisting the customer on-site in the assessment of business risks, key business requirements for security and the development of security policies and processes, which could include comprehensive security architecture assessments and design (including technology, business risks, technical risks and procedures).
- Testing and analyzing the technical and logical perimeter of the customer by performing penetration testing and vulnerability assessment services (usually these are acquired outside of the MSS offer due to regulatory reasons).
- Complying the security baseline of the infrastructure and other related security configuration based on industry regulations, standards and mandates.
Ocasionally, a MSS may have some of these services packaged together and position themselves strongly in the service market, but very frequently it ends up only offering one or two of the listed services, reason being that MSS providers themselves want to build, enable and provide the services in a cost-effective manner. By focusing upon core competencies of the company itself (e.g., a network vendor will provide H&W for network infrastructure), having framework and tools at their disposal to allow round-the-clock service (e.g., customer experience tools and site reliability contracts), it makes it so that the MSS offering ultimately is a reduced one, but on a counterpoint, a strong one. An important factor to consider in these types of dynamics, specific to MSS, is that outsourcing security service hands over critical control of the customer’s infrastructure to an outside party, the MSS company. While it doesn’t relieve the customer of having the ultimate responsibility for errors, it means that the customer of an MSS company still has the ultimate responsibility for its own security, and as such must be prepared to manage and monitor the outcome of the MSS’s actions, and hold it accountable for the services for which it is contracted. The lack of transparency in the signed contracts, which include the Service Level Agreements (SLA), Service Level Objectives (SLO), Service Level Indicators (SLI) and the covered perimeter originate a lot of heated arguments between the customer of the MSS and the MSS itself, especially when shit hits the fan. The business risks that this entails can result in information assets, upon which the business depends, not being securely configured and managed, which in turn could result in asset compromise due to violations of confidentiality, availability, and integrity. System being unavailable, data in-transit not being encryted, infrastructure being changed operationally and configuration-wise, you name it…
Let’s tackle some of the most common culprits in the MSS offering around town and, certainly, some that I have observed in my time. Let’s look at their shortcomings and their ulterior motives.
The ‘1984’ Customer Experience
Customer Experience is one of those concepts that have been throwed around for the last decade and mostly it revolves around having a better and more transparent time response and process/lifecycle of the work being done by the MSS, while having a nicer and more corteous messaging/talking with the customer, being done from the top to the bottom of the hierarchy. It is overall the betterment of the service itself and being more upfront with a lot of the backstage movements and shenanigans, as done in the past. The idea is well thought through and has good intentations, but the problem that comes with this approach is that a lot of the customers themselves are not IT-oriented, do not have the personnel (either hired or as a point of contact) and are not fair with the MSS, meaning that a lot of the processes and tools associated with this movement tend to get exploited and overused to exhaustion by the customers, provoking pressure, instability and anxiety towards the engineers in the MSS (who are not prepared for this), only so because of the customer wanting to hanker after the solution to the problem at hands, immediatelly. The MSS is supposed to serve their expectations and engage with the customer in order to show that there is committment towards them. Technical Managers, Delivery Managers, Customer Relations Spokepersons, and other people involved, most of them are weak and lack spine to go back to the customer and have a resonable and civilized discourse on how to properly work within the boundaries of the signed contracts and the service offering, and to explain why and how the customer is proceeding wrongly and mischievously. The results are that the customer mostly has the upper-hand over the MSS and handles the negotiations one-sided. Usually, the MSS people in contact with the customer comply completely and scold at their internal teams.
The other aspect of this approach is the technology used around this. The prime example are the Customer Relationship Management software, the Shift Managing software, etc. All of them serve the purpose of consistently and uniformly having a pipeline of the course of actions involved when handling different parts of the offering, as well as having the people responsible for operating within the MSS offering, doing it so in an efficient and effective manner. It is also regard as a time-controlling mechanism, workload-handled mechanism and a metrics generator. Businesses turn to MSS to alleviate the pressures they face daily related to information security and to which they cannot meet the demand, but in turn, expect the MSS offering to go far and beyond. The tools visibilly externally (public-facing) and to which the customer interact the MSS with, the numerous amount of contact channels to which they can communicate with and spam, make the work routine for a MSS engineer a living hell. It also cathes the eye of the management and people responsible and concerned with the MSS offering performance and word-of-mouth around it. These people will start hammering down when using the internal tools at their disposal, controlling their teams and making sure that their efficiency and efficacy are meeting their irrealistic standards. The amount of bathroom breaks and their duration, the lunch time, the amount of work done on a ticket, the envolvement dispended on bridge calls, everything has a number attached to it. Because of this constant systematic abuse, engineers also use these tools to their advantage and try to circumvent these mispleasent sentinels by allowing them more time to work or to rest. The internal relationship between the technical people and the non-technical people is based on diffidence, suspicion and fear, and the external relationship between MSS and customer resembles an arm-wrestling competition.