You don't have to stay in TAC

Written by Duarte Castelo Grande de Carvalho (dcgc)

PRISON

Modern slavery is the exploitation of other people for personal or commercial gain. Usually, it is hidden in plain sight and some form of slavery is more accepted by society than others. People become enclosed in these accepted ecosystems, packing goods, serving food, cleaning floors, making clothes, reaping crops, operating factory machinery and supporting IT products. Corporate entities have an enormous place in this état d’habitude and it has been among us for so long that people consider this a normality. A big Vendor will still be among us and in the core because all big players have chucked the big Vendors crap.

Life is so busy that we don’t have time to stop and think about all around us: why things are the way they are, why we do the things we do, how we spend our life. I had such an occasion in the past, with tasks up to my head, with eyes set on me every corner I turned, with unreasonable expectations from people above me and favor exchanges from several cliques. Considering my term (employee for life), I felt I had no way out. I was able to escape and, finally, I have had the time to stop and to reflect about it, but not reminisce. I am free for about three years; I was another prisoner from a TAC (Technical Assistance Center). This is an explanation of what to face when going to such a place.

Welcome to TAC: Step by step

Recruiting (Step 1): Join unknowingly a specific weird team, roughly broken into different products or disciplines, with localized flair. Of course, these are a lie, and you are being thrown into a collection of queues. All the vendor industry companies have the same strategy when it comes to product support.

The grass looks greener on the other side and the Vendor makes sure of it. When applying to a TAC team, you are given a brief and, often, confusing explanation on what exactly a TAC engineer does. “Customer Support” can be done in different ways and can mean just about anything; “Trusted Technical Authority” can come from different departments and at different stages of the acquisition and signing of a product/service by the customer; “Collaborate with top talent” does not specify the type of collaboration you will embark in; and finally, “You are not alone” couldn’t be farther from the truth. So, when you are a beginner (as often hired TAC engineers are; some even from a background completely unrealted to IT) and you are faced with such a job proposition you will get confused about a lot of what is in the description, and even if you research about it, you will simply not get the full picture of what the job is because… you simply don’t have the experience in the IT industry and you do not understand how some of the marketplace and product space works. You can’t ask questions because either you need the job badly or you want to join this reputable company no matter the cost. Only after being inside, having a few years under your belt, or someone explaining beat by beat, will you understand the structure of Vendor companies.

Either way, when you get recruited, depending on the process, you will either join a team you applied for, join a completely different team inside the domain you work for, or join a completely different team in another domain. This is due to the head count and workload that different products generate for the Vendor company (normally having to do with how faulty and crappy the product is), and the tactics usually set in place by the recruitment or hiring management revolve around arguments based on figures from studies in the IT industry or myths that never get unraveled. For example, Cybersecurity has a shortage of people, and we need cannon fodder for our Security team to deal with tickets that have nothing to do with Information Security. In terms of company culture, some of the more general tactics for persuading new people into joining have to do with benefits (parties for everyone), inclusion (show your prison ink) and other boons that are presented with a pristine and crystal coat: these align with the company’s ESG (environmental, social and corporate governance) policies. Training is another bullet point for “selling” the job, usually meaning bootcamps (bootcamps are ruining the credibility of engineer’s credentials and discrediting several reputable acronyms/badges) and online self-training (that you can get yourself).

Onboarding (Step 2): Training, TAC is the best at what they do, at least according to vendor company, vendor fanboys, security stans and beginners. You will get the best training according to presentation from HR and Onboarding. Get one to three weeks of fucking dreadful PowerPoint presentations with company template, and you will even end-up more confused due to tackling subjects that you shouldn’t, to begin with. All of this, being surrounded by new members with different level of knowledge and aspirations (some are more dirtier than others). Did I mention that most of the new workers come from university?

Broken onboarding process is no shock for someone who has been around for a few years in this wheelhouse. But it is usually one of the most surprising aspects when you start working for the first time and when you are faced by it (your university teacher has been teaching for too long and has lost the sense of reality). TAC onboarding is no exception to the rule: you get employee engagement and introduction to the Vendor company that you already read when you applied for the position months/weeks before, you waste time with safety training, harassment training and other trainings about several topics (due to ESG and local/national law) that you already possess if you have common sense and you “enwrap” yourself into technical trainings (generic or specific) because either you “came off the street empty-handed” and you are “cheap”, or you are not in line with the norm and need to be molded.

The process can take few days or weeks, depending on the level of knowledge you are in (meaning, your corporate level inside the Vendor company). Usually while you’re at it, you start wondering what the point of it is and why wasting so much time in these activities. The reasons for such onboarding have to do with teams being warned beforehand that new people are coming in and preparing themselves (usually the manager and/or team lead) for extra headaches, and, meanwhile, newcomers are being indoctrinated, formatted for the role and the company is drawing attention away from work-related questions.

The Vendor will make you an expert in a very narrow technology domain, one of their products. This expertise is useless in any other Vendor company, even on similar positions, but on other hand, it is a way of marketing yourself once you want to get into the big and well-known companies outside of the Vendor space, that are a “shop” using that Vendor. Nevertheless, the onboarding is an interregnum needed for the department and team to prepare themselves for the newcomer (it’s not about you, it’s about them).

Laboring (Step 3): Start working and devoting yourself to the cause. You are new, you are dynamic, and you are young, and you have the capability to get up to pace with everything. But you will be shown the ropes by someone senior, right? Nope. You start doing what is called “ticket picking” (handling selected “easy” tickets) and then what is called “bomb throwing” (handling “shit show” tickets that no one wants). You are put into a situation in a ticket with a very high severity (why?), you must deal with customers you never heard of, with technology you never dealt with before, and when you think you will have the place to try to replicate and emulate the issue (laboratory), you don’t have access to it, or you don’t have the necessary gear. In the end, you get told by your manager to shut the fuck up, get out or go to the Learning Platform to learn about it (the videos don’t prepare you for anything in TAC). You will have the pleasure of meeting one of the most notoriously useless management styles in corporate environment.

I was cultivated and sprinkled by engineers, not because of what they taught me but because of what they showed me inadvertently. Ticket handling is done in very different ways by TAC engineers: sometimes to escape the madness, sometimes to show to your manager that you “exist”, sometimes because is just another day in the office, sometimes to prove something to someone inside/outside the team, sometimes to learn and improve, sometimes because you have no choice, etc. Power of observation is key to survive in this kind of environment (metrics) and expect no one to lend a hand without something in return. It is not a dream job (nothing is) but it can be less painful if you are low-key. You might be able to talk to super-smart people and learn a few things, but those are far and few in-between the clutter you will find (sometimes is the people you least expect).

Reporting (Step 4): Report anomalies to developers regarding product, because you are only paid to follow Wiki articles and previous tickets to know how to solve issues. You don’t get the pleasure of working through bugs in several platforms, you are not even in the same organization/department as the developers, with no negotiation power in order to get a bug fixed.

The TAC Engineer position is a factory worker position: you learn a trade; you practice it over and over and there is no deviation from it. You didn’t invent the product, you don’t improve the product, and you don’t fix inherent issues with the product. You simply apply “bandages” and follow “handbooks” to understand what to do in different occasions. Any knowledge you would have that could even be helpful for different situations in tickets you would handle (e.g., knowledge of other Vendor products) is not welcome, in fact, it is disincentivized because it goes against the “flow of things”.

It’s not that having a deep knowledge on something is worthless, having a very deep knowledge of a single Vendor can be useful and is not necessarily a bad thing (it doesn’t hurt to have, the more the merrier), but it can’t really be the ONLY thing. And this is what entails being a TAC Engineer: you practice only one thing. If you want to tackle different things, you either move on (definitively or temporarily) to someplace else inside the Vendor, or you move to another company. Development of product is not reserved for this position, this is called BU (Business Unit), or also known as the developers.

One more thing to add regarding “TAC knowledge”, is that most people in the Enterprise world (customers) only skim the surface of knowledge for a large array of products from the Vendor, not because they are the common denominator, but because a lot of times they have scoured the Internet for an answer and to the best of their abilities they are unable to find an answer to the problem they are dealing with. The answer to this problem, this “knowledge”, mostly for troubleshooting and to understand how the product works “under-the-hood”, is “gated” with TAC and not documented at all. When you face TAC with questions regarding both these things, you are either ignored or simply not given the answer (it is the “secret sauce”).

Laureling (Step 5): Research and find “process gaps” in the team or department you are in and start developing useless tools and processes to only complicate even more, what is already complicated. Get rewarded or ignored for it, depending on who you align with.

With the rise of buzzwords regarding work methodologies, work organization, etc. you see a lot of implementations of processes (most often coming from Software Development) that are not applicable or not adjusted to other types of roles, domains and work types. Unfortunately, management and business folks oversee organizations, and most often than not, are presented with new concepts or with “what is hip” in the industry. Because most of them don’t have critical and thoughtful thinking, not really doing their research into understanding if what they learned is applicable to their setting, they are ready to “jump the gun” and apply these to their environment.

Because these processes and these concepts are tied with metrics and company culture, this means that whoever in the team comes up with new ways of “easing” things for everyone, will more than likely get treated pleasantly and might have good fortunes awaiting (e.g., promotion, bonus, etc.). The problem is that this creates another coat of competition between team members, forces engineers to dedicate part of their time into “solving” problems that should be solved differently and discourages actual rational and innovative thinking into revising processes that are outdated. The most common example is automation and agile just for the sake of it. So be prepared for a lot of meetings of “phoney baloney”…

Fleeing (Step 6): Get sick or go on a leave, so you can escape this hell for a few days or weeks.

Finally, you will get fucking exhausted and want to quit everything altogether or you will feel burned out and having to take a few days off. Workplace burnout is a real thing and TAC foments this. This has to do with the workload generated from the number of customers the Vendor signs with and provides support (as per contract signed) and with their appetite for more dollar bills, even if their technical support infrastructure cannot handle more. Besides the work itself, you have the people: some TAC engineers in some teams are the cornerstone of the department and are part of the DNA, having a voice in several matters. Those TAC engineers can be a pain in the ass and uncooperative, usually thinking that they are the best in the world in their respective domain. They definitely aren’t, sometimes even embarrassing themselves in the showboating, but because of this, they do love grilling and ridiculing others. If you don’t have toughness to handle them, you will get tired quickly.

The truth of the matter is that you are just another person passing by, if you go, they will get someone else. It is even truer for this role. The amount of people who get burned out or disappointed by TAC is so big that the rotation of personnel is high. Depending on the company and depending on your type of employment, you might have several ways out to “reset” your brain away from TAC and return to the “grind” after all is good. Refer to your contract and/or national/local labor law.

The cliques

As in a prison, TAC quality depends a lot on the product, people and pervasive engineering culture: prepare yourself. The effect of working in TAC was rewarding and encouraging growth, progress and hard work. Nowadays, you are not seen as the future of the company or as a potential employee, but as another worker (unless you align yourself appropriately). You are not encouraged to be better today than you were yesterday, and this is what changed in TAC. What the Vendor is looking for is not your approach to resolving an issue and expertise, but to resolve the issue and follow a recipe.

Disciplines

Networking: Most of the people who work on this domain usually come from the Electrical Engineering field, not having knowledge in general about IT (be it processes, standards or even other domains that operate over the network) and only having knowledge about outdated network protocols, enterprise routing and switching protocols, physical network cabling, fiber, coaxial and landline. They don’t know much about Software, Operating Systems and Security.

Systems: Most of the people who work on this domain usually come from the Computer Engineering field and were not good programmers (Software Engineering) while studying or when they started working, or decided to follow an Administration role (normally, between Windows and Linux). They have good understanding of Networking and Operating Systems, some of them are even good at scripting and automating several administration roles and are good at using Vendor applications or tools for Administration purposes (e.g., backups, monitoring, etc.).

Application: Most of the people who work on this domain usually come from the Computer Engineering field or from a reconversion program. They studied to become developers and to work in different parts of an Application (e.g., backend, frontend, etc.) or with different Applications altogether (e.g., WebApp, MobileApp, Web Services, etc.). They do not understand much of networking, but in more recent years, developers have begun acquiring an understanding of Operating Systems and Servers (because of underlying architecture serving the applications they develop).

The People

Helpers: the people who you want to work with and interact, they are the people who will help you in need of assistance. Save some of the questions (after researching and doing your part) and ask to them. They will gladly give you a “push” and direct you towards the solution. They usually are team players and care about the team and newcomers. Unfortunately, they are a minority.

Blockers: the people who you want to avoid talking on a regular basis but that you want to keep a superficial connection and keep “appearences”. These people hate what they do at work and some of them are good at what they do. Because the job sucks and because of lack of perspectives in their career (or lack of opportunities) they have to content with what they have. Because of the grudge and spite they have over the company, but more often than not, because of the envy and hate they have towards other colleagues, they will block others. This can take the form of interpersonal relationship blockage (e.g., gossips) or can take the form of professional blockage (e.g, sabotage). Fortunately, they are a minority.

Don’t Carer: the people who you can rely on most of the time, because they don’t care about consequences. Usually they are exceptional at their job, help others if asked, and do their job rather quick because they don’t care about the job. It is just something they do for money and if whatever they need to do to maintain is enough, they will do it. On the other hand, because they are very good, they can change job if they please. Because they don’t care.

Smokers/Kitcheners: the people who got a need to go smoke outside or go to the kitchen every 30 minutes, because either that is how they operate (hyperactive) to stay focused and motivated for the job, or because they need to “catch a breath”. With COVID-19 and Home Office, these people have less opportunity to do so, hiding “in plain sight”.

Assholer: the people who are assholes and need to show it to everyone. They smudge it in their colleague’s faces. Usually is because they have personal problems, are unsatisfied with their lives, or have envy of other colleagues, but cannot be better than others.

Go-To-er: the people who are always requested by their manager and are always asked questions by their colleagues because they are good enough at what they do, but are gullible to help others (and sometimes work for others) all the time.

Complainer: the people who are always complaining, rightfully so or just for the sake of it (doesn’t necessary have to be about work). They get energy from it and can inspire others to do the same if the team is in a rough spot. Usually these people pay the price for it, unless they have a favourable spot in the team.

Ghoster: the people who are always working in the shadows, delivering results (or not), not knowing where they are and at what time they arrive and leave the office.

The Partners

Vendors outsource a lot of their support all over the world, either through contractor or through the opening of lower tier centers: Manilla, Gurgaon, Budapest, San Jose (not that one), Sydney, Krakow, etc. This leads customers to experience different levels of support because outsourcing operations tend to suck badly (and sometimes you have internal employees who suck even worse).

Crème de la crème: Typically, these are the support teams around the cities or countries that have the headquarters for the Vendor company (e.g., USA).

Middle of the road: You have varying levels of success with this Partner, and they are located outside of developing countries. Some are all over the place and others will fuck you royally. Some are very responsive, and some will leave you hang out to dry. Some will jump right into a bridge call while troubleshooting and looking at the provided data, others you don’t hear from them for 3-4 days.

Bottom of the barrel: Vendor contracts these Partner companies to offload part of their TAC responsibilities. Other businesses get discounts purchasing partner support contracts than if they were to purchase support straight from TAC. The Vendor strongly dislikes Partners escalating too many cases, and there are penalties involved if many easy cases are escalated to them. Certain customers abuse the system and ask to escalate anything and everything. Also, a lot of the engineers from the Partner side, have the highest credentials, obtaining it through crash courses and/or boot camps (providing a much easier way of achieving this). The expectation to deal with these engineers is low, but sometimes you will be amazed by the level of stupidity you will face. This is part of what’s wrong with the industry: people think just because you attended some class and paid an hefty amount of money, that they instantly are better and more knowledgeable than you.

The Tickets

Tickets are usually separate by severity (easy to fool by the customer) and discipline, which in return is subdivided by type of problem. Enterprise customers dominate the market and are many customers you will deal with, having large ISP (Internet Service Provider) and Governments as the most pressing matter in tickets, at least contractually speaking (the money dropped by them for the service is huge). Any customer should be the same for you except if they have a special acronym attached to their Salesforce Account page, or if they have a special and dedicated Account Manager (from the Vendor side), Business Facilitator, or whatever you want to call them, is attached to the customer (meaning, high tier, dedicated and expensive support).

Tickets come in a queue that works in a “God knows what” scheduling order (even if you are told the contrary) and you will have shifts and specific timeframes when you will have to take in tickets and work on them. Some simple tickets are taken as a way to get yourself “busy” and have a breath of fresh air from more complex tickets, some tickets are taken as a way to deal with a mild problem and maintain your metrics in a good stand, some complex tickets are taken because you are the only one without any tickets in the queue and all other engineers are “busy”, and some “hot” tickets are taken because you are the “fireman” for the shift. In addition to undeliberate baldness, endless calming pills and smoking escapes, you will feel drowned in a multitude of tickets and seeing yourself doing the same things repeatedly.

There are also different types of tickets: most are regarding Break and Fix (something is broken, and you fix it), some are helping the customer with the product (handholding), and others are simple process tickets (e.g., RMA (Return Merchandise Authorization)). Because configuration assistance is not part of the job and RMA process can be done by the customer (depending on the Vendor), majority of the tickets are Break and Fix. If you work with specific partners (depending on the team you are in), some of these tickets will be routed to other parts of the company and to the Partners themselves. The contract type also affects how fast “support” acts or other process like RMA are treated (e.g., NBD (Next Business Day)), having several types of support contracts at the customer disposal, forcing the customer to choose a plan (e.g., Regular, Premium, Plus, etc.).

The way a ticket is handled can be seen from different angles: first, the ticket an engineer holds can be transferred to another engineer around the world, if it requires more work to be put in, so usually TAC supports FTS (Follow-the-sun) model, where the ticket is handed to the next working shift (regions like EMEA(R) (Europe, Middle East and (Russia)), APAC/APJ (Asia Pacific/Asia-Pacific and Japan), AMER (Americas), LATAM (Latin America), etc.). With this last model, engineers use it to “hot-potato” some of the undesired tickets they possess, in hopes of getting rid of it (this is very common with engineers from developing countries). Worse is when they escalate a (rightly) pissed off customer to you after 30 days of being useless (I pity the customers, to be honest). Second, case routing is another one of those processes put in place that is related to “sending off” tickets that can be interpreted in different way, in the hopes of getting rid of it. The customer must select a specific category for their problem when opening the ticket (so the ticket is routed to the appropriate team), but a lot of times the wrong category is chosen, and the ticket ends up in the wrong queue (taking more time and leaving the customer even more frustrated). The engineer usually has an amount of time to pick the ticket from the queue depending on the severity (before Duty Manager comes breathing down your neck) and sometimes, given the situation, the engineer must pick the ticket, select the correct category and “punt” the ticket to the appropriate queue. Unfortunately, some engineers who don’t want to deal with certain tickets purposefully “mis-queue” the ticket, and another engineer must deal with it (common practice). The other problem is that the category selection can be a drop-down menu with outdated values and the engineer must spend almost half an hour looking for the correct option. Fooling metrics is also part of the job: an example, tickets that were open longer than a month because of the open date not being in the calendar month, you would leave it open for over a month before closing it, so it would not be seen by the metrics.

The information put on the ticket must be relevant to the case and must contain quick and concise notes. Example of this would be to put device outputs, Bug ID’s, links to internal wiki, links to documentation that is customer-facing, and then quick actions on what was done or what to do next (next steps). This is usually done to “touch” tickets from time to time, ones that you will wait on someone (holding pattern). If you do not put even the singlest information and update to the ticket, you can have the customer calling the manager (or you), calling the Operations Manager (dedicated to the customer) and the “create-to-close” numbers being impacted. All of the above are done or created in order to have a relentless action on closure of tickets. This is because of the backlog of each engineer growing more and more, and resulting in more tickets on backlog forcing more updates, follow-ups, and resulting in less favorable numbers for the engineer. More about “create-to-close” tickets, is that these are the tickets that are easy, with known solution (not dependent on any party) and that can be solved with one email sent / call bridge. These are taken by engineers due to the metrics they need to fullfill. Often, this discourage engineers to take hard tickets, either high severity tickets or tickets that required interaction with developer team (filling new bug), but also tickets that required knowledge deeper than what was seen on the initial training or on the internal wiki (recipes).

The TAC Engineer title holds authority behind the answer and because of this there is a huge discrepancy in expectations from customers: you will face irritating customers demanding the silver bullet answer and the single source of truth. You will have to waste time negotiating and decreasing the severity of a ticket and explaining that because something doesn’t work, it doesn’t mean the ticket should have critical priority. You will also feel pressure from the customer because you will be challenged several times on your knowledge and expected to be the expert and know the answer right away. The change how customer support is done also contributes to this: instead of TAC being a technical support helpline used by Customer Engineering teams, it is becoming a customer service focused support, having the customer experience/service approach and unifying how support is delivered across all products, and the technical expertise aid being seen as an afterthought. TAC doesn’t know the specifics of a contract unless it has access to the Account of a customer on a platform like Salesforce (common among companies who have customers that subscribe their service or buy their product). Because of this, TAC doesn’t know the commitment and service cost of the Vendor for the customer besides the pure technical assistance it must provide. It is not really the job of a TAC Engineer to do sales or consulting work, but you would be surprised the amount of times TAC gets called by the customer to do so…

Finally, after some time working as a TAC engineer, you will have the nightmare ticket. The nightmare ticket is the ticket you know you’ll get stuck on for hours, on a conference bridge, escalating to other engineers, trying to find a solution to an impossible problem. For me it was NGFW (Next-Generation Firewall) HA (High-Availability).

The Tickets: Tips for a customer and for an engineer

When a support engineer from a Technical Assistance Center actually knows what they’re doing and uses good troubleshooting methodology (instead of shooting from the hip), we call it a pleasant surprise. That being said, try not to make support calls where you don’t already know the solution. If you don’t know the solution and have tried everything you know and could look for, always open a case/ticket with the utmost priority and ask to transfer the assigned engineer to an American and not to another region in the world, usually they are hard as shit to follow on the phone.
Segment your initial case/ticket text and make it perceivable, because not many engineers will bother to read your wall of text: make it short, make it concise and get to the fucking point.
Chase up TAC 10 times for a response. Because their first answer will be pasted from their notes (e.g., asking for more data, answering with the easy fix, answering with an unrelated answer, answering with bad news).
Licensing is a hell so be sure you have licensing up to date before you wait several days for an email saying you will not get support from TAC.
Products really are very dependent on TAC, there isn’t much you can configure and troubleshoot on your own, so if it is not in the documentation, forum and online KB (Knowledge base), go ahead and open a ticket.
If you are facing a problem that you do not see documented and discussed by anyone else online, it is most likely a bug. Be ready for a long wait while bug needs to get addressed until next software code release. Products’ code keeps sucking more and more.
Troubleshooting: start little first (define scope), collect logs (provided by customer), search for bugs in the relevant software (based on bug platform), look for configuration issues and then you can think about RMA (because RMA is not the end-all-be-all solution).
If you are put on hold by a TAC Engineer or if he/she asks you to wait for a moment, it means he/she is going around the floor asking for help because he/she doesn’t know what to do.
Talk is cheap: if it is not in the case/ticket notes, it didn’t happen.
To become a good engineer in TAC, you will have to work 10 hours in an 8-hour shift.
If the issue is time-sensitive, call to the hotline (the agent will create the ticket). If the issue is not time-sensitive, create the ticket through the self-service portal.
If you want shit and giggles, put the customer on the speaker phone and have a good time with the team.
As a TAC Engineer, when filling a bug seen on a ticket, make sure it was not reported previously. You want to report meaningful bugs, not junk bugs, as this will label your worth by the development team.

Collaboration and Routine

When you work at TAC, you are forced to be “on-shift” for at least 4 hours a day (meaning handling a queue). This doesn’t mean you are only working 4 hours a day, as you have to deal with backlog of tickets, calling customers, sending emails, chasing down developer team for bug fixing, doing recreations on laboratory, doing scripting/programming for things that need complete overhaul and not “bandages”, and trying to do training that was not given to you during onboarding. Usually you have 3 to 4 engineers on-shift, but this can change due to lack of people or the number of high severity tickets being handled. When a ticket comes into the queue, and nobody picks it up, you start having the duty manager pinging you on the chat and maybe the manager pinging you also. Queues are unpredictable: because you have to take tickets from all customers (global support; at least my position was), you can have any type of ticket, with any severity, at any time of the day. The hours you are on-shift are also usually random (meaning the manager decides what time and the criteria is “just because”), meaning that very often you have to eat your hindus and chinese food at your desk, and you have to count the minutes when you go pee-pee. If you ever wonder why a customer sometimes waits so long on the bridge call or on the phone, it can be due to lack of engineers, and because the engineer assigned to the ticket went to take a shit.

Some TAC tickets are mind-bogglingly difficult, involving multiple layers of help from developer teams, other TAC engineers (from different disciplines), hours in the laboratory doing recreation (that don’t work), and a lot of frustration and head-banging to the desk. Sometimes you have more people to deal on the Vendor side to add to the mess (e.g., Operations Managers and Special Engineers (dedicated support for the account), Consultants (configuration and design project), SDM’s (Service Delivery Managers) and Incident Managers (if the customer has MSS from the Vendor managing their environment)) and people on the Customer side to add to the pressure (e.g., stakeholders, managers, etc. (usually people who are not technical)). You can’t find the easy answer on the internal search engine in the Vendor company, you can’t seem to find the right people when sending an email to an internal mailer and it seems the TAC engineer who is an expert has gone on vacation or is sick (fuck me).

When collaborating with engineers inside TAC there were two main problems: bad TAC engineers and “throwing the hot potato”. TAC engineers make mistakes and a common approach for TAC engineers and customers working on a tough ticket is to just “throw hardware at it.” Sometimes this is just pure laziness: why troubleshoot a complex problem when you can send an RMA, swap out a module, and hope it works? Other times it’s a legitimate step in a complex process of elimination. RMA the faulty module and if the problem still happens, well, you’ve eliminated the module as one source of the problem. As a TAC engineer, you have to consider every single possibility, from A to Z, and most engineers I faced don’t. The other situation has to do when doing collab (collaboration) with engineers from other disciplines. You don’t know their technology and what happened often was that the engineer could be rubbish or he/she would have the same assumption as you: that you are the rubbish engineer. This created a lack of trust between both engineers, leading to “clearing the name” of their product and trying to prove that the issue was not on their side, instead of working collaboratively on the issue.

Escalations are also a crap-shoot: usually escalation engineers are not cut-out to the issue you are trying to solve. There’s nothing worse than the sinking feeling a TAC Engineer gets when realizing that he/she has no clue what to do on a ticket, and that when the internal search fails to return an answer to the problem, you contact escalation engineers, leaving you helpless and leaving the customer even more frustrated. Eventually, you solve the issue, the ticket is closed, and you move on to the next ticket. But it leaves a mark on you and shows you that some of the people who are in certain positions, leave a lot be desired.

Finally, working weekends were called “russian roullete”: a lot of the major changes are done outside of business hours by customers (organizations), meaning if there would be a fuck-up by the engineer from the customer side, or the product you supported was acting faulty again, you were in for a treat (a bridge call lasting several hours). Sometimes nothing would happen and the shift would be a breeze. Fortunately most of the time, working on weekends were a voluntary act, but sometimes you were forced to do it because you need to be a “team-player”. To be honest… fuck that, weekends are to rest or to work on yourself, not do your main job from the business week. You would make more money, sure, but if you wanted to make more money, you would change to a better job (hint: that is why you improve yourself).

Ratings and more metrics

Rating given by the customer at the end of a ticket is separated into three questions and is usually known as the “bingo” score:

How was your experience?: typically it contains a biased deposition based on the feelings and emotion of the customer, coming from the customer’s perceived engineer’s performance and the satisfaction he/she has based on the solution.
How knowledgeable was the support representative?: regularly it is answered as “very knowledgable” or “yes” due to the lack of knowledge of the customer (product “gated” knowledge).
Would you recommend support to others?: If the ticket was solved successfully and customer is satisfied, yes. If the ticket was solved successfully and customer is not satisfied, no. If the ticket was closed and the issue was not solved because it does not have solution or is not an issue, and the customer is an asshole, no.

The problem with the rating system is that it went against engineers who took tickets that were re-queues: these are tickets worked previously by another engineer, handling it awfully, and then handing it over to the next shift. Even if you resolve the ticket, because the customer is angry with the previous engineer, he gives a low score, so the engineer who closed the ticket, took the low number thanks to his/her dumbass colleague.

Why I left and never looked back

The level of engangement completely depends on the kind of TAC you’re in. Level 1 TAC (let’s call it that) - you are usually fighting fires, without any time to actually work on tickets in the backlog. Level 2 and below (something like that) - you do get spare time with the occasional rough patch of truck loads of tickets. I was put in a shitshow team and everyone was firefighting and with little time to breathe. Also, it was the team with the highest turnover and with more engineers leaving after few weeks or few months. The biggest adversity with newcomers was that a lot of them did not have the background and experience to be joining such a team (including me, for example) due to the amount of technology background you needed. Unfortunately, when taking tickets, you would deal with customers and situations where you were tackling issues where the information given showed that the customer lacked knowledge about the product. The problem was that we were being forced to handle tickets of a product we didn’t know what it did (entirely), let alone how it works. This resulted in embarrassing situations where the expert TAC engineer couldn’t help the customer and a lot of times leaving them on hold on the call. In my case, I had to spend extra hours after work, studying the product and analyze related tickets to mine, and understand a bit more of what I was working with, and not having time to study other Security topics.

My manager (in a technical team) used to be a project manager and didn’t even have a basic knowledge on what I worked with. He also liked to give career advice that he didn’t actually possess, and his goals for me didn’t get me anywhere while I was there. He had a complete lack of understanding of the things we did, what it actually meant to the customers, industry and company. There was an evaluation at the end of my trial period where we had both good and bad bullet points, based on different categories. Needless to say that the score given by the TAC Engineer guardian of each new TAC engineer on-board, was not an unbiased evaluation. Example, you have someone who can barely speak English to the customers and you have someone who can speak multiple languages with different customers, and the latter gets a lower score in communication. Because of the sheer amount of tickets and metrics, you would see bad practices, shortcuts and workarounds done by everyone (including newcomers). You need more than data to understand the performance of an engineer. Most things in life don’t lend themselves to easy quantification. Numbers always need to be in context. Unfortunately, my manager (as well as most of them) don’t look at it this way and are obsessed with quantification.

I left and never looked back, but I am very much grateful for my experience because it showed me how bad some positions can be, even under a very well-known brand, and it prepared me for the worse for the rest of my career. Financial restraints and unreasonable budgets are destroying teams and departments that are crucial to the Vendor. A lot of top talent ends up leaving or moving internally and crappy or delusional engineers stay. In going on PTO (Paid-time off; vacation), the lack of support of my team in taking some of the tickets I had pending on customer action, the attitude of some of my colleagues (chasing me when I am on a call with the hospital due to family illness), showed me that the competitiveness of some of the TAC joints as explained around (before I joined TAC) proved to be true. You always want to make a good impression and want to be a team player, but you also don’t have to be a sucker. This is what these places hope to find in new recruits. When you have a colleague rounding up everyone in a room to count how many tickets each person has and completed, you know what you are in for…

For sure, being outside of a Vendor company can be seen as being an Associate or Beginner in a Vendor company, dealing with multiple different vendors and lacking the depth that you can only obtain when on the “inside”. This holds you back from learning different technologies that only come when you have reached a certain level of depth, only attainable when you work for a Vendor. The downside is that you do not get to see the full picture as people working for the Enterprise customers do. You have overall view of what skills you need, concepts you have to master. Even if you change Vendor and go to a similar position, the thought of having to deal with the motivation to change and the effort of starting back at zero (which would be a shorter tenure for the already experienced) can be quite difficult and changing to a new place does not ensure you of learning a new technology in depth (TAC works differently and has different customers, in different places).

The bottom line for me is that a Product Support position at a Vendor company can be a great place to start your career at, because of getting thick-skin and becoming strong mentally (not for technical reasons), but it will not necessarily be the place where you will dwelve into a career of your choosing (e.g., Security). Get out when you can because the world has so much more to offer.

Blog posts about Information Technology, Information Security Industry and Life. Whatever comes to my mind.