Matt Kelley

Vendor (In)security – Inherited Risk From Your Supply Chain

The gaming and hospitality industries appear to be unique in many aspects, but in reality, share common concerns with all others. Market share, emerging trends or technologies and customer retention or satisfaction. Those are obvious concerns and businesses spend a lot of time and money working to improve in those areas.

Less obvious, however, are those things that are not directly at the forefront of our attention. Information security would certainly be one, but an aspect of that is the security in an organization’s supply chain.

How many items that are purchased are critical to business operations – your ability to sell services or entertainment – thus keeping your doors open? This is a key component of Business Continuity and Disaster Recovery, but not what I am attempting to illustrate here.

Let us take table games as an example. That department requires cards, dice, roulette balls, chips, the tables themselves, roulette wheels, associated displays and card shuffle machines. These items are only the front end; the customer-facing side of operations. On the back end, there is the table rating system, player tracking, the PC or tablet and the network. That is a lot of associated items needed just to provide operational table games to your customers, and it is not nearly an exhaustive list.

Many of these components need to be approved by your regulatory body, but not all. You depend on these vendors to supply what you need, when you need it. The “what” is not just quantitative, it is also qualitative. Does the vendor supply the quality product that you need in order to satisfy your customer expectations? Once they have met that standard, are they able to maintain it for as long as you need?

Unless you really think about the overall global market supply chain, being able to purchase what you want, whenever you want it, is taken for granted.

Did you know that there is a single U.S. based manufacturer of precision dice for casinos1? Do you know where it is located? What risks do they have?What about in their own supply chain? Do they have a limited resource pool from which to draw?

To accentuate this point, I will take a recent and current example from the healthcare industry.

The devastation that Hurricane Maria caused Puerto Rico in September of 2017 is unimaginable to most of us. It is still in the grips of a humanitarian crisis because of that storm, but it is also causing ripple effects across the American healthcare system.

According to an article in Scientific American on October 252, the Federal Drug Administration was monitoring the availability and production of 30 drugs manufactured solely or primarily in Puerto Rico, including 14 that did not have an alternative in the market. Maria crippled the island territory, directly causing the cessation of operations at the pharmaceutical manufacturers residing there.

There are, or were, 80 plants manufacturing pharmaceuticals on the island3. They are responsible for the production for 14 of the top 20 drugs in the U.S. market. Puerto Rico is, or was, the fifth largest area for that industry in the global market. At this point, nobody knows if that economy will recover with the same profile. Even now, more than six months after that hurricane, life has not returned to “normal”. Nor have the businesses. Could your organization withstand the effects of a six-month disruption in critical items for operations?

By no means am I equating the ramifications of supply chain issues between the gaming and healthcare industries. I am attempting to prove the point that logistical risks are present in every industry vertical, even those that are deemed critical.

What does any of this have to do with you, a casino or hotel executive? A lot more than you might appreciate! First, I will ask a broader question. What do you need in order to operate your business at the most basic level? What is absolutely required?

Your business is unique, but all gaming facilities must abide by Minimum Internal Control Standards4 that define certain requirements. Surveillance must remain in operation is a specific example. Within those standards, informed by your own business, are things that you can control directly.

What are those things that are indirect and out of your control though? What are your vendors bringing into your organization? What risk are you accepting, knowingly or unknowingly? Which risks are you ignoring? Are your department heads taking this into account, or at least seeking input from someone whose job it is to consider these implications? Honestly, do you even have a choice in accepting some of these risks?

I recently had the opportunity to present my first information security talk5 at the local Information Systems Security Association (ISSA)6 chapter meeting. The presentation focused on the first two Center for Internet Security’s Critical Security Controls7 on inventory. In it though, I also addressed the need to account for your organization’s risk portfolio.

You depend on vendors to supply what you need, when you need it. The “what” is not just quantitative, it is also qualitative. Does the vendor supply the quality product that you need in order to satisfy your customer expectations? Once they have met that standard, are they able to maintain it for as long as you need?

Every business compensates for external risks, especially those effecting critical services. The most basic might be power. Your hotel might be able to operate for a short time without power, but can your slot floor? To address this, you likely have a generator to provide emergency power, at least for safety and critical areas. What about that moment, or minutes, between power outage and generator provided power? Your data center likely has battery systems in place to take that momentary load, but what about the machines on the floor themselves? Did you accept that risk and impact, or did you compensate for it with a UPS for each bank or machine?

You might be able to sustain a brief power outage and still take money in. What about your customer services though? Is your cage operational? Player’s Club? Are there UPSs in place for those areas, or did those PCs hard power off? Do the underlying systems recover gracefully, or does an event like this cause chaos and a lot of required work to regain normal operation?

These are all follow on effects, however. The root cause is out of your control; the power that you receive from the grid. Your power company, just like you, attempts to compensate for risks that are fiscally feasible to do so. Those are your inherited risks.

For those risks that you simply can’t control, you compensate as much as you reasonably can. Las Vegas regularly reaches extremely high temperatures, so your business might invest in redundant cooling systems. As a resident of northern Michigan, our concerns include extreme cold, snow, ice and geography. You know it is cold when your data center cooling system has to turn off because it is only specified to handle 10 below.

An example of controllable risk would be the doors of your property. You use them every single day, barely noticing them. They are in your MICS though, and plenty of effort has been put into that area.

The National Institute of Standards and Technology, or NIST8, provides guidance on this topic through Special Publication 800-53, currently on its fourth revision9.

Who manages your access control system? It would presumably be maintenance, security or surveillance. What does IT know about door controls?On the other hand, what do any of those other departments know about operating a network and managing operating systems? Regardless of responsibility, do you have a maintenance agreement or support? Do you get updates? Or did you purchase this system and expect to never think about it again, other than the fact that it worked?

Take the time to dig a little further. Is this system currently supported, and for how long? Does the managing department subscribe to manufacturer or CERT/NIST10 notices? Is there a life cycle plan in place for projected renewal or replacement? Rinse and repeat for every system operating within your business. My professional guess is that you have unknowingly accepted risks that you may not be comfortable with.

Who cares about door controls or lighting systems though? They work as they should. Why would you ever need to devote budget or resources to such simple things? This is just an example of what could be a big deal.

It is almost guaranteed that you are running one of the Big 4 casino vendors for your slots and player tracking systems. How old is the version that you are running? Have you talked to IT or information security about what those systems require to run “supported”?

Prepare to be unhappy!

If your staff was uncomfortable with the release notes and asked the Big 4 about supported operating system and associated applications, such as Java or Adobe, this was probably the answer that they received in return: “Good news! We support Windows 7. We also support Adobe 811. However, we require JDK 612.”

That likely means nothing to you at this moment, except Windows 7 support, right? Ask your IT team about the other two requirements, then how long Windows 7 will be supported13.

If you are performing a version upgrade and they do not support the most current version of required applications, you are accepting risks in the near term. Risks that you will eventually have to deal with and compensate for.

But your gaming systems are segregated and air-gapped and completely secure, right? You don’t have any worries about the combination of Windows XP and 2003 Server. Nothing else is on that network. Your regulators and auditors have never provided a finding.

Except the staff that need to access servers for reports; and the staff that scan customer info into the player database; and the staff that connect over HTTP or another means for slot monitoring systems.

Except for the USB ports on your slot machines; the backdoor that Big 4 support requires in order to provide remote assistance; and the access to your raw data that you provide to a big data or business intelligence vendor.

This list gets longer, but I hope that is enough of our industry’s dirty laundry to get your attention.

Are you still feeling nice and secure? Still believing that your business won’t be the next headline concerning a data breach? Still hoping that you can remain unnamed like the facility compromised through an internet connected fish tank14?

This article is about inheriting risk from your vendors. This includes ALL vendors. Every time we purchase a good or service, we are accepting the same risk that the vendor has – but we generally don’t know what that risk is.

Now might be a good time to review what happened to Target, who was breached through a HVAC vendor15. Again, knowingly or unknowingly, your vendor’s risks transfer to your organization.

As an industry, we need to hold our vendors accountable for this. The Big 4 have teams of developers, but they only support X software, ever, for that version, even on support agreements. You might be able to get it to work, through your own time and effort, but it is not supported. You may even be forced to roll back versions in order to get a support ticket resolved.

Welcome to the state of perpetual Beta, where you do software compatibility testing for your vendors, at your expense, and you get to pay for that privilege.

We need to do better.
We need to demand better.

If you are fortunate enough to attend the Roundtable event this fall, bring this subject up. Please. We have let vendors dictate to us for too long. It is time that we turn the table and hold them accountable for what they are putting into our environment.

Matt Kelley, CISSP, is the IT Manager for a Tribal organization that supports Gaming and Government operations. He is also the co-founder of Korr Group, a business venture that aims to improve the Information Security of small businesses and nonprofit organizations. Matt started out in the infrastructure side of technology nearly 20 years ago, and has found his calling in Information Security, specifically GRC. To stay current with Matt, he can be followed @kelley12matt or on his blog at Contact Matt.