One question we are asked often is a comparison of public cloud vs private cloud. As an MSP, our focus has primarily shifted to Microsoft Azure over time and we have some key reasons why. Today’s article is going to focus on some of the common shortcomings we have seen with private cloud and particularly MSP owned private cloud. If done correctly, private cloud can be a great asset but some key areas should be reviewed.
Key Areas of Review
Infrastructure Age and Refresh Schedules
As an MSP, we ran private cloud for many years. We owned servers, storage area networks, network switches, firewalls, backup systems, and all of the related equipment required to run a successful cloud business. One of the ongoing areas to stay in front of was the age of the infrastructure and refresh schedules. As a company, we always had a commitment to not run equipment past end of life, but this is not always the case.
Many times, MSPs are faced with financial decisions and profitability decisions on how and when to replace equipment. Cloud infrastructure in a private cloud is always either underutilized or oversubscribed. When equipment is replaced, the MSP is always resizing the equipment for the next time period as they anticipate growth in the business. During this time period, equipment is typically sized larger than needed and underutilized. Toward the end of the lifecycle, it is very common that the business has grown and performance of that particular equipment is oversubscribed.
If the MSP does not see the anticipated growth, financial and profitability decisions can lead the MSP toward running equipment past end of life to maximize underutilized equipment. This obviously adds risk for the MSP and risk is a decision all business owners and executives must calculate. It also adds risk for the customer or end user of the MSP’s cloud infrastructure.
Qualifications of the IT Support Team
Different industries have specific requirements of their technical teams. One of the interesting facets of the IT industry is the absence of required qualification and compliance regulation. Even a bachelor’s degree from a university usually has a very unrelated curriculum to the skillset requirements of a network and cloud infrastructure engineer. Many engineers may seek certifications from companies such as Microsoft, VMWare, or even Cisco but these certifications are not required.
Hosting business applications with an MSP utilizing privately owned cloud leaves the reliability of the environment completely within the hands of their engineers. The design, implementation, and maintenance of that environment are completely controlled by these engineers who likely have evolved over time. One common risk is that a very skilled engineer originally designed the environment but is no longer with the MSP. While the MSP may have engineers quite competent in running day to day operations, they may not have retained the skillset depth over time to handle some of the most catastrophic situations such as a cyber attack or a major equipment failure.
There is zero guarantee that these engineers hold any certifications or formal training in the cloud infrastructure industry. Many engineers in this space are quite skilled but the consequences of a failure of this cloud infrastructure can be absolutely catastrophic to your business.
Cyber Attack Readiness and Recovery Plan
One very key concern with infrastructure owned privately by an MSP is their readiness for a cyber attack. Like any in office or cloud environment, their environment is under constant risk for compromise from cyber threats. An attack on these cloud environments may affect one customer. It also may affect all customers of the MSP.
It is up to the MSP to have designed and tested a readiness and recovery plan. It is up to the MSP to have outlined a plan for an acceptable amount of data loss and an acceptable recovery time. Being in an unregulated industry, the MSP has absolutely no requirement to build these readiness or recovery plans. The MSP also has absolutely no requirement to do table top exercises or ever fully test these plans.
In the event of a catastrophic cyber event, the likelihood of a swift recovery with minimal data loss is very low. Generally speaking, this recovery involves restoring virtual servers for every single customer and this restore is on a one by one basis. The cloud infrastructure likely does not have the throughput to deliver this size of restore in an acceptable timeframe. During this event, the MSP will also likely be understaffed in working through recovery tasks therefore slowing the process even further.
Backup and Disaster Recovery
Similar to cyber attack readiness, another key concern with infrastructure privately owned by an MSP is their readiness and failover plan for a catastrophic data loss or facility loss. One example is an error in a storage area network which can cause a sudden major data loss of production data for many customers. Another example is a failure in the facility which causes physical damage to equipment and potentially temporary or permanent damage to the facility itself.
Many privately owned MSP cloud environments will maintain remote backups in an alternate location in case of a catastrophic event. In the event of a total loss of the cloud infrastructure from something such as fire, flood, tornado, or hurricane, it is important to know and understand what contingency plans the MSP has to fail over to an alternate location. In this event, what is the maximum amount of data loss and what is the anticipated recovery time. In the technical industry, these terms are referred to as RPO or Recovery Point Objective and RTO or Recovery Time Objective.
It is up to the MSP to have designed redundancies into the environment and to have built and tested disaster recovery plans. It is also up to the MSP to have accurately communicated these plans to the customer and to have everyone in agreement of what RPO and RTO metrics are being tracked. It is also up to the MSP to conduct table top exercises and full failovers to ensure everything that was designed continues to function as planned. In an unregulated industry, these items are entirely within the control of the MSP and there is no regulation or requirement to ensure any standard at all is being followed.
MSP Insurance Coverage
MSPs are not required to be insured. Most MSPs will elect to carry property, general liability and workers compensation insurance at a bare minimum. Some MSPs will elect to carry errors and omission insurance and cyber liability insurance. Many carriers have become apprehensive to write these policies for MSPs running privately owned cloud infrastructure given the liabilities surrounding insuring these environments.
In the case of a catastrophic loss of equipment as outlined above, it is likely and possible that the MSP will have to replace equipment before being able to fully restore services. During this process, it is likely or possible that the MSP will have to file claims with a carrier and wait on carrier payment before being able to replace equipment. This can lead to extreme delays in restoring cloud services to all customers.
Cyber liability is also an important area to give particular detail to. The MSPs cyber liability policy is designed to reimburse the MSP for their losses during a cyber event. The MSPs policy does not cover losses to customers of the MSP during a cyber event. Given that the MSP is hosting customer data on privately owned systems, the payment of cyber liability claims can become problematic between the MSPs carrier and the customer’s carrier. In this situation, it is important to understand which party will be responsible for incident response costs, lost revenue costs, and the extreme potential of costs involved in paying a ransom to restore data.
Conclusion
In conclusion, MSPs will all manage their cloud infrastructure differently. If you have seen presentations from multiple MSPs, you have likely seen different ways each MSP will manage this. There is not a standard which leaves you wondering which option to choose.
As an MSP, we are firm believers in outsourcing that which is not core. It is why you are selecting an MSP in the first place instead of hiring an internal IT team. The experts at large public cloud providers such as Azure and Amazon have exponential amounts of experience in this area. Let’s leverage their expertise and deliver the most secure and reliable cloud infrastructure available today.