Rethinking Storage in the New World of Containers: The StorageOS Approach

Rethinking Storage in the New World of Containers

Legacy storage solutions don’t have the APIs and scalability that apps today need. Some organizations are attempting to solve this problem at the kernel layer. But our view is that storage should run at the application layer. It’s a radically different approach because if you run storage at the application layer, you can scale it like an application. It changes the way you plan, provision, secure and manage and most importantly, control your data.

StorageOS was founded by a group of professionals who came out of the London Financial Services banks. We wanted to understand how commodity hardware and software-defined storage could have a place in the enterprise. We really saw containers as the way forward because it’s a market that requires new infrastructure and deployments. It’s pushing the boundaries of the datacenter and it’s software-defined and software-aligned. We came to solve this problem having been storage professionals, operators, and builders of large IT infrastructure trying to deliver solutions we came across in our normal daily lives, every single day.

We’re entering a market that’s radically changing. IT spend on traditional disk and disk array is slowing, there’s the dramatic surge of cloud adoption and a shift of people spending on OPEX rather than CAPEX. We’re really trying to enable this idea.

Challenges for enterprise storage in the DevOps world

Most people are used to a virtual machine scheduling environment, a very mature ‘move your data around’ and schedule virtual machines. But the world is changing, there are new contenders for this scheduling and managing infrastructure. The storage environment has to cope. It needs to integrate with cloud providers (and potentially a single dominant player), physical infrastructure providers and serve as a replacement for the hypervisor in a container operating system. That level of integration is really important, but also the level of control is important.

Running a storage array, not just a gateway, but a full enterprise storage array with all the functionality you would expect within any enterprise storage product, at an app level, changes how you run your environment. That change is really liberating. It’s also really important because you need to be able to change your organization as well.

The good news is that that organizational change is already happening. If you work at a business that has developers or are embracing the DevOps movement, people are organizing and provisioning and growing their environment in very different ways.

  • You have legacy infrastructure teams that have very mature processes that do things in a way that maintain service levels and business continuity.
  • You have developers who want things instantly and want to do it very quickly and that’s a big contention point.

Most organizations struggle with this culture fit because these two teams don’t necessarily interact very often and they don’t have the context to culturally change this.

So how do you handle the people and processes? DevOps requires the same agility and ease-of-use for storage as it has for compute. But storage is fundamentally different from compute:

  • It needs to be accessible
  • It needs to be persistent
  • It needs to be protected

Existing storage solutions have long provisioning times, minimal visibility for DevOps, and no direct link between business requirements and the technical deployment. Container storage needs to provide both agility and accountability as VM storage, and leverage existing investments.

The StorageOS approach: Bridging the Great Divide

StorageOS is our solution. We combine the reliability and compliance needs of enterprise with the ease and scalability of developer tools.

We’ve built a solution that allows the developer to interact directly with the existing system – their Docker run command – and provision enterprise storage to their containers, making it persistent, durable and scalable but in the context of a rules-based engine that manages the enterprise service level and compliance.

It’s a perfect storm: developers are trying to accelerate the way things are done and infrastructure teams are trying to provide sanity and stability in a very quickly changing world.

We’ve built a solution that solves the problem of persistent storage and allows databases to be run in an enterprise-ready, mature container platform. We’ve built a technology that runs in-line for things like compression, encryption, and deduplication and automated the business rules. And it’s very easy for developers. One of our customers said it’s like a safe padded room to deploy his infrastructure in, while the infrastructure team maintains and delivers service levels and compliance.

Scaling beyond the enterprise data center

You burst into the cloud, you run your test step in the cloud, maybe you run between clouds – all of those new functionalities – and the concept of public cloud begs lots of questions about security, policy and data residency compatibility which you have to ask about. This is where the beauty of running at the application level comes back in because we have a solution that is extrapolated from the physical layer, is movable, but still leverages the rules-base that allows users to take this system into production and maintain their security and compliance.

What is StorageOS?

StorageOS is a 40 MB container that runs in a container operating system that:

  • Provides a global namespace that provisions volumes to your applications.
  • Enables you to run databases and persistent storage devices within that space.
  • Allows you to move applications around while integrating with schedulers and management systems like Swarm and Kubernetes so that devices and storage get scheduled on the same node for best performance and best latency.
  • Ensures your databases have the availability, durability and persistence you need to run enterprise applications.

We focus on delivering a developer experience that is turnkey while providing a rules-based infrastructure that allows the enterprise to define and grow their storage environment.

This has some key business benefits for the enterprise – they have enterprise-class storage combined with the management reporting and compliance that goes with it at the container level. Because we’re running at the application level, we’re running with your containers, so you have the same concepts and management and ‘cloud-like’ experience.

Our solution delivers the mobility and performance to make these systems run, and reliability in a way that’s both easy for the developer to provision while the enterprise is running on infrastructure that the developer may not know about. And it’s running in a way that meets the business rules associated with that.

We realized that if enterprises are going to use this, it needs to be at least as good, if not better, than the legacy infrastructure that we are displacing or integrating with. And it has to pay for itself immediately.

A developer needs to be able to download it to their laptop, have it up and running in minutes to test – a zero-cost POC. At the same time, an enterprise needs the ability to build rules-based policies and start to deploy a production Docker cluster in the same amount of time. It’s a very different technology adoption cycle, in a very different way to do things.

You can find out more by watching our TechFieldDay presentation on enabling enterprise storage.


Author: Chris Brandon

  • Customer Case Study: StorageOS provides MSP, Civo with Cloud Native StorageRead Now

  • Blog: Using the RabbitMQ Kubernetes Operator with Persistent DataRead Now

  • Performance Benchmarking Cloud Native Storage Solutions for KubernetesDownload Now

  • Webinar: Register for Accelerating Kubernetes Onboarding and Application Transformation on 18th August, 2021 at 4pm (BST)Register Now