PAIN POINTS
Artists, animators, and post production engineers all depend on fast storage when working on critical projects. As technology advances, the need for fast storage performance is not just a must have, it can make or break a project’s deadline and/or budget. In a typical grid environment, data can be slow making its way from Network Attached Storage within the render farm to the render nodes themselves. As the render farm workload increases, a decrease in storage performance is inevitable. This causes wait times for animators and editors who require predictible data performance to remain productive and profitable.
Typically, organizations purchase additional storage to add IOPS to the render grid in an attempt to improve storage performance. Even after substantial investments in new infrastructure, the pain of IO latency persists. The storage industry’s fastest and most expensive hardware cannot eliminate the high volume of reads needed to properly service the simplest of tasks within the render farm.
With PerfAccel, users offload 70% of the NAS IOPS from the NAS servers eliminating the need to purchase more hardware, data center real estate, power, cooling and management labor. Render farms that run PerfAccel deliver data to applications up to 10x faster within hours of initial implementation.
Read our case studies and learn more about how PerfAccel technology has delivered these kind of benefits to companies like yours.
SOLUTION BRIEF
PerfAccel delivers unparalleled performance between render nodes and network storage through a unique grid based data management console. Within minutes, the easy to use Datagres Management Console drives policy directly to thousands of nodes in order to optimize storage performance across the render farm. Through a simple, single pane of glass, PerfAccel consolidates the view of your render farm’s storage IO for real-time, IO monitoring and reporting. This helps administrators create and tune IO policies that reduce latency and increase overall IO performance.
Datagres is working alongside the M&E industry’s biggest names to accelerate application performance by managing data effectively, on local discs, SSDs, or any other storage device available on existing servers. Without disrupting the existing environment, PerfAccel software gives administrators unprecedented visibility into the IOPS requirements of their applications. Real time graphs and deep IO analytics ensure data is managed efficiently across the grid.
Looking for a real world example of how PerfAccel data intelligence works? The graph below details the cache utilization of an actual production environment of 100 high performance media servers. Within 48 hours, PerfAccel technology boosted read/write performance to the point that 80% of the movement of data within the existing network was eliminated.
Performance of IOPS
The network savings achieved can be as much as 80% of the total IO going to the backend NAS. This will completely alleviate the load on the back end storage.
The performance for the IOPS can be increased significantly. The actual performance also depends upon the caching device, workloads and speed of the network. While a 1G network provides a theoretical maximum of 128 MBPS sequential bandwidth, a SATA SSD drive provides 350 MBPS and two of them in raid 0 can provide 700 MBPS. The controller in the server is capable of much more typically 3Gbps to 6 Gbps, Some of the newer ones are at 12 Gbps. The faster SAS/SATA drives on the server also provide more speed than the 1G network.
On a 1G Network with a single SATA SSD as the caching device, we can expect a 5X+ performance of random IOPS as the NFS speed fluctuates between 40MBPS to 100 MBPS. The growth in the cache is as shown in the graph, this is taken from actual production environment of over 100 servers.
Today's need is dynamic and intelligent data management across the render farm. This is essential to ensure that your artists are able to work faster and more efficiently. To bring down the cost per frame, the expenditure on the capacity storage will begun to be less year over year.
BEFORE DATAGRES
Storage administrators deal with considerable pressure to create intelligent systems that provide seamless performance for their users. From the render farm to local workstations, IT professionals work to deliver data from storage quickly while minimizing the risk of a storage outages. As organizations continue to carefully scrutinize budgets, storage administrators are often left doing more with significantly less than they need. Many organizations have a desire to deploy SSD technology to alleviate IOPS bottlenecks but a lack of detailed application IOPS knowledge makes this investment a plug and hope proposition.
AFTER DATAGRES
Within hours of deployment, PerfAccel reduces the load to Network Attached Storage (NAS) by up to 80%. Consolidating the view of the entire render farm, PerfAccel gives administrators the ability to understand their active data at the grid level through deep IO analytics, allowing admins to get the most from their existing infrastructure for extreme IO performance.
- Reduce existing NAS workload by up to 80%
- Increase Server IO performance
- Reduce the need for additional hardware
ROI - RETURN ON INVESTMENT (Real Operational Intelligence)
Datagres is one of the few technology companies to deliver tangible ROI within hours of deployment of its PerfAccel software. As a technology that works best under pressure, PerfAccel’s easy to manage tools give administrators the power to create policies, schedules, and adjustments to policy that provide M&E operations a new standard of storage performance. PerfAccel provides a level of performance that render farms can rely on even within the most demanding of environments. For example, render farms using PerfAccel have observed an average of 70% IOPS offload from their existing NAS storage. Each of those offloaded IOPS executed with a 10X-100X latency improvement.
Through dynamic data management policies, PerfAccel allows data to be provisioned within composition consoles and workstations. Within the PerfAccel environment, high performance storage (like SSD) works closely with servers as larger and potentially slower storage systems can be used on the backend.
Get the most from your storage. Deploy a PerfAccel Proof of Concept (PoC) without any disruption to your existing environment. Speak to a Datagres representative today to get started This email address is being protected from spambots. You need JavaScript enabled to view it.
DEPLOYMENT
THE GRID INDUSTRY PROBLEM
HPC, E-CAD and Super Computing clusters observe a variety of workloads that are deployed with varying data sizes. Grid computing improves datacenter efficiency and machine utilization by optimally mixing and allocating the workload on the first available compute machine. Even though grid architectures have made huge improvements in increasing the machine utilization, they have left the gains from intelligent data management on the table.
To fulfill the vision of grid computing, it will be important to make the grid more intelligent by matching the compute machine to the workload type and availability of data. Another emerging trend is to identify and prepare the data for the workloads ahead of time. The new datacenter solutions need a close understanding of the workload and the data so that the data management can be completely automated for maximum performance gains.
CURRENT GRID ARCHITECTURE
In a large grid the data is managed away from the servers in the storage farms as pools of NAS or SAN servers. For example in the E-CAD industry, validation and testing jobs are highly regressive jobs that operate on the same data set over and over again to make sure that defects are detected early. Current grid architectures have already realized the first round of efficiencies associated with making the applications independent of the underlying compute architecture. The next round of efficiencies are going to come from efficient and intelligent data management.
Data movement is today managed using scripts and it requires that lots of moving parts are working correctly to make the right data available inside the right set of machines. Very often the mechanics of data movement break down requiring manual intervention. Most of the times, grid administrators observe that the IO is slow and the NFS machines are on crawling but without any IO control, the administrator is severely challenged to solve the problem. Buying more storage hardware is often the recourse.
DATAGRES PERFACCEL GRID DATA MANAGEMENT SOLUTION
GRID DATA MANAGEMENT FEATURE SET
A solution is required that can do the following:
- Reduce the latency of data movement to the server from the storage servers, so that CPU idling can be avoided or minimized
- Provide grid aware intelligent data movement control with data consistency
- Manage the dynamics of hot data inside a massively distributed environment to maximize the CPU utilization
- Provide data locality awareness to the job schedulers to deploy jobs that can then be deployed on machines where data is already present
- The solution needs to have inherent dynamic management capabilities spread across a very large number of computing nodes to provide adaptive intelligence and data analytics support for intelligent data movement
- Also provide some nodes for data staging so that the data is locally available in proximity of the compute nodes
- Grid Analytics to provide the IO heat map of the files and their use across thousands of jobs instantaneously
- WAN Caching: As grids get distributed around the world, the ability of the grids to access data sets in remote locations with consistency can help improve productivity
What will Datagres PerfAccel Software do with the Grid data?
Further optimization of the IO can lead to several business benefits
- Reduce Time-to-Market
- Defect-free chip design and validation requires enormous compute and storage resources
- Global spread of designers and collaborators require grids that can function globally by keeping teams productive
DATAGRES PERFACCEL SOLUTION CAPABILITIES
Productivity Gains
Productivity Gains
Optimize the shared computing cloud/grid/cluster
TECHNICAL SPECIFICATIONS
CREST CASE STUDY
Crest’s render farm is comprised of 400 servers and 2 Network Attached Storage servers within a 1GbE network. Datagres PerfAccel analytics software module helped Crest discover NAS network IO was approximately 5TB per day.
OBJECTIVE: Deploy Datagres PerfAccel across 60 servers to offload IOPS from current NAS resources and reduce IOPS latency for render farm applications.
DEPLOYMENT: After the deployment of PerfAccel, Crest administrators discovered two-thirds of storage IO was serviced from the cache. The data read from the cache on a few jobs was near 90% of the data read requests of the system. Jobs completed 20% faster. These improvements saved over 1TB of network storage traffic within a three day period. More importantly, the NAS server performance was high and predictable.
PerfAccel software gave Crest administrators flexibility and control over their render grid IOPS without risk to data consistency and integrity. Through PerfAccel’s easy to use interface console, admins created cache policies for pre-fetching, predictive caching, real-time cache size configuration, and auto-caching of hundreds of NFS mount points. Another advantage of Crest’s deployment of PerfAccel was that users could use existing storage devices on servers as a cache device. Higher performance cache devices (SSD’s) could be added later as budgets allowed. PerfAccel provided Crest the ability deploy SSD cache devices without preference to vendor type or location.
SUMMARY: After 3 months, the investment of PerfAccel reached a break-even point thanks to these important factors:
With an estimated 5TB of high speed IOPS per day on tier 1 infrastructure, Crest spent approximately 0,000 per year for systems, maintenance, and management. Acceleration from local SATA drives provided a gain of 40 servers per year at an estimated cost of 0,000 annually.
The ability to employ a vendor agnostic infrastructure could only be measured over time with Crest’s growing need for additional storage capacity. Avoiding vendor lock-in has saved companies like Crest hundreds of thousands of dollars over the lifespan of their networks.
In conclusion, PerfAccel empowered Crest’s media render farm administrators with the ability to increase network productivity and overall application performance while lowering the costs of network storage with a visible return on investment within 3 months of deployment.
Proof of Concept (PoC)
At Datagres, we know that seeing is believing. Without disruption to your current infrastructure, Datagres will work with your team to deploy, analyze, and prove immediate return on investment using PerfAccel's technology.
Email us This email address is being protected from spambots. You need JavaScript enabled to view it. for more details on how PerfAccel can drive benefits like these to your network
SOLUTION SPECS
PerfAccel – Intelligent management of Data in Motion
PerfAccel is a dynamic and intelligent grid management solution built tough for demanding industries like Media and Entertainment, ECAD, Big Data and enterprise level virtual environments. Through powerful analytics and reporting tools, PerfAccel creates smart, server side file level caching for storage IO and the management of thousands of nodes through a single pane of glass. With the ability to install invisibly on existing servers, Datagres PerfAccel deploys in minutes, without disrupting existing infrastructure. Once application IO requirements are analyzed and understood, PerfAccel dramatically reduces IO latency and offloads much (and often most) of network storage IO. Accelerate applications, scale, and manage entire grids with PerfAccel and get the most from your existing infrastructure.
Learn how some of the most data intensive users in the world rely on PerfAccel to accelerate data caching and improve network performance by up to 80%.
Testimonials
PerfAccel software delivers administrators flexibility and control to effectively manage data in motion while ensuring consistency and integrity. Through a simplified interface, storage administrators are provided with grid scale management software that enables analysis, reporting, and adjustment of IOP’s policies for predictive caching, real-time cache utilization, cache pre-fetching, and auto-caching of AutoFS mount points.
Once the performance bottleneck is alleviated from the back end NAS storage, network storage devices are measured in terms of the price of disk capacity not IOP’s. The need to continually upgrade expensive backend NAS storage for performance gains is reduced and in some cases, eliminated.
Renderfarm Analytics
