Hardware Requirements for Machine Learning
I want to read more about the hardware requirements for machine learning - since I am implementing some ML models on a rented server.
References
Related
- Video Random Access Memory
- Video Random-Access Memory (VRAM) is dedicated computer memory used to store the pixels and other graphics data as a framebuffer to be rendered on a computer monitor. It often uses a different technology than other computer memory, in order to read quickly for display on a screen.
- A framebuffer (sometimes framestore) is a portion of random-access-memory (RAM) containing a bitmap that drives a video display. It is a memory buffer containing data representing all the pixels in a complete video frame. In computing, a screen buffer is a part of computer memory used by a computer application for the representation of the content to be shown on the computer display. The information in the buffer typically consists of color values for every pixel to be shown on the display.
- Most modern GPUs rely on VRAM. In contrast, a GPU that does not use VRAM, and relies instead on system RAM, is said to have a unified memory architecture / shared graphics memory.
Notes
Given the variety of machine learning / deep learning models, the best
hardware will have unique optimal requirements. In the ML/AI domain, GPU acceleration dominates performances in most cases, but the CPU can also be the main compute engine when GOU limitations such as onboard memory (VRAM) availability require it.
The number of cores chosen will depend on the expected load for non-GPU tasks. As a rule of thumb, at least 4 cores for each GPU accelerator is recommended. However, if your workload has a significant CPU compute component then 32 or even 64 cores could be ideal.
Since the mid 2010s, GPU acceleration has been the driving force enabling rapid advancements in machine learning and AI research. For deep learning training, graphics processors often significant performance over CPUs.
Most ML/AI applications that have GPU acceleration work well with single precision (FP32). The amount of VRAM (Video memory) needed is dependent on the feature space
of the model training. Memory capacity on GPUs has been limited and ML models and frameworks have been constrained by available VRAM.
8GB of memory per GPU is considered minimal and could definitely be a limitation for lots of applications. 12 to 24GB is fairly common, and readily available on high-end video cards. For larger data problems, the 48GB available on the NVIDIA RTX 6000 Ada may be necessary.
Multiple GPUs generally improve performance in machine learning and AI.
The first rule of thumb with regards to memory is to have double the amount of CPU memory as there is total GPU memory (VRAM) in the system.