Building a GPU PC, Part 7-1: Column – The Difference Between Consumer and Professional GPUs

This time, let's consider whether consumer GPUs can be used for serious AI computation, and how they differ from professional GPUs, with a few comparisons along the way.

For an AI startup like ours (and not only startups), there are three scenarios in which we use GPUs.

1. AI research: Quick experiments with the latest models and papers.
   Small-scale training. Small-scale inference.
   
2. Training: Deep learning training in the usual sense.
   Runs can last for weeks or even months.
   
3. Commercial operation: Running an AI engine in production as the backend for
   services used by customers

The GPU machine Jun is building this time is intended for 1. AI research.

The reason is that consumer GPUs and consumer-grade DIY PCs, for example, are not well suited to long-running deep learning training.

I do occasionally see people doing it, but it makes me nervous.

Depending on the network you want to train, a single training run can take anywhere from several days to several weeks, or even months. Throughout that time, the software driving the GPU has to keep running stably. It is quite common for something that works fine for two or three days to hit a mysterious software bug in the driver layer or elsewhere after a few weeks, and for the training to have silently stopped.
You also need to take proper measures to keep the heat dissipation system stable, so that the machine doesn't overheat.

Consumer GPUs were originally designed for 3D gaming, and are not intended for weeks of continuous high load. If a training run only takes three or four days, keeping a human eye on it isn't impossible, but either way it is a lot of trouble.

That is why we position a GPU machine like the one Jun is building, assembled from parts you could pick up in Akihabara, as a "research" GPU machine, on the premise that it will not be running long, high-load computations.

So what is a professional GPU?

There are broadly two kinds of professional GPUs.

One is the workstation GPU. The other is the data center GPU.

Workstation GPUs are used in environments similar to consumer GPUs, but they are characterized by driver stability, reliability, and long-term product support, and are more reliable than consumer GPUs. They also have an advantage in computational precision for advanced calculations. Well-known and popular workstation GPUs include NVIDIA RTX A6000 and others.

Data center GPUs, as the name suggests, are designed to run in dedicated data center environments. In addition to the characteristics of workstation GPUs, they feature high energy efficiency (that is, power consumption) and a high degree of parallelism. Well-known data center GPUs include the NVIDIA RTX A100 and NVIDIA RTX H100, which cost over three million yen per card. Although they are meant for data centers, the interface is PCI Express, so they can also be run in a regular workstation.

So if you want to use GPUs "with peace of mind" for scenarios 2 or 3, you will need workstation or data center GPUs.

GPU size, cooling method, and noise

Now let's look at GPUs from the perspective of heat dissipation.

Because GPUs perform heavy computation, they generate a large amount of heat as a byproduct. So how to release the heat a GPU produces, and prevent it from overheating or burning out, becomes important.

There are broadly three methods of heat dissipation.

1. Air cooling – internal exhaust
2. Air cooling – external exhaust
3. Liquid cooling

Internal-exhaust graphics cards

Internal exhaust is a style in which the graphics card's heat is expelled inside the PC case.
Most consumer GPUs are of type 1. They generally use a large heatsink and one or more axial fans to spread heat over a wide area inside the case. This method makes it relatively inexpensive to build a graphics card, but the large heatsink and fans tend to make the card bigger. On the other hand, the fans are quieter than those on the external-exhaust cards introduced next.

External-exhaust graphics cards

External exhaust is a method in which heated air is expelled directly outside the PC case rather than into it. Workstation and data center graphics cards mostly adopt this method. The fan commonly used here is a "blower fan."

A blower fan draws air in from the center, passes it through the blades, and expels it radially, allowing it to blow the air inside the case out powerfully (in terms of flow rate) in a single direction. Older graphics cards used a "sirocco fan" (which prioritizes pressure) for the same purpose, but recently blower fans, which prioritize flow rate, have been adopted.

In this method, the fan is mounted at one end of the card and pushes air straight out; the airflow passes through the GPU's heatsink and is pushed directly outside, which improves cooling efficiency. Unlike the internal-exhaust type, which spreads heat inside the case, this method allows the graphics card to be made smaller. On the downside, blower fans can be loud, so if quiet operation is a priority, the internal-exhaust type is recommended.

Liquid-cooled graphics cards

Liquid-cooled graphics cards can occasionally be found in some consumer lineups (all-in-one liquid coolers). Large data centers may also introduce liquid cooling systems to improve the cooling efficiency of the entire server room. This makes it possible to cool large numbers of GPUs efficiently, but building a liquid cooling system is extremely costly, so it is not very common.

GPU size and noise

Having briefly looked at the cooling methods, here is a summary in terms of size and noise:
External-exhaust graphics cards are inexpensive and relatively quiet, but huge in size
Internal-exhaust graphics cards are expensive and have loud exhaust noise, but are compact in size
That is how it breaks down.

If you want to install and use multiple graphics cards, internal-exhaust graphics cards are recommended, since you won't have to worry about PCI Express slot interference.

navigation