Building a dedicated rig with 3 GPUs

[Mxyzptlk]

From the specs, that 200mm front fan only does 110cfm at peak. Is the fan running full out? That said, I am running with 2 140mm fans that each do 107cfm on the front. But they are running at about 83% atm.

I have no idea how to control the power on the GPU's w/ Linux. sorry.

[markdotgooley]

From the specs, that 200mm front fan only does 110cfm at peak. Is the fan running full out? That said, I am running with 2 140mm fans that each do 107cfm on the front. But they are running at about 83% atm.

I have no idea how to control the power on the GPU's w/ Linux. sorry.

I have all the case fans just plugged into the power supply. (I'm a bit deaf and the machine is tucked away in a far corner of the room.) Maybe I can try some fans that move more air. It's all a bit frustrating, but the machine seems to run fine if rather hot.

[Ichbin3]

nvidia-smi is what you are looking for.
Hack it into the terminal
nvidia-smi -L lists the GPUs
nvidia-smi -i 0 -pl 130 this puts your first GPU on 130W, this is almost half of the power and heat, but you will get still a decent PPD which is much more than the expected half.
nvidia-smi -h gives you an idea what else you can do.

[markdotgooley]

nvidia-smi is what you are locking for.
Hack it into the terminal
nvidia-smi -L lists the GPUs
nvidia-smi -i 0 -pl 130 this puts your first GPU on 130W, this is almost half of the power and heat, but you will get still a decent PPD which is much more than the expected half.
nvidia-smi -h gives you an idea what else you can do.

Thanks! one card is using around 149W and the other around 156W so it's no emergency, but they might run cooler if I dropped those to maybe 125W which is given as "Min. Power Limit."

[RDDyer]

Here's my 2 bits. I'm no expert, but I can offer you my experience having just built a dedicated rig with 2 GPUs. It is working pretty well, and the machine is earning about 2.7-2.9 million points on a typical day. Hopefully someone can tell me if it should be better based on the gear I have. Maybe I need to change something.

I'm a Team Red sorta guy, so it's all AMD. I chose:
- X570 Taichi motherboard to get PCIe gen 4 (for the bandwidth)
- Ryzen 9 3900X CPU
- 3600MHz C16-16-16-36 DDR4 32GB (16GB is easily enough)
- 1TB X.2 SSD PCIe gen 4
- Arctic 360 AIO for CPU
- Lian Li PC-T60B ATX Test Bench instead of a case
- 2 Gigabyte Radeon RX 5700 XT GAMING OC 8G GPUs
- Corsair 1000W Power Supply
- Windows 10 Pro

Comments:
MB - I'm fairly happy with this choice (though the B550s just came out today). It performs well, and having post code readouts for the first time, I'll never go back. (They helped me diagnose a problem with the first set of memory modules I bought.) PCIe slot arrangement is not ideal, though it is typical for ATX boards. As you know, you really have to look through the specs to see how the PCIe lanes are shared among all the slots, including the SATA ports. Just because you have 3 X16 connectors, you won't have 3 X16 slots. There just aren't enough lanes to go around. And just finding the physical room for 3 GPUs isn't easy. (See below.)
CPU - At first I thought I had wasted money here, thinking I just needed a good enough CPU to shuttle data to and from the GPUs, but it's cranking out CPU WUs with way, way higher ppd than my 2600X machine. Worth it! I may try turning off SMT as others have suggested, though I'll be curious to see how it affects CPU WU performance.
Mem - Definitely get the higher speed stuff - Ryzen likes it. I've over-volted a little so I could tighten the timings a bit. If you know what you're doing, maybe you can go higher. Again I'm not an expert. Buildzoid says this motherboard will do better if all memory slots are populated due to the PCB topology, so 4-4GB modules might be a good bang-for-buck choice.
SSD - If you want to save some money this may be a place, though mine was pretty cheap. A PCIe gen 3 250MB is probably enough. Maybe even a SATA III drive. There doesn't appear to be much traffic to and from the SSD during operation.
AIO - Cooling is the biggest problem with my setup. CPU temps are in the mid to low 80s. This AIO was pretty cheap, and I prefer water cooling, though a custom loop might be better. The main thing I don't like about air coolers is that to get lower temps you need a huge piece of hardware covering your entire motherboard and I want easy access to everything.
Case - or lack of. I would definitely stay away from a case if you don't mind the noise. I used 20X20 Aluminum Profile for my 2600X setup. It was a bit harder to build, but cheaper. The test bench is nice, but you pay for stuff you don't need like HD bays, and the AIO wasn't easy to mount. I'll also need to modify one of the aluminum pieces if I ever want more than 2 GPUs.
GPUs - I'm really happy with these GPUs. At just under 400 EUR each, they're much less than a comparable Nvidia card. They're PCIe gen 4, so you can get typical X16 bandwidth with an X8 slot. The only disadvantage is that because they're so thick (nearly 50mm) it's hard to get air between them to the cooling fans of GPU0. They have an aluminum backplate, so I've stuck a couple fairly large heatsinks on the back of GPU0 and there's a desk fan blowing 24X7 on GPU1. I used the Adrenalin software to overclock them a bit, but I had to make the fan curves very aggressive, so they are pretty noisy. GPU temps run in the low 50s with junction temps in the low 70s with clocks in the 2040-2060 MHz range.
OS - I run Windows since I don't know anything else. I have no idea if Linux would improve performance. I only paid about 15 EUR for a legal OEM version.

All that said, I've thought about adding another GPU. I think my 1000W PSU would handle it, however I don't know where to put it. Since the GPUs are so thick as I mentioned, the second video card covers the 3rd PCIe slot of a standard ATX MB. If I modify the test bench, I could move one of the GPUs to the 3rd slot and put a riser in the 2nd for another GPU, but I don't think it's worth all the work. The 3rd slot would only have a X4 data path on this MB, but gen 4 gives me an equivalent X8 bandwidth, so that's not an issue. I'm curious which motherboard you were considering?

Anyway, that's probably a lot more verbiage than you were looking for. You'll have to glean the grain from the chaff, my friend.

[markdotgooley]

From the specs, that 200mm front fan only does 110cfm at peak. Is the fan running full out? That said, I am running with 2 140mm fans that each do 107cfm on the front. But they are running at about 83% atm.

I have no idea how to control the power on the GPU's w/ Linux. sorry.

I have all the case fans just plugged into the power supply. (I'm a bit deaf and the machine is tucked away in a far corner of the room.) Maybe I can try some fans that move more air. It's all a bit frustrating, but the machine seems to run fine if rather hot.

Just bought one of these. We'll see how much it helps. Said to be noisy. https://www.amazon.com/Bgears-b-Blaster ... oniccom-20

[Mxyzptlk]

WOW! Your PC might lift off the desk with that one! Let us know how it goes.

[MeeLee]

Just bought one of these. We'll see how much it helps. Said to be noisy. https://www.amazon.com/Bgears-b-Blaster ... oniccom-20

Airflow on those high rpm fans are limited, when you plug them into the motherboard, and much higher when you connect it to a 4 or 6pin HDD or PCIE connector. They can draw up to 4 amps, which could damage the mobo..

[markdotgooley]

Just bought one of these. We'll see how much it helps. Said to be noisy. https://www.amazon.com/Bgears-b-Blaster ... oniccom-20

Airflow on those high rpm fans are limited, when you plug them into the motherboard, and much higher when you connect it to a 4 or 6pin HDD or PCIE connector. They can draw up to 4 amps, which could damage the mobo..

I worried about that so I bought an adapter so it runs off the power supply directly.

The cards actually run hotter now, after seeming at first not to. More air blows out of the case openings now, but not to much effect. Clearly I have no idea what I'm doing.

[kiore]

Just bought one of these. We'll see how much it helps. Said to be noisy. https://www.amazon.com/Bgears-b-Blaster ... oniccom-20

Airflow on those high rpm fans are limited, when you plug them into the motherboard, and much higher when you connect it to a 4 or 6pin HDD or PCIE connector. They can draw up to 4 amps, which could damage the mobo..

I worried about that so I bought an adapter so it runs off the power supply directly.

The cards actually run hotter now, after seeming at first not to. More air blows out of the case openings now, but not to much effect. Clearly I have no idea what I'm doing.

Maybe they are hotter as you on more complex projects. Maybe the ambient is hotter so the temperature exchange less efficent. Maybe you have no idea what you are doing, but think this probably the least likely cause of increased heat. Another thing you can do especially if you are in the part of the world that is in summer now is to power limit the cards using something like MSI Afterburner. This allows you to increase the card fans and decrease the thermal limits for the cards and I suggest you start by droppping the max power to 90% and see how that goes, ppd will maybe drop a bit but should reduce temps to <80c if not try 85%. I run my GTX 1660ti at 82% as it is directly beneath my GTX1080ti and gets very hot, the 1080ti is hybrid liquid cooled and stays at 55C but the twin fan 1660ti is just spinning all that heat around.

[rickoic]

I just finished upgrading a MSI X399 motherboard I had purchased several months ago.
AMD Threadripper 1900X 8 Core 3.80 GHz. Has 2 PCIe 16X and 2 PCIe 8X slots.
GTX 2080, GTX 1070, GTX 1070, and GTX 960 as well as CPU 12 running
My concern was the close proximity of the gpus to each other so I also just purchased an open air mining frame thinking of using ribbon risers to raise them so they were separated, after looking at it closer I decided to leave the gpus in PCIe slots 1 and 3 installed on the motherboard and use the ribbon risers to raise the gpus in slots 2 and 4 upward which gave me even more separation.

Been running for a couple of days now and FAHclient shows I'm getting around 3.1M points per day from the setup. This of course varies depending on wu's received.

[markdotgooley]

I just finished upgrading a MSI X399 motherboard I had purchased several months ago.
AMD Threadripper 1900X 8 Core 3.80 GHz. Has 2 PCIe 16X and 2 PCIe 8X slots.
GTX 2080, GTX 1070, GTX 1070, and GTX 960 as well as CPU 12 running
My concern was the close proximity of the gpus to each other so I also just purchased an open air mining frame thinking of using ribbon risers to raise them so they were separated, after looking at it closer I decided to leave the gpus in PCIe slots 1 and 3 installed on the motherboard and use the ribbon risers to raise the gpus in slots 2 and 4 upward which gave me even more separation.

Been running for a couple of days now and FAHclient shows I'm getting around 3.1M points per day from the setup. This of course varies depending on wu's received.

Sounds quite impressive. I might eventually try for something similar.

[HaloJones]

Sounds good. What's the breakdown between each gpu and the cpu?

[rickoic]

CPU 12 P14384 191,319 PPD
2080 X16 slot P16600 1,860,051 PPD
1070 X16 slot P13414 748,584 PPD
1070 X8 slot P14459 123,299 PPD
960 X8 slot P14468 191,319 PPD

My next thought is to move the folding folder to my M.2 card.

[HaloJones]

why is the second 1070 so slow, that doesn't look right at all.