Building a dedicated rig with 3 GPUs

A forum for discussing FAH-related hardware choices and info on actual products (not speculation).

Moderator: Site Moderators

Forum rules
Please read the forum rules before posting.

Re: Building a dedicated rig with 3 GPUs

Postby Mxyzptlk » Tue Jun 16, 2020 1:51 pm

From the specs, that 200mm front fan only does 110cfm at peak. Is the fan running full out? That said, I am running with 2 140mm fans that each do 107cfm on the front. But they are running at about 83% atm.

I have no idea how to control the power on the GPU's w/ Linux. sorry.
(2) Amd 2700X’s / (1) AMD 3900X / 2060 Super / 2060 / (2) 2070 Super
Mxyzptlk
 
Posts: 12
Joined: Wed Apr 08, 2020 9:55 pm
Location: California

Re: Building a dedicated rig with 3 GPUs

Postby markdotgooley » Tue Jun 16, 2020 3:31 pm

Mxyzptlk wrote:From the specs, that 200mm front fan only does 110cfm at peak. Is the fan running full out? That said, I am running with 2 140mm fans that each do 107cfm on the front. But they are running at about 83% atm.

I have no idea how to control the power on the GPU's w/ Linux. sorry.


I have all the case fans just plugged into the power supply. (I'm a bit deaf and the machine is tucked away in a far corner of the room.) Maybe I can try some fans that move more air. It's all a bit frustrating, but the machine seems to run fine if rather hot.
markdotgooley
 
Posts: 72
Joined: Tue Apr 21, 2020 12:46 pm

Re: Building a dedicated rig with 3 GPUs

Postby Ichbin3 » Tue Jun 16, 2020 4:18 pm

nvidia-smi is what you are looking for.
Hack it into the terminal
nvidia-smi -L lists the GPUs
nvidia-smi -i 0 -pl 130 this puts your first GPU on 130W, this is almost half of the power and heat, but you will get still a decent PPD which is much more than the expected half.
nvidia-smi -h gives you an idea what else you can do.
Last edited by Ichbin3 on Wed Jun 17, 2020 10:01 am, edited 1 time in total.
MSI B450 Tomahawk, Ryzen 5 2600, RTX 2080Ti@180W, Mint 19.3
Ichbin3
 
Posts: 69
Joined: Thu May 28, 2020 9:06 am
Location: Germany

Re: Building a dedicated rig with 3 GPUs

Postby markdotgooley » Tue Jun 16, 2020 5:00 pm

Ichbin3 wrote:nvidia-smi is what you are locking for.
Hack it into the terminal
nvidia-smi -L lists the GPUs
nvidia-smi -i 0 -pl 130 this puts your first GPU on 130W, this is almost half of the power and heat, but you will get still a decent PPD which is much more than the expected half.
nvidia-smi -h gives you an idea what else you can do.


Thanks! one card is using around 149W and the other around 156W so it's no emergency, but they might run cooler if I dropped those to maybe 125W which is given as "Min. Power Limit."
markdotgooley
 
Posts: 72
Joined: Tue Apr 21, 2020 12:46 pm

Re: Building a dedicated rig with 3 GPUs

Postby RDDyer » Tue Jun 16, 2020 5:52 pm

Here's my 2 bits. I'm no expert, but I can offer you my experience having just built a dedicated rig with 2 GPUs. It is working pretty well, and the machine is earning about 2.7-2.9 million points on a typical day. Hopefully someone can tell me if it should be better based on the gear I have. Maybe I need to change something.

I'm a Team Red sorta guy, so it's all AMD. I chose:
- X570 Taichi motherboard to get PCIe gen 4 (for the bandwidth)
- Ryzen 9 3900X CPU
- 3600MHz C16-16-16-36 DDR4 32GB (16GB is easily enough)
- 1TB X.2 SSD PCIe gen 4
- Arctic 360 AIO for CPU
- Lian Li PC-T60B ATX Test Bench instead of a case
- 2 Gigabyte Radeon RX 5700 XT GAMING OC 8G GPUs
- Corsair 1000W Power Supply
- Windows 10 Pro

Comments:
MB - I'm fairly happy with this choice (though the B550s just came out today). It performs well, and having post code readouts for the first time, I'll never go back. (They helped me diagnose a problem with the first set of memory modules I bought.) PCIe slot arrangement is not ideal, though it is typical for ATX boards. As you know, you really have to look through the specs to see how the PCIe lanes are shared among all the slots, including the SATA ports. Just because you have 3 X16 connectors, you won't have 3 X16 slots. There just aren't enough lanes to go around. And just finding the physical room for 3 GPUs isn't easy. (See below.)
CPU - At first I thought I had wasted money here, thinking I just needed a good enough CPU to shuttle data to and from the GPUs, but it's cranking out CPU WUs with way, way higher ppd than my 2600X machine. Worth it! I may try turning off SMT as others have suggested, though I'll be curious to see how it affects CPU WU performance.
Mem - Definitely get the higher speed stuff - Ryzen likes it. I've over-volted a little so I could tighten the timings a bit. If you know what you're doing, maybe you can go higher. Again I'm not an expert. Buildzoid says this motherboard will do better if all memory slots are populated due to the PCB topology, so 4-4GB modules might be a good bang-for-buck choice.
SSD - If you want to save some money this may be a place, though mine was pretty cheap. A PCIe gen 3 250MB is probably enough. Maybe even a SATA III drive. There doesn't appear to be much traffic to and from the SSD during operation.
AIO - Cooling is the biggest problem with my setup. CPU temps are in the mid to low 80s. This AIO was pretty cheap, and I prefer water cooling, though a custom loop might be better. The main thing I don't like about air coolers is that to get lower temps you need a huge piece of hardware covering your entire motherboard and I want easy access to everything.
Case - or lack of. I would definitely stay away from a case if you don't mind the noise. I used 20X20 Aluminum Profile for my 2600X setup. It was a bit harder to build, but cheaper. The test bench is nice, but you pay for stuff you don't need like HD bays, and the AIO wasn't easy to mount. I'll also need to modify one of the aluminum pieces if I ever want more than 2 GPUs.
GPUs - I'm really happy with these GPUs. At just under 400 EUR each, they're much less than a comparable Nvidia card. They're PCIe gen 4, so you can get typical X16 bandwidth with an X8 slot. The only disadvantage is that because they're so thick (nearly 50mm) it's hard to get air between them to the cooling fans of GPU0. They have an aluminum backplate, so I've stuck a couple fairly large heatsinks on the back of GPU0 and there's a desk fan blowing 24X7 on GPU1. I used the Adrenalin software to overclock them a bit, but I had to make the fan curves very aggressive, so they are pretty noisy. GPU temps run in the low 50s with junction temps in the low 70s with clocks in the 2040-2060 MHz range.
OS - I run Windows since I don't know anything else. I have no idea if Linux would improve performance. I only paid about 15 EUR for a legal OEM version.

All that said, I've thought about adding another GPU. I think my 1000W PSU would handle it, however I don't know where to put it. Since the GPUs are so thick as I mentioned, the second video card covers the 3rd PCIe slot of a standard ATX MB. If I modify the test bench, I could move one of the GPUs to the 3rd slot and put a riser in the 2nd for another GPU, but I don't think it's worth all the work. The 3rd slot would only have a X4 data path on this MB, but gen 4 gives me an equivalent X8 bandwidth, so that's not an issue. I'm curious which motherboard you were considering?

Anyway, that's probably a lot more verbiage than you were looking for. You'll have to glean the grain from the chaff, my friend.
_________________________________________________
Intel i5-3470 - (no GPU work) - Win10 Pro - v7.6.13
AMD 2600 - RX Vega 56 - Win10 Pro - v7.6.13
AMD 2600X - RX Vega 56 - Win10 Pro - v7.6.13
AMD 3900X - Dual RX5700XT - Win10 Pro - v7.6.13
RDDyer
 
Posts: 6
Joined: Sun Jun 14, 2020 8:30 pm

Re: Building a dedicated rig with 3 GPUs

Postby markdotgooley » Tue Jun 16, 2020 10:41 pm

markdotgooley wrote:
Mxyzptlk wrote:From the specs, that 200mm front fan only does 110cfm at peak. Is the fan running full out? That said, I am running with 2 140mm fans that each do 107cfm on the front. But they are running at about 83% atm.

I have no idea how to control the power on the GPU's w/ Linux. sorry.


I have all the case fans just plugged into the power supply. (I'm a bit deaf and the machine is tucked away in a far corner of the room.) Maybe I can try some fans that move more air. It's all a bit frustrating, but the machine seems to run fine if rather hot.


Just bought one of these. We'll see how much it helps. Said to be noisy. https://www.amazon.com/Bgears-b-Blaster ... oniccom-20
markdotgooley
 
Posts: 72
Joined: Tue Apr 21, 2020 12:46 pm

Re: Building a dedicated rig with 3 GPUs

Postby Mxyzptlk » Thu Jun 18, 2020 2:38 am

WOW! Your PC might lift off the desk with that one! Let us know how it goes.
Mxyzptlk
 
Posts: 12
Joined: Wed Apr 08, 2020 9:55 pm
Location: California

Re: Building a dedicated rig with 3 GPUs

Postby MeeLee » Thu Jun 18, 2020 3:00 am

Just bought one of these. We'll see how much it helps. Said to be noisy. https://www.amazon.com/Bgears-b-Blaster ... oniccom-20


Airflow on those high rpm fans are limited, when you plug them into the motherboard, and much higher when you connect it to a 4 or 6pin HDD or PCIE connector. They can draw up to 4 amps, which could damage the mobo..
MeeLee
 
Posts: 930
Joined: Tue Feb 19, 2019 11:16 pm

Re: Building a dedicated rig with 3 GPUs

Postby markdotgooley » Thu Jun 18, 2020 9:40 pm

MeeLee wrote:
Just bought one of these. We'll see how much it helps. Said to be noisy. https://www.amazon.com/Bgears-b-Blaster ... oniccom-20


Airflow on those high rpm fans are limited, when you plug them into the motherboard, and much higher when you connect it to a 4 or 6pin HDD or PCIE connector. They can draw up to 4 amps, which could damage the mobo..

I worried about that so I bought an adapter so it runs off the power supply directly.

The cards actually run hotter now, after seeming at first not to. More air blows out of the case openings now, but not to much effect. Clearly I have no idea what I'm doing.
markdotgooley
 
Posts: 72
Joined: Tue Apr 21, 2020 12:46 pm

Re: Building a dedicated rig with 3 GPUs

Postby kiore » Fri Jun 19, 2020 4:03 am

markdotgooley wrote:
MeeLee wrote:
Just bought one of these. We'll see how much it helps. Said to be noisy. https://www.amazon.com/Bgears-b-Blaster ... oniccom-20


Airflow on those high rpm fans are limited, when you plug them into the motherboard, and much higher when you connect it to a 4 or 6pin HDD or PCIE connector. They can draw up to 4 amps, which could damage the mobo..

I worried about that so I bought an adapter so it runs off the power supply directly.

The cards actually run hotter now, after seeming at first not to. More air blows out of the case openings now, but not to much effect. Clearly I have no idea what I'm doing.


Maybe they are hotter as you on more complex projects. Maybe the ambient is hotter so the temperature exchange less efficent. Maybe you have no idea what you are doing, but think this probably the least likely cause of increased heat. Another thing you can do especially if you are in the part of the world that is in summer now is to power limit the cards using something like MSI Afterburner. This allows you to increase the card fans and decrease the thermal limits for the cards and I suggest you start by droppping the max power to 90% and see how that goes, ppd will maybe drop a bit but should reduce temps to <80c if not try 85%. I run my GTX 1660ti at 82% as it is directly beneath my GTX1080ti and gets very hot, the 1080ti is hybrid liquid cooled and stays at 55C but the twin fan 1660ti is just spinning all that heat around.
Image
i7 7800x GTX 1080ti GTX1660ti, OS= win10. AMD 3700x RTX 2080ti OS= win10

Team page: http://www.rationalskepticism.org/gener ... -t616.html
kiore
 
Posts: 827
Joined: Fri Jan 16, 2009 6:45 pm
Location: USA

Re: Building a dedicated rig with 3 GPUs

Postby rickoic » Sat Jun 20, 2020 4:20 pm

I just finished upgrading a MSI X399 motherboard I had purchased several months ago.
AMD Threadripper 1900X 8 Core 3.80 GHz. Has 2 PCIe 16X and 2 PCIe 8X slots.
GTX 2080, GTX 1070, GTX 1070, and GTX 960 as well as CPU 12 running
My concern was the close proximity of the gpus to each other so I also just purchased an open air mining frame thinking of using ribbon risers to raise them so they were separated, after looking at it closer I decided to leave the gpus in PCIe slots 1 and 3 installed on the motherboard and use the ribbon risers to raise the gpus in slots 2 and 4 upward which gave me even more separation.

Been running for a couple of days now and FAHclient shows I'm getting around 3.1M points per day from the setup. This of course varies depending on wu's received.
Duel 2.8 3 250's Quad 2.4 285. 260, Quad 2.4 3 250 , i7 2.27 2 250 GPU's, i7 2.24 2 250 GPU's, i7 3.06 bigadv, duel Xeon 2.27 bigadv, AMD Phenom ][ 3 250 GPU's, Laptop GT 130M.
I'm folding because Dec 2005 I had radical prostrate surgery.
rickoic
 
Posts: 257
Joined: Sat May 23, 2009 5:49 pm
Location: Mississippi near Memphis, Tn

Re: Building a dedicated rig with 3 GPUs

Postby markdotgooley » Sun Jun 21, 2020 12:58 am

rickoic wrote:I just finished upgrading a MSI X399 motherboard I had purchased several months ago.
AMD Threadripper 1900X 8 Core 3.80 GHz. Has 2 PCIe 16X and 2 PCIe 8X slots.
GTX 2080, GTX 1070, GTX 1070, and GTX 960 as well as CPU 12 running
My concern was the close proximity of the gpus to each other so I also just purchased an open air mining frame thinking of using ribbon risers to raise them so they were separated, after looking at it closer I decided to leave the gpus in PCIe slots 1 and 3 installed on the motherboard and use the ribbon risers to raise the gpus in slots 2 and 4 upward which gave me even more separation.

Been running for a couple of days now and FAHclient shows I'm getting around 3.1M points per day from the setup. This of course varies depending on wu's received.

Sounds quite impressive. I might eventually try for something similar.
markdotgooley
 
Posts: 72
Joined: Tue Apr 21, 2020 12:46 pm

Re: Building a dedicated rig with 3 GPUs

Postby HaloJones » Sun Jun 21, 2020 2:48 pm

Sounds good. What's the breakdown between each gpu and the cpu?
1x Titan X, 5x 1070, 1x 970, 1 x Ryzen 3600

Image
HaloJones
 
Posts: 816
Joined: Thu Jul 24, 2008 11:16 am

Re: Building a dedicated rig with 3 GPUs

Postby rickoic » Sun Jun 21, 2020 7:02 pm

CPU 12 P14384 191,319 PPD
2080 X16 slot P16600 1,860,051 PPD
1070 X16 slot P13414 748,584 PPD
1070 X8 slot P14459 123,299 PPD
960 X8 slot P14468 191,319 PPD

My next thought is to move the folding folder to my M.2 card.
rickoic
 
Posts: 257
Joined: Sat May 23, 2009 5:49 pm
Location: Mississippi near Memphis, Tn

Re: Building a dedicated rig with 3 GPUs

Postby HaloJones » Sun Jun 21, 2020 10:31 pm

why is the second 1070 so slow, that doesn't look right at all.
HaloJones
 
Posts: 816
Joined: Thu Jul 24, 2008 11:16 am

PreviousNext

Return to FAH Hardware

Who is online

Users browsing this forum: Google [Bot] and 4 guests

cron