CPU & mobo requirements for 6 high powered GPUs

[fruor]

I have a 6x R9 290 system with a crappy mobo & cpu and wanted to do some upgrading in order to enter FAH.
I heard 1 cpu core per gpu is needed, but there is a lot of technical stuff in this I don't yet understand.

My plan is to upgrade my mainboard to an AMD Sockel AM3+ and filling it up with a AMD FX-8320, 8x 3.50GHz
Is that enough? overkill? will it work smoothly? what other things do I have to keep in mind (RAM etc.)

For those wondering, I come from the cryptocurrency world where these GPUs do just fine.

[PantherX]

Welcome to the F@H Forum fruor,

Depending on the AMD WHQL Driver version used, you may need 1 CPU per GPU or not, assuming that you will be getting FahCore_17 WUs. You would have to test it out and report back here.

Setting-up 6 GPUs on a single motherboard is going to be a challenge since I haven't come across many motherboard models. I would suggest you break it up into 2 systems; 3+3 or 4+2 which makes it easy to manage and maintain. There are reports that high-end Nvidia GPUs suffer performance loss when running on limited PCI-E bandwidth (viewtopic.php?f=38&t=25253) so the same could apply to AMD GPUs too.

Do note that you would need a beefy branded PSU to fold on the GPUs since folding is quite intensive when compared to games and other applications (I don't know how much stress mining puts on the GPU and how it compares to folding).

Moreover, heat generated could be an issue if the systems aren't well ventilated.

[fruor]

thanks very much PantherX, I already saw you discussing curecoin and this is exactly what I'm aiming at.
I had a rough time setting my 6 GPUs up, and all of your mentioned problems regarding 6 GPUs are true (and there are more) and it took some effort to resolve. However, I already succeeded and the only thing I'm concerned right now is getting all the cards recognized on the new motherboard, but I will handle it.

What about having 8 cpu cores for 6 GPUs, will it work? What about RAM consumation?

[fruor]

oh, I just saw that: is it true folding with GPUs require full PCIe x16 bandwith? Because most of my GPUs are on a PCIe x1 slot at the moment (using x1 to x16 risers).

[Nathan_P]

No, they will fold on an x1 slot but you may see lower performance, than if you were on a x16 slot. Not sure of the difference but I think its around 10%

[PantherX]

...What about having 8 cpu cores for 6 GPUs, will it work? What about RAM consumation?

Assuming 1 CPU per GPU for FahCore_17, you would need 6 CPUs so 8 CPUs should be sufficient. Do note that while there might be 4 FPUs shared across 8 CPUs, I haven't seen any posts which indicated that FPUs are being used by FahCore_17.

I use Process Lasso and have locked 2 instances of FahCore_17 to a single virtual core (i7-860 @ 3.8 GHz) and didn't notice any significant loss in PPD on my GTX 660 Ti and GTX 650. However, considering that your GPUs are in the high-end range, you may see a drop in performance.

RAM consumption varies on the Project being folded. IIRC, I got a maximum of ~400 MBs on a single GPU. So if we take 500 MB per GPU, you would need at least 3 GB so the standard of 4 GB or 8 GB would be sufficient, depending if the system is a dedicated folding rig or not.

If you are talking about VRAM usage, I haven't seen it cross 100 MB yet. However, this again varies on the Project so the standard of 1 GB or 2 GB would be sufficient. You would need more if you are running applications that use the GPU.

...is it true folding with GPUs require full PCIe x16 bandwith? Because most of my GPUs are on a PCIe x1 slot at the moment (using x1 to x16 risers).

While the GPUs can function in those PCI-E slots, the performance of folding could potentially be impacted. By what degree is unknown to me and would vary from system to system and the GPUs involved. A good summary can be found by reading post (viewtopic.php?p=254816#p254816).

[bruce]

Even though each GPU will use as much as a full CPU on Linux and perhaps less on Windows, I have a hunch that if each GPU doesn't get exclusive access to a dedicated CPU, there won't be a significant loss in performance. I don't have a system on which I can test it but it's worth a shot. If I had 8 Hyperthreaded CPUs and 6 GPUs, I would start by dedicating 4 CPUs to CPU folding and let the other four manage 6 GPUs. Something like Process Lasso would be needed to make sure that the 4 CPUs each had a dedicated FPU. I'd also figure out how to allocate a 16x bandwidth as evenly as possible across all GPUs.

[fruor]

why would you make only 4 CPUs manage 6 GPUs? I heard the ratio should be about 1:1, can you expand if this info is not correct?

[PantherX]

I would say that the information is correct to a certain degree. The important factor is what GPU type and Driver is being used in the system:

Nvidia GPUs
All recent drivers from Nvidia require 1 CPU per FahCore_17 instance. This is a design feature selected by Nvidia for the implementation of OpenCL. However, some have found out that you can force two instances of FahCore_17 onto a single CPU (be it virtual or real) with minimum effect on the PPD by using CPU affinity. Not sure how it is done on Linux, but for Windows, I have used Process Lasso. Moreover, only GK-110 will benefit from driver version higher than 327.23 while other GPUs will see a significant drop in PPD.

AMD GPUs
Donors have reported that the most recent WHQL Drivers by AMD will use 1 CPU per FahCore_17 instance. Previous drivers would only use the CPU occasionally (during start-up, checkpoints, verification, etc). The latest Beta brings significant improvements to a wide range of GPUs but am not sure of the CPU Usage.

Thus, if you go with the notion of 2 FahCore_17 instances per CPU, your 6 GPUs would require 3 CPUs only. If you go with 1 CPU per FahCore_17 instance, you will need 6 CPUs. In the end, once you build the system, test out what settings work best for you and you can post it here for future reference.

[davidcoton]

Sharing a core (deliberately with affinity or accidentally oversubscribing) between CPU and GPU folding has a detrimental effect on CPU performance. I have not proved (or didn't notice the figures) whether GPU performance is also affected -- that is, although Core17 keeps a CPU core busy, it is not clear whether it contributes much to the work, or whether it is just polling for completion of a frame when data needs to be transferred. If it is mainly polling, two Core17 processes on one core will only cause a slowdown when the data transfers coincide (which will happen, but not too much). Allowing four CPU cores to manage six GPUs will have less effect than using just three, although IF MY SUSPICION IS CORRECT there will be a law of diminishing returns. Also, if the eight CPU cores share just four FPUs (as I think most eight-core processors do) the extra contribution that cores 5-8 make to CPU folding is less than linear. I think also that some CPUs (Intel) have four physical CPUs for eight virtual, while some others (AMD) have eight physical CPUs and only share the FPUs.

So there are several scenarios you can test, and expect different results depending on the CPU chip, GPU manufacturer, and phase of the moon:
1) Allocate eight cores to CPU and oversubscribe with the GPUs on top: I predict this will be some way from optimum.
2) Reduce the CPU core count to six and leave two for the GPUs
3) Reduce the CPU core count to five and leave three for the GPUs
4) Reduce the CPU core count to four and leave four for the GPUs
5) Reduce the CPU core count to two and leave six for the GPUs

Run each configuration for the time it takes for four or five WUs on each slot. Note the total PPD in each configuration. Run with the best.
I wouldn't try CPU:7 because it can be troublesome. CPU:3 is worth trying if CPU:4 is otherwise best.
You may need to use Process Lasso (or equivalent) to set the affinity so that the cores used for CPU folding are associated with different FPUs, or the system may just get it right anyway. Another experiment required.

Let us all know what you find!

David

[folding_hoomer]

I´m using Catalyst 14.1 BetaV1.6 under Win7 64Bit together with a C2D6300@2600MHz and one R9 290X.
It uses an updated OpenCL-Driver that speeds up folding for all AMD/ATI-GPU´s.

Process Lasso shows an average usage of ~13% of the whole CPU (2 Cores) for folding with Core17.
The usage differs from 6-8% when folding a GPU-WU under Core 17 and 50% when starting a new GPU-WU or when checking the integrity of the data.

So i see no problem to use only 4 CPU-Cores for 6 GPU´s.

[fruor]

folding_hoomer can you give me a Link to the updated drivers?

As far as I understand the replies correctly, my hardware decision should be fine. Whatever software tuning I will make afterwards. I plan on using 3x 16x-16x risers and 3 x1-x16 risers so I will lose about 5% performance (1-(3*0,9 + 3*1)/6) to the use of 3 PCIe x1 slots.

[folding_hoomer]

Catalyst 14.1 BetaV1.6 can be downloaded at:
http://support.amd.com/en-us/download/d ... ows+7+-+64

[NookieBandit]

@ fruor, please let us know how your configuration performs. You may also want to use the third-party app HFM as a tool to more accurately measure TPF/PPD of your setup.

[fruor]

I am very sorry that I dropped this topic for such a long time. Nothing happened since my last post. It seems Curecoin will be launching these days so I am trying to get ready this week. At latest beginning of next week there will be updates on my lousy attempts.

Thanks for all the input so far, most of which is new to me so I have to do some reading and trying.