Frame Times on BigAdv A3 projects.

[Trotador]

My first 2684, so I can't complain a lot:

Average time/frame: 00:39:11
CPU: 2 x L5520 @ 2.4GHz (turbo)
# of CPU sockets: 2
# of Physical cores: 8
# of FAH CPU processes: 16
# of FAH GPU Clients: 0

RAM GB installed: 12GB
RAM Type: DDR3
RAM Speed: 1066 ?
RAM use: <2 GB (total)
OS / Linux kernel: Ubuntu 9.1 2.6.33-020633-generic + mpiexec tear
Running in VM: no
core A3 2.22

so far at 32th frame

[PantherX]

windows 7 thread handling should correctly put the load on all the real cores any thing left would go onto the HT threads evenly (win7 does it tad better then Vista they improved Thread handling due to HT and i7 for the most part as Vista did not always correctly put CPU load onto an real core that was free where as windows 7 does)

Can you please tell me where I can find more details about it?

Thanks

[Karamiekos]

Upgraded to the 2.22 A3 core and I got this right away on my Ubuntu 9.10 rig with 8 gigs ram and 4 quad core optys:

Code: Select all

[14:52:54] Verifying core Core_a3.fah...

[14:52:54] Signature is VALID

[14:52:54]

[14:52:54] Trying to unzip core FahCore_a3.exe

[14:52:54] Decompressed FahCore_a3.exe (5277040 bytes) successfully

[14:52:54] + Core successfully engaged

[14:52:59]

[14:52:59] + Processing work unit

[14:52:59] Core required: FahCore_a3.exe

[14:52:59] Core found.

[14:52:59] Working on queue slot 01 [June 19 14:52:59 UTC]

[14:52:59] + Working ...

[14:52:59] - Calling './FahCore_a3.exe -dir work/ -nice 19 -suffix 01 -np 15 -priority 96 -checkpoint 30 -verbose -lifeline 2050 -version 629'



[14:52:59]

[14:52:59] *------------------------------*

[14:52:59] Folding@Home Gromacs SMP Core

[14:52:59] Version 2.22 (June 10, 2010)

[14:52:59]

[14:52:59] Preparing to commence simulation

[14:52:59] - Looking at optimizations...

[14:52:59] - Created dyn

[14:52:59] - Files status OK

[14:53:02] - Expanded 24829290 -> 30791309 (decompressed 124.0 percent)

[14:53:02] Called DecompressByteArray: compressed_data_size=24829290 data_size=30791309, decompressed_data_size=30791309 diff=0

[14:53:02] - Digital signature verified

[14:53:02]

[14:53:02] Project: 2684 (Run 3, Clone 4, Gen 4)

[14:53:02]

[14:53:03] Assembly optimizations on if available.

[14:53:03] Entering M.D.

Starting 15 threads

NNODES=15, MYRANK=0, HOSTNAME=thread #0

NNODES=15, MYRANK=6, HOSTNAME=thread #6

NNODES=15, MYRANK=5, HOSTNAME=thread #5

NNODES=15, MYRANK=7, HOSTNAME=thread #7

NNODES=15, MYRANK=8, HOSTNAME=thread #8

NNODES=15, MYRANK=2, HOSTNAME=thread #2

NNODES=15, MYRANK=14, HOSTNAME=thread #14

NNODES=15, MYRANK=9, HOSTNAME=thread #9

NNODES=15, MYRANK=1, HOSTNAME=thread #1

NNODES=15, MYRANK=3, HOSTNAME=thread #3

NNODES=15, MYRANK=11, HOSTNAME=thread #11

NNODES=15, MYRANK=12, HOSTNAME=thread #12

NNODES=15, MYRANK=4, HOSTNAME=thread #4

NNODES=15, MYRANK=10, HOSTNAME=thread #10

NNODES=15, MYRANK=13, HOSTNAME=thread #13

Reading file work/wudata_01.tpr, VERSION 4.0.99_development_20090605 (single precision)

Making 2D domain decomposition 5 x 3 x 1

starting mdrun 'SINGLE VESICLE in water'

1250000 steps,   5000.0 ps (continuing from step 1000000,   4000.0 ps).

[14:53:15] Completed 0 out of 250000 steps  (0%)



step 1000001: Water molecule starting at atom 497223 can not be settled.

Check for bad contacts and/or reduce the timestep if appropriate.



step 1000001: Water molecule starting at atom 139446 can not be settled.

Check for bad contacts and/or reduce the timestep if appropriate.



-------------------------------------------------------

Program mdrun, VERSION 4.0.99-dev-20100610-b6a86-dirty

Source code file: /data0/FAHdev/a3_development/gromacs/src/mdlib/pme.c, line: 535



Fatal error:

1 particles communicated to PME node 10 are more than 2/3 times the cut-off out of the domain decomposition cell of their charge group in dimension xThis usually means that your system is not well equilibrated

For more information and tips for troubleshooting, please check the GROMACS

website at http://www.gromacs.org/Documentation/Errors

I have no history of problems with A3 as far as I know of and A2s are stable on this machine. Just informing the mods/PG

Fold on.

[leexgx]

windows 7 thread handling should correctly put the load on all the real cores any thing left would go onto the HT threads evenly (win7 does it tad better then Vista they improved Thread handling due to HT and i7 for the most part as Vista did not always correctly put CPU load onto an real core that was free where as windows 7 does)

Can you please tell me where I can find more details about it?

Thanks

there are others but it was official news but its quite old now
http://www.bit-tech.net/news/bits/2009/ ... hreading/1

as long as you do not mess with cpu affinity your self windows 7 should correctly place the load accordingly (VMware by default is set to use any thread but as its limited to 8 cores the other 2 threads (as you have 6 core CPU with HT) would get spread over all of the HT threads evenly or should do any way)

well i had another A3 Core Fail to Restart after i had an hang on me server (all 5 backups would Not allow to read the Stupid wudata_08.dat or what ever it fails on waste of 2.4 days) FIX the checkpoints please

[orion]

2684 with the new v2.22 core

Average time/frame: 0:30:50
CPU: 8356 @ 2.3
# of CPU sockets: 4
# of Physical cores: 16

# of FAH CPU processes: 15
# of FAH GPU Clients: 0


RAM installed: 16GB
RAM type: DDR2
RAM speed: 667
RAM timings: 5-5-5-15
OS/kernel: Ubuntu 8.04 server 2.6.24-26
Running in VM: no

[ParrLeyne]

2684 with the new v2.22 core
...
# of Physical cores: 16
# of FAH CPU processes: 15
...

You might want to run 16 FAH processes. The v2.22 core release notes suggest that the previous problem with the way that the core would create the 'helper threads' -- by running 15 vs. 16 processes, that would 'trick' the allocation algorithm to create more calculation processes.

[Note: I know have not used the appropriate terminology and that my explanation is "loose", but the idea is valid -- I saw it for myself -- so let's not get started on path of focusing on the language, it is the idea which I am trying to express that is the important item.]

[PantherX]

windows 7 thread handling should correctly put the load on all the real cores any thing left would go onto the HT threads evenly (win7 does it tad better then Vista they improved Thread handling due to HT and i7 for the most part as Vista did not always correctly put CPU load onto an real core that was free where as windows 7 does)

Can you please tell me where I can find more details about it?

Thanks

there are others but it was official news but its quite old now
http://www.bit-tech.net/news/bits/2009/ ... hreading/1

as long as you do not mess with cpu affinity your self windows 7 should correctly place the load accordingly (VMware by default is set to use any thread but as its limited to 8 cores the other 2 threads (as you have 6 core CPU with HT) would get spread over all of the HT threads evenly or should do any way)

If you are referring to me (bold part) it must be in an alternate universe ( ) as I am using 4 Core /8 thread CPU = i7-860 (The 980X isn't available here nor will be )

BTW Thanks for the link to the article.

[orion]

2684 with the new v2.22 core
...
# of Physical cores: 16
# of FAH CPU processes: 15
...

You might want to run 16 FAH processes. The v2.22 core release notes suggest that the previous problem with the way that the core would create the 'helper threads' -- by running 15 vs. 16 processes, that would 'trick' the allocation algorithm to create more calculation processes.

I'll try it the next time it picks up a 2684, it's on a 2681 now.

[Kougar]

there are others but it was official news but its quite old now
http://www.bit-tech.net/news/bits/2009/ ... hreading/1

Yes, bit sad to say but Windows 7 is the only version of Windows that recognizes the difference between a physical core and a logical core. Or recognizes what an SSD is and configures itself accordingly.

well i had another A3 Core Fail to Restart after i had an hang on me server (all 5 backups would Not allow to read the Stupid wudata_08.dat or what ever it fails on waste of 2.4 days) FIX the checkpoints please

I've had this issue myself, only one of the four backups would actually work. You might consider installing/configuring some of the additional backup scripts, including lengthen the backup intervals and increase the number of backups. Quite a few links to backup scripts/files in LinuxFAH/LinuxRouter's post for anyone looking, separate from the VM image itself: http://www.evga.com/forums/tm.aspx?&m=4464&mpage=1

[bruce]

well i had another A3 Core Fail to Restart after i had an hang on me server (all 5 backups would Not allow to read the Stupid wudata_08.dat or what ever it fails on waste of 2.4 days) FIX the checkpoints please

Why do you assume it's a checkpoint problem rather than something to do with the way you're running it? viewtopic.php?p=148470#p148470 I can't prove it either way, but I don't have checkpoint problems.

[ahu]

windows 7 thread handling should correctly put the load on all the real cores any thing left would go onto the HT threads evenly (win7 does it tad better then Vista they improved Thread handling due to HT and i7 for the most part as Vista did not always correctly put CPU load onto an real core that was free where as windows 7 does)

Can you please tell me where I can find more details about it?

Thanks

there are others but it was official news but its quite old now
http://www.bit-tech.net/news/bits/2009/ ... hreading/1

as long as you do not mess with cpu affinity your self windows 7 should correctly place the load accordingly (VMware by default is set to use any thread but as its limited to 8 cores the other 2 threads (as you have 6 core CPU with HT) would get spread over all of the HT threads evenly or should do any way)

I don't think this is true. I can give you two examples on my 12/24-core Windows 7 system that would seem invalidate this:
- VMWare with 8-core VM: On my system I get MUCH better performance by either disabling HT or assigning the OS threads manually in VMWare configuration file to cores 0, 2, 4, 6, 8, 10, 12 & 14.
- 3DMark Vantage CPU-test: This test seems to be capped at 14 threads according to Task Manager. When I disable HT I have all 12 physical cores utillized insted of a mix of 14 physical/logical cores and get significantly better result.

I'm talking about a performance improvement of around 30% - 50%. So the automatic OS thread optimization simply isn't happening here. Of course there can be app specific oddities, but I would really like to see an example where this would actually work.

[Kougar]

Unless you are maxing out all the logical cores/threads, you WILL see better performance by disabling HT. Especially if you are only using few to none of the spare logical cores. As you said you were using two that is 12 physical and 4 logical cores, with 8 more unused. The best case for Hyperthreading is when you are maxing out all available threads, only then does performance tend to ratchet up 15-30% more than the same CPU with HT disabled. Look back at past results where users tested a Core i7 920 with and without HT enabled... with HT they would run a second 4-thread SMP core. TPF's for both would drop as both had to split/share some common core resources, but overall the final PPD was still higher.

If you're not maxing out the threads (or hardly using any at all) then dedicating all of the pipeline's resources to a single thread would give you the better performance, that's always been true. Understand how HT actually works... it either splits or shares each core's resources, L2 cache, and other stages of the execution pipeline into two. I'm oversimplifying slightly because I'm not an architecture guru, but basically, it means if only one thread is executing, it still has less resources to work with because HT either shares, reserves, or splits half the resources for the idle logical core. If you want a general list of the resources: http://www.anandtech.com/show/2658/8 He also has benches that illustrate how single-core tasks drop in performance with HT enabled, but overall performance is increased if you max out all threads, such as the Cinebench 1xCPU vs xCPU.

I'm talking about a performance improvement of around 30% - 50%. So the automatic OS thread optimization simply isn't happening here. Of course there can be app specific oddities, but I would really like to see an example where this would actually work.

That's a tougher one to quantify. Anand La Shimpi said that for the most part Windows 7 does a good job of this, but it wasn't perfect. He also said he was still working on a way to test it. Link. The Tech Report briefly mentions this as well, but I don't recall them giving empirical results for testing it. Link

[leexgx]

any way you have to use HT or you do not get Bigadv work units (6 core or 4 core cpus) unless you have an dual socket setup (and even then quite sure HT does have an good impact on frame times)

[stevew]

Project: 2684 (Run 11, Clone 13, Gen 1)

Average time/frame: 00:42:39 (after 2 frames)
CPU: Xeon E5520 @ 2.26 GHz
# of CPU sockets: 2
# of Physical cores: 8 (+ 8 HT = 16)

# of FAH CPU processes: 1 (@ 1590%, all 16 cores flat out)
# of FAH GPU Clients: 0

RAM GB installed: 12
RAM Type: DDR3
RAM Speed: 1066
OS / Linux kernel: OS X 10.6.4
Running in VM: No
----
RAM used by F@h ~1950 MB
Total RAM used 2.91 GB (including OS, w/Activity Mon + Safari+ 3 Terminals open.)
Top shows 1 FahCore_a3.exe @ 1,590% CPU use.
top = Load Avg: 16.19, 16.14, 16.09 CPU usage: 95.32% user, 4.67% sys, 0.0% idle (A2s ran loads of about 13,13,13)
Activity Monitor -> CPU History shows all 16 cores maxed.

[ParrLeyne]

After starting this thread, I have finally got my first 2684 WU (took me a while to rebuild my 980x system with OC, after frying my previous 980x CPU)

Project: 2684 (Run 4, Clone 2, Gen 6)

Average time/frame: 00:31:56 (after 52 frames)
CPU: 980X @ 4.256 GHz (32 x 133)
# of CPU sockets: 1
# of Physical cores: 6

# of FAH CPU processes: 1 (-smp 11)
# of FAH GPU Clients: 0

RAM GB installed: 6
RAM Type: DDR3
RAM Speed: 1866
OS / Linux kernel: 2.6.31-19
Running in VM: No