Problems with Visualization 7.4.4 (Yes, known problems)

Moderators: Site Moderators, FAHC Science Team

Problems with Visualization 7.4.4 (Yes, known problems)

Postby CrissT » Mon Oct 17, 2016 12:52 am

Hi All,

I've got a problem with the viewer. When I view the CPU slot it works fine. When I try to view the GPU slot it doesn't display the protein. It seems that it connects to the slot because I get in the info boxes the project number, the progress bar is updated as well as the time left. the FahCore reads: UNKNOWN_ENUM 0x21
Staus shows: snapshot 0 of 0 with connection connected, Protein: loading and slot 1
Except viewing the GPU slot there are no other issues with it. Both GPU and CPU seems to run the work units without a problem.

This is the beginning of the log file:
Code: Select all
*********************** Log Started 2016-10-09T21:02:09Z ***********************
21:02:09:************************* Folding@home Client *************************
21:02:09:      Website: http://folding.stanford.edu/
21:02:09:    Copyright: (c) 2009-2014 Stanford University
21:02:09:       Author: Joseph Coffland <joseph@cauldrondevelopment.com>
21:02:09:         Args: --open-web-control
21:02:09:       Config: C:/ProgramData/FAHClient/config.xml
21:02:09:******************************** Build ********************************
21:02:09:      Version: 7.4.4
21:02:09:         Date: Mar 4 2014
21:02:09:         Time: 20:26:54
21:02:09:      SVN Rev: 4130
21:02:09:       Branch: fah/trunk/client
21:02:09:     Compiler: Intel(R) C++ MSVC 1500 mode 1200
21:02:09:      Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
21:02:09:               /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT /Qmkl
21:02:09:     Platform: win32 XP
21:02:09:         Bits: 32
21:02:09:         Mode: Release
21:02:09:******************************* System ********************************
21:02:09:          CPU: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
21:02:09:       CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
21:02:09:         CPUs: 8
21:02:09:       Memory: 15.95GiB
21:02:09:  Free Memory: 14.34GiB
21:02:09:      Threads: WINDOWS_THREADS
21:02:09:   OS Version: 6.2
21:02:09:  Has Battery: false
21:02:09:   On Battery: false
21:02:09:   UTC Offset: 11
21:02:09:          PID: 6944
21:02:09:          CWD: C:/ProgramData/FAHClient
21:02:09:           OS: Windows 10 Pro
21:02:09:      OS Arch: AMD64
21:02:09:         GPUs: 1
21:02:09:        GPU 0: NVIDIA:5 GP106 [GeForce GTX 1060]
21:02:09:         CUDA: 6.1
21:02:09:  CUDA Driver: 8000
21:02:09:Win32 Service: false
21:02:09:***********************************************************************
21:02:09:<config>
21:02:09:  <!-- Network -->
21:02:09:  <proxy v='10.132.72.41:8080'/>
21:02:09:  <proxy-enable v='true'/>
21:02:09:
21:02:09:  <!-- User Information -->
21:02:09:  <user v='CrissT'/>
21:02:09:
21:02:09:  <!-- Folding Slots -->
21:02:09:  <slot id='0' type='CPU'/>
21:02:09:  <slot id='1' type='GPU'/>
21:02:09:</config>
21:02:09:Trying to access database...
21:02:09:Successfully acquired database lock
21:02:09:Enabled folding slot 00: READY cpu:6
21:02:09:Enabled folding slot 01: READY gpu:0:GP106 [GeForce GTX 1060]
21:02:09:WU02:FS00:Starting
21:02:09:WU02:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 02 -suffix 01 -version 704 -lifeline 6944 -checkpoint 15 -np 6
21:02:09:WU02:FS00:Started FahCore on PID 1856
21:02:09:WU02:FS00:Core PID:4140
21:02:09:WU02:FS00:FahCore 0xa4 started
21:02:09:WU01:FS01:Starting
21:02:09:WU01:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/ProgramData/FAHClient/cores/web.stanford.edu/~pande/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 01 -suffix 01 -version 704 -lifeline 6944 -checkpoint 15 -gpu 0 -gpu-vendor nvidia
21:02:09:WU01:FS01:Started FahCore on PID 6844
21:02:09:WU01:FS01:Core PID:7036
21:02:09:WU01:FS01:FahCore 0x21 started
21:02:10:WU02:FS00:0xa4:
21:02:10:WU02:FS00:0xa4:*------------------------------*
21:02:10:WU02:FS00:0xa4:Folding@Home Gromacs GB Core
21:02:10:WU02:FS00:0xa4:Version 2.27 (Dec. 15, 2010)
21:02:10:WU02:FS00:0xa4:
21:02:10:WU02:FS00:0xa4:Preparing to commence simulation
21:02:10:WU02:FS00:0xa4:- Ensuring status. Please wait.
21:02:10:WU01:FS01:0x21:*********************** Log Started 2016-10-09T21:02:10Z ***********************
21:02:10:WU01:FS01:0x21:Project: 9195 (Run 2, Clone 35, Gen 38)
21:02:10:WU01:FS01:0x21:Unit: 0x00000030ab40415c57cb2f00314f4771
21:02:10:WU01:FS01:0x21:CPU: 0x00000000000000000000000000000000
21:02:10:WU01:FS01:0x21:Machine: 1
21:02:10:WU01:FS01:0x21:Digital signatures verified
21:02:10:WU01:FS01:0x21:Folding@home GPU Core21 Folding@home Core
21:02:10:WU01:FS01:0x21:Version 0.0.17
21:02:10:WU01:FS01:0x21:  Found a checkpoint file
21:02:15:WU01:FS01:0x21:Completed 500000 out of 2500000 steps (20%)
21:02:15:WU01:FS01:0x21:Temperature control disabled. Requirements: single Nvidia GPU, tmax must be < 110 and twait >= 900
21:02:19:WU02:FS00:0xa4:- Looking at optimizations...
21:02:19:WU02:FS00:0xa4:- Working with standard loops on this execution.
21:02:19:WU02:FS00:0xa4:- Previous termination of core was improper.
21:02:19:WU02:FS00:0xa4:- Files status OK
21:02:19:WU02:FS00:0xa4:- Expanded 825997 -> 1402860 (decompressed 169.8 percent)
21:02:19:WU02:FS00:0xa4:Called DecompressByteArray: compressed_data_size=825997 data_size=1402860, decompressed_data_size=1402860 diff=0
21:02:19:WU02:FS00:0xa4:- Digital signature verified
21:02:19:WU02:FS00:0xa4:
21:02:19:WU02:FS00:0xa4:Project: 9038 (Run 178, Clone 10, Gen 65)
21:02:19:WU02:FS00:0xa4:
21:02:19:WU02:FS00:0xa4:Entering M.D.
21:02:25:WU02:FS00:0xa4:Using Gromacs checkpoints
CrissT
 
Posts: 6
Joined: Mon Oct 17, 2016 12:33 am

Re: Problems with Visualization 7.4.4

Postby Joe_H » Mon Oct 17, 2016 2:34 am

Welcome to the folding support forum.

This is a known problem with the current client and viewer. The newer GPU cores do not provide the necessary information about the protein to FAHViewer for it to display it. So for now it can only display results from the CPU core.

Updates to fix this have been requested, and as I understand it are planned to be released at some time in the future. However I do not have further information as to when that will be.
Image

iMac 2.8 i7 12 GB smp8, Mac Pro 2.8 quad 12 GB smp6
MacBook Pro 2.9 i7 8 GB smp3
Joe_H
Site Admin
 
Posts: 6547
Joined: Tue Apr 21, 2009 5:41 pm
Location: W. MA

Re: Problems with Visualization 7.4.4

Postby ChristianVirtual » Mon Oct 17, 2016 2:39 am

For whatever reason currently all GPU cores don't deliver the required data via FAHControl to FAHViewer.

If you are open for some experimentation you can try a little python script I wrote last year:
https://github.com/ChristianVirtual/FAH ... Trajectory

It read the content from the working files and create the needed data stream towards the viewer.
Last edited by ChristianVirtual on Mon Oct 17, 2016 4:53 am, edited 1 time in total.
ImageImage
Please contribute your logs to http://ppd.fahmm.net
User avatar
ChristianVirtual
 
Posts: 1596
Joined: Tue May 28, 2013 1:14 pm
Location: Tokyo

Re: Problems with Visualization 7.4.4

Postby CrissT » Mon Oct 17, 2016 2:55 am

Thanks Joe for the technical details of this issue. I did a little searching on google about this problem and didn't find anything useful.

ChristianVirtual: I'll try your script, I just have to install python and see what else is required on the FAH PC. I'll come back with the outcome of this endeavor. :)
CrissT
 
Posts: 6
Joined: Mon Oct 17, 2016 12:33 am

Re: Problems with Visualization 7.4.4

Postby CrissT » Mon Oct 17, 2016 3:23 am

ChristianVirtual:

I've got the script and ran it. When trying to launch the viewer for the GPU i get:
Code: Select all
2016-10-17:01:15:35:new connection for ('10.132.72.103', 58286) established
2016-10-17:01:15:35:map {'01': '00', '00': '01'}
2016-10-17:01:15:37:get trajectory
2016-10-17:01:15:37:get trajectory for slot 00 with WU 01
2016-10-17:01:15:37:working folder C:/ProgramData/FAHClient\work\01\01
2016-10-17:01:15:37:no position yet known, send empty data


and nothing is shown :)
had a look at the path C:/ProgramData/FAHClient\work\01\01 and there are files in there.
In case it helps this is the log file for that slot:
Code: Select all
*********************** Log Started 2016-10-17T01:45:20Z ***********************
************************** Gromacs Folding@home Core ***************************
       Type: 0xa7
       Core: Gromacs
    Website: http://folding.stanford.edu/
  Copyright: (c) 2009-2016 Stanford University
     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
       Args: -dir 01 -suffix 01 -version 704 -lifeline 4196 -checkpoint 15 -np 6
     Config: <none>
************************************ Build *************************************
    Version: 0.0.11
       Date: Sep 21 2016
       Time: 01:43:48
 Repository: Git
   Revision: 957bd90e68d95ddcf1594dc15ff6c64cc4555146
     Branch: master
   Compiler: GNU 4.2.1 Compatible Clang 3.9.0 (trunk 274080)
    Options: -std=gnu++98 -O3 -funroll-loops -ffast-math -mfpmath=sse
             -fno-unsafe-math-optimizations -msse2 -I/mingw64/include
             -Wno-inconsistent-dllimport -Wno-parentheses-equality
             -Wno-deprecated-register -Wno-unused-local-typedef
   Platform: linux2 4.6.0-1-amd64
       Bits: 64
       Mode: Release
       SIMD: avx_256
************************************ System ************************************
        CPU: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
     CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
       CPUs: 8
     Memory: 15.95GiB
Free Memory: 13.16GiB
    Threads: WINDOWS_THREADS
 OS Version: 6.2
Has Battery: false
 On Battery: false
 UTC Offset: 11
        PID: 3096
        CWD: C:\ProgramData\FAHClient\work
         OS: Windows 10 Pro
    OS Arch: AMD64
********************************************************************************
Project: 11920 (Run 93, Clone 0, Gen 13)
Unit: 0x00000016ab40412957bff7401115e9ed
Reading tar file core.xml
Reading tar file frame13.tpr
Digital signatures verified
Calling: mdrun -s frame13.tpr -o frame13.trr -cpt 15 -nt 6
Steps: first=1040000 total=80000
Completed 1 out of 80000 steps (0%)
Completed 800 out of 80000 steps (1%)
Completed 1600 out of 80000 steps (2%)
Completed 2400 out of 80000 steps (3%)
Completed 3200 out of 80000 steps (4%)
CrissT
 
Posts: 6
Joined: Mon Oct 17, 2016 12:33 am

Re: Problems with Visualization 7.4.4

Postby ChristianVirtual » Mon Oct 17, 2016 4:24 am

I think you need to wait a bit until all files are written. Mainly viewerTop.json and viewerFrame<nn>.json

# viewerTop.json for the atoms
# viewerFrame1.json for the coordinates
# system.xml for the bonds

Can you list the files in the folder which follows the naming as above ? If I'm not wrong they might be written at every checkpoint, which for GPU projects is predefined during setup of projects without chance to overwrite from our side. Some WU write checkpoint only every 20% or 25% progress
Last edited by ChristianVirtual on Mon Oct 17, 2016 4:58 am, edited 1 time in total.
User avatar
ChristianVirtual
 
Posts: 1596
Joined: Tue May 28, 2013 1:14 pm
Location: Tokyo

Re: Problems with Visualization 7.4.4

Postby CrissT » Mon Oct 17, 2016 4:57 am

Hey Guys,

I think I found the problem with the GPU visualization.
After digging into Christian's python script I kind of figured that the visualization app is looking for some json/xml files.
It seems that the default location where those files are meant to be saved by the working-client in <FAH data folder>/work/<wu>/01/
Looking at the CPU working client, that is the case.
The problem with the GPU working client is that it saves the json files one folder out <FAH data folder>/work/<wu>
and the system.xml file doesn't exist at all

this is the content of C:\ProgramData\FAHClient\work\00:
Code: Select all
17/10/2016  02:58 PM    <DIR>          01
17/10/2016  02:28 PM             2,844 logfile_01-20161017-032852.txt
17/10/2016  02:58 PM             3,471 logfile_01.txt
17/10/2016  02:28 PM           522,525 viewerFrame0.json
17/10/2016  02:15 PM           522,599 viewerFrame1.json
17/10/2016  02:22 PM           522,599 viewerFrame10.json
17/10/2016  02:23 PM           522,599 viewerFrame11.json
17/10/2016  02:24 PM           522,599 viewerFrame12.json
17/10/2016  02:24 PM           522,599 viewerFrame13.json
17/10/2016  02:25 PM           522,599 viewerFrame14.json
17/10/2016  02:26 PM           522,599 viewerFrame15.json
17/10/2016  02:27 PM           522,599 viewerFrame16.json
17/10/2016  02:27 PM           522,599 viewerFrame17.json
17/10/2016  02:28 PM           522,525 viewerFrame18.json
17/10/2016  02:29 PM           522,525 viewerFrame19.json
17/10/2016  02:16 PM           522,599 viewerFrame2.json
17/10/2016  02:30 PM           522,525 viewerFrame20.json
17/10/2016  02:31 PM           522,525 viewerFrame21.json
17/10/2016  02:32 PM           522,525 viewerFrame22.json
17/10/2016  02:33 PM           522,525 viewerFrame23.json
17/10/2016  02:34 PM           522,525 viewerFrame24.json
17/10/2016  02:35 PM           522,629 viewerFrame25.json
17/10/2016  02:36 PM           522,629 viewerFrame26.json
17/10/2016  02:36 PM           522,629 viewerFrame27.json
17/10/2016  02:37 PM           522,629 viewerFrame28.json
17/10/2016  02:38 PM           522,629 viewerFrame29.json
17/10/2016  02:16 PM           522,599 viewerFrame3.json
17/10/2016  02:39 PM           522,629 viewerFrame30.json
17/10/2016  02:40 PM           522,629 viewerFrame31.json
17/10/2016  02:41 PM           522,629 viewerFrame32.json
17/10/2016  02:42 PM           522,629 viewerFrame33.json
17/10/2016  02:43 PM           522,629 viewerFrame34.json
17/10/2016  02:43 PM           522,629 viewerFrame35.json
17/10/2016  02:44 PM           522,580 viewerFrame36.json
17/10/2016  02:45 PM           522,580 viewerFrame37.json
17/10/2016  02:46 PM           522,580 viewerFrame38.json
17/10/2016  02:47 PM           522,580 viewerFrame39.json
17/10/2016  02:17 PM           522,599 viewerFrame4.json
17/10/2016  02:47 PM           522,580 viewerFrame40.json
17/10/2016  02:48 PM           522,580 viewerFrame41.json
17/10/2016  02:49 PM           522,580 viewerFrame42.json
17/10/2016  02:50 PM           522,580 viewerFrame43.json
17/10/2016  02:51 PM           522,580 viewerFrame44.json
17/10/2016  02:52 PM           522,580 viewerFrame45.json
17/10/2016  02:52 PM           522,580 viewerFrame46.json
17/10/2016  02:53 PM           522,580 viewerFrame47.json
17/10/2016  02:54 PM           522,580 viewerFrame48.json
17/10/2016  02:55 PM           522,580 viewerFrame49.json
17/10/2016  02:18 PM           522,599 viewerFrame5.json
17/10/2016  02:56 PM           522,539 viewerFrame50.json
17/10/2016  02:56 PM           522,539 viewerFrame51.json
17/10/2016  02:57 PM           522,539 viewerFrame52.json
17/10/2016  02:58 PM           522,539 viewerFrame53.json
17/10/2016  02:59 PM           522,488 viewerFrame54.json
17/10/2016  02:19 PM           522,599 viewerFrame6.json
17/10/2016  02:20 PM           522,599 viewerFrame7.json
17/10/2016  02:20 PM           522,599 viewerFrame8.json
17/10/2016  02:21 PM           522,599 viewerFrame9.json
17/10/2016  02:28 PM           898,331 viewerTop.json
17/10/2016  02:14 PM        21,982,720 wudata_01.dat
17/10/2016  02:28 PM                 0 wudata_01.lock
17/10/2016  02:59 PM               512 wuinfo_01.dat
              61 File(s)     51,629,944 bytes


and this is the content of C:\ProgramData\FAHClient\work\00\01:
Code: Select all
17/10/2016  02:58 PM    <DIR>          .
17/10/2016  02:58 PM    <DIR>          ..
17/10/2016  02:58 PM               840 checkpt.crc
17/10/2016  02:14 PM               119 core.xml
17/10/2016  02:28 PM             1,796 ener.edr
17/10/2016  02:14 PM        21,979,448 frame16.tpr
17/10/2016  02:28 PM         4,117,896 frame16.trr
17/10/2016  02:58 PM            31,469 md.log
17/10/2016  02:28 PM             6,381 science.log
17/10/2016  02:58 PM         4,119,864 state.cpt
17/10/2016  02:43 PM         4,119,864 state_prev.cpt
17/10/2016  02:58 PM         1,928,728 traj_comp.xtc
              10 File(s)     36,306,405 bytes
CrissT
 
Posts: 6
Joined: Mon Oct 17, 2016 12:33 am

Re: Problems with Visualization 7.4.4

Postby ChristianVirtual » Mon Oct 17, 2016 5:04 am

Not sure if the viewer read the files directly, to my understanding it use the API for 3rd party apps. You can find the source of FAHViewer also on GitHub.

One more major issue is also the that data formats for the GROMACS based CPU projects and OpenMM based GPU projects are different. It's not only file location; the content is also different.
User avatar
ChristianVirtual
 
Posts: 1596
Joined: Tue May 28, 2013 1:14 pm
Location: Tokyo

Re: Problems with Visualization 7.4.4

Postby ChristianVirtual » Mon Oct 17, 2016 5:08 am

Ok, I see now your edited posting ... interesting ... might need to tweak the script based on OS or assigned projects further :-(
User avatar
ChristianVirtual
 
Posts: 1596
Joined: Tue May 28, 2013 1:14 pm
Location: Tokyo

Re: Problems with Visualization 7.4.4

Postby CrissT » Mon Oct 17, 2016 5:20 am

I've added some debugging lines in your python script and it is indeed looking for the files in the wrong place. Didn't look in the code of FAHViewer, but i suspect all the working units are supposed to save their files in the /01 subfolder of the WU data folder. Furthermore the system.xml file is missing. Without that one, there is no way for FAHViewer to draw the structure. I reckon that the GPU plugin is not finished, or at least the part that generates the files for FAHViewer isn't finished. In the end is not a vital part of the project. As long as the simulated model is running to the end correctly, it's doing it's job. As Joe said earlier, the functionality isn't working for the newer cores and it was requested to be added in the future release.
CrissT
 
Posts: 6
Joined: Mon Oct 17, 2016 12:33 am

Re: Problems with Visualization 7.4.4

Postby ChristianVirtual » Mon Oct 17, 2016 5:53 am

Actually I just read once more your log file and saw it's a 11920; that's a CPU project ... and based on new core A7 it might have a different issue.

Please try once more with a GPU project.
User avatar
ChristianVirtual
 
Posts: 1596
Joined: Tue May 28, 2013 1:14 pm
Location: Tokyo

Re: Problems with Visualization 7.4.4

Postby CrissT » Mon Oct 17, 2016 6:31 am

Gotta go home now.
I'll try tomorrow first thing in the morning.
CrissT
 
Posts: 6
Joined: Mon Oct 17, 2016 12:33 am

Re: Problems with Visualization 7.4.4

Postby Porkey » Wed Jul 12, 2017 3:31 am

Sorry for reviving an old thread, but did we ever find a solution to this?

I've been trying to get this wrapper/viewer combo working for about an hour now, and I realized I had to add "localhost" or "127.0.0.1" to the hostnameWrapper line of the script source, but I haven't had much success.
It appears now the FAHViewer is working, but it is displaying a demo protein and the connection status continually says "Trying." Before adding something to the hostnameWrapper line, the viewer would stay blank, as it would normally.
I'm not too familiar with Python, but my guess is that the wrapper may be working now, but might not be able to find the working directory of any of my folding slots. I have tried editing the workingPath line, with the only results being either the script crashing, or no effect.
Porkey
 
Posts: 6
Joined: Wed Jul 12, 2017 3:21 am

Re: Problems with Visualization 7.4.4

Postby ChristianVirtual » Wed Jul 12, 2017 4:36 am

What project are you trying ... there are still projects out there not delivering full data (e.g. Stop after 19 atoms). Was not able to spend much time in the last half year on checking further but next week I have a few days off ... and hope
User avatar
ChristianVirtual
 
Posts: 1596
Joined: Tue May 28, 2013 1:14 pm
Location: Tokyo

Re: Problems with Visualization 7.4.4

Postby Porkey » Wed Jul 12, 2017 6:12 am

Currently I have one of two GPUs folding, running project 10490.
Porkey
 
Posts: 6
Joined: Wed Jul 12, 2017 3:21 am

Next

Return to V7.4.4 Public Release Windows/Linux/MacOS X (deprecated)

Who is online

Users browsing this forum: No registered users and 1 guest

cron