Folding Forum

Community driven support forum for Folding@home
https://foldingforum.org/

Error folding on GPU [OpenSUSE]

https://foldingforum.org/viewtopic.php?t=28751

Page 2 of 5

Re: Error folding on GPU

Posted: Tue Apr 19, 2016 8:10 pm
by DJViking
7im wrote:
DJViking wrote:...snip...

I have now also tried to remove the old slot and created a new gpu slot. Values for gpu core, opencl and cuda was set to -1. That didn't work. Still getting the problem.
Unless you know those new index values to be accurate, you will have made the troubleshooting much more difficult. The index settings are not the issue. Please return them to their default values.
I haven't changed them. The default value was -1.

Re: Error folding on GPU

Posted: Tue Apr 19, 2016 8:19 pm
by DJViking
These are the packages I have installed from Nvidia
nvidia-computeG04
nvidia-gfxG04-kmp-default
nvidia-glG04
x11-video-nvidiaG04

There is no xorg-x11-drv-nvidia-*.rpm

This is the log output from when I reinstalled the nvidia drivers:

Code: Select all

(1/4) Installerer: nvidia-gfxG04-kmp-default-361.42_k4.1.12_1-21.1.x86_64 ..........................................................................[fullført]
Flere rpm-resultater:
make: Entering directory '/usr/src/linux-4.1.20-11-obj/x86_64/default'
  Building modules, stage 2.
  MODPOST 0 modules
make: Leaving directory '/usr/src/linux-4.1.20-11-obj/x86_64/default'
/usr/src/kernel-modules/nvidia-361.42-default /
make "CC=cc" KBUILD_OUTPUT=/usr/src/linux-obj/x86_64/default KBUILD_VERBOSE= -C /usr/src/linux M=/usr/src/kernel-modules/nvidia-361.42-default ARCH=x86_64 NV_KERNEL_SOURCES=/usr/src/linux NV_KERNEL_OUTPUT=/usr/src/linux-obj/x86_64/default NV_KERNEL_MODULES="nvidia nvidia-uvm nvidia-modeset" INSTALL_MOD_DIR=kernel/drivers/video modules
make[1]: Entering directory '/usr/src/linux-4.1.20-11'
make[2]: Entering directory '/usr/src/linux-4.1.20-11-obj/x86_64/default'
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-frontend.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-instance.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-acpi.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-chrdev.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-cray.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-dma.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-drm.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-gvi.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-i2c.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-mempool.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-mmap.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-p2p.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-pat.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-procfs.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-usermap.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-vm.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-vtophys.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/os-interface.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/os-mlock.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/os-pci.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/os-registry.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/os-usermap.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-modeset-interface.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv_uvm_interface.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nvlink_linux.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nvlink_pci.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/ebridge_linux.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia/ibmnpu_linux.o
  LD [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_utils.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_common.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_linux.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_page_migration.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_page_migration_kepler.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_page_migration_maxwell.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_page_migration_pascal.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_channel_mgmt.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/nvstatus.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_perf.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_common_test.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_channel_directed_tests.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_channel_basic_sanity_tests.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_kernel_events.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_kernel_counters.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_debug_session.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_gpu_ops_tests.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_lite.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_page_cache.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_lite_api.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_lite_prefetch.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_lite_region_tracking.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_mmu_mgmt_pascal.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_mmu_mgmt.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_full.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_full_api.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_full_device_mgmt.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_full_test.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_full_fault_buffer.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_full_fault_buffer_pascal.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_full_ctx_mgmt.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_hashmap.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_full_pa_mgmt.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_full_unit_test.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_full_va_trie.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_full_pagetbl_mgmt.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/mmu_fmt.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/gmmu_fmt.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_full_fault_handler.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_full_identity_map.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm_full_perf.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_tools.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_global.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_gpu.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_procfs.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_va_space.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_gpu_semaphore.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_mem.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_rm_mem.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_channel.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_lock.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_hal.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_range_tree.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_range_allocator.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_va_range.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_va_block.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_range_group.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_gpu_page_fault.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_perf_events.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_perf_module.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_mmu.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_pte_batch.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_tlb_batch.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_push.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_pushbuffer.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_thread_context.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_tracker.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_kepler.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_kepler_ce.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_kepler_host.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_kepler_mmu.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_maxwell.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_maxwell_host.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_pascal.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_pascal_ce.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_pascal_host.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_pascal_mmu.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_pascal_fault_buffer.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_policy.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_perf_utils.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_kvmalloc.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_pmm_gpu.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_migrate.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_map_external.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_user_channel.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_test.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_test_rng.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_range_tree_test.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_range_allocator_test.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_gpu_semaphore_test.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_mem_test.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_rm_mem_test.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_page_tree_test.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_tracker_test.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_push_test.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_channel_test.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_ce_test.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_lock_test.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_perf_utils_test.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_kvmalloc_test.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_pmm_test.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_perf_events_test.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_perf_module_test.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_get_rm_ptes_test.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_fault_buffer_flush_test.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm/uvm8_mmu_test.o
  LD [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm.o
  CC [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-modeset/nvidia-modeset-linux.o
  LD [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-modeset.o
ld -r -o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-interface.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-frontend.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-instance.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-acpi.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-chrdev.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-cray.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-dma.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-drm.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-gvi.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-i2c.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-mempool.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-mmap.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-p2p.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-pat.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-procfs.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-usermap.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-vm.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-vtophys.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/os-interface.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/os-mlock.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/os-pci.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/os-registry.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/os-usermap.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv-modeset-interface.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nv_uvm_interface.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nvlink_linux.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/nvlink_pci.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/ebridge_linux.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia/ibmnpu_linux.o
ld -r -o /usr/src/kernel-modules/nvidia-361.42-default/nvidia-modeset/nv-modeset-interface.o /usr/src/kernel-modules/nvidia-361.42-default/nvidia-modeset/nvidia-modeset-linux.o
  Building modules, stage 2.
  MODPOST 3 modules
  CC      /usr/src/kernel-modules/nvidia-361.42-default/nvidia-modeset.mod.o
  LD [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-modeset.ko
  CC      /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm.mod.o
  LD [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia-uvm.ko
  CC      /usr/src/kernel-modules/nvidia-361.42-default/nvidia.mod.o
  LD [M]  /usr/src/kernel-modules/nvidia-361.42-default/nvidia.ko
make[2]: Leaving directory '/usr/src/linux-4.1.20-11-obj/x86_64/default'
make[1]: Leaving directory '/usr/src/linux-4.1.20-11'
ld -T /usr/src/linux/scripts/module-common.lds -r -o nv-linux.o \
  nvidia.mod.o nvidia/nv-interface.o
/

Modprobe blacklist files have been created at /etc/modprobe.d to prevent Nouveau from loading. This can be reverted by deleting /etc/modprobe.d/nvidia-*.conf.

*** Reboot your computer and verify that the NVIDIA graphics driver can be loaded. ***



(2/4) Installerer: nvidia-glG04-361.42-21.1.x86_64 .................................................................................................[fullført]
(3/4) Installerer: nvidia-computeG04-361.42-21.1.x86_64 ............................................................................................[fullført]
(4/4) Installerer: x11-video-nvidiaG04-361.42-21.1.x86_64 ..........................................................................................[fullført]
Flere rpm-resultater:
Using MD5DIR="/var/adm/SuSEconfig/md5"...
No changes for /etc/X11/xdm/Xservers
No changes for /etc/X11/xdm/xdm-config
/sbin/mkinitrd
Creating initrd: /boot/initrd-4.1.12-1-default
Executing: /usr/bin/dracut --logfile /var/log/YaST2/mkinitrd.log --force --force-drivers "pata_marvell ata_piix ata_generic" /boot/initrd-4.1.12-1-default 4.1.12-1-default
*** Including module: bash ***
*** Including module: warpclock ***
*** Including module: i18n ***
*** Including module: ifcfg ***
*** Including module: drm ***
*** Including module: plymouth ***
*** Including module: kernel-modules ***
Omitting driver i2o_scsi
*** Including module: resume ***
*** Including module: rootfs-block ***
*** Including module: terminfo ***
*** Including module: udev-rules ***
Skipping udev rule: 91-permissions.rules
Skipping udev rule: 80-drivers-modprobe.rules
*** Including module: haveged ***
*** Including module: systemd ***
*** Including module: usrmount ***
*** Including module: base ***
*** Including module: fs-lib ***
*** Including module: shutdown ***
*** Including module: suse ***
*** Including modules done ***
*** Installing kernel module dependencies and firmware ***
*** Installing kernel module dependencies and firmware done ***
*** Resolving executable dependencies ***
*** Resolving executable dependencies done***
*** Hardlinking files ***
*** Hardlinking files done ***
*** Stripping files ***
*** Stripping files done ***
*** Generating early-microcode cpio image ***
*** Constructing GenuineIntel.bin ****
*** Store current command line parameters ***
Stored kernel commandline:
rd.driver.pre=pata_marvell
rd.driver.pre=ata_piix
rd.driver.pre=ata_generic
 resume=UUID=0697d8b7-1e9f-4a2b-ae66-df5ba8bc7d32
root=UUID=8067d677-395e-4075-88c9-12f3822d0dce rootflags=rw,relatime,data=ordered rootfstype=ext4
*** Creating image file ***
*** Creating image file done ***
Some kernel modules could not be included
This is not necessarily an error:
swap
Creating initrd: /boot/initrd-4.1.13-5-default
Executing: /usr/bin/dracut --logfile /var/log/YaST2/mkinitrd.log --force --force-drivers "pata_marvell ata_piix ata_generic" /boot/initrd-4.1.13-5-default 4.1.13-5-default
*** Including module: bash ***
*** Including module: warpclock ***
*** Including module: i18n ***
*** Including module: ifcfg ***
*** Including module: drm ***
*** Including module: plymouth ***
*** Including module: kernel-modules ***
Omitting driver i2o_scsi
*** Including module: resume ***
*** Including module: rootfs-block ***
*** Including module: terminfo ***
*** Including module: udev-rules ***
Skipping udev rule: 91-permissions.rules
Skipping udev rule: 80-drivers-modprobe.rules
*** Including module: haveged ***
*** Including module: systemd ***
*** Including module: usrmount ***
*** Including module: base ***
*** Including module: fs-lib ***
*** Including module: shutdown ***
*** Including module: suse ***
*** Including modules done ***
*** Installing kernel module dependencies and firmware ***
*** Installing kernel module dependencies and firmware done ***
*** Resolving executable dependencies ***
*** Resolving executable dependencies done***
*** Hardlinking files ***
*** Hardlinking files done ***
*** Stripping files ***
*** Stripping files done ***
*** Generating early-microcode cpio image ***
*** Constructing GenuineIntel.bin ****
*** Store current command line parameters ***
Stored kernel commandline:
rd.driver.pre=pata_marvell
rd.driver.pre=ata_piix
rd.driver.pre=ata_generic
 resume=UUID=0697d8b7-1e9f-4a2b-ae66-df5ba8bc7d32
root=UUID=8067d677-395e-4075-88c9-12f3822d0dce rootflags=rw,relatime,data=ordered rootfstype=ext4
*** Creating image file ***
*** Creating image file done ***
Some kernel modules could not be included
This is not necessarily an error:
swap
Creating initrd: /boot/initrd-4.1.15-8-default
Executing: /usr/bin/dracut --logfile /var/log/YaST2/mkinitrd.log --force --force-drivers "pata_marvell ata_piix ata_generic" /boot/initrd-4.1.15-8-default 4.1.15-8-default
*** Including module: bash ***
*** Including module: warpclock ***
*** Including module: i18n ***
*** Including module: ifcfg ***
*** Including module: drm ***
*** Including module: plymouth ***
*** Including module: kernel-modules ***
Omitting driver i2o_scsi
*** Including module: resume ***
*** Including module: rootfs-block ***
*** Including module: terminfo ***
*** Including module: udev-rules ***
Skipping udev rule: 91-permissions.rules
Skipping udev rule: 80-drivers-modprobe.rules
*** Including module: haveged ***
*** Including module: systemd ***
*** Including module: usrmount ***
*** Including module: base ***
*** Including module: fs-lib ***
*** Including module: shutdown ***
*** Including module: suse ***
*** Including modules done ***
*** Installing kernel module dependencies and firmware ***
*** Installing kernel module dependencies and firmware done ***
*** Resolving executable dependencies ***
*** Resolving executable dependencies done***
*** Hardlinking files ***
*** Hardlinking files done ***
*** Stripping files ***
*** Stripping files done ***
*** Generating early-microcode cpio image ***
*** Constructing GenuineIntel.bin ****
*** Store current command line parameters ***
Stored kernel commandline:
rd.driver.pre=pata_marvell
rd.driver.pre=ata_piix
rd.driver.pre=ata_generic
 resume=UUID=0697d8b7-1e9f-4a2b-ae66-df5ba8bc7d32
root=UUID=8067d677-395e-4075-88c9-12f3822d0dce rootflags=rw,relatime,data=ordered rootfstype=ext4
*** Creating image file ***
*** Creating image file done ***
Some kernel modules could not be included
This is not necessarily an error:
swap
Creating initrd: /boot/initrd-4.1.20-11-default
Executing: /usr/bin/dracut --logfile /var/log/YaST2/mkinitrd.log --force --force-drivers "pata_marvell ata_piix ata_generic" /boot/initrd-4.1.20-11-default 4.1.20-11-default
*** Including module: bash ***
*** Including module: warpclock ***
*** Including module: i18n ***
*** Including module: ifcfg ***
*** Including module: drm ***
*** Including module: plymouth ***
*** Including module: kernel-modules ***
Omitting driver i2o_scsi
*** Including module: resume ***
*** Including module: rootfs-block ***
*** Including module: terminfo ***
*** Including module: udev-rules ***
Skipping udev rule: 91-permissions.rules
Skipping udev rule: 80-drivers-modprobe.rules
*** Including module: haveged ***
*** Including module: systemd ***
*** Including module: usrmount ***
*** Including module: base ***
*** Including module: fs-lib ***
*** Including module: shutdown ***
*** Including module: suse ***
*** Including modules done ***
*** Installing kernel module dependencies and firmware ***
*** Installing kernel module dependencies and firmware done ***
*** Resolving executable dependencies ***
*** Resolving executable dependencies done***
*** Hardlinking files ***
*** Hardlinking files done ***
*** Stripping files ***
*** Stripping files done ***
*** Generating early-microcode cpio image ***
*** Constructing GenuineIntel.bin ****
*** Store current command line parameters ***
Stored kernel commandline:
rd.driver.pre=pata_marvell
rd.driver.pre=ata_piix
rd.driver.pre=ata_generic
 resume=UUID=0697d8b7-1e9f-4a2b-ae66-df5ba8bc7d32
root=UUID=8067d677-395e-4075-88c9-12f3822d0dce rootflags=rw,relatime,data=ordered rootfstype=ext4
*** Creating image file ***
*** Creating image file done ***
Some kernel modules could not be included
This is not necessarily an error:
swap
Update bootloader...


Output of nvidia-gfxG04-kmp-default-361.42_k4.1.12_1-21.1.x86_64.rpm %posttrans script:
    Creating initrd: /boot/initrd-4.1.12-1-default
    Executing: /usr/bin/dracut --logfile /var/log/YaST2/mkinitrd.log --force --force-drivers "pata_marvell ata_piix ata_generic" /boot/initrd-4.1.12-1-default 4.1.12-1-default
    *** Including module: bash ***
    *** Including module: warpclock ***
    *** Including module: i18n ***
    *** Including module: ifcfg ***
    *** Including module: drm ***
    *** Including module: plymouth ***
    *** Including module: kernel-modules ***
    Omitting driver i2o_scsi
    *** Including module: resume ***
    *** Including module: rootfs-block ***
    *** Including module: terminfo ***
    *** Including module: udev-rules ***
    Skipping udev rule: 91-permissions.rules
    Skipping udev rule: 80-drivers-modprobe.rules
    *** Including module: haveged ***
    *** Including module: systemd ***
    *** Including module: usrmount ***
    *** Including module: base ***
    *** Including module: fs-lib ***
    *** Including module: shutdown ***
    *** Including module: suse ***
    *** Including modules done ***
    *** Installing kernel module dependencies and firmware ***
    *** Installing kernel module dependencies and firmware done ***
    *** Resolving executable dependencies ***
    *** Resolving executable dependencies done***
    *** Hardlinking files ***
    *** Hardlinking files done ***
    *** Stripping files ***
    *** Stripping files done ***
    *** Generating early-microcode cpio image ***
    *** Constructing GenuineIntel.bin ****
    *** Store current command line parameters ***
    Stored kernel commandline:
    rd.driver.pre=pata_marvell
    rd.driver.pre=ata_piix
    rd.driver.pre=ata_generic
     resume=UUID=0697d8b7-1e9f-4a2b-ae66-df5ba8bc7d32
    root=UUID=8067d677-395e-4075-88c9-12f3822d0dce rootflags=rw,relatime,data=ordered rootfstype=ext4
    *** Creating image file ***
    *** Creating image file done ***
    Some kernel modules could not be included
    This is not necessarily an error:
    swap
    Update bootloader...
    Creating initrd: /boot/initrd-4.1.13-5-default
    Executing: /usr/bin/dracut --logfile /var/log/YaST2/mkinitrd.log --force --force-drivers "pata_marvell ata_piix ata_generic" /boot/initrd-4.1.13-5-default 4.1.13-5-default
    *** Including module: bash ***
    *** Including module: warpclock ***
    *** Including module: i18n ***
    *** Including module: ifcfg ***
    *** Including module: drm ***
    *** Including module: plymouth ***
    *** Including module: kernel-modules ***
    Omitting driver i2o_scsi
    *** Including module: resume ***
    *** Including module: rootfs-block ***
    *** Including module: terminfo ***
    *** Including module: udev-rules ***
    Skipping udev rule: 91-permissions.rules
    Skipping udev rule: 80-drivers-modprobe.rules
    *** Including module: haveged ***
    *** Including module: systemd ***
    *** Including module: usrmount ***
    *** Including module: base ***
    *** Including module: fs-lib ***
    *** Including module: shutdown ***
    *** Including module: suse ***
    *** Including modules done ***
    *** Installing kernel module dependencies and firmware ***
    *** Installing kernel module dependencies and firmware done ***
    *** Resolving executable dependencies ***
    *** Resolving executable dependencies done***
    *** Hardlinking files ***
    *** Hardlinking files done ***
    *** Stripping files ***
    *** Stripping files done ***
    *** Generating early-microcode cpio image ***
    *** Constructing GenuineIntel.bin ****
    *** Store current command line parameters ***
    Stored kernel commandline:
    rd.driver.pre=pata_marvell
    rd.driver.pre=ata_piix
    rd.driver.pre=ata_generic
     resume=UUID=0697d8b7-1e9f-4a2b-ae66-df5ba8bc7d32
    root=UUID=8067d677-395e-4075-88c9-12f3822d0dce rootflags=rw,relatime,data=ordered rootfstype=ext4
    *** Creating image file ***
    *** Creating image file done ***
    Some kernel modules could not be included
    This is not necessarily an error:
    swap
    Update bootloader...
    Creating initrd: /boot/initrd-4.1.15-8-default
    Executing: /usr/bin/dracut --logfile /var/log/YaST2/mkinitrd.log --force --force-drivers "pata_marvell ata_piix ata_generic" /boot/initrd-4.1.15-8-default 4.1.15-8-default
    *** Including module: bash ***
    *** Including module: warpclock ***
    *** Including module: i18n ***
    *** Including module: ifcfg ***
    *** Including module: drm ***
    *** Including module: plymouth ***
    *** Including module: kernel-modules ***
    Omitting driver i2o_scsi
    *** Including module: resume ***
    *** Including module: rootfs-block ***
    *** Including module: terminfo ***
    *** Including module: udev-rules ***
    Skipping udev rule: 91-permissions.rules
    Skipping udev rule: 80-drivers-modprobe.rules
    *** Including module: haveged ***
    *** Including module: systemd ***
    *** Including module: usrmount ***
    *** Including module: base ***
    *** Including module: fs-lib ***
    *** Including module: shutdown ***
    *** Including module: suse ***
    *** Including modules done ***
    *** Installing kernel module dependencies and firmware ***
    *** Installing kernel module dependencies and firmware done ***
    *** Resolving executable dependencies ***
    *** Resolving executable dependencies done***
    *** Hardlinking files ***
    *** Hardlinking files done ***
    *** Stripping files ***
    *** Stripping files done ***
    *** Generating early-microcode cpio image ***
    *** Constructing GenuineIntel.bin ****
    *** Store current command line parameters ***
    Stored kernel commandline:
    rd.driver.pre=pata_marvell
    rd.driver.pre=ata_piix
    rd.driver.pre=ata_generic
     resume=UUID=0697d8b7-1e9f-4a2b-ae66-df5ba8bc7d32
    root=UUID=8067d677-395e-4075-88c9-12f3822d0dce rootflags=rw,relatime,data=ordered rootfstype=ext4
    *** Creating image file ***
    *** Creating image file done ***
    Some kernel modules could not be included
    This is not necessarily an error:
    swap
    Update bootloader...
    Creating initrd: /boot/initrd-4.1.20-11-default
    Executing: /usr/bin/dracut --logfile /var/log/YaST2/mkinitrd.log --force --force-drivers "pata_marvell ata_piix ata_generic" /boot/initrd-4.1.20-11-default 4.1.20-11-default
    *** Including module: bash ***
    *** Including module: warpclock ***
    *** Including module: i18n ***
    *** Including module: ifcfg ***
    *** Including module: drm ***
    *** Including module: plymouth ***
    *** Including module: kernel-modules ***
    Omitting driver i2o_scsi
    *** Including module: resume ***
    *** Including module: rootfs-block ***
    *** Including module: terminfo ***
    *** Including module: udev-rules ***
    Skipping udev rule: 91-permissions.rules
    Skipping udev rule: 80-drivers-modprobe.rules
    *** Including module: haveged ***
    *** Including module: systemd ***
    *** Including module: usrmount ***
    *** Including module: base ***
    *** Including module: fs-lib ***
    *** Including module: shutdown ***
    *** Including module: suse ***
    *** Including modules done ***
    *** Installing kernel module dependencies and firmware ***
    *** Installing kernel module dependencies and firmware done ***
    *** Resolving executable dependencies ***
    *** Resolving executable dependencies done***
    *** Hardlinking files ***
    *** Hardlinking files done ***
    *** Stripping files ***
    *** Stripping files done ***
    *** Generating early-microcode cpio image ***
    *** Constructing GenuineIntel.bin ****
    *** Store current command line parameters ***
    Stored kernel commandline:
    rd.driver.pre=pata_marvell
    rd.driver.pre=ata_piix
    rd.driver.pre=ata_generic
     resume=UUID=0697d8b7-1e9f-4a2b-ae66-df5ba8bc7d32
    root=UUID=8067d677-395e-4075-88c9-12f3822d0dce rootflags=rw,relatime,data=ordered rootfstype=ext4
    *** Creating image file ***
    *** Creating image file done ***
    Some kernel modules could not be included
    This is not necessarily an error:
    swap
    Update bootloader...

Re: Error folding on GPU

Posted: Tue Apr 19, 2016 8:50 pm
by jimerickson
in my experience with nvidia gpus you need an external opencl implentation. nvidia opencl is deficient. ocl-icd fills that role. though there may be some quirk in opesuse that i do not currently understand.

Re: Error folding on GPU

Posted: Tue Apr 19, 2016 9:10 pm
by toTOW
If you don't install the OpenCL package provided with the NV drivers, FAH will never work ... (same applies to ATI GPUs and drivers, and to Windows OS ... sorry to being unable to help more, I don't know how to use a GPU on Linux).

Re: Error folding on GPU

Posted: Tue Apr 19, 2016 11:42 pm
by jimerickson
i urge you to give ocl-icd another try. now that you have rebooted and the driver is active it should do the trick.

Re: Error folding on GPU

Posted: Wed Apr 20, 2016 9:32 am
by DJViking
I just tried again and installed ocl-icd (libOpenCL1), and it didn't solve the problem. I still get the same error.
Nvidia drivers has their own package with OpenCL nvidia-computeG04. Whether I use that or libOpenCL1 it still fails.

Package nvidia-computeG04 can't be removed as it would also remove x11-video-nvidiaG04

Code: Select all

The following 2 packages are going to be REMOVED:
  nvidia-computeG04 x11-video-nvidiaG04
The OpenCL library is installed on the system with nvidia-computeG04.

Code: Select all

djviking@mintaka:~> ll /usr/lib64/libOpenCL*    
lrwxrwxrwx 1 root root    14 20.04.2016 11:44:39 /usr/lib64/libOpenCL.so -> libOpenCL.so.1
lrwxrwxrwx 1 root root    18 20.04.2016 11:44:39 /usr/lib64/libOpenCL.so.1 -> libOpenCL.so.1.0.0
-rwxr-xr-x 1 root root 26328 06.04.2016 14:59:49 /usr/lib64/libOpenCL.so.1.0.0

Re: Error folding on GPU

Posted: Wed Apr 20, 2016 10:24 am
by DJViking
I removed/uninstalled the nvidia drivers completely, then reinstalled them, and performed a reboot. I made another go at the folding, but got the same error.

Re: Error folding on GPU

Posted: Wed Apr 20, 2016 11:09 am
by DJViking
I read somewhere that Fermi based graphic cards was no longer supported. Though I am using a Kepler base graphic card I thought this looked like a problem
10:22:41:WU00:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/web.stanford.edu/~pande/Linux/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21 -dir 00 -suffix 01 -version 704 -lifeline 3129 -checkpoint 15 -gpu 0 -gpu-vendor nvidia

Re: Error folding on GPU

Posted: Wed Apr 20, 2016 1:06 pm
by DJViking
There must be someone of the millions of folders out there who use OpenSUSE and Nvidia GPU.
Can't find any client statistics for Linux based on distributions: http://fah-web.stanford.edu/cgi-bin/mai ... e=osstats2

Re: Error folding on GPU

Posted: Wed Apr 20, 2016 2:08 pm
by DJViking
I found several post from people who where having the same problem.
viewtopic.php?f=96&t=27321

viewtopic.php?f=80&t=28144

viewtopic.php?f=80&t=26981

viewtopic.php?f=89&t=26639

viewtopic.php?f=61&t=26696

Found one thread though with OpenSUSE that looked like it has good answers. That the fahclient user could not run on GPU because it lacked video in group.
OpenSUSE 13.1
viewtopic.php?f=80&t=26417
viewtopic.php?f=80&t=26417&p=265631&hilit=Bad+platformId+size#p265482

Re: Error folding on GPU

Posted: Wed Apr 20, 2016 2:18 pm
by Joe_H
There is also this topic - viewtopic.php?f=80&t=27597&hilit=suse. There are some suggestions from poster everyman that might help.

Also, could you post the first 100 or so lines of your log file. That will show the system information and folding configuration, and can also give us an idea of what the client does detect.

Re: Error folding on GPU

Posted: Wed Apr 20, 2016 2:58 pm
by DJViking

Code: Select all

*********************** Log Started 2016-04-20T13:18:33Z ***********************
13:18:33:************************* Folding@home Client *************************
13:18:33:    Website: http://folding.stanford.edu/
13:18:33:  Copyright: (c) 2009-2014 Stanford University
13:18:33:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
13:18:33:       Args: --child --lifeline 12869 /etc/fahclient/config.xml --run-as
13:18:33:             fahclient --pid-file=/var/run/fahclient.pid --daemon
13:18:33:     Config: /etc/fahclient/config.xml
13:18:33:******************************** Build ********************************
13:18:33:    Version: 7.4.4
13:18:33:       Date: Mar 4 2014
13:18:33:       Time: 12:01:17
13:18:33:    SVN Rev: 4130
13:18:33:     Branch: fah/trunk/client
13:18:33:   Compiler: GNU 4.1.2 20080704 (Red Hat 4.1.2-46)
13:18:33:    Options: -std=gnu++98 -O3 -funroll-loops -mfpmath=sse -ffast-math
13:18:33:             -fno-unsafe-math-optimizations -msse2
13:18:33:   Platform: linux2 2.6.18-164.11.1.el5
13:18:33:       Bits: 64
13:18:33:       Mode: Release
13:18:33:******************************* System ********************************
13:18:33:        CPU: Intel(R) Core(TM)2 Duo CPU E7400 @ 2.80GHz
13:18:33:     CPU ID: GenuineIntel Family 6 Model 23 Stepping 10
13:18:33:       CPUs: 2
13:18:33:     Memory: 7.80GiB
13:18:33:Free Memory: 104.85MiB
13:18:33:    Threads: POSIX_THREADS
13:18:33: OS Version: 4.1
13:18:33:Has Battery: false
13:18:33: On Battery: false
13:18:33: UTC Offset: 2
13:18:33:        PID: 12871
13:18:33:        CWD: /var/lib/fahclient
13:18:33:         OS: Linux 4.1.20-11-default x86_64
13:18:33:    OS Arch: AMD64
13:18:33:       GPUs: 1
13:18:33:      GPU 0: NVIDIA:3 GK106 [GeForce GTX 650 Ti]
13:18:33:       CUDA: Not detected
13:18:33:***********************************************************************

Re: Error folding on GPU

Posted: Wed Apr 20, 2016 3:19 pm
by Joe_H
The log shows that the client did not detect CUDA being present. As mentioned in the thread I linked, that should also be seen. You may need to install the CUDA support, nVidia has its OpenCL support connected to CUDA as well. So even though the folding core does not use it, CUDA appears to be a necessary part of the install to fold on nVidia GPU's.

P.S. A PM reminded me that there is still one GPU core that uses CUDA, and it might be used again in the future. However that Core_15 only runs on Windows and has been announced to be end-of-life. The few projects that still use it are approaching end of life as well.

Re: Error folding on GPU

Posted: Wed Apr 20, 2016 3:39 pm
by DJViking
I have installed support for both CUDA and OpenCL along with the nvidia drivers.

I found the problem. The user fahclient running the FAHClient does not have access to the GPU. I added the user fahclient to group video in /etc/group, but that did not work (perhaps I would need an restart for /etc/group to take affect).
When I started FAHClient as my own user I had no problem folding with the GPU.

Output from the log when running FAHClient with my own user.

Code: Select all

*********************** Log Started 2016-04-20T16:25:57Z ***********************
16:25:57:************************* Folding@home Client *************************
16:25:57:    Website: http://folding.stanford.edu/
16:25:57:  Copyright: (c) 2009-2014 Stanford University
16:25:57:     Author: Joseph Coffland <joseph@cauldrondevelopment.com>
16:25:57:       Args: --child --lifeline 28159 --run-as djviking
16:25:57:             --pid-file=/var/run/djviking.pid --daemon
16:25:57:     Config: /var/lib/djviking/config.xml
16:25:57:******************************** Build ********************************
16:25:57:    Version: 7.4.4
16:25:57:       Date: Mar 4 2014
16:25:57:       Time: 12:01:17
16:25:57:    SVN Rev: 4130
16:25:57:     Branch: fah/trunk/client
16:25:57:   Compiler: GNU 4.1.2 20080704 (Red Hat 4.1.2-46)
16:25:57:    Options: -std=gnu++98 -O3 -funroll-loops -mfpmath=sse -ffast-math
16:25:57:             -fno-unsafe-math-optimizations -msse2
16:25:57:   Platform: linux2 2.6.18-164.11.1.el5
16:25:57:       Bits: 64
16:25:57:       Mode: Release
16:25:57:******************************* System ********************************
16:25:57:        CPU: Intel(R) Core(TM)2 Duo CPU E7400 @ 2.80GHz
16:25:57:     CPU ID: GenuineIntel Family 6 Model 23 Stepping 10
16:25:57:       CPUs: 2
16:25:57:     Memory: 7.80GiB
16:25:57:Free Memory: 446.27MiB
16:25:57:    Threads: POSIX_THREADS
16:25:57: OS Version: 4.1
16:25:57:Has Battery: false
16:25:57: On Battery: false
16:25:57: UTC Offset: 2
16:25:57:        PID: 28161
16:25:57:        CWD: /var/lib/djviking
16:25:57:         OS: Linux 4.1.20-11-default x86_64
16:25:57:    OS Arch: AMD64
16:25:57:       GPUs: 1
16:25:57:      GPU 0: NVIDIA:3 GK106 [GeForce GTX 650 Ti]
16:25:57:       CUDA: 3.0
16:25:57:CUDA Driver: 8000
16:25:57:***********************************************************************

Re: Error folding on GPU

Posted: Wed Apr 20, 2016 4:53 pm
by jimerickson
glad its working!
All times are UTC
Page 2 of 5

Powered by phpBB® Forum Software © phpBB Limited