Upload to 66.170.111.50:8080 failing repeatedly

Moderators: Site Moderators, FAHC Science Team

jrsynow
Posts: 7
Joined: Thu Dec 22, 2022 3:28 am

Re: Upload to 66.170.111.50:8080 failing repeatedly

Post by jrsynow »

jcoffland wrote: Mon Jun 05, 2023 1:24 pm ... The first major question is, does the problem lie in a) the WS software, b) the Linux kernel or c) the VMWare network.
...
I noticed the same issues with the Temple University servers back in November of last year(viewtopic.php?t=38869, Discord support channel 11/10/2022), so that could rule out point C. Up until ~Nov. 2022 uploads were always rock solid on my end.
Lazvon
Posts: 105
Joined: Wed Jan 05, 2022 1:06 am
Hardware configuration: 4080 / 12700F, 3090Ti/12900KS, 3090/12900K, 3090/10940X, 3080Ti/12700K, 3080Ti/9900X, 3080Ti/9900X

Re: Upload to 66.170.111.50:8080 failing repeatedly

Post by Lazvon »

I am happy to take a look. But sounds like server side capture is same as client side. Drop a PM on where to grab it.

Things I would look at:
  • VMWare: are there 2 (or more) virtual network cards listed in the virtual machine itself? If so, can it plumbed for a single NIC as far as Linux is concerned?
  • VMWare: what packet stats does vCenter or ESXi directly show for the virtual and physical NICs?
  • Linux: I assume you have already looked at the Linux interface stats (netstat -i (and -s, and -s -t), ethtool -S, ip -s link show (never used this myself)), to validate you aren’t dropping in Linux itself?
  • Linux: does sar or iostat show any IOPS to the disk problems? VMWare view of IO? (And CPU and Mem too?)
  • Switch: any drops/overruns/discard counters increasing for the VMWare host?
  • Switch/Router/Firewall/ISP: assume none of those components are set for per packet load balancing, and they are set for per flow, right?
Folding since Feb 2021. 1) 4090/12900KS, 2) 4080/12700F, 3) 4070Ti/9900X, 4) 3090/12900K, 5) 3090/10940X, 6) 3080Ti/12700K, 7) 3080Ti/9900X

Image
BobWilliams757
Posts: 491
Joined: Fri Apr 03, 2020 2:22 pm
Hardware configuration: ASRock X370M PRO4
Ryzen 2400G APU
16 GB DDR4-3200
MSI GTX 1660 Super Gaming X

Re: Upload to 66.170.111.50:8080 failing repeatedly

Post by BobWilliams757 »

I know nothing at all about the proper ways to troubleshoot such things, but would gladly help comb through data if anyone can break it down to what I need to look for within that data. :oops:

In my case, it has been inconsistent and occasional slowdowns, and they often have "bursts" of decent file transfer followed by periods of very slow transfer.

I'll refrain from posting any logs unless it shows a longer slowdown that is having a negative impact on the returns. I haven't had delays any longer than a few minutes total lately, and no more issues of the one hour until it throws an error and switches to a new server.
Fold them if you get them!
Lazvon
Posts: 105
Joined: Wed Jan 05, 2022 1:06 am
Hardware configuration: 4080 / 12700F, 3090Ti/12900KS, 3090/12900K, 3090/10940X, 3080Ti/12700K, 3080Ti/9900X, 3080Ti/9900X

Re: Upload to 66.170.111.50:8080 failing repeatedly

Post by Lazvon »

Is this on VMWare Cloud? Is it a donated VM?

I spend about $60m a year with VMWare. Couple of the folks who work me are on their CAB. Generally a good partner for me and they like to keep me happy so I don’t go somewhere else. I can probably get one of their better engineers responsible for their cloud instances to help me troubleshoot from ESXi to the physical host to the network to the Internet of so.

Their ISP’s (NTT) routers are certainly busy (slow ICMP TTL Expired messages while tracerouting) assuming this is in Seattle and doesn’t go somewhere else. So could totally just be an ISP issue.

Again, happy to take a look at the PCAP. It is something I have been doing since the mid-90s (well, not PCAP, was Network General Sniffers back then). Not sure it will show anything different than I have seen on client side.

Essentially, either there are 2+ links and per packet load balancing is turned on so a longer path and shorter path get used, and thus Out of Order packets. Or, there is severe caching along the path with congestion and buffers aren’t being served in order for the OOO issues, and/or some kind of random early discard dropping packets on the floor to try and sliver congestion, but if congestion is too bad… this is what we get.
Folding since Feb 2021. 1) 4090/12900KS, 2) 4080/12700F, 3) 4070Ti/9900X, 4) 3090/12900K, 5) 3090/10940X, 6) 3080Ti/12700K, 7) 3080Ti/9900X

Image
Lazvon
Posts: 105
Joined: Wed Jan 05, 2022 1:06 am
Hardware configuration: 4080 / 12700F, 3090Ti/12900KS, 3090/12900K, 3090/10940X, 3080Ti/12700K, 3080Ti/9900X, 3080Ti/9900X

Re: Upload to 66.170.111.50:8080 failing repeatedly

Post by Lazvon »

Was able to do 50M PPD yesterday. Past week has been 40-45M, from what I can tell just because of the constant transfer rate problems with this server. Hopefully things are improving and I get back to 53-55M again.
Folding since Feb 2021. 1) 4090/12900KS, 2) 4080/12700F, 3) 4070Ti/9900X, 4) 3090/12900K, 5) 3090/10940X, 6) 3080Ti/12700K, 7) 3080Ti/9900X

Image
Lazvon
Posts: 105
Joined: Wed Jan 05, 2022 1:06 am
Hardware configuration: 4080 / 12700F, 3090Ti/12900KS, 3090/12900K, 3090/10940X, 3080Ti/12700K, 3080Ti/9900X, 3080Ti/9900X

Re: Upload to 66.170.111.50:8080 failing repeatedly

Post by Lazvon »

So, slow transfers still very much going on. This capture was started on June 2nd at 7:23pm EDT. I mention the time, in case you want to figure out the "Relative Start Time" which is seconds for any of these connections over the last 6-days. I only show about 70 of the connections out of the "134" a fair number of the 134 were 0-bytes though since I had the FW blocking for a while. I'm gonna leave it open, but certainly slower than all the other servers which are instant (or nearly instant) download/uploads versus 10-30 minutes.

There are a few fast downloads in there, but don't think there are ever any fast uploads.

Image
Folding since Feb 2021. 1) 4090/12900KS, 2) 4080/12700F, 3) 4070Ti/9900X, 4) 3090/12900K, 5) 3090/10940X, 6) 3080Ti/12700K, 7) 3080Ti/9900X

Image
Lazvon
Posts: 105
Joined: Wed Jan 05, 2022 1:06 am
Hardware configuration: 4080 / 12700F, 3090Ti/12900KS, 3090/12900K, 3090/10940X, 3080Ti/12700K, 3080Ti/9900X, 3080Ti/9900X

Re: Upload to 66.170.111.50:8080 failing repeatedly

Post by Lazvon »

3 out of 7 folders all downloading during this particular 10-minute poll...

Image
Folding since Feb 2021. 1) 4090/12900KS, 2) 4080/12700F, 3) 4070Ti/9900X, 4) 3090/12900K, 5) 3090/10940X, 6) 3080Ti/12700K, 7) 3080Ti/9900X

Image
toTOW
Site Moderator
Posts: 6293
Joined: Sun Dec 02, 2007 10:38 am
Location: Bordeaux, France
Contact:

Re: Upload to 66.170.111.50:8080 failing repeatedly

Post by toTOW »

Lazvon wrote: Thu Jun 08, 2023 12:10 am Is this on VMWare Cloud? Is it a donated VM?
Yes, it's a donated server.
Image

Folding@Home beta tester since 2002. Folding Forum moderator since July 2008.
FrancoV78
Posts: 42
Joined: Sat May 04, 2019 12:35 pm

Re: Upload to 66.170.111.50:8080 failing repeatedly

Post by FrancoV78 »

I also opened a topic, I generally have problems with uploading these days.
Post Reply