TerraSync issues

10 Jul 2018 08:35 #39276 by enrogue
Just thought I'd start a new thread:

Problem:

Massive framerate slowdown when Terrasync is enabled for some people

Possible cause:

TerraSync when enabled in FGFS appears to spend a large amount of time & CPU syncing the Airports subdirectory at every startup which causes the frame rate & general responsiveness of the sim to suffer until it's finished - on systems with low spec CPU/storage this is a pain in the arse, as they are hit even if they have a decent graphics card.

There are changes in the works for how TerraSync is structured & stored, so these issues may go away but my aim here is to show the results of some tests & either find a workaround or suggest some changes to the developers
The following user(s) said Thank You: Algernon

Please Log in or Create an account to join the conversation.

10 Jul 2018 08:58 #39278 by enrogue
Test Setup:

Host machine: Core i5-3320M 16GB ram, 500G MX300 SSD, Ubuntu 18.04

Windows VM: VMware Player, 2 vCPU, 6GB ram, Windows 10 1803, Windows Defender AV, FlightGear 2018.2.2 standard download, FG is set to run in a window at 1024x600, multithreading is set to AutomaticSelection, terrasync when used is set to my local (home) mirror running on an Ubuntu box via http, when terrasync is disabled it is accessing the same mirror via an SMB/Windows share

Please Log in or Create an account to join the conversation.

10 Jul 2018 09:18 - 10 Jul 2018 09:20 #39279 by enrogue
At all times the Aircraft used to startup is the F-86 from FGaddon, and starting point is EGOD, Renderer is default, no AA

1st Test: AV is enabled with no exceptions, terrasync disabled, getting scenery via SMB share

Time from start to get to runway: 45 seconds
Time to settle (cpu settled, scenery loaded, framerate settled): 78 seconds
framerate: 35fps when settled, while scenery is loading is at around 25-28fps
CPU utilisation: 50% when settled, while loading scenery is 85%

Please Log in or Create an account to join the conversation.

10 Jul 2018 13:04 #39285 by enrogue
2nd Test: terrasync using local mirror via http, AV on with no exceptions

time to runway: 100s, fps 20-25
time to settle: (finish sync check): 17 minutes!
CPU is at 95-100% during this time, about 15-20% is taken by AV
unfortunately the graphics thread locks after about 6 minutes so I can't give the FPS or CPU when settled

Please Log in or Create an account to join the conversation.

10 Jul 2018 14:51 #39292 by enrogue
With AV but excluding Documents/FlightGear:

time to runway 1:40, fps 25, cpu 95-100%, AV only taking 1-2% cpu
time to settle 14 minutes, cpu when finished 57%

With AV but excluding Documents/FlightGear - default terrasync:

time to runway 1:40 fps 23-25, cpu 93-100%, AV only takng 1-2%cpu
time to settle (finish sync) 14 minutes, cpu when finished 60%, fps 30-34

With AV but excluding Documents/FlightGear and excluding fgfs/fgfs.exe process, default terrasync:

time to runway 1:35, fps 24-27, cpu 95-100%, AV <1% cpu
time to finish sync 12 minutes, graphics locked up

AV off, local terrasync mirror:

time to runway 1:40, fps 23-26, cpu 100%
time to finish sync 17 minutes, cpu 65% when finished, fps 30-35

AV off, default terrasync:

time to runway 1:40, fps 23-26, cpu 95-100%
time to finish sync 17 minutes, cpu 65% when finished, fps 30-35

I didn't clear the TerraSync directory between runs, so the issue is that it is checking every file in the Models & Airports directory (Airports is the one that takes the longest) on very startup if you use TerraSync

Running in a VM isn't the best performance wise obviously, but I only use to to compile & check stuff, but there is a clear performance impact here with TerraSync enabled for 12-17 minutes after startup if you are I/O bound, probably worse if CPU is old as well

Please Log in or Create an account to join the conversation.

10 Jul 2018 15:31 #39293 by enrogue
Final Note (I promise!)

The effects of AV are variable, and I suspect this will vary more with Windows version & AV vendor. For my tests, disabling AV or excluding both the flightgear process and the terrasync directory made the CPU load a bit lower but mainly made things more stable

I'll have a look if there are any changes since 2018.2.2 that affect terrasync (especially the airports directory), if I don't find anything I'll ask on the mailing list about the sync issue
The following user(s) said Thank You: StuartC

Please Log in or Create an account to join the conversation.

10 Jul 2018 15:57 #39295 by Algernon
Thanks for your efforts on this! :)

Please Log in or Create an account to join the conversation.

13 Jul 2018 16:09 #39333 by enrogue
Still working on this, have duplicated the issue on a Netbook (slow, no emulation, but at least the graphics thread doesn't lock)

I've been going through the Terrasync & HttpRepository code in simgear to familiarise myself with whats going on - there's some debug stuff in there I'll turn back on to see if it highlights anything unusual

Please Log in or Create an account to join the conversation.

13 Jul 2018 16:16 #39334 by timi
By the way I just remembered a tool you can use to scope out what is hogging performance in the OS. That's the Windows Performance Toolkit:
docs.microsoft.com/en-us/windows-hardware/test/wpt/
The following user(s) said Thank You: Vodoun da Vinci

Please Log in or Create an account to join the conversation.

16 Jul 2018 11:10 #39385 by enrogue
Thanks for that, hopefully I won't have to use it just yet

I reenabled the debug messages in the httprepository code for hash mismatches this morning to check Linux, Mac & Windows behaviour

I suspected that Windows hash checking was failing & I was right - it's having to traverse the whole tree & check all the hashes which is CPU/time consuming

timi you can check if it's happening in your build by commenting out the #if 0 at line 336, and the #endif at line 343 of simgear code at simgear\io\HTTPRepository.cxx

I would recommend changing the SG_DEBUG to SG_WARN within the commented #if... #endif section so you can just set --log-level=warn (I wasn't able to get the --log-class stuff to filter properly)

according to the logic of the checks, a hash exists, but is incorrect, which means it has to traverse deeper into the tree, meaning more checks, more cpu & more time

I'll post an email onto the dev list about this - I've just tried a cleaned out appdata directory as well & it still fails, and changing antivirus did nothing either

Please Log in or Create an account to join the conversation.

16 Jul 2018 12:05 #39386 by timi
Ok, I'll make a new build for this. Good find by the way. :)

Please Log in or Create an account to join the conversation.

16 Jul 2018 13:23 - 16 Jul 2018 13:27 #39387 by enrogue
Just in case - heres the small change needed to enable the hash debug messages & set them to SG_WARN:
diff --git a/simgear/io/HTTPRepository.cxx b/simgear/io/HTTPRepository.cxx
index 690d5622..400241b1 100644
--- a/simgear/io/HTTPRepository.cxx
+++ b/simgear/io/HTTPRepository.cxx
@@ -333,14 +333,14 @@ public:
             if (c == children.end()) {
                 orphans.push_back(it->file());
             } else if (c->hash != hash) {
-#if 0
-                SG_LOG(SG_TERRASYNC, SG_DEBUG, "hash mismatch'" << it->file() );
+//#if 0
+                SG_LOG(SG_TERRASYNC, SG_WARN, "hash mismatch'" << it->file() );
                 // file exists, but hash mismatch, schedule update
                 if (!hash.empty()) {
-                    SG_LOG(SG_TERRASYNC, SG_DEBUG, "file exists but hash is wrong for:" << it->file() );
-                    SG_LOG(SG_TERRASYNC, SG_DEBUG, "on disk:" << hash << " vs in info:" << c->hash);
+                    SG_LOG(SG_TERRASYNC, SG_WARN, "file exists but hash is wrong for:" << it->file() );
+                    SG_LOG(SG_TERRASYNC, SG_WARN, "on disk:" << hash << " vs in info:" << c->hash);
                 }
-#endif
+//#endif
                 toBeUpdated.push_back(it->file() );
             } else {
                 // file exists and hash is valid. If it's a directory,

This is not a fix - it's just telling you when the hash checks are failing

Please Log in or Create an account to join the conversation.

16 Jul 2018 13:39 #39388 by timi
My simgear build started failing after uncommenting though...

Please Log in or Create an account to join the conversation.

16 Jul 2018 13:46 #39389 by enrogue
Odd - what error?

I did rerun cmake & regenerate after making the changes - building 2018.2.2 (the 2018.2 tree current)

Please Log in or Create an account to join the conversation.

16 Jul 2018 13:49 #39390 by timi
Seems to fail on some tests so disabled them and try again.

Please Log in or Create an account to join the conversation.

16 Jul 2018 13:54 #39391 by timi
Didn't help. But I just remembered I might have tried to run some other patches to sources as well.
So I'll unpack a fresh set of sources...

Please Log in or Create an account to join the conversation.

16 Jul 2018 14:03 #39392 by timi
Also setting BOOST_ROOT wasn't enough I had to define BOOST_INCLUDEDIR as well.
Trying to rebuild now...

Please Log in or Create an account to join the conversation.

16 Jul 2018 14:40 #39393 by timi
Got simgear built but now flightgear is failing too.

Please Log in or Create an account to join the conversation.

16 Jul 2018 14:42 #39394 by timi
Hmm... there were some pending windows updates. Perhaps they messed things up...

Please Log in or Create an account to join the conversation.

16 Jul 2018 15:10 #39395 by enrogue
Well I've posted to the dev list, lets see what comes back

Please Log in or Create an account to join the conversation.

Time to create page: 0.138 seconds
Powered by Kunena Forum

PM Mailbox

You are not logged in.

Forum Search

Keyword