Wikia

OpenArena

Multithreaded map compiling

370pages on
this wiki
Talk0

It is possible to take advantage of modern multi-core processors to compile maps much faster by using multithreading.

A quick way to see if your machine has got multiple cores (or multiple processors), is:

  • Microsoft Windows: to take a look to the "Performance" tab of the "Task manager" tool (from Ctrl+Alt+Del). If the "CPU utilization history" is divided into two or more graphs, your machine as got at least two or more cores (physical or logical).
    Note: up to Windows 7, under "View" -> "CPU History" menu from Task manager "Perfomance" tab, you can find two options: "One Graph, All CPUs" and "One Graph per CPU": the second one is the one enabled by default required for this test. In case you have "One Graph, ALL CPUs" checked instead, you will NOT be able to find out the number of cores, unless you modify the option. If you are using Windows 8/10, you will not find the "CPU history" item under the "View" menu: you have to right-click on the CPU graph and choose "Change graph to" -> "Logical processors" instead (by default, a single graph is showed there!). However Windows 8/10 Task Manager, under "Performance" -> "CPU" screen, lists the number of physical processors, cores and logical processors as simple text right under the graph.
  • GNU/Linux: run the "top" command from a Linux "shell" (command line interface). After pressing the "1" key, in the upper part of the screen, the "Cpu(s)" line should change into multiple lines ("Cpu0", "Cpu1", "Cpu2"...), one for each core (physical or logical).

If you have got more than one core, you may find useful to compile maps (q3map/q3map2) using more threads, in other words splitting the process into separate sections that can run at the same time on multi-core systems. This way, you should see higher CPU load in Task Manager/top while compiling, and you should be able to compile maps much faster than using a single thread, because each thread should work on a separate core (with a single thread, only one core would be used). So, choose the number of threads (1, 2, 3, 4, 5, 6...) depending from how many cores you have got and how many of them you want to use for this purpose: using all available cores should give you faster compiling. On the other hand, leaving at least one physical core "free" should allow you to do not experience slowdowns using the computer do to other things (e.g. playing a game or surfing the web) while you are compiling a map; it's your decision.[1] You may wish to create various custom pre-sets using different numbers of threads, and choose each time which one to use depending from how much processing power you wish to have free for other applications.

How to setup multithreaded compilingEdit

Q3MAP (map compiler that comes with the original Q3Radiant) and Q3MAP2 (map compiler that comes with GTK Radiant, with some extra features) are command line tools. The map editor launches q3map (or q3map2) with various command-line parameters, depending from the Bsp option you choose from the menu. Q3MAP and Q3MAP2 support the "-threads n" parameter to tell how many threads they have to use ("n" is the number of threads, e.g. 1, 2, etc.).

  • While running on Microsoft Windows OS, Q3MAP2 auto detects the number of total cores (physical + logical) your machine has, and automatically selects the number of threads to use accordingly. However you can use the "-threads" parameter to manually force the number of threads to use.
  • While running the compiler on GNU/Linux OS, instead, you have to manually specify the number of threads to use.
  • Also the old Q3MAP can automatically use a thread for each core it detects on Windows, if you don't specify the "-threads" parameter... but unfortunately some options in Q3Radiant "Bsp" menu do actually contain "-threads 1" by default, and so you will have to manually modify them if you want to use multiple threads.

Note: map compiler doesn't know the difference between complete processors, real cores and Hyper-threading technology's logical cores.[2] It'll assume that each "CPU" it finds is a complete CPU.
In reality, each physical processor can have one or more physical cores. Each core may have hyperthreading feature, that would show it as two cores. So, a dual-core processor with hyperthreading enabled would look like a four-processors machine, but the performances would be lower than a four-processors machine![3] While a dual-core CPU has got two physical cores that share some components, Hyper-threading technology shows an extra (logical) core by simply doubling few components in one physical core, thus it is not as efficient: depending from the different applications and environments, in a few cases HT may even hurt performances; fortunately, it looks like this is not our case, and that you can actually take advantage of HT.

There is no advantage in trying to force more threads than the number of total cores you have. While using all your physical dual (or quad, or more) cores should speed up a lot, the effect of hyperthreading (logical) cores may vary; anyway it seems HT is quite good for map compiling, and using all of your cores should give you better results. If you wish, you can do tests compiling the same map specifying different number of threads, comparing the time needed.[4]

BSPCEdit

Also BSPC (the tool that creates "bot navigation" .AAS files) supports multiple threads, but it does not auto-recognize the numer of threads to use (not even on Windows), and will use one thread by default. You can however specify the number of threads to use by adding the -threads <n> option, gaining compiling speed on multi-core machines. Just an example, if you lauch BSPC from a batch file that executes bspc -bsp2aas mymap.bsp, you can modify it to execute bspc -threads 2 -bsp2aas mymap.bsp[5] to run it with two threads instead of one. However, the time adavantage it gives is not as evident as with map compiling: test it and then decide... probably you can leave it without that option and still live happy (especially in case you notice enabling it causes BSPC to crash more often in your machine; however, switching to a different version of BSPC may fix the issue and allow you to use multithreading without crashing, hence allowing you faster .aas compiling.[6]).
You may also specify the -threads <n> option to the batch file you use to optimize the .AAS file you previously created -the operation that makes the .aas file smaller- (example: bspc -threads 2 -aasopt mymap.bsp), but that operation is already so quick you don't need multithreading at all for it. By the way, it is possible (and probably advisable) to create aas file and optimize its size with a single command, using a command line like bspc -grapplereach -optimize -bsp2aas mymap.bsp
Note: in this example, we also specified the "-grapplereach" option because it causes the "calulating reachability" phase to take some more time and you may take advantage of this to better compare BSPC performances when using different values for the "-threads" option. For your information, "-grapplereach" option is required for bots to use the grappling hook, if the server allows them to use it (bot_grapple 1). However in-game bot grapple support is incomplete and buggy since Q3A, and currently (OA 0.8.8) it has not been fixed yet.

Q3RadiantEdit

Note: please read all this section before trying to apply it: as later said, a bug in the editor may mess up your compiling options even if you didn't actually change them! In short, to avoid that bug, you will need to manually tweak a file called "quake.qe4" with a text editor (but it's worth the effort!).

Let's take the case we are using the old Q3Radiant editor, configured for the original Quake III Arena game, on Microsoft Windows[7].

In the "Bsp" menu, you can find the bsp_FullVis (light -extra) option. This is probably the one that takes more time to compile, used also for the "final" map, and unfortunately this menu option specifies "-threads 1" parameter,[8] so it is the one we will tweak. You may notice great time benefits after the tweak.

Go to "File" -> "Project settings" menu. In the "Menu commands" section, select bsp_FullVis (light -extra) line and click "Change...".
In the window that opens, select all the text in the "Command" field, and copy it to your clipboard (CTRL+C or right button, copy), then click on "Cancel", to be sure we will not damage the original option.
Now click on "Add...", and paste from your clipborard into the "Command" field (CTRL+V or right button, paste). Modify from -threads 1 to -threads 2 (or other number, e.g. 3, 4, 5, 6...) in the two places where the "-threads" option appears. Enter bsp_FullVis (light -extra) Multithread (or whatever you prefer, but anyway the name has to begin with bsp, and it should not contain the " character) in the "Menu text" field, then click OK.[9]

You should have started from something similar to this (of course your Q3 installation path may vary):

! C:\PROGRA~1\QUAKEI~1\Tools\q3map $ && ! C:\PROGRA~1\QUAKEI~1\Tools\q3map -vis -threads 1 $ && ! C:\PROGRA~1\QUAKEI~1\Tools\q3map -light -extra -threads 1 $

And you should have got something like this (in case you want 2 threads):

! C:\PROGRA~1\QUAKEI~1\Tools\q3map $ && ! C:\PROGRA~1\QUAKEI~1\Tools\q3map -vis -threads 2 $ && ! C:\PROGRA~1\QUAKEI~1\Tools\q3map -light -extra -threads 2 $

Still in the "Project Settings" windows, select the newly created option and click "Change...". If everything is still as you set a minute ago, you are ready and you can go compiling your map... but if the line in the "Command filed" appears incomplete (where many options misteriously disappeared), e.g. like this:

! C:\PROGRA~1\QUAKEI~1\Tools\q3map $ && ! C:\PROGRA~1\QUAKEI~1\Tools\q3map 

you enconountered a bug of the editor (this is another reason we suggest to never modify the original menu options) and thus you have to manually modify the file that contains the menu to fix the menu option (compiling would not run correctly without the missing part of the command line).

In this case, close Q3Radiant. Then go to your Quake III Arena installation folder (if you have multiple installations of the game, the one you use with Q3Radiant), then enter the "BASEQ3" folder, and the "scripts" folder in it. There, use a plain text editor (such as notepad) to edit the "quake.qe4" file. Locate the line that contains your newly created menu option (in our example, bsp_FullVis (light -extra) Multithread).
As you can guess, the format of a line is simply "menu text" "command". Correct the command line part, copy-pasting from the original menu option and changing the value in the two places of the line where the -threads option is. Save and close the file, and start Q3Radiant again.

At this point, if you select the newly created option from the Bsp menu, you should notice your map compile in less time.

Of course, you may have directly edited the quake.qe4 file and added a line (somewhere between the { and } lines) for the new menu option, if you prefer. Also in this case, remember that the name of the new menu option has to begin with bsp and should not contain the " character (that is instead used to separe the menu option name from the command line).

Hint: also considering the editor bug, you may want to make a backup copy of that quake.qe4 file before starting to make any changes to it through Q3Radiant or through text editor.

Tip: in our example, we specified the number of threads we want. Anyway you can set q3map to automatically use the same number of threads as the total number of CPU cores Windows finds, by removing the "-threads n" option from the desired line in quake.qe4 file, in the two places of the line where it appears.

GTKRadiantEdit

<Someone please write this section>

NetRadiantEdit

<Someone please write this section>

Q3Map2BuildEdit

With Q3Map2Build (a small but useful tool that allows to compile a map -and optionally, also its .aas file- without launching a Radiant editor, with full control over compiling options), you can specify the number of threads to use by clicking on the "Options" buttons corresponding to each stage (BSP, VIS, LIGHT, BSPC). Such "Options" buttons are active only if you have chosen "Custom" for that stage. While in selected stage options window, just check "-threads" option and specify the desired number of threads in the apposite field, and click "Save".

After you set options for all stages, build (compile) the map.

NotesEdit

  1. Another way to do not experience slowdowns while doing other things at the same time of compiling is to use tools as Task Manager or Process Explorer to mange process priority, by giving lower priority to the q3map/q3map2 process, or giving higher priority to the other process you are interested to. Howerver, this is not very comfortable, because you would have to manually repeat it every time q3map/q3map2 is invoked.
  2. "Hyper-threading" (HT) is a technology by Intel. It allows to use multi-threading, but it's not a synonym of "Multi-threading".
  3. It's just an example: real multiprocessor machines are usually used only in servers, not in end-user machines. It's unlikely you would buy such hardware to compile OpenArena maps! And a today's high end single processor computer may be better than a ten-years-old double or even quad processors machine.
  4. It may even be possible to completely disable HT technology from your system's BIOS/UEFI (this would make the system show half cores than before), and then do more tests.
    Other than the application in use, HT perfomances may also vary depending from the OS in use: if the Operating System is aware that some cores are physical and others are logical, it can better distribute the load; Microsoft reports that Windows is HT-Aware since Windows XP/Windows Server 2003, and we can guess later versions should be even more optimized. Some tests with the old q3map (Bsp_fullvis -light -extra from Q3Radiant, with tweaked threads number) on a Windows 7 machine with a quad-core Intel Core i7 (8 total cores when HT is enabled; 4 total cores when HT is disabled), showed up the compiling of a test map in:
    • about 3 min 5 sec using 4 threads with HT disabled in BIOS
    • about 3 min 5 sec using 8 threads with HT disabled in BIOS
    • about 3 min 13 sec using 4 threads with HT enabled in BIOS
    • about 2 min 11 sec using 8 threads with HT enabled in BIOS.
    This seems to prove that compiling maps using as many threads as the total number of cores you see is good.
    For your information, the same compiling, using the option with the original "-threads 1" parameter, took 10 minutes 32 seconds.
    A note about the tests above: sometimes, q3map tended to give a crash error in the middle phase: in that case, we interrupted the compiling and re-launched it again, until the map compile completed without crashes.
  5. You may place the -threads <n> option at the end of the BSPC command line, if you prefer, it's the same.
  6. In a Windows 7 64 bit test, "BSPC version 2.0, Jun 21 2000 16:12:44 by Mr Elusive" -provided with Q3Radiant 202- did crash often with more than 1 thread, while "BSPC version 2.1h, May 6 2001 16:15:03 by Mr Elusive" -provided with GTKRadiant 1.5- and "BSPC version 2.1h, Oct 7 2012 15:47:47" -provided with GTKRadiant 1.6- did not.
  7. Hint: it looks like that Q3Rradiant (or Q3MAP, which is invoked by Q3Radiant when compiling) is more stable if run as administrator: if you encounter crashes while running it without administrative privledges, try launching Q3Radiant as administrator.
  8. While without it, you may achieve noticeably lower compiling time, if you have multiple processors, multiple cores or HyperThreading.
  9. Alternatively, when you click on "Modify..." after selecting the original menu option, edit the "threads" parameters there and MODIFY the "menu text", then click OK: this will cause the program to automatically create a new menu option, without the need of copy-pasting (we suggest to do not apply any change directly to the original menu options, but to create new options, because you may risk to ruin them, also due to a Q3Radiant bug that may "cut" the command line!).

See alsoEdit

Around Wikia's network

Random Wiki