✔️ NEW FEATURE: New Build System (No more Visual Studio!)

Eggheads talking about bytes and stuff.
User avatar
Krishty
Site Admin
Posts: 1271
Joined: 2022-Jan-09, 00:59

✔️ NEW FEATURE: New Build System (No more Visual Studio!)

Post by Krishty »

So, the TFXplorer repository didn’t have updates for a while. Here’s why.

Aside from TAW stuff, my family, and my job (I’m left with only one this year), and my other/other other game modding stuff, I maintain a few programs from my time in previous jobs.

These share huge parts of TFXplorer’s code base, which is heavily based on Microsoft’s Visual Studio.

Microsoft’s continued descent towards insanity has just brought us interesting corporate innovations like a video chat client built directly into the development environment. My mixed feelings towards their toolchain have deeper roots, though.

So I’ve been planning for long to become independent from Microsoft’s toolchain. Which means breaking free from Visual Studio, its compiler, and its build system.

TFXplorer will soon get a new build system with …
  • Ninja instead of Visual Studio projects/solutions;
  • Clang support (not a full move away from Visual C++ yet, but a first step);
    and – last but not least –
  • a fully portable toolchain with Clang and Visual C++ right out of the repo, i.e. no need to install Visual Studio / Windows SDK / whatnot.
That last thing is something I’m especially proud of, as you no longer need to install Microsoft’s bloatware or open a MSN/Live/Hotmail/MSDN/Microsoft/Azure account to compile TFXplorer. I can literally download the repo to my mom’s laptop, double-click the build script, and have TFXplorer compiled not even half a minute later.

The only downside is that the repo will grow in size. The Clang compiler, for example, is alone 100 MiB large.

Great! What’s taking so long?
As I said, TFXplorer shares a codebase with my other tools. These are a bit more advanced in the sense that they have actual setups, changelogs tied to websites, etc. I will commit this change after porting all projects to the new build system to be sure we’re future-proof.

Can only be a matter of weeks. (Famous last words!)
User avatar
Krishty
Site Admin
Posts: 1271
Joined: 2022-Jan-09, 00:59

Re: New build system underway (No more Visual Studio!)

Post by Krishty »

For the past two years, TFXplorer’s Clang build was plagued by internal compiler errors (i.e. bugs in Clang). I was only ever able to build debug versions. It was something about my pretty highly optimized SSE2 vector code throwing off the compiler.

With Clang 16.0.0, the problem was finally fixed. I can, for the first time, build optimized TFXplorer with Clang.

The performance is pretty good indeed. Clang performs a few more aggressive optimizations than MSVC in many places, albeit it loses in others.

There are odd bugs now like the 2nd digit of the seconds in the clock disappearing for the first ten seconds in a minute 😆 But we’re getting there!
mikew
Data Genius
Posts: 558
Joined: 2022-Jan-09, 20:21

Re: New build system underway (No more Visual Studio!)

Post by mikew »

Well, I suppose that's a bit less Microsoft-y, so that's good.

These days, I'm having trouble deciding which PC to start up as win10, Linux and MacOS all suck in different ways.
It feels like the bright future of computing is being denied to us.
User avatar
Krishty
Site Admin
Posts: 1271
Joined: 2022-Jan-09, 00:59

Re: ✔️ NEW FEATURE: New Build System (No more Visual Studio!)

Post by Krishty »

The new build system is now in the repository.

Quick Start

To build and run from the command line (works without Visual Studio installed):
  1. Open a command prompt. I don’t know if PowerShell and Windows Terminal are compatible because I don’t have them installed, so feel free to try and let me know.
     
  2. Navigate to the repository, e.g. cd C:\projects\TFXplorer
     
  3. Run build.cmd.
     
  4. This should start Ninja to build all projects in all configurations. After several seconds, all jobs should be complete, i.e. output should state something like [1023/1023] without red errors.
     
  5. Navigate to !build/UAW (note that some shells may have problems with the exclamation mark; use quoting or escaping if so) and select the configuration you’d like to run. Note that msvc-debug can be 50 times slower than msvc-release.
     
  6. Inside the according configuration, start TFXplorer.exe.
     
  7. Done!
To build with Visual Studio:
  1. Start Visual Studio.
     
  2. On the start page, select Open a local folder and navigate to the the repository, e.g. cd C:\projects\TFXplorer.
     
  3. In Solution Explorer, right-click the repository root and choose Build. This should start Ninja to build all projects in all configurations.
    image.png
    image.png (10.21 KiB) Viewed 2245 times
    After several seconds, all jobs should be complete, i.e. output should state Build succeeded..
    image.png
    image.png (2.21 KiB) Viewed 2245 times
     
  4. To start/debug the game, select the proper project configuration from the menu bar:
    image.png
    image.png (5.34 KiB) Viewed 2245 times
     
  5. In Solution Explorer, right-click TFX/TFXplorer and select Debug:
    image.png
    image.png (9.82 KiB) Viewed 2245 times
     
    Done!
Notes

I didn’t push the Clang build yet, for a simple reason: Microsoft’s compiler is 160 MiB large; Clang is 500 MiB. I won’t force such a huge download upon you until there’s a strong reason for it.

Neither did I push the deployment scripts for generating setups or archives. I don’t think anyone will need it, and it’s easier for me to maintain this stuff without synchronization.

If the build breaks, you can clean it by deleting the !build directory. I added a shortcut to Visual Studio: Right-click the repository root in Solution Explorer, select Clean.
mikew
Data Genius
Posts: 558
Joined: 2022-Jan-09, 20:21

Re: ✔️ NEW FEATURE: New Build System (No more Visual Studio!)

Post by mikew »

I might be too tired, but I don't see 'build.cmd' anywhere...
User avatar
Krishty
Site Admin
Posts: 1271
Joined: 2022-Jan-09, 00:59

Re: ✔️ NEW FEATURE: New Build System (No more Visual Studio!)

Post by Krishty »

I hope you didn’t check out the repository too early. I clicked Submit on this one topic too early … please try git pull --rebase again
mikew
Data Genius
Posts: 558
Joined: 2022-Jan-09, 20:21

Re: ✔️ NEW FEATURE: New Build System (No more Visual Studio!)

Post by mikew »

Ah, yes. I was too keen. There it is. :)
mikew
Data Genius
Posts: 558
Joined: 2022-Jan-09, 20:21

Re: ✔️ NEW FEATURE: New Build System (No more Visual Studio!)

Post by mikew »

Only had a Linux PC yesterday, and Wine has a good attempt at build.cmd but ultimately unsuccessful.

That build method works fine in Win10, but where are the scenarios? Has the registry system changed?
User avatar
Krishty
Site Admin
Posts: 1271
Joined: 2022-Jan-09, 00:59

Re: ✔️ NEW FEATURE: New Build System (No more Visual Studio!)

Post by Krishty »

mikew wrote: 2023-Oct-07, 19:23Only had a Linux PC yesterday, and Wine has a good attempt at build.cmd but ultimately unsuccessful.
Oh, send me the output and I’ll see what I can do!
mikew wrote: 2023-Oct-07, 19:23That build method works fine in Win10, but where are the scenarios? Has the registry system changed?
Yes they have. Writing about it rn!
mikew
Data Genius
Posts: 558
Joined: 2022-Jan-09, 20:21

Re: ✔️ NEW FEATURE: New Build System (No more Visual Studio!)

Post by mikew »

As usual, I might not be doing things the same way as everybody else...

So, I'm using a laptop that came with Win11 Home on it. After some effort to overcome the SecureBoot stuff, I could get it running Linux but left Win11 there so could boot it if necessary but never let it go online.
That works well with this method of building TFXplorer, as if things don't work with Wine, I still have Windows to fall back on.

The question is how best to get around the 'DirectX runtime 2010 not installed, so there won't be any sound' warning with the least possible contact with Microsoft?
I suppose I could download the 2010 SDK installer and run that, but why should I do that in 2023?
User avatar
Krishty
Site Admin
Posts: 1271
Joined: 2022-Jan-09, 00:59

Re: ✔️ NEW FEATURE: New Build System (No more Visual Studio!)

Post by Krishty »

Do you get this message only in debug builds or also in release builds?

Because the debug builds use the debug version of the runtime, which is not usually shipped with Windows but requires said installer. The release version should use the runtime that comes pre-installed with most Windows versions …
mikew
Data Genius
Posts: 558
Joined: 2022-Jan-09, 20:21

Re: ✔️ NEW FEATURE: New Build System (No more Visual Studio!)

Post by mikew »

It's release builds as well.
User avatar
Krishty
Site Admin
Posts: 1271
Joined: 2022-Jan-09, 00:59

Re: ✔️ NEW FEATURE: New Build System (No more Visual Studio!)

Post by Krishty »

Ouch!

Okay. For minimal MS interfacing:
  1. Download the DirectX Redistributable (June 2010)
  2. Extract the .exe to a new directory, e.g. via 7-Zip.
  3. From the new directory, delete all .cab files except dxupdate.cab, Jun2010_XAudio_x64.cab and Jun2010_XAudio_x86.cab. Be sure to keep the installer files like DXSETUP.exe, DSETUP.dll, etc.
  4. Run DXSETUP.exe and complete the setup.
This should give you only the June 2010 XAudio runtime without the other 300 D3D versions.

And if you need a specific version of D3D/D3DX/XAudio/etc. for an old game, you now know the drill.
mikew
Data Genius
Posts: 558
Joined: 2022-Jan-09, 20:21

Re: ✔️ NEW FEATURE: New Build System (No more Visual Studio!)

Post by mikew »

Thanks! I'll do that...later. :)
mikew
Data Genius
Posts: 558
Joined: 2022-Jan-09, 20:21

Re: ✔️ NEW FEATURE: New Build System (No more Visual Studio!)

Post by mikew »

Well, later was a bit sooner than I thought.

It took a bit of effort, downloading that and the 7zip installer with Linux to the Win11 partition etc...but it works!

By the way, the debug versions won't run due to these missing things:
deberr.png
deberr.png (8.46 KiB) Viewed 2052 times
deberr2.png
deberr2.png (6.88 KiB) Viewed 2052 times
I won't be using these versions on this PC, but it might be interesting to you to see what happens with a reasonably 'oot of the box' Win11 installation.
User avatar
Krishty
Site Admin
Posts: 1271
Joined: 2022-Jan-09, 00:59

Re: ✔️ NEW FEATURE: New Build System (No more Visual Studio!)

Post by Krishty »

Good observation! These are the debug version of the Visual C++ runtime library. They are normally installed by Visual Studio, that’s why I must have missed them.

Whether I should provide them … interesting question. If you don’t have VS installed, there’s likely no debugger that could be used with the debug versions.
mikew
Data Genius
Posts: 558
Joined: 2022-Jan-09, 20:21

Re: ✔️ NEW FEATURE: New Build System (No more Visual Studio!)

Post by mikew »

A couple more observations:

The debug builds are crashing even on a PC with VS installed. No error box, but the gui is visible for about a second, then disappears.

Also, (and this may be standard behaviour) these files stay active for a long time after I exit the game. I know this because I can't delete the 'tfxplorer' folder until they exit which is maybe after 15 minutes or so. Not something I do often though.
hangms.PNG
hangms.PNG (13.23 KiB) Viewed 1841 times
User avatar
Krishty
Site Admin
Posts: 1271
Joined: 2022-Jan-09, 00:59

Re: ✔️ NEW FEATURE: New Build System (No more Visual Studio!)

Post by Krishty »

mikew wrote: 2023-Nov-09, 10:45The debug builds are crashing even on a PC with VS installed. No error box, but the gui is visible for about a second, then disappears.
Have you tried to run them with a debugger attached, e.g. from the Visual Studio IDE? I know that there is a place in extension setup that does not give proper error messages but just runs into hard-coded breakpoints on error. Could be it. I’ll create a ticket so I don’t forget to clean this up …
Also, (and this may be standard behaviour) these files stay active for a long time after I exit the game. I know this because I can't delete the 'tfxplorer' folder until they exit which is maybe after 15 minutes or so. Not something I do often though.
hangms.PNG
Okay, this one is a bit more complicated.

So the compiler generates debug information, i.e. which instruction in the executable maps to which line in the code, how our data structures are laid out, etc. This debug information goes to a PDB file (Program DataBase or some other stone age term) and that’s pretty straightforward if your program consists of a single C++ file.

But if you have multiple C++ files and compile in parallel, you’ll have 10 compiler instances trying to write the debug information from 10 C++ files at the same time. The first instinct is to synchronize access. But this introduces more performance problems, as every compiler instance needs to open the existing PDB, parse it entirely, then place its new information there, and close it.

That became a bottleneck and so Microsoft decided they’d have a background service. That’s mspdbsrv.exe – MicroSoft Program DataBase SeRVice. Now the first compiler instance launches the service and instead of opening the PDB file itself, just passes on the debug information to the service. And so do all other instances. The service opens the PDB only once, so the O(n²) becomes O(n).

But that only works if the service sticks around for a while (doesn’t shut down during compilation) – or at least MS thinks so. So MS gave this service a timeout of several minutes before it closes.

And that is why you can’t delete the repository right after compiling: mspdbsrv.exe still sticks around, in case that someone needs to write more debug information for the EXE.

So usually it’s enough to open the Task Manager and kill mspdbsrv.exe. But here’s the catch: Due to some Microsoft black magic, it sometimes doesn’t show up in Task Manager. (Really interesting topic, I’d love to talk about that with an IT security pro one day!) So if that’s the case for you, go to the Performance tab, click Open Resource Monitor at the very bottom, and terminate it from there.
image.png
image.png (34.53 KiB) Viewed 1840 times

Sorry for the text, but this is pretty much undocumented and it’s probably better to understand the problem instead of having me say “Just terminate that EXE from my repository that’s invisible in your Task Manager and it’ll be fine, trust me bro!” And for the paranoid (like me): That EXE is digitally signed by Microsoft, as you can check in the file properties. So it’s no malware, or at least not more harmful than other Microsoft products.
mikew
Data Genius
Posts: 558
Joined: 2022-Jan-09, 20:21

Re: ✔️ NEW FEATURE: New Build System (No more Visual Studio!)

Post by mikew »

Thanks for the insight!
So, it's the compilation process that causes that. I thought it may be because I tried to run the exe.
The x86 directory deleted OK, but it may have already timed out I guess.

I'm only using build.cmd these days and don't normally open VS at all.
User avatar
Krishty
Site Admin
Posts: 1271
Joined: 2022-Jan-09, 00:59

Re: ✔️ NEW FEATURE: New Build System (No more Visual Studio!)

Post by Krishty »

mikew wrote: 2023-Nov-09, 12:15The x86 directory deleted OK, but it may have already timed out I guess.
On 64-bit systems, only the x64 version of the compiler is used, even for compiling 32-bit binaries (cross compiling).

These design decisions have interesting effects on the toolchain size, as we’ll see when I introduce Clang to the mix 🙂
Post Reply