Latest download: ZHLT 1.7p15
Earlier versions: ZHLT 1.7p13
Cagey's full source code: ZHLT 17p15.5 source Note: when using the source code, you should refer to my notes on page 33 of this topic.

For those who aren't aware, this is an update of Merl's 1.7 Custom Build of Zoner's tools. All of the features that Merl and Zoner added are still available in this distribution.

See the red notes below for important information on the latest build.

ok i just did a test with my map (not maxplanes reachd zet tough) but reducing planes. it went (according to the tool) from 6622 to 2047 and ran without any problem. XP-Cagey, i just can say WOAH! i hope i don´t run into any problems with this later on. but currently it seems to be absolutely amazing and gives me MUCH more room to build what i want. still nodes and such to keep an eye on but hell, this really is GREAT! Now i need to build a shrine.. you want red white or black candles?

Keep it up, this is one of the best additions to zoners! lets hope it gets official with the next release of ZHLT. You .

Im now compiling with the 1.7p toops

my OS has missing the file
MSVCR70.DLL
too bad that the hammer-compiler does not tell me that, it just says "file not found".
Just another batch compiler told be about the missing dll, so I asked google about it...

QUOTE

This DLL is needed by applications compiled to use the Microsoft Foundation Classes (MFC) as a DLL, and compiled on a machine that has the Microsoft .NET SDK installed on it.  This is no different than older MFC runtime DLL requirements; it's just that this is a very new runtime DLL and most folks don't have it.  The system running this application (mapserver, compiled this way) also must have Microsoft Internet Explorer 4.0 or higher installed on it, since the runtime uses components from IE as well.

You can get this file anywere on the internet.
Copy it in your main system directory or in system32 .

well,
the compiling succeded:

QUOTE

Object names  Objects/Maxobjs  Memory / Maxmem  Fullness ------------  ---------------  ---------------  -------- models            196/400        12544/25600    (49.0%) planes          31086/65535     621720/1310700  (47.4%) vertexes         4765/65535      57180/786420   ( 7.3%) nodes            2504/32767      60096/786408   ( 7.6%) texinfos         6504/32767     260160/1310680  (19.8%) faces            3681/65535      73620/1310700  ( 5.6%) clipnodes       12193/32767      97544/262136   (37.2%) leaves           1771/8192       49588/229376   (21.6%) marksurfaces     3926/65535       7852/131070   ( 6.0%) surfedges       16234/512000     64936/2048000  ( 3.2%) edges            8181/256000     32724/1024000  ( 3.2%) texdata          [variable]       2380/4194304  ( 0.1%) lightdata        [variable]          0/4194304  ( 0.0%) visdata          [variable]          0/2097152  ( 0.0%) entdata          [variable]      17394/524288   ( 3.3%) 54 textures referenced === Total BSP file data space used: 1357738 bytes === 119.95 seconds elapsed [1m 59s]

the map is half-completed (raw hallways) and already touches the 90% limit, because its BIG.
now its 50% and I hope the compression tool cuts enough planes from the void.

You **** genius! GENIUS!

This is absolutely superb. I love you.

Makes you wonder what the official maps would have been like if they hadn't hit the plane limit...

Now we may find out.

I was quite sure there was more to it than changing a preprocessor directive in the compiler source, since Merl had so much trouble working with it. Oh well.

So far it seems to work. A test map shrunk from 2080 planes to 484, and seems to run fine in-game.

QUOTE (Yamazaki @ Jan 26 2003, 03:31 PM)

I was quite sure there was more to it than changing a preprocessor directive in the compiler source, since Merl had so much trouble working with it. Oh well.

Merl had bumped up the defined amount before, then knocked it back down when it caused in-game problems. This is a source comment embedded at the point of the directive:

CODE

#define MAX_MAP_PLANES      32767 // (from email): I have been building a rather complicated map, and using your latest // tools (1.61) it seemed to compile fine.  However, in game, the engine was dropping // a lot of faces from almost every FUNC_WALL, and also caused a strange texture // phenomenon in software mode (see attached screen shot).  When I compiled with v1.41, // I noticed that it hit the MAX_MAP_PLANES limit of 32k.  After deleting some brushes // I was able to bring the map under the limit, and all of the previous errors went away.

The problem has always been game engine based rather than something with the compiler. I suspect that the half-life engine uses a 32K sized static array for storing the loaded planes, and loses information when this is exceeded. There isn't anything magic about the 32K plane limit (it's not a hardware concern like the 4MB texture limit, for instance), and I'm willing to bet it was a semi-arbitrary size placed by the programmers when 400 was considered a high r_speeds value (akin to "nobody will ever need over 640K of memory" ).

Oh i almost forgot, could this get a STICKY maybe? pretty please?

sticky good. - so edited

Im going to split this page off into a new topic and sticky it. God I love admin tools

thanks for the tools, ive started to add alot more detail in now. theyre so helpful, where the hell did u learn to code on that sort of level!? youd think valve would know how to do this, personally i reckon you should send the link to planethalflife.com - im sure they would get out the word about the download as it helps all the community!

Its a great set of tools, helped me out a bunch! Brings my map from 4k to 1k planes

that is quite amazing, not to mention incredibly useful.

i can even detail my map further now... god i wub j00

I'm not at home right now so I can't try this out, but has anyone tried this on a map which has already exceeded the max planes (which is why it's there, obviously)?

In my experience, if the map has too many planes in it then you can't fully compile the map anyway. If you've removed this 'lock', then kudos to you and congratulations on producing such a useful tool. But, if you still can't compile maps which originally have too many planes (before the optimisation), then that makes this kinda useless.

QUOTE (Merkaba @ Jan 27 2003, 10:31 AM)

If you've removed this 'lock', then kudos to you and congratulations on producing such a useful tool.

Yep, it's been moved up to about double what it was before (65535 now). If that's still not high enough, another recompile can shift it upwards again... the only limitation for the compiler tools (I'm not talking about running the game itself here) is the amount of memory required to store the information while it's being processed.

EDIT: shifting the limit upward again will require storing the information outside of the bsp during the compile because the bsp face structure's plane number field can only hold 16 bits.

Amazing, congrats! This may lead to a whole new level of complex levels

Give us Big Hera. Now. Please?

QUOTE (Merkaba @ Jan 27 2003, 10:17 PM)

Amazing, congrats! This may lead to a whole new level of complex levels

Damn straight!

http://www.cyscape.com/download/msvcr70.dll

Theres an URL to the .DLL if you can't find it or want it instantly.

QUOTE (Revenant @ Jan 27 2003, 10:19 PM)

http://www.cyscape.com/download/msvcr70.dll Theres an URL to the .DLL if you can't find it or want it instantly.

Thanks for tracking that down Rev -- added a copy of the link to the first post so people will see it

This will be mighty usefull to all mappers in here , really appreciated thx alot!

I also added it to the HLRally forums, the original post and the files aswell. I figured they could use it aswell, then again all mappers in the community could. As I want to make a HUGE winter map for HLR, Its likely I would go over the plane limitwhen making my map detailed.

http://forum.hlrally.net/index.php?act=ST&f=3&t=2117 - the url for u to look at, tell me if u want anything added as you are the official "compiler tool owner" seen as u made them

QUOTE (Revenant @ Jan 28 2003, 11:34 AM)

tell me if u want anything added

The only thing I would like added to that thread is a mention of the optimizer -- if you forget to use it and you're over the old planes limit, the map still breaks in the game... running the optimizer is really what allows the new flexibility-- ZHLT has had the higher limit before, and I don't want to cause any confusion. The extra program (opt_plns.exe) isn't part of the official tools, but I've submitted the source to Merl for inclusion in future builds.

actualy it didn't break it for me, I just got A REALLY bad case of FPS-Lows so you should make sure to optimize!

Send this mofo to Valve so it can make this official so everyone can benefit from them!

well... i actualy have a little question about all that.. (im a mapper, i know most of the HL-engine stuff). What are those planes? i have a little idea, but im not shure about it.

I'm not the most knowledgeable person on this, but the HL engine has a limited number of planes that can be stored in memory. If you break this limit, your map will not run correctly. A plane is a mathematical model of a flat, two-dimensional surface, and each face of the map must be represented by part of a plane. As far as I know, more than one face can rest on the same plane, but there is still a limit.

does this mean when you reduce the planes you reduce the r_speeds? i made a little test it went down from 342 to 320. Not much but it does... any information on that?

Generally it does bring down r_speeds

This may seem minor... but after testing the optimizer on all of the full-sized existing Natural Selection map's, I'd recommend raising the 'new' limit on planes to at least 98304, triple the original limit. Below is a list of how each of the original maps changed when ran through the optimizer:

NS_Bast 63.803212606119831429792951810908
32746 Before
11853 After

NS_Tanith 62.659141619635122719516996981231
30476 Before
11380 After

NS_Caged 59.534332219467500269483669289641
18554 Before
7508 After

NS_Eclipse 65.626992561105207226354941551541
18820 Before
6469 After

NS_Europa 62.807934307347765809960541751093
18754 Before
6975 After

NS_Hera 66.903478686918177364037236648702
32656 Before
10808 After

NS_Nancy 63.984801534602331415080419064483
27108 Before
9763 After

NS_Nothing 63.547775872958585220608138590634
22166 Before
8080 After

Average amount reduced: ~63%
Minimum amount reduced: ~59%
Maximum amount reduced: ~66%

Based on this, I'm seeing just shy of a 2/3rds reduction in the larger maps. Note that Hera and Eclipse, two of the maps that ran into the most problems in development as I recall, and likely had the most plane-optimization done to them already, benefited the most from the optimizer, while Caged didn't break 60% benefit, unlike all other maps in the test-set.

So, to be able to truly 'unlock' the abilities of higher plane-counts, I'd recommend supporting enough planes that in real-world work, even with the optimizer, it's possible to end up with too many planes. This may sound counter-productive, but right now you're possibly artificially-limiting the planes available now, with how effecient the optimizer actually is.

Also, this is a good example of just how much of a reduction this one tool is actually able to make in-game.

What exactly does the exectuable do? At this point it's like PFM...pop the bsp in at 8mb...it comes out at 7.5MB and with a nice little reduction in overall r_speeds and avg visible leafs and the nice little command window tells me I've reduced my planes from 32k to around 13K. Ok...I'm still a bit unsure how all this works...what exactly is removed? Planes that extend out to the horizon? I'm considering using the method on several of my maps and need to know what this "little blue pill" does. Thx.

Specifically, it does a modified mark-and-sweep garbage collection of planes before writing the result back to the file.

In layman's terms, the algorithm works like this:

1) read the file into memory
2) make an array of integers that is the same size as the array of planes on the map
3) peek at every face on the map, and make a mark on the integer array that corresponds to the plane being used (all faces have a plane that they reference by index, so if face #928 uses plane #124, I mark the integer array at #124 to say that plane has been used). To keep track of which plane is which, I use an increasing count for each match (so if it's the 10th plane I've marked, it's #10 in the new ordering... if it's used again, which is common, it stays #10 in each use).
4) repeat step #3, but look at nodes instead of faces
5) repeat step #3, but look at clipnodes instead of faces
6) using the integer array as a reference, copy all of the used planes to temporary storage (in this step they are rearranged to match the order they were marked)
7) copy the sorted planes back to the planes array (now used planes take up the front of the array)
8) tell the tools that the number of planes on the map is actually the number of used planes, so it ignores the garbage at the back of the array
9) copy the map information back into the file

The garbage at the back can be ignored in step #8 due to the way the tools allocate memory--each array is created at startup to be the maximum possible size, and then they aren't ever resized. To keep track of actual data sizes, the tools use a second variable that keeps track of the count in the array. I don't need to resize the array, just fix the size variable.

I guess I answered, "what does it do" literally in the previous post, but here's why it makes a difference:

The hlcsg and hlbsp steps that are responsible for creating level geometry (planes, nodes, clipnodes) have a "fill" step where the outside of the world is stripped away from the node lists. This is one reason why maps that leak have significantly higher r_speeds even if you don't run vis on the leakproof copy. Many of the faces on each brush are also thrown out if they aren't part of the final visible hull--stuff like faces that are flush with another brush that is part of the worldspawn.

Although faces and nodes have always been removed by the tools, the planes that help to define them (by saying which way each face or node portal points) are never removed even if all faces and nodes that use the plane are removed. Many planes are artifacts of faces and nodes that were already removed by the tools. The extra information provided by the removed planes is required by the tools, so the optimizer shouldn't be run if you're going to do extra lighting work or compile with --onlyents to tweak your map, but the game itself only references the planes when looking up face and node information -- if there isn't a face or node that needs it, a plane is just taking up space when running the game.

Hopefully the explanation above combined with the literal "whatsitdo?" will give you enough background to consider using the optimizer.

Well...you are entirely too smart. Thanks...I have a map in dod beta 4 and I think this helps in the bsp size reduction and r_speed arenas...though I'm not smart enough to understand why my r_speeds got a little better other than to think the engine was attempting to draw some of the planes pre-optimization that got removed after optimization that it didn;t need to draw anymore...but the map looks fine (no gaping holes...lol) and seems to run a bit smoother now so whatever intelligent magic was involved I appreciate it.

I have some doubt that this optimizer is so good. I compiled my map with those tools and dragged my bsp and it optimized it but I noticed that the planes were ok and r_speeds lower but I think the map lagged much more than ever before maybe FPS was dropped. or what I am doing wrong.

hmm should not happen. just test it a bit. compile the map without optimizing it, run around and note r_speeds and FPS, then optimize it and check again. that simple if you can proof there is a problem all the better so cagey has the time to fix it.

Hmmm, I just tried using the optimizer, having added a load of extra detail to a map that was already pushing the (original) planes limit; it compiled OK with the new compile tools, but when it came to optimizing, it crashed (Windows XP reported a problem and closed it down) :/

Here's the dialog box that came up, along with the compile/optimising info:



Any ideas about what could be wrong - could it be because the clipnodes limit is also perilously full? Or perhaps it's because of too many entities or something (I managed to run the map in NS, despite having gone over the regular planes limit without optimising, and it only seemed to be func_illusionary entities that were b0rky)? My map properties are here, in case it's any use:



Any advice/help will be much appreciated, as I don't want to have to scrap my new biodome brushwork to get back below the planes limit :/

I noticed that opt_plns.exe crashes if it can't find the file you specify.. might wanna check to make sure it's correct

I'm doing it by dragging the .bsp file onto the optimizer program, and they're both in the same directory, so I don't think it's that.

Windows XP can cause spurious crashes at random times with a LOT of command-line tools, so it may just be Windows XP trying to 'protect' your system from the utility. On the other hand, maybe the optimizer doesn't support larger than 32k arrays? *grins mischeviously* Would be amusing at the very least.

QUOTE (WolfWings @ Feb 4 2003, 06:47 PM)

On the other hand, maybe the optimizer doesn't support larger than 32k arrays? *grins mischeviously* Would be amusing at the very least.

That would be pretty funny . The hard limit for the optimizer is set to 64K planes, same as the 1.7p build of the tools... I just hardcoded the display to be relative to 32K since that's the point where the game starts acting strange.

Cadaver, would you be willing to let me use your bsp to try to duplicate the crash? PM me if you'd like me to check it over.

I'm on Win2000 here, so it could be a difference in the OS, but that wouldn't be my first inclination. The fact that the map information is displayed before the program dies means that it's reading the file from disk successfully. I'm guessing the crash is probably happening at either step 2 (see my earlier post in this thread for my description of the steps involved) when I'm attempting to allocate a large array to the stack or step 6 when I'm allocating a large chunk of additional memory. Since it's not displaying the optimized plane count, it's definately crashing before step 8.

If anybody else is having crash issues, I'd like to get as detailed a bug report as you can give... Just PM me the details and I'll see if I can track down the problems. I'll fix the missing file crash ASAP, and post v1.2 of opt_plns.exe here.

Version 1.2 of opt_plns.exe is now available, fixing the missing file crash and also fixing a bug which may cause data corruption at plane #0. This is a recommended upgrade for anybody who has downloaded a previous version.

EDIT: By missing file crash I was referring to the crash on bsp file not found. I should say what I mean more often Apologies about any confusion, and as venomus notes in the next post, the .Net framework DLLs are still required.

EDIT: this version has been superceded by version 1.4, which is available on the 5th page of this topic. Version 1.4 removes texdata, visdata, and lightdata limits and contains cumulative bugfixes from 1.2 and 1.3.

Hi, I'm still trying to get the tools work so I can resurrect an ancient map of mine, abandoned when it became too complex. I'll post more details on these problems later.

However, I'm just pointing out that although version 1.2 of opt_plns fixes the missing file crash, the modified tools still seem to require that file. If anyone is looking for the the two dlls needed and is running Windows XP, try looking in WINDOWS\Microsoft.NET\Framework\v1.0 . I just searched my hard drive for the files. Then copy the files (msvcr70.dll and msvcp70.dll) to your WINDOWS\system or WINDOWS\system32 folder (for me it was the latter).

Also, this is just a minor inconvenience, opt_plns requires you to press a key after it finishes, which seems to screw up my batch files. At the moment the batch file looks something like this:

hlcsg.exe bigmap.map
hlbsp.exe bigmap.map
hlvis.exe bigmap.map
hlrad.exe bigmap.map
opt_plns.exe bigmap.bsp
copy bigmap.bsp C:\HL\valve\maps
C:\HL\hl.exe +map bigmap.bsp

The sequence stops after I have to press the key. Can the batch be scripted to avoid this, or the program changed to have the key pressing optional or something?

QUOTE (venomus @ Feb 5 2003, 08:04 AM)

The sequence stops after I have to press the key. Can the batch be scripted to avoid this, or the program changed to have the key pressing optional or something?

I'll add a command line switch to make the key press optional in the next public release (which should happen very soon, possibly later today. I've asked Cadaver to double-check a bugfix for his problem before I make it public).

OK, turns out the map I was trying to resurrect also had too many clipnodes, but for a variety of reasons this was hard to detect. Still I have noticed as much as 4/5 planes were culled on some other maps I tried the tool on, and clipnodes can be economised relatively easily. These tools are excellent .

Version 1.3 is now available... this version fixes a crash bug (discovered by Cadaver) that may occur on maps that are over the original 32K plane limit, and also introduces a command line switch, -nopause, that turns off the keypress prompt when the program finishes. The -nopause option was requested for using opt_plns in a batch script.

If you haven't had any problems with 1.2, you can wait before upgrading. People using 1.1 should upgrade to 1.3 to fix the plane #0 bug corrected in 1.2.

EDIT: this version has been superceded by version 1.4, which is available on the 5th page of this topic. Version 1.4 removes texdata, visdata, and lightdata limits and contains cumulative bugfixes from 1.2 and 1.3.

i found that there is an error when trying to compile a map with -subdivide 256. so i coded this tool which fixes this problem. just
run it before the compile process starts like "fixmap mymap" and there wont be any errors

now you can truely reduce your w_poly a lot as you can see here:

http://www.jobvdsande.com/~lumpn/de/tutori...ials/speeds.htm

gnah i know it's german, sorry folks but the text just sais nothing else than i did here, just look at the images

/me lumps you!

hmm, seems like there is gonna get lots of tools soon, maybe code LumpNs program into the end of XP-Cageys optplns.exe(If thats the right exe name)?

One more request (thanks for implementing the last one so quickly). Could you make it so the opt_plns output can be appended to mapname.log?