GL4ES: another OpenGL over OpenGLES2 emulation - some tech. info and porting progress

Quote:
Correct. No need.

Quote:
Please reply in the bug ticket. It gets too complicated if things are communicated all over the place.

Hans

@thread

If all else fails try : PVS-Studio Analyzer

https://www.viva64.com/en/pvs-studio/

It catches hard to find bugs, inefficiency.

@angelheart
is it free?

Update : https://www.viva64.com/en/b/0457/

@all
There was another round of fixes in warp3dnova, so issues with sky overlaping in quake3 and wrong alpha colors in neverball are gone.

Through that issue which we have when enable extensions and have mess all over the screen still here, and probably i will just try to strip down source of the quake3 till the moment when bug still triggered, but sources left will be small as possible.

But glBegin/glEnd route works fine, and all renders correctly now. And so i compare minigl version and latest gl4es one , and in the same 1600xx1200, with all on maximum, 32 bit , etc i have in timedemo1/demofour for minigl: 44.5 fps, for gl4es version 40.5 fps. So, almost on pair, but not faster surely.

But (!), rendering in the gl4es version is more accurate and better in compare with minigl version. For example, there is places in minigl version where it just all renders wrong. Like first level where we watch over the mirror.

See there in minigl version at first no mirroring, and when we come close to mirror, there ugly black triangles on floor.


(press on open in new tab for fullsize)


(press on open in new tab for fullsize)


And thats gl4es version, where all looks correct:


(press on open in new tab for fullsize)


(press on open in new tab for fullsize)


Or for example some portal: In, minigl version, there again ugly black trianges visibly, and rendering of what we see ghosted in portal not correct in whole:


(press on open in new tab for fullsize)

And gl4es version, where all renders correctly:


(press on open in new tab for fullsize)

Surely that all not big deal today, but just point out on those tasty little details which some of us love :)

Also ptitSeb (gl4es author), works almost every day on improve gl4es, so good for us.

What is left now, is to deal with that strange bug when we have mess on screen when enable extensions in quake3 (and the same bug we have in irrlicht engine), and, if after that one fixed, there still will be memory trashing about which we talk few pages before, then that one as well.

@kas1e

Perhaps for the MiniGL bug you may contact Daytona, recently he still worked on the MiniGL side

http://www.hyperion-entertainment.biz ... niGL/branches/updates-kc/

@samo
Quote:

Perhaps you may contact Daytona yourself, if you need those fixes in minigl :)

Quote:

I can read his Facebook posts too :)

Quote:

And i know where minigl placed, too :))

@kas1e

Mmm how ironic you are ...
I asked you because you are involved more than me and for sure you know exactly what to report in detail .. while i'm not

And MiniGL is aniway usefull for the entire community ..

>ptitSeb (gl4es author), works almost every day on improve gl4es

ptitSeb ? (=small Sebastian) so he is french

Found an article about him
"Meet an ODROIDIAN Sebastien Chevalier (@ptitseb)"
page 32
https://magazine.odroid.com/wp-content ... DROID-Magazine-201610.pdf

I fully agree about "Learning programming"

@samo79
Quote:
Actually the recent bug fixes were only a side-effect of what I was really doing:
adding the std.PPC<->SPE-ABI bridge to the libs so that programs compiled for std. PPC could attach to (at least internally) native SPE MiniGL / mglut libs.
Such a bridge is always necessary if std.PPC-code shall talk to PPC-SPE-code and float-parameters are involved, because the PPC-FPU- and PPC-SPE-ABIs are incompatible when it comes to floats, naturally, because usually float parameters are passed via FPU-registers, which simply don't exist on PPC-SPE (on such systems such parameters are expected to be in GPRs instead).
Anyway, I simply fixed what I saw when doing some cleanup. I don't plan to otherwise invest much time into regular MiniGL, especially because my own MiniGL replacement lib is progressing well (some of the MiniGL fixes also came up because I noticed differences when comparing my lib with the output of std. MGL, which in some cases was due to std.MGL bugs).

That Q3-issue with the broken geometry in the mirrors / portals here smells like clipping-bugs in MGL. However, I noticed that W3D for e.g. old Radeons and W3D for SI behave different when it comes to clipping (and I don't mean R200-guard-band-related effects). It may also be that this here is an incompatibility (compared to other earlier W3D implementations) inside the W3DSI-driver.

EDIT: no, I just checked it on my sam440, it happens there too, so you can safely asume that it's a MiniGL-bug, most likely related to clipping.

@all

I just noticed someone on amiga-ng.org claiming that GL4ES is slower than MiniGL. It might look that way based on what kas1e has said, but let's look deeper.

Quake III on GL4ES may have achieved 40.5 fps in the time demo vs 44.5 in MiniGL. However, his test uses the slow glBegin()/glEnd() render path due to a bug in GL4ES preventing the use of compiled vertex arrays. It'll be interesting to see what the performance is once that bug is squashed and compiled vertex arrays are used.

Which one is faster will probably depend on what's being rendered. I expect GL4ES will have the edge when large meshes with lots of vertices. When it comes to rendering lots of tiny meshes, then the bottleneck is how many render ops/s the system can manage. The W3D_SI Warp3D driver has code to accumulate lots of tiny meshes into batches precisely reduce the effect of that bottleneck. So, it may have an edge rendering smaller meshes. Exactly what size mesh is the threshold where it flips in GL4ES' favour is uncertain at this point (and is likely to change over time).

Hans

EDITED: Corrected based on kas1e's response.

@Hans
40.5 vs 44.5 its of course when both versions runs without enabled extensions, so glBegin/glEnd in both cases. With enabled extensions minigl scored about 90 fps if i remember right, but i dont point on it, as gl4es version cant be running with enabled extensions due to that hardcore bug, which still unclear where to look, as seems from ogles2 side all is ok. I only can try to reduce q3 code.

From another side, if we take neverball/neverputs games, they faster a lot in compare with minigl versions. I.e when minigl give 20 fps, gl4es give 40-50 fps

@kas1e

Okay, I've corrected my post above. It's interesting that Neverball/NeverPutt is faster with GL4ES, as I assumed that its models also have a low-polygon count.

BTW, any progress in figuring out why compiled vertex arrays get all messed up?

Hans

@Hans
Quote:

When ptitSeb checking it all before, we come to conclusion that it's the glVertexAttrib with GL_UNSIGNED_BYTE and Normalize TRUE that break things. It's use for the Colors (and it's converted to GL_FLOAT when using glBegin()/glEnd() code path)..

To check that theory we just made a workoround like this:



And it make everything works. And q3 with enabled extensions and irrlicht engine.

Then, i bring that all info to Daniel, but that didn't help us, as from ogles2 side still everything looks correct. And, Daniel explain as well that if it was issue with normalisation, distortions can be of color only, not of geometry.

But that again, can be not only normalisation, but normaisation + something else + whatever else..

In end of all, i just create another bz on the gl4es page (just to summorize everything about that issue):

https://github.com/ptitSeb/gl4es/issues/61

Plz, take a look on that.

If you want to check it for real yourself, there is 2 test cases.

1. Quake3

That one you can download from Daniel's ftp from REPORTS_HERE/q3_test.lha , also take q3_batching_fixed.zip from the same place. Then, once you will enable extensions in q3 (and quit/run again), you will see everything yourself (don't forget to hit "enter" few times after first anim movie disappear).

2. Irrlicht engine's hello-world

You can download it from the same Daniel's ftp, from REPROT_HERE/irrlicht_example.lha. There you will find 2 binaries. One called "non_hacked", its original, which show bug. And another one, with that workoround i post above, which render all correct. So you may compare them too. Maybe, irrlicht engine's test will be easer than q3 one.

Don't forget to unpack them as it, and run it from the same "bin/amigaos4" directory, as it will search for data from ../../media/.

That test case probably will be more easy than q3. At least size is smaller.

@kas1e

I can't comment on the internals of GL4ES and ogles2.library. However, the W3DN_SI driver currently can't handle anything other than 32-bit datatypes (it says so in the readme). So that excludes GL_UNSIGNED_BYTE.

The problem is that Southern Islands and newer GPUs are little-endian only while our CPUs are big-endian. So the driver has to convert the endianness as its copied to the GPU. Right now it assumes that everything is 32-bit, and it probably returns an error if you try to use 8/16-bit vertex data.

Writing a system to perform the correct endianness swapping for the data is on the to-do list. It'll be a bitch to get right, because it's got to handle interleaved data, structures with different datatypes, etc. It's one of those things I wish I could get someone else to write...

Hans

@Hans
In case with gl4es: everything converted to 32 (you may remember i post that info in alpha bug bz). So, that 8/16 little/big endian differences there also not what cause those distortions. Its something else. You may try to run those test cases, maybe you will see something in nova over your debug environment

@kas1e
Sorry, but I really don't have the time to be debugging other people's code right now.

All I can say is that the Warp3D Nova driver code handling vertex inputs is pretty straightforward. So long as you're feeding it with floats and have given it the right pointers, strides, etc., it works.

Hans

Hello
The "quake3 mirror distorsion bug" happen with waZp3d too so certainly is a minigl bug
Next week I will try to find what cause it as wazp3d got lots of debugging features

@Hans
I ask ptitSeb about "does gl4es convert eveything for 32bit?", and yes, except for this one (and technically, it's only accepted when size=4, so RGBA color fit in a 32bits data space, but yeah, it's 8 bits data basically).

But, GL_UNSIGNED_BYTE are no "endian-sensitive": there are the same in big endian and little endian, so it should not be an issue, unless GL_UNSIGNED_BYTE are indeed not supported at all by Warp3d driver because only 32bits data is implemented.

ptitSeb can add workoround to gl4es for example, so to be sure no GL_UNSIGNED_BYTE are used (of course, that will convert a few data, so it would be better if Warp3d accept 8bits datas, speed-wise).

Btw, there is also another case of non 32bits data in gl4es: the elements indices used in glDrawElements can be GL_UNSIGNED_SHORT (so 16bits), but this seems supported (?) or nothing would work at all. Strange through, if , as you say only 32bit supported.

So.. as 8bit are the same for little/big endians and if warp3d nova have no support for, probably adding just 8 bit will be easy ?

There are the question to awnser to understand what the next step should be.

Does OGLES2 transfert GL_UNSIGNED_BYTE as is or does it transform to GL_FLOAT ?

if NO: can GL_UNSIGNED_BYTE (with normalization) be implemented in Warp3d.

If YES: there is something odd in that conversion

if NO: ptitSeb can made a workaround in gl4es (unless OGLES2 handled that, as GL_UNSIGNED_BYTE support by GLES2 driver is mandatory).

If YES: lets wait for it.

PS. But any workoround mean loose of speed, of course.

EDIT:
taking another close look at one of the two readmes, in the "known limitations" section of the one, reveals that there is indeed information about such limitations. Namely only 32bit float vertex data is supported, which renders 4x8 bit ubyte color VAs offcially unsupported indeed.

Okay, I have no idea how I could overlook this. Most likely because a) the reason mentioned regarding endianness makes no sense in this case, b) because it (seemed to) worked so far, c) because it's something so basic and d) because we had lots of talk about ubyte colors and the idea of this not being supported at all didnt pop up until now.

So, due to the fact that I overlooked it and did not take care of this limitation (if it exists for real, as being said, so far it seemed to work), I will add a workaround for it to ogles2.
This will probably result in a severe performance penalty, especially if using the lib in a non-VBO-way.

However, now that I became aware of this limitation:
IMHO this should be the very very first entry on the Nova todo-list!
4x8bit color (or similar) VAs are used extremely often to have compact vertex structures, so this can be considered an absolute basic building block.

Also the doc should be edited to not show completely misleading / false information. The doc is what people use when coding and this limitation requires a fat warning note in VBOSetArray "only supports this and that for now".


Before EDIT:
@Hans
Quote:
The very latest Nova 1.53's readme from two days ago suddenly contains a new information saying:

Quote:
So from this it now sounds as if e.g. VA colors composed out of 4 unsigned char / GL_UNSIGNED_BYTE suddenly became unsupported now, excluded en passant with this readme entry.

But: This is real news. OGLES2 uses ubyte VA colors with normalization since day one without issues (well, at least I thought so...).
And so far no readme or doc of Nova contained info saying that this was not supported!
Also, although I'm not using it: until two days ago I also believed that 32bit VA ints would work. The docs didn't give any reason not to think so. So what's presented as news in the latest readme is sth. everybody reading the docs until now expected to work anyway.

The only limitations of Nova of that style that were known until two days ago regarding 32bit was if using the data as indices for vertex data. However, even that limitation has been lifted long ago, so that 16bit indices are possible natively.
And then there was / is the "recommendation" to internally align every VA to 32bit.

That was it. There was zero hint until now that unsigned byte VAs and thus the typical RGBA8 colors won't work or won't work reliably.

Quote:
As kas1e correctly said, there is no endian problem with single unsigned bytes. There is also no endian problem with 4 RGBA unsigned bytes. The client application has to take care that the components are in the correct order, neither ogles2 nor Nova have to be "smart" here.
Endian-issues are no explanation for missing / broken unsigned byte VA support.
Besides that (although not of interest in this case here) the doc for VBOSetArray explicitely says that calling it before VBOLock would "allow the driver to know what endianness conversion to perform beforehand". It does not mention that such endian conversion is not implemented or broken.

Quote:
"Probably"? Until now Nova did not return an error. It simply swallowed such VA colors / VBO layouts and it worked. Well, unless the semi-random vertex-trash e.g. Q3-gl4es generates under certain circumstances and which keeps us puzzled for a long time now is such a "probable" effect...
Which in turn would raise the question why we weren't informed that such VA attributes are potentially broken in Nova despite the tons of discussions / reports on this topic.

Quote:
Now it's even floats only all of a sudden?


Sorry, man, clarification please! Apparently you cannot rely on the information in the docs at all?
Simple question: which are the allowed / (not just probably) functional values for the following parameters of VBOSetArray?

W3DN_ElementFormat elementType:
so far the doc didn't forbid any (the before-mentioned index limitation aside). Nothing about 32bit float only.

BOOL normalized:
for which elementTypes is it working? So far the doc said that it worked for ALL signed / unsigned integer types; actually for everything but floats.

Hopefully this is all just some kind of weird misunderstanding

Thanks,
Daniel

@kas1e
Quote:

ogles2 followed the Nova docs and what seemed to work until now...
As such it does not do any conversion of GL_UNSIGNED_BYTE VAs, of course.

If it turns out that Nova cannot handle this, then I am going to interprete this as a severe Nova bug. I won't add any workarounds in ogles2 for this but wait until it gets fixed, sorry.

EDIT:
see EDIT of previous post.
In the meantime I found out that I really overlooked a limitation mentioned in one of the readmes.
So in contrast to what I said above I will add an internal uchar8 converter to ogles2.
This will stay active until this gets fixed inside Nova.
Don't add such a conversion to gl4es, it really makes no sense to mess around there.

EDIT #2:
The first workaround has been added, check it out on my FTP, ogles2_wtf_ubyte_1.zip
This version will internally convert every GL_UNSIGNED_BYTE VA for client memory usage. For safety I am converting to float internally, I'm not relying on this new 32bit integer support, because I better prefer to do my own normalization and because that way it should work with previous Nova versions too.
Note that I'm only patching GL_UNSIGNED_BYTE for now since this is the one that's used in 99% of the non-32bit use-cases.

Next to come is the patching of VBOs supplied by the client application. This will be really ugly since internally a totally different looking VBO has to be created and maintained as soon as there's at least one GL_UNSIGNED_BYTE VA. :P