MineCraft (MineTest) work in progress help need it

Posted on: 2021/1/4 12:09 #1

Home away from home

@All

Tried for a while to make Minecraft working on AmigaOS4 (that one use Irrlicht which I port a year ago and which used in HCraft, SuperTuxKart, and NightOfTheZombies games).

Thanks to Salas00 for a fix in our GCC I can go further now, and so, there is at least something on the screen (click open in a new window for full size):

Resized Image

So far there are major issues I have for now (and If anyone can help me with it (i can donate for of course), that will be very helpful!) :

~~Issue #1. Semaphores and Threading + exit() call from the wrong process + other related bits.~~ FIXED !!

Fixed by Salas00's POSIX-on-os4 semaphore implementation: https://www.dropbox.com/s/43b4g16kvhd7 ... emaphore-amigaos4.7z?dl=0

+ by few fixes to amigaos4-native-threading implementation for our GCC (one by salas00 and one by Capehill)

~~Issue #2. Invalid UTF8-string everywhere~~ FIXED !

Fixed by replacing WCHAR_T on UCS-4 in string.cpp (c) salas00

]~~Issue #3. Failing unit-tests for serialization.~~ FIXED !!

by the usage of proper GCC flags(added -fno-math-errno -fno-trapping-math -ffinite-math-only -fno-signed-zeros).

Issue #4. Failing unit-tests for killthread()
That one cause lot of "Waiting for thread" on serial.

Issue #5. Failing unit-tests for servermodmanager

Issue #8. Port for real network-based code. Need to remove ipv6 and use directly Roadshow (so proto/bsdsocket.h), as threaded curl use it as well.

It seems it still give us some issues with threading (crashes in the __gthread_entry() + _ZL15eh_globals_dtorPv(), following by lot of "Waiting for thread" on serial).

~~Issue #9. On running have "please insert volume minetest_menu_footer.png in any drive" and "please insert volume minetest_menu_header.png in any drive"~~. FIXED !! By getting rid of -lunix and replacing posix rename()/delete()

~~Issue #10. Make RecursiveDelete() works~~ FIXED !!
That one: https://github.com/minetest/minetest/b ... ster/src/filesys.cpp#L290

fixed by getting as base win32 version, and replace few bits so IDOS->Delete() is used for.

Edited by kas1e on 2021/1/4 19:26:02
Edited by kas1e on 2021/1/4 21:59:33
Edited by kas1e on 2021/1/6 16:12:01
Edited by kas1e on 2021/1/7 15:20:47
Edited by kas1e on 2021/1/7 15:23:48
Edited by kas1e on 2021/1/7 15:24:21
Edited by kas1e on 2021/1/9 12:38:19
Edited by kas1e on 2021/1/9 15:04:34
Edited by kas1e on 2021/1/10 20:21:37
Edited by kas1e on 2021/1/10 20:27:08
Edited by kas1e on 2021/1/10 20:28:11
Edited by kas1e on 2021/1/13 16:17:12
Edited by kas1e on 2021/1/13 16:19:14
Edited by kas1e on 2021/1/13 20:44:48

Join us to improve dopus5!
AmigaOS4 on youtube

Re: MineCraft (MineTest) work in progress help need it

Posted on: 2021/1/4 12:36 #2

Just can't stay away

@kas1e

I wrote a quick posix semaphore implementation for a game port that I haven't finished.

As I didn't get the game fully compiled it is completely untested.

You can find the code for it here:

https://www.dropbox.com/s/43b4g16kvhd7 ... emaphore-amigaos4.7z?dl=0

I'm not sure if the locking in sem_getvalue() is needed. I added it just to be safe, but this particular operation doesn't really seem safe to me even with the locking (because of multitasking there is no guarantee that the value is still correct after the function has returned).

Re: MineCraft (MineTest) work in progress help need it

Posted on: 2021/1/4 15:23 #3

Home away from home

@Salas00

Oh, thanks a bunch, pretty good start!

Through it missing (at least for that game) those functions:

sem_destroy() and sem_timedwait().

Join us to improve dopus5!
AmigaOS4 on youtube

Re: MineCraft (MineTest) work in progress help need it

Posted on: 2021/1/4 15:49 #4

Home away from home

@salas00
As for threading, just including of our pthread.h mostly did the trick, but we seems do not have those 2 functions from "sched":

sched_get_priority_min() and sched_get_priority_max().

EDIT: Oh, new SDK files from update2 (pthread.h and sched.h) did the trick with the sched functions!

So, what only left is sem_destroy() and sem_timedwait().

Join us to improve dopus5!
AmigaOS4 on youtube

Re: MineCraft (MineTest) work in progress help need it

Posted on: 2021/1/4 16:21 #5

Home away from home

@Salas00
Despite missing those 2 functions, i tried to compile semaphore.c and it brings me a little bunch of undefs, which i fixed by adding those include :
proto/dos.h
errno.h
limits.h

And i left with:

Quote:

$ ppc-amigaos-g++ semaphore.c -c -o semaphore.o
semaphore.c:13:15: error: expected constructor, destructor, or type conversion before ‘(’ token
_Static_assert(sizeof(sem_t) == sizeof(isem_t), "Internal and public structures are not the same size!");
^
semaphore.c: In function ‘int sem_wait(sem_t*)’:
semaphore.c:54:12: error: ‘EBREAK’ was not declared in this scope
errno = EBREAK;
^~~~~~

EBREAK can't find in our includes, while EOVERFLOW is in sys/errno.h, but not EBREAK.

And the first errors seem code error? I tried with -std=c++11 and -std=c++17 just in case, but it seems there some typing error or something? Sure not the line which is important for code itself, but just curious wtf.

Join us to improve dopus5!
AmigaOS4 on youtube

Re: MineCraft (MineTest) work in progress help need it

Posted on: 2021/1/4 17:15 #6

Just can't stay away

@kas1e

I made some code fixes and added the sem_destroy() and sem_timedwait() functions.

Because sem_init() doesn't allocate anything the sem_destroy() is not needed and can be implemented as a no-op.

Link is same as above:
https://www.dropbox.com/s/43b4g16kvhd7 ... emaphore-amigaos4.7z?dl=0

Re: MineCraft (MineTest) work in progress help need it

Posted on: 2021/1/4 17:25 #7

Home away from home

@salas00
Tried to compile those ones, and:


$ ppc-amigaos-g++ -c semaphore.c -o semaphore.o

semaphore.c:23:15: error: expected constructor, destructor, or type conversion before ‘(’ token

 _Static_assert(sizeof(sem_t) == sizeof(isem_t),

               ^

semaphore.c: In function ‘int sem_timedwait(sem_t*, const timespec*)’:

semaphore.c:97:33: error: invalid conversion from ‘APTR’ {aka ‘void*’} to ‘MsgPort*’ [-fpermissive]

   mp = IExec->AllocSysObjectTags(ASOT_PORT,

        ~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~

    ASOPORT_Signal, SIGBREAKB_CTRL_C,

    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    TAG_END);

    ~~~~~~~~

semaphore.c:100:33: error: invalid conversion from ‘APTR’ {aka ‘void*’} to ‘TimeRequest*’ [-fpermissive]

   tr = IExec->AllocSysObjectTags(ASOT_IOREQUEST,

        ~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~

    ASOIOR_ReplyPort, mp,

    ~~~~~~~~~~~~~~~~~~~~~

    ASOIOR_Size,      sizeof(struct TimeRequest),

    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    TAG_END);

    ~~~~~~~~

Join us to improve dopus5!
AmigaOS4 on youtube

Re: MineCraft (MineTest) work in progress help need it

Posted on: 2021/1/4 17:30 #8

Just can't stay away

@kas1e

That's because you use g++ on non-C++ code.

Re: MineCraft (MineTest) work in progress help need it

Posted on: 2021/1/4 17:31 #9

Just can't stay away

@kas1e

-fpermissive issues should be simple to fix. You need to to cast those pointers to actual types. You can use:

mp = (struct MsgPort *)IExec->Alloc...

Or (C++):

mp = static_cast<MsgPort *>(IExec->Alloc...)

Re: MineCraft (MineTest) work in progress help need it

Posted on: 2021/1/4 17:32 #10

Home away from home

@Salas00
Probably i can just "fix it" like this:


mp = (MsgPort *)IExec->AllocSysObjectTags(ASOT_PORT,

            ASOPORT_Signal, SIGBREAKB_CTRL_C,

            TAG_END);

        tr = (TimeRequest *)IExec->AllocSysObjectTags(ASOT_IOREQUEST,

            ASOIOR_ReplyPort, mp,

            ASOIOR_Size,      sizeof(struct TimeRequest),

            TAG_END);

Join us to improve dopus5!
AmigaOS4 on youtube

Re: MineCraft (MineTest) work in progress help need it

Posted on: 2021/1/4 17:37 #11

Just can't stay away

@kas1e

If you compile semaphore.c as C++ (using g++ instead of gcc) you need to remove the 'extern "C" { ... }' from semaphore.h as well.

Re: MineCraft (MineTest) work in progress help need it

Posted on: 2021/1/4 19:03 #12

Home away from home

@Salas00, Capehill

Yeah, got it for c++ as well. All compiles fine now! Thanks a bunch!

Through, I still have that strange issue #3, with 4 error windows from newlib.library saying that exit() of process xx called from wrong process xx, using IExec->RemTask(NULL).

Maybe that side back of the issue #2, as in the log i have exactly 4 strings of such kind:

Quote:

2021-01-02 21:59:37: INFO[Main]: Couldn't convert UTF-8 string 0x4d61696e204d656e75 into wstring
2021-01-02 21:59:37: INFO[Main]: Couldn't convert UTF-8 string 0x4d696e6574657374 into wstring
2021-01-02 21:59:37: INFO[Main]: Couldn't convert UTF-8 string 0x352e302e31 into wstring
2021-01-02 21:59:38: INFO[Main]: Couldn't convert UTF-8 string 0x into wstring

Imho firstly need to deal with those errors, and if errors from newlib didn't go, then investigate after.

And that happens because of that string.cpp's code:
https://github.com/minetest/minetest/b ... aster/src/util/string.cpp

See there at top bool convert(...) function which use iconv() for conversion like iconv(cd, &inbuf_ptr, inbuf_left_ptr, &outbuf_ptr, outbuf_left_ptr);

And that function is failed when it called from std::wstring utf8_to_wide(const std::string &input){...}
in the same file when doing:

Quote:

if (!convert(DEFAULT_ENCODING, "UTF-8", outbuf, outbuf_size, inbuf, inbuf_size)) {
infostream << "Couldn't convert UTF-8 string 0x" << hex_encode(input)
<< " into wstring" << std::endl;
delete[] inbuf;
delete[] outbuf;
return L"<invalid UTF-8 string>";
}

And I have exactly that "invalid UTF-8 string".

I put some printfs right before call to that "convert()" function and "inbuf" there surely correct, i can just prinfs("%s\n") it. The same it correct if I put printf in the convert() function itself. But something going wrong exactly when we call "convert()" and it didn't translate the things as expected.

Maybe iconv() call is different on our side? Strangely that i have no needs to do -liconv on linking, it takes from somewhere else in SDK, so maybe just different implementations?

Btw, i also had to add in the string.h that:


#ifdef __amigaos4__



#include <string>

#include <sstream>

 

namespace std { 

    template <typename T> string to_string(const T &n) { 

        ostringstream strm; 

        strm << n; 

        return strm.str(); 

    } 



    typedef basic_string<wchar_t> wstring; 

    typedef basic_ostream<wchar_t> wostream;

    typedef basic_ostringstream<wchar_t> wostringstream;

    typedef basic_stringstream<wchar_t> wstringstream;

    

} 

#endif

But not sure it may be related there...

Any suggestions are very welcome of course :) Just want this damn Minecraft come to us.

Join us to improve dopus5!
AmigaOS4 on youtube

Re: MineCraft (MineTest) work in progress help need it

Posted on: 2021/1/4 20:54 #13

Just can't stay away

@kas1e

Try changing the value of DEFAULT_ENCODING from "WCHAR_T" to "UCS-4" or "UCS-4BE".

Re: MineCraft (MineTest) work in progress help need it

Posted on: 2021/1/4 21:58 #14

Home away from home

@Salas00
Quote:

Try changing the value of DEFAULT_ENCODING from "WCHAR_T" to "UCS-4" or "UCS-4BE".

Yeah, UCS-4 did the trick! Messages all everywhere and looks correct.

I was in the hope it will auto-fix those newlib errors as well, but nope :( Check out how they look like:

Resized Image

Strange that exactly 4 windows. And exit() of process XX for all 4 are the same XX. But from a wrong process are different (increased +1 for each).

All I can see in the console is :

Quote:

terminate called after throwing an instance of 'std::system_error'
What() : terminate called recursively
Device or resource busy

Wtf it means dunno

EDIT: what i find is that "std::system_error" is coming from the thread.cpp in the Thread::start():


bool Thread::start()

{

    MutexAutoLock lock(m_mutex);



    if (m_running)

        return false;



    m_request_stop = false;



    // The mutex may already be locked if the thread is being restarted

    m_start_finished_mutex.try_lock();



    try {

        m_thread_obj = new std::thread(threadProc, this);

    } catch (const std::system_error &e) {

        return false;

    }



    // Allow spawned thread to continue

    m_start_finished_mutex.unlock();



    while (!m_running)

        sleep_ms(1);



    m_joinable = true;



    return true;

}

So it indeed a catch, is it mean some threads can't be created? And newlib just throw errors about RemTask(NULL) being called from the wrong process?

EDIT2: Find out that minecraft can be running with "--run-unittests" so can be checked all the code. And when it just tryied to test module TestThreading, it then shows me the same error from newlib.library:

Quote:

"exit() of process 214 called from wrong process 216, using IExec->RemTask(NULL). Process: "New Process".

and in the console i have:

Quote:

======== Testing module TestThreading
terminate called after throwing an instance of 'std::system_error'
what(): Device or resource busy

And testthreading code are there:

https://github.com/minetest/minetest/b ... ittest/test_threading.cpp

Its uses and threads and semaphores. So I assume something there still.

And interestingly it didn't skip that error. I.e. test stops here, without bringing "fail" and just halt the console.

Edited by kas1e on 2021/1/5 8:32:51
Edited by kas1e on 2021/1/5 8:58:05
Edited by kas1e on 2021/1/5 9:05:41

Join us to improve dopus5!
AmigaOS4 on youtube

Re: MineCraft (MineTest) work in progress help need it

Posted on: 2021/1/5 9:07 #15

Just can't stay away

@kas1e


try {

        m_thread_obj = new std::thread(threadProc, this);

        // try logging here

    } catch (const std::system_error &e) {

        // try logging here

        return false;

    }

The exception was catched here so I think the problem is somewhere else (an uncaught exception: try-catch missing). If you put a log after thread creation, will it appear or not?

Device or Resource busy: would this refer to errno EBUSY somewhere?

Re: MineCraft (MineTest) work in progress help need it

Posted on: 2021/1/5 9:10 #16

Home away from home

@Capehill

Check my last edit in previous post as well : i found there some simple test case for threading/semaphores, so not involving game's code : https://github.com/minetest/minetest/b ... ittest/test_threading.cpp

And it also brings me such an error. So it definitely threading/semaphores issue by itself.

Will try to rebuild that test as standalone, so we can then step by step reduce things and find out the bastard

Edited by kas1e on 2021/1/5 9:30:40

Join us to improve dopus5!
AmigaOS4 on youtube

Re: MineCraft (MineTest) work in progress help need it

Posted on: 2021/1/5 14:38 #17

Home away from home

@Capehill
Making standalone example a bit harder than i expect, so firstly going your way, doing just that:


try {

        m_thread_obj = new std::thread(threadProc, this);

        printf("log 1\n");        

    } catch (const std::system_error &e) {

        printf("log 2\n");

        return false;

    }

And I have in return when this newlib. library error comes up only "log 1" printf, never see "log 2" one. I.e.

Quote:

=============Testing module TestThreading
log 1
terminate called after throwing an instance of 'std::system_error'
what(): Device or resource busy

+ error from newlib.library saying that "exit() of process 341 called from wrong process 343, using IExec->RemTask(NULL). PRocess: "New PRocess".

Quote:

Device or Resource busy: would this refer to errno EBUSY somewhere?

Dunno, but i just search on that word, and that what find in their semaphore.cpp:


Semaphore::~Semaphore()

{

#ifdef _WIN32

    CloseHandle(semaphore);

#else

    int ret = sem_destroy(&semaphore);

#ifdef __ANDROID__

    // Workaround for broken bionic semaphore implementation!

    assert(!ret || errno == EBUSY);

#else

    assert(!ret);

#endif

    UNUSED(ret);

#endif

}

Maybe something about our sem_destroy() which just "return 0;" ?

Join us to improve dopus5!
AmigaOS4 on youtube

Re: MineCraft (MineTest) work in progress help need it

Posted on: 2021/1/5 15:20 #18

Just can't stay away

@kas1e

std::thread::join (for example) can also throw std::system_error, and join is called without try-catch block:

https://github.com/minetest/minetest/b ... threading/thread.cpp#L151

You could try to surround the "join" call with before/after logs, or just add the try-catch.

https://en.cppreference.com/w/cpp/thread/thread/join

EDIT: EBUSY might trigger from the GCC thread patch. In try_lock, is there also increment missing on succesful attempt:

https://github.com/sba1/adtools/blob/m ... s-thread-model.patch#L732

?

Edited by Capehill on 2021/1/5 16:32:12

Re: MineCraft (MineTest) work in progress help need it

Posted on: 2021/1/5 17:53 #19

Home away from home

@Capehill
Quote:

std::thread::join (for example) can also throw std::system_error, and join is called without try-catch block:

https://github.com/minetest/minetest/b ... threading/thread.cpp#L151

You could try to surround the "join" call with before/after logs, or just add the try-catch.

Added printfs around "join" in Thread::wait as you say, and there is output:

Quote:

=============Testing module TestThreading
log 1
terminate called after throwing an instance of 'std::system_error'
what(): Device or resource busy
before join
<newlib.library error about exit() from wrong process using IEXEC->RemTask(NULL)>

And I never see "after join" printf, output just stops
So .. in the end, it join() fail?

Quote:

EDIT: EBUSY might trigger from the GCC thread patch. In try_lock, is there also an increment missing on successful attempt:

https://github.com/sba1/adtools/blob/m ... s-thread-model.patch#L732

You mean mx->u.i.acquired++; ? Can't say (suck at such low-level things), but have an idea how to test? I can easily re-build my GCC with any change.

Join us to improve dopus5!
AmigaOS4 on youtube