My Experience of Cross Compiling; In This Case: Coreutils to Amiga OS 4 on an X1000

===TLDR added at end of this experience===
Go to post #10 for the final result of the particular use case: CoreUtils
Go to post #14 for the questions that I formed after this whole experience.

===Start of original post===
===Goal===
Cross compile and build CoreUtils on an x86 for use on my Amiga X1000 OS 4 FE U2.

===Some Info===
I will not post all commands in here, some obvious ones like copying directories I will ommit. The reason for posting all of this information is purely because of my own experiences: I hate it when searching for information and coming across a potentially great thread that does not contain enough information.

I also create this quasi-step-by-step guide because someone will probably go: NO! Don't do it like that! For example, I may be using an out-of-date guide or using a git-hub location that is way out of date, or, just doing something plain-wrong!

Lines that begin with "PRESUMPTION:" document my thought process and general questions.

===Host Machine===


===Host OS===


===Source Code for CoreUtils===
https://github.com/adtools/coreutils

===Guide(s) I am using for my goal===
I am really only using README.os4 / README and browsing the configure file to glean information.

===Start Doing Things===
I am going to create a CHROOT using Xenial 32-bit because we want to use 32-lib libraries and it comes with GCC version 5.X***; all the packages mentioned as requisites for compiling the cross tool chain match up with Xenial. I do not want to mess around with multi-arch etc and having a CHROOT seems a lot cleaner. It has been a long time since I did this, so there maybe easier or better methods.

*** - I mention this because I need to compile the cross-compiler toolchain, and the prerequisites mentions a need for GCC 5.X 32-bit.

===CHROOT a Xenial 32-bit===
I will create a directory: /var/2xenial32 which will be the root for the 32-bit Xenial.
I will use debootstrap to create a minimal image:


Copy the apt/sources and rename focal to xenial so that I can access the right repos for packages.

===Okay, now I need to get a cross compiler before building CoreUtils===
I will use: https://github.com/adtools/amigaos-cross-toolchain
And the guide for that: https://github.com/adtools/amigaos-cro ... ain/blob/master/README.md.

We will create the gcc 4.2.4 cross-compiler for OS 4 FE U2.

Let me CHROOT into the 2xenial32 and start building the cross toolchain. Before I can even start, let's get all the prerequisites in the README.md file. This is essentially running apt-get install for each of these packages. It also includes running install for a few other packages that are not mentioned in the README file that must have been left out by mistake; I needed to also install subversion and flex.

Also, during step 2 of the README.md:
./toolchain-ppc --prefix=/opt/ppc-amigaos build
the script (toolchain-ppc) will hang when trying to download http://isl.gforge.inria.fr/isl-0.12.2.tar.bz2 because this file no longer exists. So, I just searched for it and hacked in a new url.

Then, I run: ./toolchain-ppc --prefix=/opt/ppc-amigaos build
and wait for a long time, since the host machine is real slow!

And, when it is finished I test the default paths etc for the compiler:

And, to test with -mcrt=clib2:


And:

And, for -mcrt=clib2:


===Peforming a basic test of hello world===
As the heading suggest, I now create a simple application printing out hello world, scp to amiga and run it to ensure that it works so far.

Since the compiler defaults to newlib I compile helloWorld both by default and by appending -mcrt=clib2.

Both binaries work.

===Get back to building CoreUtils===
Since the README.os4 for CoreUtils says:

Quote:

I should probably get libwide and the "current" clib2 from the github. So, let's do that.

===libwide===
Just extract it into the SDK directory.

===clib2===
The issue is that for the documentation readme file for building http://github.com/adtools/clib2 says:
Quote:
So, I am not sure how to get the "netinclude".
Quote:

===start compiling CoreUtils===
Since the default for the X-compiler is not to use CLIB, I am assuming I need to override that and add CFLAGS=-mcrt=clib2 as well as set the --host=ppc-amigaos. The X-compiler needs to be on the path also!

Quote:

But, just to see what happens I try:

And this generates okay, but when I try

I get:


And, when you look at that line you can see:


So, as I already saw in the readme, we should be building with clib2!

Okay, so now I try (after cleaning out directory completely. Essentially, deleting it and copying a new one):

in this case, we get a Makefile again, but:

results in:

So, function 'gethostname' was already:

But in the core-utils directory that I am trying to build:

already exists.
Quote:

===Help===
So, I am just wondering if someone can help point me in the right direction.

Thanks all.

I meant to post this in OS4 Developer.

If a moderator can move it.

Thanks.

Thread moved.

I am really interested in the progress of it. I have a couple of questions:

Quote:

Coreutils are available at http://os4depot.net/?function=showfil ... ent/utility/coreutils.lha
Do you want to make a new compilation because they do not cover any of your needs? Or for some other reason?

Quote:

If you are going to use a cross compiler, why not use a newer version of GCC?

@rjd324
Not of a bIg help but as George poInt out: 4.2.4 Is veeeeery old. Everyone who actIve enough today uses 8.x at a minimum but some of us are already on 10.x and 11.x (everything already in adtools repo on SBA1's github page).

Hey both,

the reason for using GCC 4.2.4 is only because when I download the latest SDK version that is the version that is supplied. I know that a new SDK came out a few days ago, but this all started a week ago.

I assumed that was the version I needed to use. When reading the READMEs I was also getting pointed to 4.2.4.

The reason for compiling CoreUtils from source is purely for educational purposes.

I can also contact Olaf, but I am not sure if he still uses the email address that is referenced in the github page (GMX address).

==EDIT==

I also see with the latest sdk:

@rjd324
Quote:

Official SDK is very much dated. If not George (Walkero) we even will not have that official update of SDK released a few days ago. The reason is that for the "official" release of the new SDK there need to spend a lot of time and testing and stuff and before George stepped in there wasn't anyone willing to do so.

So we all go the easy route: we build new GCC for ourselves from ADTools repo where Sebastian and others made necessary OS4 patches etc. The GCC that goes inside of SDK (that 4.2.4 one) is the same taken from adtools repo.

The only good thing from the current public SDK is AmgaOS4 special includes. They need it much to be in sync with current components. But for GCC and Coreutils : adtools repo is the one to be up2date with GCC and for developers has the ability to use all modern stuff include all those things like c++11 and more.

Quote:

What we need now is updated CoreUtils as well (current ones while works just too old). But that is about making some codework on os4 specific parts to make them work, not just recompilng.


Also if you in interest to build a cross-compiler for Cygwin or msys2 or something to build os4 bins you can check my article about:

https://os4coding.net/blog/kas1e/how-b ... utils-2232-gcc-830-cygwin

@kas1e

Thanks for the info. And is @Walkero the youtuber? Who demonstrates with his X5000. I think he is Greek also...

Okay, so what you guys do is to get the latest SDK but just forget about the old compiler provided and use the new one you mentioned? Personally, I could not care less about C++, but I am wondering why you bother to use the newer C compiler. Of course using a newer compiler means what using a newer compiler always means: that clever people have improved it and better PPC code is generated etc... I am wondering if the real reason is for the ability to use C++.

I am not saying that using the newer compiler is bad! It makes total sense, and it probably helps ports newer software over to OS 4 and allows the C++ obsessed to be happy 😊.

Still, at some point, someone got that version of CoreUtils compiled with 4.2.4 I believe, and I cannot do it because I am metaphorically trying to drill a square peg into a round hole. That is, somewhere with my set up mentioned in the original post, I am not using the correct version of something or missing something in the initial configuration just before generating the Makefile.

I can hack my way passed the conflict of the gethostname function, but then I run into problems with general time/date structures. Like "struct timespec" not having a member "tv_secs".

So somewhere something is not fitting into the round hole and I guess it's either something I missed out in the configuration or a bad version of the SDK against the version of coreutils or something.

So why am I wanting to do it so badly? Because I am like all of you, an engineer, and it was done before.

And, why do I want to do this anyway? Because with the CoreUtils I have from os4 depot, RM -RF does not act as it should. If I have a directory and something inside it then even RM -RF will not work. I had to write my own RM program to get around it and I wanted to contribute by trying to fix it since I have done a bit of classic system development and have used the NDK 1.3 quite a lot. Yes, 1.3 - it's just a rule I have.

And, I could not just use delete because I needed rm since I was porting over a project with rm and different options to rm all over the Makefile. It makes porting an older POSIX project very difficult when RM is used almost all the time.

C++ is pretty pointless on Amiga

@rjd324
Quote:

Yes


Quote:

Yeah )

Quote:

For me personally, C++ needs it because i am more in "semi-modern" things. All the games or 3D stuff i port from wn32/Linux/etc are on C++ long ago. And not just C++ but C++11/C++17 and more. Odyssey is also C++ (and any later versions of Webkit as well). So for me, this is a must.

But for myself i never like C++. it's just for things i am interested in C++ is necessary.

For Alfkil (who does the port of QT) C++ is the preferable way of coding (he wrote some sort of basic debugger for us on C++ and QT port full of C++ of course too).

For others ts a matter of taste. Ones who want to use C other ones use C++.

i can be wrong but if i remember right Hans (2d/3D drivers developer) also does all his work on C++ (but there i am not 100% sure).

For sake of fun and education purposes, i did my local GCC with Objective-C Fortran and ADA support added in. Not for any necessary but for education purposes. See my howto about how to make Fortran work on os4:

https://www.amigans.net/modules/xforum ... m=25&topic_id=7885&order=

But!

Of course, C++ is just ONE language supported by GCC while there is a gazillion of changes in all languages/compilers/generators/linkers and what most important optimization parts in the whole GCC. LTO (which i use too and that for any language be it C or C++) as an example. Another example is much better optimization in the latest GCC: for example, build Odyssey with GCC 8.x in comparison with GCC 10.x saves 3MB of binary data. Taking into account that newer GCC is more "bug-find" better (just switching from one GCC to another helped me to find a few hardcore bugs which weren't noticed in gcc4.x).

But anyway that all depends on what you need. Somebody loves assembler still. Or StormC for old Amigas and continue to blame GCC which today builds so good and optimized code that is of nowhere how it was 20 years ago.

Quote:

Well with native building you always will have issues with core utils sadly. Just they are not Cygwin, and they will be "half wrong" most of the time. For these exact reasons almost all of us use cross-compilers: where everything works as expected and then uploads via FTP (for example) ready binary to amiga for testing.

But f you want t all for sake of education/engineering purposes then yeah that needs to be done for sure :)

Okay, so here is the update:

======Update======
So, the issue regarding gethostname is solved by using -D__NO_NET_API, and Olaf uses this regularly in his other projects.

This means that it should be:

Getting us further to:

We still have this issue regarding "tv_secs" to solve.
Quote:

*** TODO: Figure out the issue with 'tv_secs'.
*** __USE_OLD_TIMEVAL__ is used in a lot of Amiga projects I can see on GitHub. Look at https://wiki.amigaos.net/wiki/Programm ... _-_The_Data_Administrator and search for OLD_TIME - You can see that the Amiga is incompatible. Again, somewhere in the configuration I need to sort something out!
---> Okay, I noticed that at some point (2017-ish) struct timespec was added to CLIB2 include/time.h. I decided to keep the old clib2 as it was and just copy over that particular change to the time.h file. So, at this point we have solved the gethostname and the tv_secs issue. (Hack NUMBER -1)

Clearly, some version of something is incompatible. After that, I received an error about the usage of C99 type of for loop, this can be corrected. But, if the compiler is not C99 then surely coreutils was not built with 4.2.4!

So, I will hack in the pre c-99 change to the for loop in: lib/secure_getenv.c:87 (Hack always needed NUMBER 0)

Now we get the error:

To get around that, I just defined TZSET in or around 30 of time_rz. (Hack always needed NUMBER 1).

Then,
undefined referenced to some wide char function. This confirms that you really do need to link with libwide.a as specified in the readme file.

***Currently trying to add -lwide to the LIBS passed to ./configure.
So, this gets passed the error regarding strings, but the WIDE library relies on 'strnlen' which is not even included in the generated SDK version created via the X-compile tool chain build! This means that we have to update the version of the SDK/clib2 library since newer version include 'strnlen'.

*** So, we will no longer need hack NUMBER -1 since a newer clib2 will also include a definition for struct timespec (something that was added to clib2 also). AND SO, we end up going back to the fact of the bottom of README.os4 for CoreUtils which says: you need libwide and the "latest" clib2 version on github [although, the latest version at the time of that writing for 2017].

==So, let's use the latest clib2 - this may include compiling it===
NOTE: Even the latest clib2 from 53.34 (latest SDK) does not include strnlen. But, the git-hub master version does.

***TODO: Compile the latest clib2 from the git-hub master as referenced in the readme guide. Expect to fix NUMBER 0 and perhaps NUMBER 1 and then rebuild and see what happens. [[[DONE]]]

***** It is clear that this was never compiled with GCC 4.2.4 and since the wide library (libwide.a) uses strnlen it is also clear that we need the non-official clib2 included in even the latest SDK. clib2 on the github does include strnlen. Fact is, when you put the pieces together, I have realised that you need 1. a newer compiler with c99 and 2. the github version of CLIB2 3. yeah, libwide.a also.

So now: let's compile the latest CLIB2 (github), with the old 4.2.4 GCC compiler (we need to remove force aggressive loop option) and see what happens when we just drop the newly compiled CLIB2 into the older SDK location in /opt/ppc-amigaos. This will not result in not needing to do all of the hacks since, at the very least, we are using a non-c99 compiler!

===Compiling new Clib2 from GitHub===
This seemed to compile, but I am using the old 4.2.4 compiler which I should not! Anyway, it just compiled and I can see the libraries

***TODO: Put the built clib2 into the ppc-amigaos directory in chroot /opt/ppc* and see what the hell happens! I am only replacing the clib2 within the SDK directory, that may be an issue ... ? If that does not work, then put the latest SDK in there as well.

==> And so, this seems to have improved things. But there are more undefined references requiring the addition of "-lnet -lunix" so far. Also, using the latest clib2 resulted in me not needing to worry about non-C99 compliancy.

But this leads to: __huge_val being undefined. I think we are missing "-lm" also. So, let's try that...

===SUMMARY===
So, I finally got CoreUtils to compile. I wanted to do that because I noticed that with the latest CoreUtils.lha from OS4Depot RM -RF was not working. I would like someone else to confirm this though. So, if anyone can just create a directory and touch a file in that directory and then run RM -RF on that directory, can you confirm that this does not work. It was not for me; see also (https://github.com/adtools/coreutils/issues/1). What I wanted to do was to fix this issue. Since this required the source code I set about finding the CoreUtils source code and I found it on GitHub. At this point I was under the niave assumption that downloading the latest SDK would be enough, but I was quite wrong! During this experiment, I have realised that people are using a version of CLIB2 that is not within the latest SDK. I have also noticed that people are not using the version of GCC provided with the latest SDK; so, I ask myself, should I have known about this already? For someone new coming to the Amiga scene (niche, I know) would they know that the latest SDK is not really enough? If not, how would they know other than doing what I have done which is to ask on the forums? Further, the reason I want to fix CoreUtils for native usage on the AmigaOne is because I like to do everything on my Amiga. I have since heard that people are happy to develop for the Amiga using alternative machine and are happy to X-Compile. Maybe I am just insane, but I like the idea that if I only ever had one computer remaining after a nuclear apocolypse and it was my AmigaOne X1000: I would have the resources to develop to new software for it!

When I noticed that RM -RF was not working, I tried to just download CoreUtils and build it on my Amiga. But, the very use of RM -RF in the Makefile itself caused issues.

So, what did I do?
- I said, let's use my Linux system which already has the tools on it (albeit, a very slow machine)
- Downloaded CoreUtils from https://github.com/adtools/coreutils/
- Used the README.os4 in there and took inspiration from the last entry which said:
Quote:
- Realised that I would need a CrossCompiler to do this: I assumed that "adtools" was the main place. And I saw https://github.com/adtools/amigaos-cross-toolchain.
- Realised in the https://github.com/adtools/amigaos-cross-toolchain/ README.md says: you shall use a GCC 5.X compiler with 32-bit libs
- Decided to create a CHROOT on my Linux machine using Xenial 32 bit since that also came with GCC 5.x out of the box
- Built the ppc-amigaos-* tool chain and installed to the default /opt/ppc-amigaos
- The above point means that I have the default GCC 4.2.4 with an older version of the SDK which is purely from Hyperion.
- Then, I cloned the CoreUtils from the GitHub
- ./configure --host=ppc-amigaos
- Bumped into a number of issues along the way (see above)
- required -D__NO_NET_API
- required fixing C99 for-loops since the build process believed that this was a non-C99 compiler
- Realised that I really did need libwide.a as suggested in the README.os4 fle from the CoreUtils
- Realised that libwide was using 'strnlen' that was not even in CLIB2 using the latest SDK, and so:
- Realised that I needed to upgrade the CLIB2 version using the CLIB2 on the GitHub [https://github.com/adtools/clib2] (since the Hyperion official one did not have the necessary features)
- Compiled the latest CLIB2 from GitHub
- So, I kept the original /opt/ppc-amigaos/ etc files that were installed via the cross-compiler build above, but simply replaced SDK/CLIB2 with the newly compiled one
- Realised that I needed to use "-lnet -lunix -lm -lwide -lc"

- Finally compiled CoreUtils

===RESULT===
So, I SCP the freshly compiled RM to my Amiga X1000 and just try it out out of curiosity and it works! Now, my RM compiled will actually work correctly if there is something in a test directory.

=======

So, I am wondering if someone can test out RM using CoreUtils supplied through AmigaOS 4 Depot.

Regards.

===EDIT===
I may have just realised that the latest SDK does not need CoreUtils. Looks like the latest SDK has separate binary for RM, CP etc vs the older SDK (as of last week) which all went through the "coreutils" binary.

Actually, confirmed that RM does now work with the latest SDK (53.34) vs SDK (53.30 for which you needed to separately download CoreUtils and all RM, CP etc went through the "coreutils" binary). I literally just wasted a few days of my life. But, I guess I learned a lot.

@rjd324

LOL!
Anyway it's a must to have native modern/full working compiler solution on Amiga.
Maybe your experience can be helpful for next SDK release.
Thanks for sharing your experience.

@rjd324
just in case:

we https://github.com/sba1/adtools not https://github.com/adtools/amigaos-cross-toolchain/
(just to not mess things out latest stuff on SBA1's page)

@flash
I agree. Let me quote the relevant part I created:
Quote:
I understand how using an cross-tool-chain is more convenient. Also, how CoreUtils cannot necessarily be trusted when ported to the Amiga. But, there is just something about being able to to do it on the Amiga that I like. This meant the need to have RM -RF working in Makefiles when porting over other projects. Hence why this whole thing started.

It now means that I need my favourite editor / cscope / ctags. So, trying to get a version of Emacs is something that I have to have. I have used Emacs for too long to try anything else because my fingers have now been molested after 12 years of use.

So, what I will try and do in the upcoming days is - on my X1000:
- get the latest SDK installed
- replace the 4.2.4 compiler with a much later version of a compiler,and,
- forcefully overwrite the SDK/CLIB2 with latest one I compiled

@kas1e
Thanks, but this is where I am a little confused. Why is it that I did not know about that? Let me quote my relevant part in the wall of text above:
Quote:
You buy an AmigaOne and you download the SDK from Hyperion. Generally, you would assume that is all you need. You do not have any idea that at this point in time everyone is using a totally different version of CLIB2 and different versions of this and that.

I am not criticising, I am just wondering if I have missed something obvious and if I have not I am wondering how we can make it easier for new people coming onboard.

===EDIT===
Actually, there is: https://forum.hyperion-entertainment.com/. This has some interesting information which I could have also used. I will bookmark that page also.

@rjd324
Sorry, but I was under the impression you know what you buy :) Especially since it used x1000 :) (which, btw one of the best amigaNG's ever done).

With amgaos4 there is nothing easy if you expect from it all to be logical and sane in all the aspects. The companies behind amigaos4 have their issues like lacking money, resources, sometimes knowledge, sometimes just pure logic. Sometimes things are just insane. So spending energy on "why oh why they not xxxxxx" only will hurt.

You will surely find more unpleasant surprises so better low the expectation or you will be disappointed fast by any of the reasons :) Don't expect it to be a fully supported platform or that we all there are sane enough (sometimes very much not) and all will be fine then :)

@kas1e

You sum it up nicely. Thank you.

I am quite happy for the upcoming challenges.