Hyperion blog update.

Following the recent thread regarding lack of update news on AmigaOS4, I felt compelled to post this very interesting snippet.

"Breaking the Memory Barrier"
http://blog.hyperion-entertainment.biz/?p=1131

A very welcome development and a superbly written news update.

Many thanks for the project and for taking the time to inform and explain the concept so clearly! (Really helps Luddites like me understand abstract ideas )

Heh! Extmem! Grazy! But cool as well.

may I suggest:
-make RAD drive(s) that can be mapped to ext mem
-enable shoft booting from RAD (grazy but cool)
-enable SWAP file allocation from extmem

Q1: When do we get this ext mem feature?

Q2: So what is now the new maximum memory limit? Same as for 64bit file systems on hard drives?
IIRC, e5500 cores can address 32GB RAM, would all that be available via extmem?

Q3: Is extmem memory protected? So only one application can access it's extmem?

Q4: Have I understood correctly: existing 32bit apps are still limited to running in the ~2GB space? (or is it 4GB)

Q5: New 32bit apps can have up to 2GB directly mapped RAM and tens of gigabytes of extmem?

Q6: Is ext mem ever swapped to HDD? (swap partition #2?)

Q7: Is it possible to run two legacy 32bit apps using 2GB per application? (some memory is swapped to HDD / ext mem?)



Q8: Grazy "extmemextender" idea... would it be possible to have a application that has control part in the 32bit space and processing part in pure 64bit space (it would require something like WarpUP-NG and access to OS services would need to be accessed via the 32bit control instance ... )
(and while at it ... WarpUP-NG could use multiple cores...)
//ok, better come back to earth//


Q9: Would this extmem be usefull also for disc read/write cache? (I doubt)

Q10: can this extmem system use also the unused GFX card memory? Or some other memory on PCIe card?


I imagine we see games/game ports that buffer/cache data to extmem, so that once loaded data does not ever need to be loaded again.

Nice reading and interesting possibilities! While we wait for 64bit, this could be a solution for some demanding app and also a nice thing for RAM disk!

@ddni

If its efficient to use it can be used in emulators to move this into the upper 4Gbytes, (emulators can easily allocate 128mb or 256mb of ram.), Saving RAM in lower 4GBytes, but it depends on the implementation.

1#
Is the upper memory shareable between process and child processes, this is a requirement for efficient memory handling, a emulator has many child processes that handle different things like sound, graphics, Ethernet and so on.

2#
Do just set a page number some where in the code, and just poke memory like normally, or do you access the memory trow functions? The latter might be too slow for emulators.

3#
can also virtual private memory be allocated in the upper memory beyond 4GBytes?

This are the questions I have.

Are emulators of 8/16bit hardware really lacking in memory space?

I'm guessing not, this feature is for real software!

@broadblues

32bit MacOS System 8 maybe. or x86 PC emulator(s).

Not really but maybe you want to use that memory space for some thing else, that's what I'm thinking. A emulator has to emulate memory anyway, that's way I think maybe this can work, but only if gain is bigger then the costs.

As I see it if more applications use EMO the better, the possibility of running out of RAM (below 4GBytes) decrees.

@KimmoK

I don't know all of the details, but...

Quote:
Yes. If you read the second paragraph, you'll see that PCI devices (incl. VRAM), the kernel and other memory buffers take up space outside that 2GB.

Quote:
Yes.

Quote:
I assume that it could be.

Quote:
AmigaOS uses a shared memory model, so no.

Quote:
That would be a bad idea. With ExtMem, you map in the extra memory when you need it, and unmap it when you don't. Since you're (probably) using a cache all the time, you'd leave it permanently mapped, so you might as well just allocate memory directly.

Quote:
No. That additional memory is accessible by the GPU only.

Hans

@Hans

Quote:

So does that effect just current process or all processes.

Lets say program x maps memory from EMO a, and program y maps memory from EMO b, can program x and program y use the same <32bit memory for mapping or will they need to be unique until unmapped.

Case 1:

Process x : a = map(EMOA)
Process y : b = map(EMOB)
Process x : unmap(EMOA)
Process y : unmap(EMOB)

Case 2:

Process x : a = map(EMOA)
Process x : unmap(EMOA)
Process y : b = map(EMOB)
Process y : unmap(EMOB)


Can one or more process/child process access the same EMO at once?

@Hans

Quote:Wrong, for example the diskcache.library caches are mapped on demand already anyway (while they aren't in use the pages are write protected), the only reason ExtMem isn't usable for disk caches in the current implementation is that it can be paged out and there is no option to disable it.

@joerg

Quote:
Okay, I stand corrected then. However, do note that flicking the write protect flag on a memory page that's already mapped requires much less effort than mapping in new memory (you don't have to find free address space, or add the pages to the relevant tables). The extra overhead of mapping in new memory would still be far less than having to load the data from disk, though.


@LiveForIt

Quote:
I guess that would depend on if the ExtMem is marked as private or not. However, you'd have to make absolutely sure that all tasks/processes know when the EMO is mapped and when it isn't.

Hans

@Hans

Quote:Only if you would change the CPU registers (or MMU tables in memory, depending on the CPU) directly, but not with the OS functions: The pages are unmapped and then mapped again with the new flags.

Quote:There is no difference with ExtMem (unless you use delayed mapping), the pages are there already, just without a virtual address as long as they aren't in the currently mapped window.

@joerg

I think you misunderstood. I did not mean effort on the part of the application developer, but computational effort; i.e., the extra overhead involved. Of course a developer doesn't have to manually find free virtual address space and manually map their ExtMem object in.

Hans

@Hans

Quote:Me neither.

Quote:There is no difference between changing the flags, for example read-write<->read-only, and changing the physical addresses of pages (changing the ExtMem offset by remapping the pages inside the ExtMem virtual address window to different physical addresses).
I don't know the hardware details, maybe the access flags are only 16 or 32 bit while the physical address is 64 bit, which would make changing flags a little bit faster, but except for that there is no difference, the page table entry has to be found by it's virtual address (trees, hashes, or whatever) in the MMU table, modified and maybe a MMU cache for it invalidated. The slow part is finding the page table entry, the changes done in it don't make a difference.

@joerg

Quote:

Sorry, but are you suggesting that the process of finding a large enough block of free virtual addressing space to map to has zero overhead? That makes no sense.

Hans

@Hans

Quote:Always creating new ExtMem windows and destroying them again after you accessed the contents doesn't make sense indeed
Changing the offset of an already mapped ExtMem window doesn't need to find new virtual addresses, they stay the same and only the physical addresses of the pages in the virtual address window have to be updated.

@joerg

Quote:

This still doesn't make any sense. In the blog post Hans-Joerg says that "there is no limit to the number of mappings an application can do." Since multiple ExtMem objects cannot be mapped to the same virtual address at the same time, there has to be some process to decide where to locate them in virtual address space. Sure, you have a predefined ExtMem window but, when multiple ExtMem objects are mapped into that window simultaneously, they still each need their own unique virtual address range (within that window).

Hans

Alright, let me try to answer a few of the questions.

1) There is no limit on the maximum size of the ExtMem blob. It just needs to fit in the physical memory.
2) Currently, nothing enforces sharability of the memory, since there is no per-task MMU setup yet. Rules for that will be laid out later. Right now the rule is that tasks and sub-tasks can share the memory.
3) Memory in ExtMem objects is currently not subject to swapping, but there isn't a reason it cannot be swapped.
4) ExtMem works with both 32 and 64 bit CPU's. It theoretically works on the classic too, but you rarely have enough memory on the classic for something like that.

About remapping, I plan to add that ASAP. Remapping means that the offset into the ExtMem blob changes without the virtual address window changing. As was rightly pointed out, finding virtual address space for a map isn't free, so re-setting the start address into the blob is an optimization that will be useful.

Another point I haven't raised in the blog (yet) is how this can evolve. Right now, there are three possible ways to map the pages for the blob: Immediately (meaning the whole memory is allocated when the blob is created, for performance reasons), by mapping (meaning once a range is mapped for the first time it is stacked up with pages), or by access (meaning no pages are allocated until they are really accessed). Each of these methods is successively more expensive at runtime in favor of being more efficient with pages.

There is more that can be done with the system once it evolves. For example, it would be possible to not even have memory backing up the blob. Mapping a range could mean memory-mapping a file, and unmapping would write the file back to disk (or buffer in a memory buffer until a commit writes it back), to allow for easy memory-mapped files. Another example is mapping a large graphics card area. Right now, you are limited to the size of the PCI address space; graphics card drivers will make sure the right stuff is copied in the right addresses. However, ExtMem could be used to access the memory directly for cards capable of 64 bit PCI access, which would be interesting for things like Compute shaders that might need to work on a large dataset.

As you can see, we do believe that there is potential for future functionality in this. We won't be able to make AmigaOS a 64 bit OS any time soon, as I wrote on the dev blog; the 32 bit pointers are depply woven into the API, and only by throwing the whole API away and replacing it with a new one would this be cured. But at the very least, this allows for using much more memory than we have right now, and the performance overhead, while not being zero, is relatively low.

Hope that answers some questions.

Good Day,

I like a logical simple layout. My thoughts on this are, If we could Have a 4Gb chunk dedicated to graphics card and other. So lets say you have 2 Gb graphics card, you could then add 2 more Gb to use as Clipboard/Ram disk thus leaving all 4 Gb system memory alone.

If the OS knows what all the graphics processes are, this can be done. Handle the graphics first then add a manager later to use what's left for "other".

I wish but, not likely to happen.

Chris

@QuikSanz

Graphic card memory and system memory is not the same, but I remember that Hans talked about swaping of textures in and out of the graphic card, maybe this is what your think about, using using 64bit memory for texture swap memory.

@LiveForIt,

"Rogue,
Another example is mapping a large graphics card area. Right now, you are limited to the size of the PCI address space; graphics card drivers will make sure the right stuff is copied in the right addresses."

Did I misread this?

Chris