I've done some "simple" scripting in AREXX for a while now.
I've hit the borders of what's possible in AREXX when it comes to processing *large* files and changing them (strip, substr etc.) and writing them back to another file.
I get the dreaded "ram usage goes beyond ~70% silent system freeze" every time i want to process a 1MB+ sized text file.
I'm doing a simple task like opening the file, reading in line for line and writing it back into another file ... not possible with AREXX as it will eat away my 2 GB of RAM and freeze solid
Now, i'd like to switch to one of the other supported scripting languages.
Can anyone point me to a good online tutorial (a printed manual would be even better) or at least tell me which of the two languages would be easier to switch to coming from AREXX?
I tried both already with copy and paste and opening and reading the same file and at least both doesn't eat up all my ram (albeit it seems Perl - or was it python? - doesn't free all of it after the script finished)
Just for the sake of completeness, here's what i want to do (and no, i don't want anyone to write it for me)
1) Open a file 2) read the file line by line 3) check if a special condition is met 4) if so, change the line accordingly (AREXX SUBSTR) 5) write it back to another file 6) exit
The goal is to fetch a file from the net, change it accordingly to that it can be used in AmigaOS
Sorry you originally asked about this in PM and I sent you and extansive reply, which somehow disppeared and I didn;t get round to re replying....
Quote:
I've done some "simple" scripting in AREXX for a while now.
I've hit the borders of what's possible in AREXX when it comes to processing *large* files and changing them (strip, substr etc.) and writing them back to another file.
I get the dreaded "ram usage goes beyond ~70% silent system freeze" every time i want to process a 1MB+ sized text file.
Really with 1Mb files? I was imagining that you were processing simething much larger when you PMed me.
/* slurp file */
options results
if open("FH","SAM_Ram Disk:webhitdata.csv","r") then do
if open("OUT","ram:processed","W") then do
do while ~eof("FH")
line = readln("FH")
call writeln("OUT",line)
end
end
end
Loads and writes out a 1.5 Mb CSV file, with about 11000 lines, with barely a ripple in memory usage.
Quote:
I'm doing a simple task like opening the file, reading in line for line and writing it back into another file ... not possible with AREXX as it will eat away my 2 GB of RAM and freeze solid
Okay see above, what else are you doing that uses up the memory? Are you trying to hold all the data in memory before writing it out?
Quote:
Now, i'd like to switch to one of the other supported scripting languages.
Can anyone point me to a good online tutorial (a printed manual would be even better) or at least tell me which of the two languages would be easier to switch to coming from AREXX?
python is easier than perl but perl may be nore suited to what you are doing, though without fine detail it hard to say exdactly.
Quote:
I tried both already with copy and paste and opening and reading the same file and at least both doesn't eat up all my ram (albeit it seems Perl - or was it python? - doesn't free all of it after the script finished)
Just for the sake of completeness, here's what i want to do (and no, i don't want anyone to write it for me)
1) Open a file 2) read the file line by line 3) check if a special condition is met 4) if so, change the line accordingly (AREXX SUBSTR) 5) write it back to another file 6) exit
The goal is to fetch a file from the net, change it accordingly to that it can be used in AmigaOS
Thanks a lot in advance
Perl frame work is as follows.
#!perl
open IN, "<", "SAM_Ram Disk:webhitdata.csv" or die "Couldn't open input file";
open OUT, ">", "ram:processed" or die "Couldn't open outputfile";
while (<IN>) {
print OUT $_;
};
or
#!perl
open IN, "<", "SAM_Ram Disk:webhitdata.csv" or die "Couldn't open input file";
open OUT, ">", "ram:processed" or die "Couldn't open outputfile";
[quote]Just for the sake of completeness, here's what i want to do (and no, i don't want anyone to write it for me)
1) Open a file 2) read the file line by line 3) check if a special condition is met 4) if so, change the line accordingly (AREXX SUBSTR) 5) write it back to another file 6) exit [quote]
Here's the same thing in python:
#!C:Python
f1=open('load.txt', 'r')
f2=open('save.txt', 'wb')
for line in f1:
line = line.strip()
line = line.replace('original text', 'amiga text')
f2.write(line + '\\n')
f1.close()
f2.close()
if os.path.exists(loadfile):
f1=open('loadfile', 'r')
f2=open('savefile', 'wb')
for line in f1:
line = line.strip()
line = line.replace('original text', 'amiga text')
f2.write(line + '\n')
f1.close()
f2.close()
Edited by Severin on 2016/11/28 17:58:37
Amiga user since 1985 AOS4, A-EON, IBrowse & Alinea Betatester
open a file read a line process the line write it back to another file rinse repeat
doing it like above though, it will fill up my ram constantly until it freezes the system
here is the (broken and very rudimentary) script and the link to the file i let it run on. Download the file and use it as argument
/*
Blacklist Creator
*/
PARSE ARG hosts_txt
x=0
y=0
OPEN(hosts_read,hosts_txt,'R')
/* LASTPOS('.',working_line)-POS('.',working_line)+5 */
DO WHILE EOF(hosts_read)=0
working_line=READLN(hosts_read)
IF SUBSTR(working_line,1,7)='0.0.0.0' THEN DO
working_line=SUBSTR(working_line,9)
x=x+1
IF POS('.', working_line)~=LASTPOS('.',working_line) THEN
probable_individual_link=SUBSTR(working_line,POS('.',working_line)+1)
ELSE
probable_individual_link=working_line
ENDIF
IF probable_individual_link~=individual_link THEN DO
individual_link=probable_individual_link
SAY individual_link
y=y+1
END
END
IF y=2000 THEN
LEAVE
ENDIF
END
SAY "Links checked:" x
SAY "Individual links found:" y
EXIT 0
For now the script is supposed to only count individual links (speaking of the main sites, like google.com where the probable links can alter from ads1.google.com to ads99.google.com or whatever, it should just pick google.com and discard the rest etc.)
As i said, the script doesn't work like i want it to right now and i don't want a ready script written from any of you, i want to script it myself
But if you know how i can manage AREXX to not fill up the ram i'd be grateful
But just for the fun of it, try it and see how it eats away your ram.
I added a safe case where it stops after finding 2000 links, remove that and it will goe beyond the ram barrier that freezes the system
I removed the 200 line limit and ran your script on that data on my SAM, did not run out of memory, in fact didn't ever use much memory.
Is that the exact script you have been having issues with?
Quote:
and no comments on my scripting, please
Okay I won't mention that there is no such keyword as ENDIF in ARexx
the syntax is simply
if <condition> then <expression> else <other expression>
is all you need expression may be a do block if you need more than one statement
ie
if FOO = BAR THEN DO /* lots of stuff */ END ELSE DO /* different stuff */ END
your not seeing an error as when ENDIF is uset it simply evaluates to 'ENDIF' (the string) rexx searches for a function in all the available hosts and does nothing if not found, since you don't have a function host added (ie address is REXX) it simply ignores it, if you had a host address set you would likely see an error
BTW Exit returns a string not a number. EXIT 0 is not meaningful in this context.
It does seem to leak in FE: But in an odd way, memeory usage as displayed by the WB header goes right down to 480Mb free then bounces back to 1400 Mb free and the machine remains usable except that all rexx related stuff then hangs.
Unsure what the chnage from FE: to beta is as all the obvious rexx related components appear the same RX Rexxmast and rexxsyslib being the most obvious.
Edit I removed the say statements to eliminate the console from enquiries..
Unsure what the chnage from FE: to beta is as all the obvious rexx related components appear the same RX Rexxmast and rexxsyslib being the most obvious.
Ram-handler? Elf.library? DOS?
Anyway, the good news is that the problem is (for us)/will be (for everybody) solved.
#!C:Python
X=Y=0
I=open('hosts','r')
O=open('blacklist','wb')
for L in I:
X+=1
if L.startswith('0.0.0.0'):
L=L[9:]
if L.count('.')>1:
L=L[L.find('.')+1:]
O.write(L)
Y+=1
I.close()
O.close()
print "Lines checked:"+str(X)+"\\nIndividual links found:"+str(Y)
edit: changed indenting from tabs to spaces to make it more readable,
Edited by Severin on 2016/12/25 21:48:03 Edited by Severin on 2017/1/1 16:26:22
Amiga user since 1985 AOS4, A-EON, IBrowse & Alinea Betatester
News feed
OS4 Feedback forum
OS4Depot Feedback forum
Amigabounty forum
OpenAmiga forum
AmiCygnix forum
ABC shell forum
AmiKit forum
Cinnamon Writer forum
CodeBench forum
Digital Universe forum
Dopus 5 forum
E-UAE forum
Gnash forum
Ibrowse forum
JAmiga forum
Odyssey forum
OWB forum
Qt forum
SmartFileSystem forum
Timberwolf forum
TouchDevice forum
TuneNet forum
Unsatisfactory Software forum
Donate
