Random access Gigabyte size text file
I'd like to support reading very large text files. For some reason I cannot get /skip to work:
text: to-string read/binary/part/skip (to-file full-path) 20 10
I do get 20 characters and running the command without the third parameter gives an error.
However the returned string starts at the file first character.
I worked around doing
myfileport: open/direct/binary/read to-file full-path
copy/part myfileport 1000000000 ; Giga range
text: copy []
text: to-string copy/part myfileport grid-line-max-display
This solution takes about 5 seconds to read about one gigabyte, too slow to browse forward and backward.
Any idea what I could try?
posted by: VIDpuzzle 25-Feb-2019/17:58:41-8:00
As I understand it, random-access file ports are broken in Rebol 2 and unlikely to be fixed.
http://www.rebol.com/article/0198.html
posted by: Chris 26-Feb-2019/9:51:10-8:00
I tested read/part/seek on Rec-C and it works as expected.
I am considering using Ren-C through a TCP socket
http://www.rebol.net/cookbook/recipes/0028.html
to work around R2 limitations, in this case requesting a large file line range to a Ren-C server app to display within a R2 screen.
The general idea is that the missing Ren-C GUI might be right under our noses.
Would it not be possible to use R2 as a "dumb terminal",pushing block layouts from R3/Ren-C to some sort of R2 loop?
posted by: VIDpuzzle 1-Mar-2019/16:51:25-8:00
Should be possible. Note that I have an HTTPD server scheme for both Rebol 2 and Ren-C—thus you could use HTTP as your transfer protocol in either direction. Or just use your own TCP connection directly.
posted by: Chris 1-Mar-2019/17:23:04-8:00
My Rebol 2 HTTPd script is at:
https://github.com/rgchris/Scripts-for-Rebol-2
For Ren-C, you can just do:
import
wait srv: open [scheme: 'httpd 8000 [render "Hello"]]
posted by: Chris 1-Mar-2019/17:25:18-8:00
Wouldn't a client application pulling data from a server require a synchronous connection rather than the asynchronous connection a web server can offer?
In other words, the client needs to be garanteed a complete response in order to be sure the data set received is complete.
The event loop seems to have its own thread since in the following example the clock keeps ticking on regardless of the WAIT PORT.
The following code attempts to maintain a connection and a message stream regardless of the server or the client shutting down and coming back on-line.
The original code
http://www.rebol.net/cookbook/recipes/0058.html
errors when the receiving end isn't online.
Unfortunately I get an Internal Error: Stack overflow after 200 messages or so.
I was not able to receive more than a message per connection, so the code keeps spawning a new one for each message.
This is the sending part.
; port-receive
connection: off
data-wait: off
port-connect: does [
port: open/binary/no-wait tcp://:60111
wait port
client: first port
connection: on
wait client
]
port-close: does [
if connection [close client]
]
view/title layout [
origin 0
do [port-connect]
banner 200x32 rate 50 effect [gradient 0x1 0.0.150 0.0.50]
feel [engage: func [f a e] [
set-face f now/time
if connection [
data: to-string copy client
if (length? data) > 0 [
print data
close client
connection: off
close port
port-connect
]
]
]
]
below
button "Quit" [port-close quit]
button "Halt" [port-close halt]
] "R2 Client"
;-- Start listening for connections:
print "waiting for connection"
port: open/binary/no-wait tcp://:60111
wait port
In order
;-- Wait for a client to connect:
client: first port
probe client
print "client connected, waiting for data"
wait client
;-- Read what we've got so far and decode the header:
data: to-string copy client
; start: find data #""
print data
close port
posted by: VIDpuzzle 3-Mar-2019/21:32:55-8:00
; port-receive
connection: off
data-wait: off
port-connect: does [
port: open/binary/no-wait tcp://:60111
wait port
client: first port
connection: on
wait client
]
port-close: does [
if connection [close client]
]
view/title layout [
origin 0
do [port-connect]
banner 200x32 rate 50 effect [gradient 0x1 0.0.150 0.0.50]
feel [engage: func [f a e] [
set-face f now/time
if connection [
data: to-string copy client
if (length? data) > 0 [
print data
close client
connection: off
close port
port-connect
]
]
]
]
below
button "Quit" [port-close quit]
button "Halt" [port-close halt]
] "R2 Client"
posted by: VIDpuzzle 3-Mar-2019/21:34:43-8:00
; both sides in R2 for now
; port-send
data: "Hello )R(en-C"
message: 0
;-- Open the binary TCP socket:
print "Opening to send..."
connection: off
port-connect: does [
if not error? connect-error: try [
server: open/binary/no-wait tcp://localhost:60111
] [connection: on]
]
port-close: does [
if connection [close server]
]
blue-220: 0.0.220
blue-163: 0.0.163
view/options layout [
backcolor blue-220
origin 0
label h1 "R2 Server" white
return
across
below
banner 200x32 rate 50 effect [gradient 0x1 0.0.150 0.0.50]
feel [engage: func [f a e] [
set-face f now/time
either not connection [port-connect]
[
if connection [
insert server form reduce [(message) data]
++ message
connection: false
]
]
]
]
below
button "Quit" [port-close quit]
button "Halt" [port-close halt]
] [no-title]
posted by: VIDpuzzle 3-Mar-2019/21:36:55-8:00
|