Home   Archive   Permalink



Random access Gigabyte size text file

I'd like to support reading very large text files. For some reason I cannot get /skip to work:
    
text: to-string read/binary/part/skip (to-file full-path) 20 10
    
I do get 20 characters and running the command without the third parameter gives an error.
However the returned string starts at the file first character.
    
I worked around doing
    
myfileport: open/direct/binary/read to-file full-path
    
copy/part myfileport 1000000000 ; Giga range
text: copy []
text: to-string copy/part myfileport grid-line-max-display
    
This solution takes about 5 seconds to read about one gigabyte, too slow to browse forward and backward.
    
Any idea what I could try?

posted by:   VIDpuzzle       25-Feb-2019/17:58:41-8:00



As I understand it, random-access file ports are broken in Rebol 2 and unlikely to be fixed.
    
http://www.rebol.com/article/0198.html

posted by:   Chris       26-Feb-2019/9:51:10-8:00



I tested read/part/seek on Rec-C and it works as expected.
    
I am considering using Ren-C through a TCP socket
    
http://www.rebol.net/cookbook/recipes/0028.html
    
to work around R2 limitations, in this case requesting a large file line range to a Ren-C server app to display within a R2 screen.
    
The general idea is that the missing Ren-C GUI might be right under our noses.
    
Would it not be possible to use R2 as a "dumb terminal",pushing block layouts from R3/Ren-C to some sort of R2 loop?

posted by:   VIDpuzzle       1-Mar-2019/16:51:25-8:00



Should be possible. Note that I have an HTTPD server scheme for both Rebol 2 and Ren-C—thus you could use HTTP as your transfer protocol in either direction. Or just use your own TCP connection directly.

posted by:   Chris       1-Mar-2019/17:23:04-8:00



My Rebol 2 HTTPd script is at:
https://github.com/rgchris/Scripts-for-Rebol-2
    
For Ren-C, you can just do:
    
     import <httpd>
     wait srv: open [scheme: 'httpd 8000 [render "Hello"]]


posted by:   Chris       1-Mar-2019/17:25:18-8:00



Wouldn't a client application pulling data from a server require a synchronous connection rather than the asynchronous connection a web server can offer?
    
In other words, the client needs to be garanteed a complete response in order to be sure the data set received is complete.
    
The event loop seems to have its own thread since in the following example the clock keeps ticking on regardless of the WAIT PORT.
    
The following code attempts to maintain a connection and a message stream regardless of the server or the client shutting down and coming back on-line.
    
The original code
    
http://www.rebol.net/cookbook/recipes/0058.html
    
errors when the receiving end isn't online.
    
Unfortunately I get an Internal Error: Stack overflow after 200 messages or so.
    
I was not able to receive more than a message per connection, so the code keeps spawning a new one for each message.
    
This is the sending part.
    
; port-receive
connection: off
data-wait: off
    
port-connect: does [
        
            port: open/binary/no-wait tcp://:60111
            wait port
            client: first port
            connection: on            
            wait client
]
    
port-close: does [
     if connection [close client]
     ]
    
    view/title layout [
    origin 0
    do [port-connect]
    banner 200x32 rate 50 effect [gradient 0x1 0.0.150 0.0.50]
        feel [engage: func [f a e] [
         set-face f now/time
            if connection [
                data: to-string copy client
                if (length? data) > 0 [
                    print data
                    close client
                    connection: off                    
                    close port
                    port-connect
                 ]
             ]
     ]
    ]
    below
    button "Quit" [port-close quit]
    button "Halt" [port-close halt]    
] "R2 Client"
    
    ;-- Start listening for connections:
    print "waiting for connection"
    port: open/binary/no-wait tcp://:60111
    wait port
In order    
    
    
    ;-- Wait for a client to connect:
    client: first port
    probe client
    print "client connected, waiting for data"
    wait client
    
    ;-- Read what we've got so far and decode the header:
    data: to-string copy client
    ; start: find data #""
    print data
    close port

posted by:   VIDpuzzle       3-Mar-2019/21:32:55-8:00



; port-receive
connection: off
data-wait: off
    
port-connect: does [
        
            port: open/binary/no-wait tcp://:60111
            wait port
            client: first port
            connection: on            
            wait client
]
    
port-close: does [
     if connection [close client]
     ]
    
    view/title layout [
    origin 0
    do [port-connect]
    banner 200x32 rate 50 effect [gradient 0x1 0.0.150 0.0.50]
        feel [engage: func [f a e] [
         set-face f now/time
            if connection [
                data: to-string copy client
                if (length? data) > 0 [
                    print data
                    close client
                    connection: off                    
                    close port
                    port-connect
                 ]
             ]
     ]
    ]
    below
    button "Quit" [port-close quit]
    button "Halt" [port-close halt]    
] "R2 Client"

posted by:   VIDpuzzle       3-Mar-2019/21:34:43-8:00



; both sides in R2 for now
    ; port-send
    data: "Hello )R(en-C"
    message: 0
    
    ;-- Open the binary TCP socket:
print "Opening to send..."
connection: off
    
port-connect: does [
         if not error? connect-error: try [
            server: open/binary/no-wait tcp://localhost:60111
         ] [connection: on]
]
    
port-close: does [
     if connection [close server]
     ]
    
    blue-220: 0.0.220
    blue-163: 0.0.163    
    
    view/options layout [
    backcolor blue-220
    origin 0
    label h1 "R2 Server" white
    return
    across
    below
    banner 200x32 rate 50 effect [gradient 0x1 0.0.150 0.0.50]
        feel [engage: func [f a e] [
         set-face f now/time
         either not connection [port-connect]
             [
                if connection [        
                    insert server form reduce [(message) data]
                    ++ message
                    connection: false
                 ]
             ]
     ]
    ]
    below
    button "Quit" [port-close quit]
    button "Halt" [port-close halt]    
] [no-title]

posted by:   VIDpuzzle       3-Mar-2019/21:36:55-8:00



Name:


Message:


Type the reverse of this captcha text: "h t a p - r e s w o r b - t e s"



Home