Introduction

RangesIO is a basic class for wrapping another IO object allowing you to arbitrarily reorder slices of the input file by providing a list of ranges. Intended as an initial measure to curb inefficiencies in the Dirent#data method just reading all of a file‘s data in one hit, with no method to stream it.

This class will encapuslate the ranges (corresponding to big or small blocks) of any ole file and thus allow reading/writing directly to the source bytes, in a streamed fashion (so just getting 16 bytes doesn‘t read the whole thing).

In the simplest case it can be used with a single range to provide a limited io to a section of a file.

Limitations

No buffering. by design at the moment. Intended for large reads

TODO

On further reflection, this class is something of a joining/optimization of two separate IO classes. a SubfileIO, for providing access to a range within a File as a separate IO object, and a ConcatIO, allowing the presentation of a bunch of io objects as a single unified whole.

I will need such a ConcatIO if I‘m to provide Mime#to_io, a method that will convert a whole mime message into an IO stream, that can be read from. It will just be the concatenation of a series of IO objects, corresponding to headers and boundaries, as StringIO‘s, and SubfileIO objects, coming from the original message proper, or RangesIO as provided by the Attachment#data, that will then get wrapped by Mime in a Base64IO or similar, to get encoded on-the- fly. Thus the attachment, in its plain or encoded form, and the message as a whole never exists as a single string in memory, as it does now. This is a fair bit of work to achieve, but generally useful I believe.

This class isn‘t ole specific, maybe move it to my general ruby stream project.

Methods

<< close eof? gets inspect new open pos= ranges= read readline rewind seek size= truncate write

External Aliases

pos -> tell

Attributes

 io
 mode
 pos
 ranges
 size

Public Class methods

new(io, mode='r', params={})

`io`:	the parent io object that we are wrapping.
`mode`:	the mode to use
`params`:	hash of params.

:ranges - byte offsets, either:
1. an array of ranges [1..2, 4..5, 6..8] or
2. an array of arrays, where the second is length [[1, 1], [4, 1], [6, 2]] for the above (think the way String indexing works)
:close_parent - boolean to close parent when this object is closed

NOTE: the ranges can overlap.

[Source]

    # File lib/ole/ranges_io.rb, line 54
54:         def initialize io, mode='r', params={}
55:                 mode, params = 'r', mode if Hash === mode
56:                 ranges = params[:ranges]
57:                 @params = {:close_parent => false}.merge params
58:                 @mode = IO::Mode.new mode
59:                 @io = io
60:                 # initial position in the file
61:                 @pos = 0
62:                 self.ranges = ranges || [[0, io.size]]
63:                 # handle some mode flags
64:                 truncate 0 if @mode.truncate?
65:                 seek size if @mode.append?
66:         end

open(*args) {|ranges_io| ...}

add block form. TODO add test for this

[Source]

    # File lib/ole/ranges_io.rb, line 69
69:         def self.open(*args, &block)
70:                 ranges_io = new(*args)
71:                 if block_given?
72:                         begin;  yield ranges_io
73:                         ensure; ranges_io.close
74:                         end
75:                 else
76:                         ranges_io
77:                 end
78:         end

Public Instance methods

<<(data)

Alias for write

close()

[Source]

     # File lib/ole/ranges_io.rb, line 147
147:         def close
148:                 @io.close if @params[:close_parent]
149:         end

eof?()

[Source]

     # File lib/ole/ranges_io.rb, line 151
151:         def eof?
152:                 @pos == @size
153:         end

gets()

i can wrap it in a buffered io stream that provides gets, and appropriately handle pos, truncate. mostly added just to past the tests. FIXME

[Source]

     # File lib/ole/ranges_io.rb, line 235
235:         def gets
236:                 s = read 1024
237:                 i = s.index "\n"
238:                 self.pos -= s.length - (i+1)
239:                 s[0..i]
240:         end

inspect()

[Source]

     # File lib/ole/ranges_io.rb, line 243
243:         def inspect
244:                 "#<#{self.class} io=#{io.inspect}, size=#{@size}, pos=#{@pos}>"
245:         end

pos=(pos, whence=IO::SEEK_SET)

[Source]

     # File lib/ole/ranges_io.rb, line 110
110:         def pos= pos, whence=IO::SEEK_SET
111:                 case whence
112:                 when IO::SEEK_SET
113:                 when IO::SEEK_CUR
114:                         pos += @pos
115:                 when IO::SEEK_END
116:                         pos = @size + pos
117:                 else raise Errno::EINVAL
118:                 end
119:                 raise Errno::EINVAL unless (0..@size) === pos
120:                 @pos = pos
121: 
122:                 # do a binary search throuh @offsets to find the active range.
123:                 a, c, b = 0, 0, @offsets.length
124:                 while a < b
125:                         c = (a + b) / 2
126:                         pivot = @offsets[c]
127:                         if pos == pivot
128:                                 @active = c
129:                                 return
130:                         elsif pos < pivot
131:                                 b = c
132:                         else
133:                                 a = c + 1
134:                         end
135:                 end
136: 
137:                 @active = a - 1
138:         end

ranges=(ranges)

[Source]

     # File lib/ole/ranges_io.rb, line 80
 80:         def ranges= ranges
 81:                 # convert ranges to arrays. check for negative ranges?
 82:                 ranges = ranges.map { |r| Range === r ? [r.begin, r.end - r.begin] : r }
 83:                 # combine ranges
 84:                 if @params[:combine] == false
 85:                         # might be useful for debugging...
 86:                         @ranges = ranges
 87:                 else
 88:                         @ranges = []
 89:                         next_pos = nil
 90:                         ranges.each do |pos, len|
 91:                                 if next_pos == pos
 92:                                         @ranges.last[1] += len
 93:                                         next_pos += len
 94:                                 else
 95:                                         @ranges << [pos, len]
 96:                                         next_pos = pos + len
 97:                                 end
 98:                         end
 99:                 end
100:                 # calculate cumulative offsets from range sizes
101:                 @size = 0
102:                 @offsets = []
103:                 @ranges.each do |pos, len|
104:                         @offsets << @size
105:                         @size += len
106:                 end
107:                 self.pos = @pos
108:         end

read(limit=nil)

read bytes from file, to a maximum of limit, or all available if unspecified.

[Source]

     # File lib/ole/ranges_io.rb, line 156
156:         def read limit=nil
157:                 data = ''
158:                 return data if eof?
159:                 limit ||= size
160:                 pos, len = @ranges[@active]
161:                 diff = @pos - @offsets[@active]
162:                 pos += diff
163:                 len -= diff
164:                 loop do
165:                         @io.seek pos
166:                         if limit < len
167:                                 s = @io.read(limit).to_s
168:                                 @pos += s.length
169:                                 data << s
170:                                 break
171:                         end
172:                         s = @io.read(len).to_s
173:                         @pos += s.length
174:                         data << s
175:                         break if s.length != len
176:                         limit -= len
177:                         break if @active == @ranges.length - 1
178:                         @active += 1
179:                         pos, len = @ranges[@active]
180:                 end
181:                 data
182:         end

readline()

Alias for gets

rewind()

[Source]

     # File lib/ole/ranges_io.rb, line 143
143:         def rewind
144:                 seek 0
145:         end

seek(pos, whence=IO::SEEK_SET)

Alias for pos=

size=(size)

using explicit forward instead of an alias now for overriding. should override truncate.

[Source]

     # File lib/ole/ranges_io.rb, line 191
191:         def size=      size
192:                 truncate size
193:         end

truncate(size)

you may override this call to update @ranges and @size, if applicable.

[Source]

     # File lib/ole/ranges_io.rb, line 185
185:         def truncate size
186:                 raise NotImplementedError, 'truncate not supported'
187:         end

write(data)

[Source]

     # File lib/ole/ranges_io.rb, line 195
195:         def write data
196:                 return 0 if data.empty?
197:                 data_pos = 0
198:                 # if we don't have room, we can use the truncate hook to make more space.
199:                 if data.length > @size - @pos
200:                         begin
201:                                 truncate @pos + data.length
202:                         rescue NotImplementedError
203:                                 raise IOError, "unable to grow #{inspect} to write #{data.length} bytes" 
204:                         end
205:                 end
206:                 pos, len = @ranges[@active]
207:                 diff = @pos - @offsets[@active]
208:                 pos += diff
209:                 len -= diff
210:                 loop do
211:                         @io.seek pos
212:                         if data_pos + len > data.length
213:                                 chunk = data[data_pos..-1]
214:                                 @io.write chunk
215:                                 @pos += chunk.length
216:                                 data_pos = data.length
217:                                 break
218:                         end
219:                         @io.write data[data_pos, len]
220:                         @pos += len
221:                         data_pos += len
222:                         break if @active == @ranges.length - 1
223:                         @active += 1
224:                         pos, len = @ranges[@active]
225:                 end
226:                 data_pos
227:         end