Skip to content

Commit

Permalink
Fall back to opening files using 'r' if 'rb' mode fails
Browse files Browse the repository at this point in the history
Code to support 'rb' (or non-utf8 decoding of 'r' files) is
not yet available in pywandio publicly, which was causing
issues for external users.
  • Loading branch information
salcock committed Apr 21, 2021
1 parent fb53434 commit 59627ee
Showing 1 changed file with 25 additions and 4 deletions.
29 changes: 25 additions & 4 deletions src/pyavro_stardust/baseavro.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -330,10 +330,31 @@ cdef class AvroReader:
def start(self):
if self.fh is not None:
return
try:
self.fh = wandio.open(self.filepath, 'rb')
except:
raise

# Try 'rb' mode if pywandio supports it, else fallback to 'r'
# and hope that we're not reading a file off local disk (that
# pywandio will try to "decode" into utf-8)
#
# Future versions of pywandio may allow us to override the
# decoding method, in which case we can rework this code to be
# less clunky.
mode = 'rb'
saved = None
while mode != 'fail':
try:
self.fh = wandio.open(self.filepath, mode=mode)
except ValueError as e:
if mode == 'rb':
mode = 'r'
else:
mode = 'fail'
raise
except Exception:
raise

if self.fh is not None:
break


if self.syncmarker is None:
self._readAvroFileHeader()
Expand Down

0 comments on commit 59627ee

Please sign in to comment.