BinaryReader.PeekChar fails when checking for end of file

BinaryReader.PeekChar fails when checking for end of file

• k1dugar

  This is a belated message, but I wanted to say thank you. We ran into this issue tonight, and hung us up for 2 hours, but your message "unstuck us".

  We're working with .NET20, C#, BinaryReader, and it works for everyone except for one developer on his PC -- weird issue. The following code threws the "output char buffer is too small" half-way through a binary mesh file, for only the one developer:

  BinaryReader reader;

  return (reader.PeekChar() == -1);

  So we changed it to this, and it seems to work like a charm:

  return (reader.BaseStream.Position == reader.BaseStream.Length);

  I'm not sure if it's legit, but it works for us, mostly because the stream "Length" is known to us.

  
  

• Philip Lee

  Thank you for the work around - it looks like doing a br.Read and checking the returned length instead of using br.PeekChar works for me.  That is, in the file that was causing the error, I can now read past where the error occurred, presumably to the real end of file.

  If you take my original code sample (first post in this thread) and apply it to the "C:\NTLDR" ReadOnly Hidden System file, you should get the original error that I got (I am on Win XP Pro SP-2).  If it doesn't happen, I can email you the file, but I hesitate to post it here (in fact, I do not know how to post it here!).

   

  Thanks again for your interest.

  
  

• Michael Cullina

  I received this error also. Specifying ASCII encoding in the BinaryReader constructor resolved the problem for me.

  BinaryReader binaryReader = new BinaryReader(memoryStream, Encoding.ASCII);

  
  

• zhonglixunni

  Yes, I managed to repro this issue. Thank you very much for reporting this problem.

  Thanks

  Lakshan

  BCL team

  
  
  

• Floax

  Thanks for you reply.

  Before I go sending a sample file, I am a little (a lot ) confused.  I thought a BinaryReader would read byte by byte without regard for what is in each byte or its significance in the stream and so, with the "ReadByte" method, I would have thought encoding is irrelevant.  Perhaps encoding is important for "ReadChar" but I don't see its relevance for "ReadByte", hope you understand   In other words, I simply want to read a byte at a time and get the 8 bit value into my program as a byte data type - is this not possible

  
  

• Mahidhar

  Hi all

  The Position and Length checking does not work for my case becuase I got this same error in the middle of reading a image file. We have a 3 gig file and the error happens at about 2400 char position from the head of file.

  BinaryReader br =new BinaryReader( new FileStream(filename, FileMode.Open, FileAccess.Read));

  char ch=0;

  long k=0;

  while (ch = br.readChar() != -1)

  {

  ....

  k++;

  }

  when k is about 2400, we get the error. The file size is about 3 gig.
  
  

• santosh_nm

  I works for me too

  Thanks

  
  

• The Strul

  Hello,

  Seems it is a little .NET bug. I have the same problem on a PC with windows Vista. The application was tested on few machines with XP and worked well.

  The error:

  The output char buffer is too small to contain the decoded characters, encoding 'Unicode (UTF-8)' fallback 'System.Text.DecoderReplacementFallback'

  
  

• MKMahesh

  Hello,

  
  You are right, ReadByte() will work without any encoding considerations. The problem is with PeekChar(). This method will try to peek at a char and can get into an error state.

  Is it possible for you to use something like below to read the file I would still like to get the repro of your original problem if possible to follow up on it.

          Dim fileName As String = "<fileName>"
          Dim fs As New FileStream(fileName, FileMode.Open)
          Dim br As New BinaryReader(fs)
          Dim done As Boolean = False
          Dim bufferSize As Int32 = 1024 'you can specify the size you want here. the performance would improve with large sizes since we would buffer internally
          Dim bits(bufferSize) As Byte
          Dim count As Integer

          While Not done
              count = br.Read(bits, 0, bufferSize)
              If count = 0 Then
                  done = True
              Else
                  'process the bytes that have been read
              End If
          End While
          br.Close()

  Thanks
  Lakshan
  BCL team

  
  
  

• G. Karthikeyan

  We are actively looking to obsolete BinaryReader.PeekChar in a future release since it has some design issues. In the meantime, can you look at the workaround that I mention up a couple of threads earlier pl.

  Thank you

  Lakshan Fernando

  BCL Team

  
  

• Nenad Lakinski

  I have a program that's been working fine until today it threw

  "The output char buffer is too small to contain the decoded characters"

  i might of changed some code but don't think i did.

  anyway i'm persisting doubles in groups of 5 and am about to try modifying your work around to

  Dim bufferSize As Int32 = 1024 * 40

  Dim bits(buffersize) As Double

  hope it works. will keep you posted

  here's my original code

  any suggestions for improvement are welcome

  i'm not sure about the finally

  steve

  Public Function LoadBarsFromFile(ByVal fn As String) As Integer

  Dim c As Integer

  Dim FS As FileStream

  Dim BR As BinaryReader

  Try

  FS = New FileStream(fn, FileMode.Open, FileAccess.Read)

  BR = New BinaryReader(FS)

  BR.BaseStream.Seek(0, SeekOrigin.Begin) ' redundant

  While BR.PeekChar <> -1

  ReadBar(BR)

  c += 1

  End While

  Console.WriteLine("Load1minBarsFromFile: BarCount = " & c)

  Catch ex As EndOfStreamException

  Console.WriteLine("LoadBarsFromFile: exception caught and ignored. " & ex.GetType().Name)

  Finally

  If Not (BR Is Nothing) Then

  BR.Close()

  End If

  If Not (FS Is Nothing) Then

  FS.Close()

  End If

  ' if the file doesn't open properly BR will be null

  End Try

  Return c

  End Function

  Sub ReadBar(ByVal br As BinaryReader)

  Dim b As Bar

  Dim n As Integer

  For n = 0 To nFXub

  b = New Bar

  b.HiAsk = br.ReadDouble

  b.HiBid = br.ReadDouble

  b.LoAsk = br.ReadDouble

  b.LoBid = br.ReadDouble

  b.Mean = br.ReadDouble

  db.AddBar(n, b)

  Next

  End Sub
  

• Vlcan S

  ASCII encoding works because a Char with ASCII encoding is only one byte long. The default encoding of UTF8 uses more than one byte.

  
  
  

• Yarik

  Hello,
  

  Sorry for the delay in responding. In general, this arises when the incorrect encoding is set on the BinaryReader (the default one is UTF8 which will be used in your code) from what is encoded in the stream. PeekChar() (and ReadChar()) methods in BinaryReader will attempt to decode the bytes in the stream to fit in with the specified encoding and can fault if they are not compatible. These methods also fail when trying to read a surrogate character.

  
  StreamReader is better if you are only dealing with text and has members that are more text friendly.

  
  If the stream does contain the right bytes for the encoding that you have specified, can you attach a small repro (including the stream bytes) for me to take a closer look pl.
  

  Thanks
  Lakshan
  

   

  
  
  
• BinaryReader.PeekChar fails when checking for end of file