c3f53854b6
Fix #2897. We have been relying on iconv/mb_convert_encoding to detect invalid UTF-8, but all techniques designed to validate UTF-8 seem to accept FFFE and FFFF. This PR explicitly converts those characters to FFFD (Unicode substitution character) before validating the rest of the string. It also substitutes one or more FFFD when it detects invalid UTF-8 character sequences. A comment in the code being change stated that it doesn't handle surrogates. It is right not to do so. The only case where we should see surrogates is reading UTF-16. Additional tests are added to an existing test reading a UTF-16 Csv to demonstrate that surrogates are handled correctly, and that FFFE/FFFF are handled reasonably. |
||
|---|---|---|
| .. | ||
| NumberFormatTest.csv | ||
| NumberFormatTest.de.csv | ||
| backslash.csv | ||
| contains_html.csv | ||
| csv_without_extension | ||
| empty.csv | ||
| enclosure.csv | ||
| encoding.iso88591.csv | ||
| encoding.utf8.csv | ||
| encoding.utf8bom.csv | ||
| encoding.utf16be.csv | ||
| encoding.utf16le.csv | ||
| encoding.utf32be.csv | ||
| encoding.utf32le.csv | ||
| escape.csv | ||
| issue.2232.csv | ||
| line_break_escaped_32le.csv | ||
| line_break_in_enclosure.csv | ||
| line_break_in_enclosure_with_escaped_quotes.csv | ||
| no_delimiter.csv | ||
| premiere.utf8.csv | ||
| premiere.utf8bom.csv | ||
| premiere.utf16be.csv | ||
| premiere.utf16bebom.csv | ||
| premiere.utf16le.csv | ||
| premiere.utf16lebom.csv | ||
| premiere.utf32be.csv | ||
| premiere.utf32bebom.csv | ||
| premiere.utf32le.csv | ||
| premiere.utf32lebom.csv | ||
| premiere.win1252.csv | ||
| semicolon_separated.csv | ||
| sep.csv | ||
| utf16be.line_break_in_enclosure.csv | ||