oleibman c3f53854b6 Php/iconv Should Not Treat FFFE/FFFF as Valid (#2910) ... Fix #2897. We have been relying on iconv/mb_convert_encoding to detect invalid UTF-8, but all techniques designed to validate UTF-8 seem to accept FFFE and FFFF. This PR explicitly converts those characters to FFFD (Unicode substitution character) before validating the rest of the string. It also substitutes one or more FFFD when it detects invalid UTF-8 character sequences. A comment in the code being change stated that it doesn't handle surrogates. It is right not to do so. The only case where we should see surrogates is reading UTF-16. Additional tests are added to an existing test reading a UTF-16 Csv to demonstrate that surrogates are handled correctly, and that FFFE/FFFF are handled reasonably.		2022-07-02 08:53:39 -07:00
..
NumberFormatTest.csv	Work on setting up locale-aware formatted number conversion for the Csv Reader	2022-03-02 08:53:29 +01:00
NumberFormatTest.de.csv	Work on setting up locale-aware formatted number conversion for the Csv Reader	2022-03-02 08:53:29 +01:00
backslash.csv	Allow CSV escape character to be set	2018-05-23 10:31:41 +09:00
contains_html.csv	Could not open CSV file containing HTML fragment	2018-06-25 11:12:27 +09:00
csv_without_extension	Could not open CSV file containing HTML fragment	2018-06-25 11:12:27 +09:00
empty.csv	Check for MIME type to know if CSV reader can read a file	2018-02-05 21:33:23 +09:00
enclosure.csv	Better auto-detection of CSV separators	2017-12-28 12:25:37 +09:00
encoding.iso88591.csv	Improve Coverage for CSV (#1475)	2020-05-17 18:15:18 +09:00
encoding.utf8.csv	Improve Coverage for CSV (#1475)	2020-05-17 18:15:18 +09:00
encoding.utf8bom.csv	Improve Coverage for CSV (#1475)	2020-05-17 18:15:18 +09:00
encoding.utf16be.csv	Improve Coverage for CSV (#1475)	2020-05-17 18:15:18 +09:00
encoding.utf16le.csv	Improve Coverage for CSV (#1475)	2020-05-17 18:15:18 +09:00
encoding.utf32be.csv	Improve Coverage for CSV (#1475)	2020-05-17 18:15:18 +09:00
encoding.utf32le.csv	Improve Coverage for CSV (#1475)	2020-05-17 18:15:18 +09:00
escape.csv	CSV - Guess Encoding, Handle Null-string Escape (#1717)	2020-12-25 17:47:29 +01:00
issue.2232.csv	Work on setting up locale-aware formatted number conversion for the Csv Reader	2022-03-02 08:53:29 +01:00
line_break_escaped_32le.csv	CSV Sample File Was Miscoded (#1489)	2020-05-24 19:57:39 +09:00
line_break_in_enclosure.csv	Fix CSV delimiter detection on line breaks	2018-10-21 18:23:55 +11:00
line_break_in_enclosure_with_escaped_quotes.csv	CSV Sample File Was Miscoded (#1489)	2020-05-24 19:57:39 +09:00
no_delimiter.csv	Csv reader avoid notice when the file is empty	2018-10-28 14:16:53 +11:00
premiere.utf8.csv	CSV - Guess Encoding, Handle Null-string Escape (#1717)	2020-12-25 17:47:29 +01:00
premiere.utf8bom.csv	CSV - Guess Encoding, Handle Null-string Escape (#1717)	2020-12-25 17:47:29 +01:00
premiere.utf16be.csv	CSV - Guess Encoding, Handle Null-string Escape (#1717)	2020-12-25 17:47:29 +01:00
premiere.utf16bebom.csv	CSV - Guess Encoding, Handle Null-string Escape (#1717)	2020-12-25 17:47:29 +01:00
premiere.utf16le.csv	Php/iconv Should Not Treat FFFE/FFFF as Valid (#2910)	2022-07-02 08:53:39 -07:00
premiere.utf16lebom.csv	CSV - Guess Encoding, Handle Null-string Escape (#1717)	2020-12-25 17:47:29 +01:00
premiere.utf32be.csv	CSV - Guess Encoding, Handle Null-string Escape (#1717)	2020-12-25 17:47:29 +01:00
premiere.utf32bebom.csv	CSV - Guess Encoding, Handle Null-string Escape (#1717)	2020-12-25 17:47:29 +01:00
premiere.utf32le.csv	CSV - Guess Encoding, Handle Null-string Escape (#1717)	2020-12-25 17:47:29 +01:00
premiere.utf32lebom.csv	CSV - Guess Encoding, Handle Null-string Escape (#1717)	2020-12-25 17:47:29 +01:00
premiere.win1252.csv	CSV - Guess Encoding, Handle Null-string Escape (#1717)	2020-12-25 17:47:29 +01:00
semicolon_separated.csv	…
sep.csv	Improve Coverage for CSV (#1475)	2020-05-17 18:15:18 +09:00
utf16be.line_break_in_enclosure.csv	Improve Coverage for CSV (#1475)	2020-05-17 18:15:18 +09:00