PR #2110 added some documentation for an unexpected observation when formula pre-calculation was set to false. I had suggested adding a unit test to demonstrate the observation, but I couldn't find any existing tests for PreCalc. This PR rectifies that omission.
While researching an issue, I noticed that coverage of Document/Properties was poor. Further, the use of int timestamps will eventually lead to problems for 32-bit PHP (see issue #1826).
Coverage Changes:
- Many property types with no special handling are enumerated but not tested. These are removed, but will continue to function as before.
- Existing code theoretically allows property to be set to an object, but there is no means to read or write such a property, and, even if there were, I don't believe Excel supports it. Setting a property to an object will now be changed to a no-op (can throw an exception if preferred).
- Since the Properties object now has no members which are themselves objects, there is no need for a deep clone. The untested __clone method is removed.
- Large switch statements are replaced with associative arrays. Scrutinizer will like that.
- Coverage is now 100%.
<!-- end of coverage changes list -->
Timestamp Changes:
- Timestamps will be stored as int if possible, or float if not. This is, or will soon be, needed for 32-bit systems. Tests have been added for beyond-epoch dates, and run successfully with 32-bit.
- LibreOffice doesn't quite get the Created/Modified properties correct. These are written to the file as a string which includes offset from UTC, but LibreOffice ignores the offset portion when displaying them. Code had been generating these in UTC, but now generates them in default timezone, which should meet user's expectations.
<!-- end of timestamp changes list -->
Other Changes:
- Custom properties added to ODS Writer.
- Samples had not been generating any ODS files. One is now generated.
- Ods uses a single 'keywords' property rather than multiple 'keyword' properties.
- Breaking change - default company is changed to null string from Microsoft Corporation.
- Breaking change of sorts - PropertiesTest incorrectly tested a custom date property against a string, Reader/XlsxTest correctly tested against a timestamp converted to a string. PropertiesTest was defective, and will no longer work as coded; anyone using it as a model will likewise have a problem.
- PHP8.1 has been complaining for weeks about a time zone conversion test. I have now downloaded a version, and changed the code so that it will work in 8.1 as well as prior releases. (It is still likely that the existing code should work in 8.1, but I haven't yet figured out how to file a bug report.) In the course of testing, 3 additional 8.1 problems were reported (all along the lines of "can't pass null to strpos"), and are fixed with null coercion.
- Two Calculation tests failed because of large results on 32-bit system. These are corrected by allowing the functions involved to return float|int rather than int. I suspect that there are other functions with this problem, and will investigate as a follow-up activity.
- See issue #2090. I believe that changes between 17.1 and master will merely cause the problematic spreadsheet to fail in a different way. I believe that enclosing in quotes some variables passed to Document/Properties by Reader/Xlsx will eliminate the problem, but, in the absence of an example file, cannot say for sure.
- Properties tests are now separated out from Reader/XlsxTest and Reader/OdsTest, and now test both Read and Write (via reload).
<!-- end of other changes list -->
Miscellaneous Notes:
- There remains no support for Custom Properties in Xls Reader or Writer.
- We now have default timezones for all of PHP itself, Shared/Date, and Shared/Timezone. That is least one too many. I was unable to disentangle the latter two for this change, but will look into deprecating one or the other in future.
Fix for #2082. Xlsx Writer was writing a cell which is a formula which evaluates to boolean false as an empty XML tag. This is okay for Excel 365, but not for Excel 2016-. Change to write the tag as a value of 0 instead, which works for all Excel releases. Add test.
19_NamedRange.php was not changed to use absolute addressing when that was introduced to Named Ranges. Consequently, the output from this sample has been wrong ever since, for both Xls and Xlsx.
There was an additional problem with Xls. It appears that the Xls Writer Parser does not parse multiple concatenations using the ampersand operator correctly. So, `=B1+" "+B2` was parsed as `=B1+" "`. I believe that this is due to ampersand being treated as a condition rather than an operator; `A1>A2>A3` isn't valid, but `A1&A2&A3` is. My original PR (#1992, which I will now close) only partially resolved this, but I think moving ampersand handling from `condition` to `expression` is fully successful.
There are already more than ample tests for Named Ranges, so I did not add a new one for that purpose. However, I did add a new test for the Xls parser problem.
As issue #2042 documents, SUM behaves differently with invalid strings depending on whether they come from a cell or are used as literals in the formula. SUM is not alone in this regard; COUNTA is another function within this behavior, and the solution to this one is modeled on COUNTA. New tests are added for SUM, and the resulting tests are duplicated to confirm correct behavior for both cells and literals.
Samples 16 (CSV), 17 (Html), and 21 (PDF) were adversely affected by this problem. 17 and 21 were immediately fixed, but 16 had another problem - Excel was not interpreting the UTF8 currency symbols correctly, even though the file was saved with a BOM. After some experimenting, it appears that the `sep=;` line generated by setExcelCompatibility(true) causes Excel to mis-handle the file. This seems like a bug - there is apparently no way to save a UTF-8 CSV with non-ASCII characters which specifies a non-standard separator which Excel will open correctly. I don't know if this is a recent change or if it is just the case that nobody noticed this problem till now. So, I changed Sample 16 to use setUseBom rather than setExcelCompatibility, which solved its problem. I then added new tests for setExcelCompatibility, with documentation of this problem.
* Defined names/formulae in ODS are prefixed by $$ when used in a formula; so we need to strip this out to fully convert them to an Excel formula
* Test for ODS Writer for DefinedNames
* First steps in the implementation of AutoFilters for ODS Reader and Writer, starting with reading a basic AutoFilter range (ignoring row visibility, filter types and active filters for the moment).
And also some additional refactoring to extract the DefinedNames Reader into its own dedicated class as a part of overall code improvement... on the principle of "when working on a class, always try to leave the library codebase in a better state than you found it"
* Provide a basic Ods Writer implementation for AutoFilters
* AutoFilter Reader Test
* AutoFilter Writer Test
* Update Change Log
* Fix for Issue #1887 - Lose Track of Selected Cells After Save
Issue #1887 reports that selected cells are lost after saving Xlsx. Testing indicates that this applies to the object in memory, though not to the saved spreadsheet.
Xlsx writer tries to save calculated values for cells which contain formulas. Calculation::_calculateFormulaValue issues a getStyle call merely to retrieve the quotePrefix property, which, if set, indicates that the cell does not contain a formula even though it looks like one. A side-effect of calls to getStyle is that selectedCell is updated. That is clearly accidental, and highly undesirable, in this case. Code is changed to save selectedCell before getStyle call and restore it afterwards.
The problem was reported only for Xlsx save. To be on the safe side, test is made for output formats of Xlsx, Xls, Ods, Html (which basically includes Pdf), and Csv. For all of those, the object in memory is tested after the save. For Xlsx and Xls, the saved file is also tested. It does not make sense to test the saved file for Csv and Html. It does make sense to test it for Ods, but the necessary support is not yet present in either the Ods Reader or Ods Writer - a project for another day.
* Move Logic Out of Calculation, Add Support for Ods ActiveSheet and SelectedCells
Mark Baker thought logic belonged in Worksheet, not Calculation.
I couldn't get it to work in Worksheet, but doing it in Cell works,
and that has already been used to preserve ActiveSheet over call to
getCalculatedValue, so this just extends that idea to SelectedCells.
Original tests could not completely support Ods because of a lack of support
for ActiveSheet and SelectedCells in Ods Reader and Writer.
There's a lot missing in Ods support, but a journey of 1000 miles ...
Those two particular concepts are now supported for Ods.
* Treat inline strings like strings in Open Document because it has no specific inline-string format
* implement data-type error
Co-authored-by: Mark Baker <mark@lange.demon.co.uk>
* Delete Temporary Files In XssVulnerabilityTest
They need not exist after the test. Some of them are placed in
current directory, which means Git thinks they are needed.
* Replace voku/anti-xss with ezyang/htmlpurifier. Despite anti-xss being a smaller footprint dependency, an a better license fit with our MIT license, there are issues with it's automatic it sanitisation of global variables causing side effects
* Additional unit tests for xss in html writer cell comments
* Initial work modifying the way named ranges are stored, and handled by the calculation engine
This should provide better support for:
- both union and intersection operators in composite named range values
- MS Excel implementation of the union operator duplicating values
- named formulae
- named ranges and formulae that reference other named ranges and formulae
- ranges and formulae that reference multiple ranges across multiple worksheets
* Initial work on handling defined names (named ranges and named formulae) correctly
- UTF-8 names (already extracted as a separate PR and merged)
- distinction between named ranges and named formulae
- correct handling of union and intersection operators in named ranges
- correct evaluation of named range operators in calculations
- calculation support for named formulae
- support for nested ranges and formulae (named ranges and formulae that reference other named ranges/formulae) in calculations
* Minor tweaks before resolving merge conflicts
* Fix extractSheetTitle() method to work on the last ! in a cell reference rather than the first
* Throw exception if a the reference to a defined name in a formula doesn't exist as a defined name
* Properly assess scope for defined names in calculation engine
* Elimination of some redundant code
* Minor tweaks to simplify entries o the stack where we need to check type
* Ensure correct scoping rules are applied when evaluating named ranges and formulae
* Adjustments to Gnumeric Reader for new defined names structure
* Initial work modifying the Ods Reader to handle named ranges, they weren't actually supported previously... this is still ongoing work
* Handle Ranges formatted as 3-d ranges, as long as the references are both to the same worksheet
* Additional testing for Named Ranges formatted as 3-d ranges, as long as the references are both to the same worksheet
* Skip composite named range tests for the moment
* Clean handling for `undefined name` exception when thrown in the calculation engine. Catch and replace with `#NAME?`
* Adjust method we use to determine whether a defined name is a range or a formula
* PHPCS Recommendations
* PHP doesn't support `mixed` yet, at least not at the minium version that we're working with
* More phpcs fixes
* More phpcs appeasements
* Final phpcs fixes for the moment
Still have a lot of echo and var_dump() statements in the code that scrutinizer will hate, but they stay for the moment while this is still WIP
* Please let this be the last of the phpcs fixes
* Unit tests to determine whether a defined name value is a range value or a formula
* phpcs appeasement
* Named tests from provider
* Initial steps for named ranges and formulae in the Ods Reader
* Reading pseudo-3d range addresses in Ods; treat second sheet reference as being identical to the first, which is the majority of cases where this will occur
* Initial work on Gnumeric reader for named ranges and formulae
* Suppress debug logging again
* Remove more debugging displays
* Last minor tweaks before phase two
* Minor refinements
* And all for the want of a space
* A little tidying up
* More tidying up
* phpcs fix
* Modify defined names in rebindParent()
* Renaming variables
* Resolve an issue with locally scoped defined names that don't contain any worksheet reference
* Keep phpcs happy
* Fix quote handling in regexp
* Fix a couple of scrutinizer issues
* Fix a couple of scrutinizer issues
* Update Xlsx Writer to work with the new defined name internal definition
Additional validation checks
* When adding new defined names through the readers, worksheet may not exist if we're only loading selected sheets rather than the full spreadsheet
* If the only thing that phpcs can pickup on is strings in double quotes instead of single quotes, then I know I'm getting close to ready
* Refactor Defined Names logic for Xlsx Writer into its own class
* phpcs keeping me on my toes
* Restore a couple of files that I managed to change without intending to
* Initial work on Ods Write to provide support for saving named ranges and formulae
* Resolve commas to semi-colons s argument separator when writing named formulae for Ods
* Extract Named Expression Writer for Ods into its own class
* Keep phpcs happy
* Refactoring of formula conversion when reading SpreadsheetML; preparation for reading named ranges because they will also need to use the same conversion method
* First pass at reading Named Ranges/Formulae from SpreadsheetML format xml files
* Remove unused namespace reference
* Defined names being written correctly for Xls; but not yet writing cell formulae that reference those defined names... that's the next big step
And I anticipate that defined names that reference other defined names will also be a problem
* Just to keep phpcs happy
... and yes, I know that there are still diagnostic echo statements in the code
* I had to miss some of the phpcs issues didn't I
* Work on the Xls Writer's Parser Tree to identify named range tokens in a formula, and to distinguish them from function tokens
* Still working on packing that d*** defined name reference in the writer
* Throw an exception in the Parser for saving Xls output if we encounter a defined name in a formula... writer will simply write the calculated cell value, and not the formula as at present
Strip out diagnostic output
* Some phpcs appeasement
* Fix a couple of Scrutinizer issues
* Additional verifications to differentiate a formula from a range value
Add explicit getters/setters for named ranges, named formulae and defined names
Additional unit tests
* Styling for closures
* Remove redundant docblocks
* Spaces
* Gah! Namespace use complaints
* Consistency of making calls to DefinedName rather than NamedRange; NamedRange should now be used only for Named Ranges, and should exclude Named Formulae
* Styling
* spurious newline
* No need to test for variable === null when we're typing it in the function argument definition
* Additional unit tests for local/global scoped named ranges and formulae; and a fix to getNamedFormula()
* Fix silly typo that led to breaking test
* Void return signature for unit tests
* Why weren't these picked up in the last pass?
* Refactoring of getNamedRange()/getNamedFormula()
* Eliminate unused constants, and defaults for private method parameters when always called with a value
* Use strict comparisons when comparing object hash codes
* Initial update to documentation for working with named formulae
* Fix for calculation of relative cell references in named ranges/formulae
* Fix current named range tests, because we should be using absolute references; tests for relative named ranges to be added later
* Fix for calculation of relative cell references in named ranges/formulae
* Updates to changelog and documentation for handling of absolute/relative references in named ranges
* Fix last remaining unit test with a named range reference
* Refactor formula conversion for Ods into a separate class; I hadn't realised that it previously wrote formulae as the MS Excel syntax without any conversion to Ods format
* Fix Ods Writer test xml to reflect Ods-native format for formula
* Docblocks
* Drop dollar prefix from Ods formulae and ranges unless it's necessary
* Set the formula convertor in the content writer constructor
* Documentation update
* Minor updates
* Remove var_dumps from file
* Fix the spurious single quote that was breaking named expressions in the Ods Writer... big sigh of relief that I finally spotted it
* Starting work on documentation for Defined Names, and some examples of using Named Ranges and Formulae
* Starting work on documentation for Defined Names, and some examples of using Named Ranges and Formulae
* Example of a relative named range for the documentation
* Mustn't have phpcs problems in sample code either
* More updates to the documentation
* That should conclude the documentation for Named Ranges, now time to move on to documenting Named Formulae
* That should conclude the documentation for Named Ranges, now time to move on to documenting Named Formulae
* PHPCS appeasement in sample code
* Initial documentation on Named Formulae
* PHPCS appeasements
* Additional comments in the documentation, and modify the named range name validation to support a \ as the first character in a name
* Fix breaking build
* Make defined names case-insensitive
* Fix case-insensitivity
* Improved documentation, and additional unit tests
* Additional unit tests, and a fix for removing a globally scoped defined name even if a worksheet is specified in the method call
* Fix unit test for removing named formulae
* Use assertCount instead of assertSame
* Forgotten voids
* Fix arguments for assertCount
* Unit tests for removing defined names, and a fix for removing locally scoped names
* Unit tests for absolute and relative named ranges in calculation engine, and fix an issue with worksheet name in the offset adjustments for relative references
* PHPCS Appeasement
* Additional unit tests, more documentation, and a fix to the calculation engine when no worksheet reference is provided with a named formula
* PHPCS appeasements
* Additional documentation and examples of using Named Formulae
* Additional examples to go with documentation
* A few minor phpcs appeasements
* Minor refactor of updateFormulaReferencesAnyWorksheet() method
* Discard an unused method argument
* Additional unit tests
* Additional unit tests
* Remove unused argument
* Stricter typing
* Fix return typehinting from remove named range/formula; should return the Spreadsheet object
* Use return typehint of self rather than explicit object type
* Redundant code just to keep scrutinizer happy
* Minor change to handle merge conflict
* phpcs fixes after merge
* Namespace usage ordering
* Please let this be the last phpcs fix needed
Co-authored-by: Adrien Crivelli <adrien.crivelli@gmail.com>
This request does not change any source code, only tests.
For a change on which I was working, a test passed when run on its own,
but failed when run as part of the full test suite. It turned out that
an existing test had changed a static value,
thousands separator in this case, and failed to restore it.
The test turned out to be AdvancedBinderTest.
The search for the offending test was more difficult than it should have
been because 26 test scripts which had nothing to do with thousands
separator nevertheless changed that value. They all changed
decimal separator, currency code, and compatibility mode as well,
again for no reason. I changed all of those to eliminate those operations.
I changed the following tests, which actually do change the static
properties identified above for a reason, to restore them as part of teardown.
- CalculationTest sets compatibilityMode and locale
- DayTest sets compatibilityMode, returnDateType, and excelCalendar
- CountTest sets compatibilityMode
- FunctionsTest sets compatibilityMode and returnDateType
- AdvancedValueBinderTest sets currencyCode, decimalSeparator, thousandsSeparator
- StringHelperTest sets currencyCode, decimalSeparator, thousandsSeparator
- NumberFormatTest sets currencyCode, decimalSeparator, thousandsSeparator
- HtmlNumberFormatTest sets currencyCode, decimalSeparator, thousandsSeparator
If row or column dimensions are accessed, then HTML writer would
still generate lots of empty cells, to show nothing at all. This
now ignore row and column dimensions to only output cell that
actually exists (even if those cells are empty).
Fixes#1235Close#1537
* Xls Writer - Correct Timestamp Bug, Improve Coverage
I believe that Xls Writer is 100% covered now.
The Xls Writer sets its timestamp incorrectly. The problem is actually
in Shared/Ole::localDateToOLE, which converts its timestamp using
gmmktime; mktime is correct. If I save a file at 3:00 p.m. in San Francisco,
this bug means the time is actually recorded as 3:00 p.m. UTC.
A consequence of this is that if you use Phpspreadsheet to read the
file and save it as a new Xls, the creation timestamp goes further
and further back in time with each generation (or further forward
if east of Greenwich). One of the tests added confirms that
the creation timestamp is consistent with the start and end times
of the test.
The major change in coverage is adding tests to save GIF and BMP
images, which aren't supported in Xls, but are converted to PNG
in the PhpSpreadsheet code.
Mark Baker
Owen Leibman
xandros15
Adrien Crivelli
Akira Taniguchi
Vivek Kumar
Alexander Gunkel