PR #2110 added some documentation for an unexpected observation when formula pre-calculation was set to false. I had suggested adding a unit test to demonstrate the observation, but I couldn't find any existing tests for PreCalc. This PR rectifies that omission.
This change restored behavior from PHP7 in PHP8. In PHP7 calling
setSize(0) resulted in font size being set to 10. The fix addresses
change to equal comparisons in PHP8. Extra comparison is added to keep
result from PHP7 in PHP8 for the setSize(0) case.
While researching an issue, I noticed that coverage of Document/Properties was poor. Further, the use of int timestamps will eventually lead to problems for 32-bit PHP (see issue #1826).
Coverage Changes:
- Many property types with no special handling are enumerated but not tested. These are removed, but will continue to function as before.
- Existing code theoretically allows property to be set to an object, but there is no means to read or write such a property, and, even if there were, I don't believe Excel supports it. Setting a property to an object will now be changed to a no-op (can throw an exception if preferred).
- Since the Properties object now has no members which are themselves objects, there is no need for a deep clone. The untested __clone method is removed.
- Large switch statements are replaced with associative arrays. Scrutinizer will like that.
- Coverage is now 100%.
<!-- end of coverage changes list -->
Timestamp Changes:
- Timestamps will be stored as int if possible, or float if not. This is, or will soon be, needed for 32-bit systems. Tests have been added for beyond-epoch dates, and run successfully with 32-bit.
- LibreOffice doesn't quite get the Created/Modified properties correct. These are written to the file as a string which includes offset from UTC, but LibreOffice ignores the offset portion when displaying them. Code had been generating these in UTC, but now generates them in default timezone, which should meet user's expectations.
<!-- end of timestamp changes list -->
Other Changes:
- Custom properties added to ODS Writer.
- Samples had not been generating any ODS files. One is now generated.
- Ods uses a single 'keywords' property rather than multiple 'keyword' properties.
- Breaking change - default company is changed to null string from Microsoft Corporation.
- Breaking change of sorts - PropertiesTest incorrectly tested a custom date property against a string, Reader/XlsxTest correctly tested against a timestamp converted to a string. PropertiesTest was defective, and will no longer work as coded; anyone using it as a model will likewise have a problem.
- PHP8.1 has been complaining for weeks about a time zone conversion test. I have now downloaded a version, and changed the code so that it will work in 8.1 as well as prior releases. (It is still likely that the existing code should work in 8.1, but I haven't yet figured out how to file a bug report.) In the course of testing, 3 additional 8.1 problems were reported (all along the lines of "can't pass null to strpos"), and are fixed with null coercion.
- Two Calculation tests failed because of large results on 32-bit system. These are corrected by allowing the functions involved to return float|int rather than int. I suspect that there are other functions with this problem, and will investigate as a follow-up activity.
- See issue #2090. I believe that changes between 17.1 and master will merely cause the problematic spreadsheet to fail in a different way. I believe that enclosing in quotes some variables passed to Document/Properties by Reader/Xlsx will eliminate the problem, but, in the absence of an example file, cannot say for sure.
- Properties tests are now separated out from Reader/XlsxTest and Reader/OdsTest, and now test both Read and Write (via reload).
<!-- end of other changes list -->
Miscellaneous Notes:
- There remains no support for Custom Properties in Xls Reader or Writer.
- We now have default timezones for all of PHP itself, Shared/Date, and Shared/Timezone. That is least one too many. I was unable to disentangle the latter two for this change, but will look into deprecating one or the other in future.
This PR came about as I pondered how feasible it was to change the default escape character from backslash to null string, since the latter emulates Excel's own actions. Also, surveying issues relating to CSV, it seems that people are often in a situation where the current defaults aren't optimal for them (e.g. they are in a region where semicolon rather than comma is a better default delimiter). My case and that case can both be handled by methods after a reader is constructed. However, the issues also show that many use `IOFactory::load` rather than `new Csv()`, and the methods to affect the defaults are not available in that case.
Adding a static callback that can be invoked by the constructor addresses all these problems. This can be set as part of the user application's normal initialization, and no special attention needs to be paid to CSV loads thereafter, no matter how they are invoked.
This also makes it feasible to use 'guess' as inputEncoding, by providing a new setFallbackEncoding (default CP1252) method to use if none of the heuristic tests pass. There was already the ability to guess the encoding before `$reader->load()`, but not before `IOFactory::load`.
Almost all typehints in Reader/Csv and Reader/Csv/Delimiter are now part of the function signature rather than in the DocBlock. The exceptions are one method in Delimiter which uses a `resource` parameter, and the `canRead` and `load` methods, which must match the signature in IOFactory. I will look into changing those later.
The Csv Reader tests are moved into their own directory. All Phpstan baseline entries involving Csv Reader are eliminated.
Fix for #2082. Xlsx Writer was writing a cell which is a formula which evaluates to boolean false as an empty XML tag. This is okay for Excel 365, but not for Excel 2016-. Change to write the tag as a value of 0 instead, which works for all Excel releases. Add test.
19_NamedRange.php was not changed to use absolute addressing when that was introduced to Named Ranges. Consequently, the output from this sample has been wrong ever since, for both Xls and Xlsx.
There was an additional problem with Xls. It appears that the Xls Writer Parser does not parse multiple concatenations using the ampersand operator correctly. So, `=B1+" "+B2` was parsed as `=B1+" "`. I believe that this is due to ampersand being treated as a condition rather than an operator; `A1>A2>A3` isn't valid, but `A1&A2&A3` is. My original PR (#1992, which I will now close) only partially resolved this, but I think moving ampersand handling from `condition` to `expression` is fully successful.
There are already more than ample tests for Named Ranges, so I did not add a new one for that purpose. However, I did add a new test for the Xls parser problem.
* Additional unit tests for VLOOKUP() and HLOOKUP()
* Additional unit tests for CHOOSE()
* Unit tests for HYPERLINK() function
* Fix CHOOSE() test for spillage
* Let's see if the tests now pass against PHP8; output file looks to be good
* Font can't be both superscript and subscript at the same time, so we use if/else rather than if/if
* Gnumeric Better Namespace Handling
There have been a number of issues concerning the handling of legitimate but unexpected namespace prefixes in Xlsx spreadsheets created by software other than Excel and PhpSpreadsheet/PhpExcel.I have studied them, but, till now, have not had a good idea on how to act on them. A recent comment https://github.com/PHPOffice/PhpSpreadsheet/issues/860#issuecomment-824926224 in issue #860 by @IMSoP has triggered an idea about how to proceed.
Although the issues exclusively concern Xlsx format, I am starting out by dealing with Gnumeric. It is simpler and smaller than Xlsx, and, more important, already has a test for an unexpected prefix, since, at some point, it changed its generic prefix from gmr to gnm. I added support and a test for that some time ago, but almost certainly not in the best possible manner. The code as changed for this PR seems simpler and less kludgey, both for that exceptional case as well as for normal handling.
My hope is that this change can be a template for similar Reader changes for Xml, Ods, and, especially, Xlsx.
All grandfathered Phpstan issues with Gnumeric are fixed and eliminated from baseline as part of this change.
* Namespace Handling using XMLReader
Adopt a suggestion from @IMSoP affecting listWorkSheetInfo, which uses XMLReader rather than SimpleXML for its processing.
* Update GnumericLoadTest.php
PR #2024 was pushed last night, causing a Phpstan problem with this member.
* Update Gnumeric.php
Suggestions from Mark Baker - strict equality test, more descriptive variable names.
As issue #2042 documents, SUM behaves differently with invalid strings depending on whether they come from a cell or are used as literals in the formula. SUM is not alone in this regard; COUNTA is another function within this behavior, and the solution to this one is modeled on COUNTA. New tests are added for SUM, and the resulting tests are duplicated to confirm correct behavior for both cells and literals.
Samples 16 (CSV), 17 (Html), and 21 (PDF) were adversely affected by this problem. 17 and 21 were immediately fixed, but 16 had another problem - Excel was not interpreting the UTF8 currency symbols correctly, even though the file was saved with a BOM. After some experimenting, it appears that the `sep=;` line generated by setExcelCompatibility(true) causes Excel to mis-handle the file. This seems like a bug - there is apparently no way to save a UTF-8 CSV with non-ASCII characters which specifies a non-standard separator which Excel will open correctly. I don't know if this is a recent change or if it is just the case that nobody noticed this problem till now. So, I changed Sample 16 to use setUseBom rather than setExcelCompatibility, which solved its problem. I then added new tests for setExcelCompatibility, with documentation of this problem.
* Defined names/formulae in ODS are prefixed by $$ when used in a formula; so we need to strip this out to fully convert them to an Excel formula
* Test for ODS Writer for DefinedNames
* First steps in the implementation of AutoFilters for ODS Reader and Writer, starting with reading a basic AutoFilter range (ignoring row visibility, filter types and active filters for the moment).
And also some additional refactoring to extract the DefinedNames Reader into its own dedicated class as a part of overall code improvement... on the principle of "when working on a class, always try to leave the library codebase in a better state than you found it"
* Provide a basic Ods Writer implementation for AutoFilters
* AutoFilter Reader Test
* AutoFilter Writer Test
* Update Change Log
* Pattern Fill style should default to 'solid' if there is a pattern fill style for a conditional; though may need to check if there are defined fg/bg colours as well; and only set a fill style if there are defined colurs
* Fix for Issue 2029 (Invalid Cell Coordinate A-1)
Fix for #2021. When Html Reader encounters an embedded table, it tries to shift it up a row. It obviously should not attempt to shift it above row 1. @danmodini reported the problem, and suggests the correct solution. This PR implements that and adds a test case.
Performing some additional testing, I found that Html Reader cannot handle inline column width or row height set in points rather than pixels (and HTML writer with useInlineCss generates these values in points). It also doesn't handle border style when the border width (which it ignores) is omitted. Fixed and added tests.
* Completion of refactoring for Excel Lookup and Reference functions
* Fix LookupRef tests checking for cell existence
* Fix a couple of now invalid callable references in the Calculation Engine lookup table
Both methods used to optionally return null if passed a
second argument. This second argument was removed entirely and the
method always returns a RowDimension or ColumnDimension respectively
(possibly creating it if needed).
This make the API more predictable and easier to do static analysis
with tools such as PHPStan.
If you relied on that second parameter, you should instead use the
`Worksheet::getRowDimensions()` or `Worksheet::getColumnDimensions()` and
check for existence yourself before calling the getters.
* DateTimeExcel - Change Names of funcWhatever to evaluate
Per discussions while MathTrig was being broken up, this would help standardize the code. This PR applies that standardization to the DateTimeExcel family of functions.
The deprecation messages in DateTime.php are changed to match the style used in PR #2005.
All Phpstan grandfathered errors (about 25) in DateTimeExcel are fixed and removed from baseline. A small number (about 5) of phpstan annotations in the source members in that directory are also fixed and eliminated.
* MathTrig - Fix Phpstan Accomodations
This should be the last of my mass changes to MathTrig. All he Phpstan violations found in baseline which are part of MathTrig are now fixed and removed from baseline. There were about 20 of these.
* MathTrig - Change Names of funcWhatever to evaluate
Per discussions while MathTrig was being broken up, this would help standardize the code. That idea was adopted partway through the breakup. This PR applies that standardization to the earlier efforts. A similar effort is required for DateTime; that will come later. This PR replaces #2006.
The only 2 remaining funcWhatevers in MathTrig are both in SUM, which required two different methods depending on whether or not string parameters were to be ignored. It seems appropriate to leave those method names non-standardized in order to require a decision about which is to be used if they are invoked internally.
3 Phpstan grandfathered errors were eliminated as part of this change, and its baseline has changed accordingly.
Co-authored-by: Mark Baker <mark@lange.demon.co.uk>
* Improved Support for INDIRECT, ROW, and COLUMN Functions
This should address issues #1913 and #1993. INDIRECT had heretofore not supported an optional parameter intended to support addresses in R1C1 format which was introduced with Excel 2010. It also had not supported the use of defined names as an argument. This PR is a replacement for #1995, which is currently in draft status and which I will close in a day or two.
The ROW and COLUMN functions also should support defined names. I have added that, and test cases, with the latest push. ROWS and COLUMNS already supported it correctly, but there had been no test cases. Because ROW and COLUMN can return arrays, and PhpSpreadsheet does not support dynamic arrays, I left the existing direct-call tests unchanged to demonstrate those capabilities.
The unit tests for INDIRECT had used mocking, and were sorely lacking (tested only error conditions). They have been replaced with normal, and hopefully adequate, tests. This includes testing globally defined names, as well as locally defined names, both in and out of scope.
The test case in 1913 was too complicated for me to add as a unit test. The main impediments to it are now removed, and its complex situation will, I hope, be corrected with this fix.
INDIRECT can also support a reference of the form Sheetname!localName when localName on its own would be out of scope. That functionality is added. It is also added, in theory, for ROW and COLUMN, however such a construction is rejected by the Calculation engine before passing control to ROW or COLUMN. It might be possible to change the engine to allow this, and I may want to look into that later, but it seems much too risky, and not nearly useful enough, to attempt to address that as part of this change.
Several unusual test cases (leading equals sign, not-quite-as-expected name definition in file, complex indirection involving concatenation and a dropdown list) were suggested by @MarkBaker and are included in this request.
Openpyxl can generate the xml tag `<patternFill/>`, possibly even as a default style. Excel has no problem with this, treating it as "fill none", but PhpSpreadsheet has a glitch because it treats it as "fill solid white". So, when PhpSpreadsheet loads and saves such a file, the result at first appears as if gridlines are disabled; in fact, the gridlines are merely invisible behind the cells with their solid white fill. This PR makes PhpSpreadsheet behave the same as Excel in this circumstance.
Co-authored-by: Mark Baker <mark@lange.demon.co.uk>
`Worksheet::getCell()` used to optionnaly return null if passed a
second argument. This second argument was removed entirely and the
method always returns a Cell (possibly creating it if needed).
This make the API more predictable and easier to do static analysis
with tools such as PHPStan.
If you relied on that second parameter, you should instead use the
`Worksheet::cellExists()` before calling `getCell()`.
* More Financial function extracts, this time looking at the Periodic Cashflow functions
* Initial extract of Constant Periodic Interest and Payment functions
* Continue MathTrig Breakup - Completion!
Continuing the process of breaking MathTrip.php up into smaller classes. This round takes care of everything that was left:
- ABS
- DEGREES
- EXP
- RADIANS
- SQRT
- SQRTPI
- SUMSQ, SUMX2MY2, SUMX2PY2, SUMXMY2
The only notable logic change was that the 3 SUMX* functions had accepted arrays of unlike length; in that condition, they now return N/A, as Excel does. There had been no tests for this condition.
All the functions in MathTrig.php are now deprecated. Except for COMBIN, the test suite executes them only from MathTrig MovedFunctionsTest. COMBIN is still directly called by some Statistics Binomial functions which have not yet had the opportunity to be re-coded for the new location.
Co-authored-by: Mark Baker <mark@lange.demon.co.uk>
* Let's start with some appeasements to phpstan, just to reduce the baseline
* Appeasements to phpstan, taking the number of reported errors down to just 61
* Extract Normal and Standard Normal Distributions from the Statistical Class
* Extract ZTest from the Statistical Class, and move it to the Standard Normal Distribution class
Additional unit tests for NORMINV()
* Extract LogNormal distribution functions from Statistical
* Continue MathTrig Breakup - Penultimate?
Continuing the process of breaking MathTrip.php up into smaller classes. This round takes care of about half of what is left, so perhaps one round after this one will finish the job:
- ARABIC
- COMBIN; also implemented COMBINA
- FACTDOUBLE
- GCD (which accepts and ignores empty cells as arguments, but returns VALUE if all the arguments are that way; LCM does the same)
- LOG_BASE, LOG10, LN
- implemented MUNIT
- MOD
- POWER
- RAND, RANDBETWEEN (RANDARRAY is too complicated to implement with this ticket)
As you can see from the description, there are some functions which were combined in a single class. When not combined, I adopted PowerKiki's suggestion of using "execute" as the function name.
Co-authored-by: Mark Baker <mark@lange.demon.co.uk>
* Extract a few more Distribution functions from Statistical; this time EXPONDIST() and HYPGEOMDIST()
* Extract the F Distribution (although only F.DIST() is implemented so far
* Updae docblocks
* PHPCS
* Extract Binomial Distribution functions from Statistical
Replace the old MS algorithm for CRITBINOM() (which has now been replaced with te BINOM.INV() function) with a brute force approach - I'll look to refine it later. The MS algorithm is no longer documented, and the implementation produced erroneous results anyway
* Exract the NEGBINOMDIST() function as well; still need to add a cumulative flag to support the additional argument for the newer NEGBINOM.DIST() function
* Rationalise validation of probability arguments
* Difference in variance calculations between Excel/Gnumeric and Open/LibreOffice
* Simplify STDEV() function logic by remembering that STDEV() is simply the square root of VAR(), so we can simply use the VAR() calculaion rather than duplicating the basic logic... and also allow for the differences between Excel/Gnumeric and Open/LibreOffice
* Start implementing Newton-Raphson for the inverse of Statistical Distributions, starting with the two-tailed Student-T
* Additional unit tests and validations
* Use the new Newton Raphson class for calculating the Inverse of ChiSquared
* Extract Weibull distribution, and provide unit tests
* Extract ACCRINT() and ACCRINTM() Financial functions into their own class
Implement additional validations, with additional unit tests
Add support for the new calculation method argument for ACCRINT()
* Additional tests for Amortization functions
Continuing the process of breaking MathTrip.php up into smaller classes. This round takes care of all functions which might be an impediment to installing due to either uncovered code or "complexity":
- BASE
- FACT
- LCM
- MDETERM, MINVERSE, MMULT
- MULTINOMIAL
- PRODUCT
- QUOTIENT
- SERIESSUM
- SUM
- SUMPRODUCT
MathTrig and the members in directory MathTrig are now 100% covered. Many tests have been added, and some edge-case bugs are corrected. Some cases where PhpSpreadsheet had rejected numeric values stored as strings have been changed to accept them whenever Excel does; there had been no tests for that condition.
Boolean arguments are now accepted as arguments wherever Excel accpets them. Taking a cue from what has been done in Engineering, the parameter validation now happens in a routine which issues Exceptions for invalid values; this simplifies the code in the functions themselves. Thank you for doing that; I did not foresee how useful that was when I first looked at it.
Consistent with earlier changes of this nature, the versions in the MathTrig class remain, with a doc block indicating deprecation, and a stub call to the new routines.
All tests except for MINVERSE and MMULT are now handled in the context of a spreadsheet rather than a direct call to the calculation function which implements it. PhpSpreadsheet would need to handle dynamic arrays in order to test MINVERSE and MMULT in a spreadsheet context. Implementing that looks like it might be *very* challenging. It is not something I plan to look at, at least not in the near future.
One parsing problem turned up in the test conversion. It is in one of the SUMIF tests. It takes me to an area in Calculation where the comment says "I don't even want to know what you did to get here". It did not show up in the previous incarnation because, by using a direct call, the previous test managed to bypass the parsing. I have confirmed that this problem shows up in earlier releases of PhpSpreadsheet, so the changes in this PR did not cause it - they merely exposed it. I have left the test intact, but marked it "incomplete" for documentation purposes. I have not been able to get a handle on what's going wrong yet. I will probably open an issue on it if I can't resolve it soon. However, the test in question isn't a "real world" issue, and the error wasn't caused by this change, so I see no reason to delay this pending a resolution of the problem.
SUM had an idiosyncratic moment of its own. It had been ignoring non-numeric values, but Excel returns VALUE in that situation. So I changed it and wrote some new tests, which worked, but ... SUMIF uses several levels of indirection to get to SUM, and SUMIF *does* ignore non-numeric values, so a SUMIF test broke. SUM is a really simple function; the most practical approach seemed to be to clone it, with the string-accepting version being used by the Legacy version (which is called by SUMIF), and the non-string-accepting version being used in the Calculation Function table. That seems far easier and more practical than, for instance, adding a boolean parameter to the variable parameter list. As a follow-up, I will change SUMIF to explicitly call the appropriate new version, but I did not want to add that to this already large change.
SUM again - although it was fully covered beforehand, there was not a specific test member for it. There is now.
FACT had been coded to fail Gnumeric requests where the numeric argument has a decimal portion. However, Gnumeric does accept such an argument, and, unlike Excel and ODS, does not truncate it, but returns the result of a Gamma function call instead. This has been corrected.
When LCM included arguments which contained both 0 and a negative number, it returned 0 or NUM, whichever it found first. It is changed to always return NUM in that circumstance, as Excel does.
QUOTIENT had been documented as taking a variadic list of arguments. In fact, it takes exactly 2 - numerator and denominator - and the docblock and signature is fixed, even in the deprecated version.
The SERIESSUM docbock and signature are more accurate, even in the deprecated version. It is changed to ignore nulls, as Excel does, rather than return VALUE, and is one of the routines which previously rejected numbers in string form.
SUBTOTAL tests had used mocking for some reason. These are replaced with normal tests. And SUBTOTAL had a big surprise in store. That part of it which deals with hidden cells cares only whether the row is hidden, and doesn't care about the column's visibility.
I struggled with whether it should be SubTotal or Subtotal. I think the latter is correct, so that's how I proceeded. I don't think there are likely to be any other capitalization controversies.
* New Bessel Algorithm, providing a higher degree of precision (12 decimal places) and still matching/exceeding MS Excel's precision across the range of values
* Complete Breakup Of Calculation/DateTime Functions
In conjunction with parallel breakups happening in other areas of Calculation, this change breaks up all the DateTime functions into their own classes. All methods remaining in DateTime itself have a doc block deprecation notice, and consist only of stub code to call the replacement methods. Coverage of DateTime itself and all the replacement methods is 100%.
There is only one substantive change to the code (see next paragraph). Among the non-substantive changes, it now adopts the same parsing technique (throwing and catching exceptions) already in use in Engineering and MathTrig. Boolean parameters are allowed in lieu of numbers when Excel allows them. Most of the code changes involve refactoring due to the need to avoid Scrutinizer "complexity" failures in what it will consider to be new methods.
Issue #1936 was opened just as I was staging this. It is now fixed. One existing WORKDAY test was wrong (noted in a comment in the test data file), and a bunch of new tests are added.
I found it confusing to use DateTime as a node of the the class name since most of the methods invoke native DateTime methods. So, everything is moved to directory DateTimeExcel, and that is what is used in the class names.
There are several follow-up activities that I am planning to undertake if this PR is merged.
- ODS supports dates well before 1900. There are exactly 2 assertions for this functionality. More are needed (and some functions might have to change to accept this).
- WEEKDAY has some poorly documented extra options for "style" which are not yet implemented.
- Most tests have been changed to use a formula as entered on a spreadsheet rather than a direct call to the method which implements the formula. There are 3 exceptions at this time. WORKDAY and NETWORKDAYS, which include arrays as part of their parameters, are more complicated than most. YEARFRAC was just too large to deal with now.
- There are direct calls to the now-deprecated methods in both source code and tests, mostly in Financial code, but possibly in others as well. These need to be changed.
- Some constants, none "officially" documented, remain in the original class. These should be either deleted or marked deprecated. I wasn't sure if deprecation was even possible (or desirable), and did not want that to be something which would cause Scrutinizer to fail the change.
* Deprecate Now-unused Constants, Fix Yearfrac bug, Change 3 Tests
Add new DateTime/Constants class, initially populated with constants used in Weeknum.
MS has another inconsistency with how it handles null cells in Yearfrac. Change PhpSpreadsheet to behave compatibly with this bug.
I have modified YearFrac, WorkDay, and NetworkDays tests to be more to my liking. Many tests added to YearFrac because of the bug above. Only minor modifications to the existing tests for the others.
* Start work on breaking down some of the Financial Excel functions
* Unhappy path unit tests for Treasury Bill functions
* Codebase for Treasury Bills includes logic for a different days between settlement and maturity calculation for OpenOffice; but Open/Libre Office now uses the Excel days calculation, so this discrepancy between packages is no longer required
* We've already converted the Settlement and Maturity dates to Excel timestamps, so there's no need to try doing it again when calculating the days between Settlement and Maturity
* Add Unit Tests for the Days per Year helper function
* Extract Interest Rate functions - EFFECT() and NOMINAL() - with additional validation, and unhappy path unit tests
* First pass at extracting the Coupon Excel functions
* Simplify the validation methods
* Extended unit tests to cover all combinations of frequency and basis, including leap years
Fix for COUPDAYSNC() when basis is US 360 and settlement date is the last day of the month
* Ensure that all Financial function code uses the new Helpers class for Days Per Year
Owen Leibman
Matjaž Drolc
MarkBaker
Nathan Dench
xandros15
Adrien Crivelli
Tiago Fernandes
Tiago Fernandes
Akira Taniguchi
Vivek