* jpgraph seems to be finally dying with PHP. Until we have a valid alternative, disabling this run for PHP because it errors
https://github.com/HuasoFoundries/jpgraph looks like a natural successor, but it isn't BC so it will require some work to integrate
* Bitwise Functions and 32-bit
When running the test suite with 32-bit PHP, a failure was reported in BITLSHIFT.
In fact, all of the following are vulnerable to problems, and didn't report
any failures only because of a scarcity of tests:
- BITAND
- BITOR
- BITXOR
- BITRSHIFT
- BITLSHIFT
Those last 2 can be resolved fairly easily by using multiplication by a power of 2
rather than shifting. The first 3 are a tougher nut to crack, and I will continue
to think how they might best be approached. For now, I have added skippable tests
for each of them, which at least documents the problem.
Aside from adding many new tests, some bugs were correctd:
- The function list in Calculation.php pointed BITXOR to BITOR.
- All 5 functions allow null/false/true parameters.
- BIT*SHIFT shift amount must be numeric, can be negative, allows decimal portion
(which is truncated to integer), and has an absolute value limit of 53.
- Because BITRSHIFT allows negative shift amount, its result can overflow
(in which case return NAN).
- All 5 functions disallow negative parameters (except ...SHIFT second parameter).
This was coded, but the code had been thwarted by an earlier is_int test.
* Full Support for AND/OR/XOR on 32-bit
Previous version did not support operands 2**32 through 2**48.
* Improve Coverage of BIN2DEC etc.
The following functions have some special handling
depending on the Calculation mode:
- BIN2DEC
- BIN2HEX
- BIN2OCT
- DEC2BIN
- DEC2HEX
- DEC2OCT
- HEX2BIN
- HEX2DEC
- HEX2OCT
- OCT2BIN
- OCT2DEC
- OCT2HEX
Ods accepts boolean for its numeric argument.
This had already been coded, but there were no tests for it.
Gnumeric allows the use of non-integer argument where Excel/Ods do not.
The existing code allowed this for certain functions but not for others.
Gnumeric consistently allows it, so there is no need for parameter
gnumericCheck in convertBase::ValidateValue.
Again, there were no tests for this.
There were some minor changes needed:
- In functions where you are allowed to specify the numnber of "places" in the
result, there is an upper bound of 10 which had not been enforced.
- Negative values were not handled correctly in some cases.
- There was at least one (avoidable) error on a 32-bit system.
- Some upper and lower bounds were not being enforced. In addition to enforcing
those, the bounds are now defined as class constants in ConvertDecimal.
Many tests have been added, so that Engineering is now almost 100% covered.
The exception is some BESSEL code. There have been some recent changes to
BESSEL which are not yet part of my fork, so I could not address those now.
However, I freely admit that, when I looked at the uncovered portion, it seemed
like it might be a difficult task, so I probably wouldn't have tackled it anyhow.
In particular, the uncovered code seemed to deal with very large numbers,
and, although PhpSpreadsheet and Excel both give very large results for these
conditions, their answers are not particularly close to each other. I think
we're dealing with resuts approaching infinity. More study is needed.
* Coverage for Helper/Samples
I was perplexed by the fact that Helper/Samples seemed to be entirely uncovered when running the test suite, since I know all the samples are run as part of the test. I think that what must be happening is that the Helper code is invoked mostly as part of a Data Provider (and therefore not counted), not as part of the test proper (which would count). So, this change adds a small number of tests which result in Samples being 100% covered.
Covering one statement was tricky - simulating the inability to create a test directory. Mocking, a technique I have not used before, solves this problem admirably.
* Suggestions From Mark Baker
Tests changed from assertEquals to assertSame.
Added @covers annotation to test class.
Validate parameter for method being mocked.
* First step extracting INDIRECT() and OFFSET() to their own classes
* Start building unit tests for OFFSET() and INDEX()
* Named ranges should be handled by the Calculation Engine, not by the implementation of the Excel INDIRECT() function
* When calling the calculation engine to get the range of cells to return, INDIRECT() and OFFSET() should use the instance of the calculation engine for the current workbook to benefit from cached results in that range
There's a couple of minor bugfixes in here; but it's basically just refactoring of the INDIRECT() and OFFSET() Excel functions into their own classes - still needs a lot of work on unit testing; and there's a lot more that could be improved in the code itself (including handling of the a1 flag for R1C1 format in INDIRECT()
* Continue MathTrig Breakup - Trig Functions
Continuing the process of breaking MathTrip.php up into smaller classes.
This round takes care of the trig and hyperbolic functions, plus a few others.
- COS, COSH, ACOS, ACOSH
- COT, COTH, ACOT, ACOTH
- CSC, CSCH
- SEC, SECH
- SIN, SINH, ASIN, ASINH
- TAN, TANH, ATAN, ATANH, ATAN2
- EVEN
- ODD
- SIGN
There are no bug fixes in this PR, except that boolean arguments are now
accepted for all these functions, as they are for Excel.
Taking a cue from what has been done in Engineering, the parameter validation
now happens in a routine which issues Exceptions for invalid values;
this simplifies the code in the functions themselves.
Consistent with earlier changes of this nature, the versions in the
MathTrig class remain, with a doc block indicating deprecation,
and a stub call to the new routines.
I think several more iterations will be needed to break up MathTrig completely.
* Fix SpreadsheetML (xml) detection (#1916)
Replace the unrequired product signature by the
required namespace definition for XML Spreadsheet.
* Add summary to changelog (#1916)
Co-authored-by: Christof Bachmann <c.bachmann@pro54.com>
* Replace manual wildcard logic in MATCH() function with the new WildcardMatch methods
* Additional unit tests
* Refactor input validations
* Refactor actual search logic into dedicated methods
* Eliminate redundant code
* Extract LookupRef\INDEX() into index() method of LookupRef\Matrix class
Additional tests
* Bugfix for returning a column using INDEX()
* Some improvements to ROW() and COLUMN()
* Simplify some of the INDEX() logic, eliminating redundant code
* Fix for Issue #1887 - Lose Track of Selected Cells After Save
Issue #1887 reports that selected cells are lost after saving Xlsx. Testing indicates that this applies to the object in memory, though not to the saved spreadsheet.
Xlsx writer tries to save calculated values for cells which contain formulas. Calculation::_calculateFormulaValue issues a getStyle call merely to retrieve the quotePrefix property, which, if set, indicates that the cell does not contain a formula even though it looks like one. A side-effect of calls to getStyle is that selectedCell is updated. That is clearly accidental, and highly undesirable, in this case. Code is changed to save selectedCell before getStyle call and restore it afterwards.
The problem was reported only for Xlsx save. To be on the safe side, test is made for output formats of Xlsx, Xls, Ods, Html (which basically includes Pdf), and Csv. For all of those, the object in memory is tested after the save. For Xlsx and Xls, the saved file is also tested. It does not make sense to test the saved file for Csv and Html. It does make sense to test it for Ods, but the necessary support is not yet present in either the Ods Reader or Ods Writer - a project for another day.
* Move Logic Out of Calculation, Add Support for Ods ActiveSheet and SelectedCells
Mark Baker thought logic belonged in Worksheet, not Calculation.
I couldn't get it to work in Worksheet, but doing it in Cell works,
and that has already been used to preserve ActiveSheet over call to
getCalculatedValue, so this just extends that idea to SelectedCells.
Original tests could not completely support Ods because of a lack of support
for ActiveSheet and SelectedCells in Ods Reader and Writer.
There's a lot missing in Ods support, but a journey of 1000 miles ...
Those two particular concepts are now supported for Ods.
* Start refactoring the Lookup and Reference functions
- COLUMN(), COLUMNS(), ROW() and ROWS()
- LOOKUP(), VLOOKUP() and HLOOKUP()
- Refactor TRANSPOSE() and ADDRESS() functions into their own classes
* Additional unit tests
- LOOKUP()
- TRANSPOSE()
- ADDRESS()
- Move TREND() functions into the Statistical Trends class
- Unit tests for TREND()
- Create Confidence class for Statistical Confidence functions, and the CONFIDENCE() method
I ran the test suite using 32-bit PHP. There were 2 places where changes
were needed due to 32-bit timestamps.
Reader\\Xml.php was using strtotime as an intermediate step in converting
a string timestamp to an Excel timestamp. The XML file type stores pure timestamps
(i.e. no date portion) as, e.g., 1899-12-31T02:30:00.000, and that value
causes an error using strtotime on a 32-bit system. However, it is sufficient
to use that value in a DateTime constructor, and that will work for 32- and 64-bit.
There was no test for that particular cell, so I added one to the XML read test.
And that's when I discovered the getFormattedValue bug. The cell's format
is `hh":"mm":"ss`. The quotes around the colons are disrupting the formatting.
PhpSpreadsheet formats the cell by converting the Excel format
to a Php Date format, in this case `H\:m\:s`.
That's a problem,
since Excel thinks 'm' means *minutes*, but PHP thinks it means *months*.
This is not a problem when the colon is not quoted; there are ample tests for that.
I added my best guess as to how to recognize this situation,
changing `\:m` to `:i`. The XML read test
now succeeds, and no other tests were broken by this change.
Test Shared\\DateTest had one test where the expected result of converting to a
Unix timestamp exceeds 2**32. Since a Unix timestamp is strictly an int,
that test fails on a 32-bit system. In the discussion regarding recently merged
PR #1870, it was felt that the user base might still be using the functions
that convert to and from a timestamp. So, we should not drop this test, but,
since it cannot succeed on a 32-bit system, I changed it to be skipped
whenever the expected result exceeded PHP_INT_MAX. There are 3 "toTimestamp"
functions within that test. Only one of these had been affected, but I thought
it was a good idea to add additional tests to the others to demonstrate this
condition.
In the course of testing, I also discovered some 32-bit problems with
bitwise and base-conversion functions. I am preparing separate PRs to
deal with those.
* Start splitting some of the basic Statistical functions out into separate classes containing just a few similar functions
* Splitting some of the basic Statistical functions out into separate classes containing just a few similar functions - MAX(), MAXA(), MIN() and MINA()
* Splitting some more of the basic Statistical functions out into separate classes containing just a few similar functions - StandardDeviations and Variances
* _setPageSize's strtoupper() on array argument
PhpSpreadsheet/Writer/Pdf.php Class defines a protected static mixed array called $paperSizes, this array contains string values along with array values. 'strtoupper() expects parameter 1 to be string, array given' error happens due to array passed to $paperSize variable from that $paperSizes mixed array on the Mpdf Class where Pdf extends
Examples of cases, when a 'Letter' paper size is chosen, then no problem occurs since the index in that value for the array is a string value, but when 'Tabloid' paper size is chosen the value in the index for that paper size is an array, that's when the strtoupper() error happens
* _setPageSize's strtoupper() on array argument
PhpSpreadsheet/Writer/Pdf.php Class defines a protected static mixed array called $paperSizes, this array contains string values along with array values. 'strtoupper() expects parameter 1 to be string, array given' error happens due to array passed to $paperSize variable from that $paperSizes mixed array on the Dompdf Class where Pdf extends
Examples of cases; when a 'Letter' paper size is chosen, then no problem occurs since the index in the array for that value a string, but when 'Tabloid' paper size is chosen the value in the index for that paper size is an array, that's when the strtoupper() error happens.
* Avoid the performance/memory overheads of "clone on modify" of $args when building the condition set/database for AVERAGEIFS(), MAXIFS() and MINIFS()
* Avoid the performance/memory overheads of "clone on modify" of $args when building the condition set/database for COUNTIFS()
* 100% Coverage for Calculation/DateTime
The code in DateTime is now completely covered.
Along the way, some errors were discovered and corrected.
- The tests which have had to be changed at the start of every year are
replaced by more robust equivalents which do not require annual changes.
- Several places in the code where Gnumeric and OpenOffice were thought to differ
from Excel do not appear to have had any justification.
I have left a comment where such code has been removed.
- Use DateTime when possible rather than date, time, or strftime functions to avoid
potential Y2038 problems.
- Some impossible code has been removed, replaced by an explanatory comment.
- NETWORKDAYS had a bug when the start date was Sunday. There had been no tests
of this condition.
- Some functions allow boolean and null arguments where a number is expected.
This is more complicated than the equivalent situations in MathTrig because
the initial date for these calculations can be Day 1 rather than Day 0.
- More testing for dates from 1900-01-01 through the fictitious
everywhere-but-Excel 1900-01-29.
- This showed that there is an additional Excel bug - Excel evaluates
WEEKNUM(emptycell) as 0, which is not a valid result for
WEEKNUM without a second argument.
PhpSpreadsheet now duplicates this bug.
- There is a similar and even worse bug for 1904-01-01 in 1904 calculations.
Weeknum returns 0 for this,
but returns the correct value for arguments of 0 or null.
- DATEVALUE should accept 1900-02-29 (sigh) and relatives.
PhpSpreadsheet now duplicates this bug.
- Testing bootstrap sets default timezone. This appears to be a relic from
the releases of PHP where the unwise decision, subsequenly reversed,
was made to issue messages for
"no default timezone is set" rather than just use a sensible default.
This was a disruptive setting for some of the tests I added.
There is only one test in the entire suite which is default-timezone-dependent.
Setting and resetting of default timezone is moved to that test
(Reader/ODS/ODSTest), and out of bootstrap.
- There had been no testing of NOW() function.
- DATEVALUE test had no tests for 1904 calendar and needs some.
- DATE test changed 1900/1904 calendar in use without restoring it.
- WEEKDAY test had no tests for 1904 calendar and needs some.
- Which revealed a bug in Shared/Date (excelToDateTimeObject was not
recognizing 1904-01-01 as valid when 1904 calendar is in use).
- And an additional bug in that legal 1904-calendar values in the 0.0-1.0
range yielded the same "wrong" answers as 1900-calendar (see "One note" below).
Also the comment for one of the calendar-1904 tests was wrong in attempting
to identify what time of day the fraction represented.
I had wanted to break this up into a set of smaller modules, a process already
started for Engineering and MathTrig.
However the number of source code changes was sufficient that I wanted
a clean delta for this request.
If it is merged, I will work on breaking it up afterwards.
One note - Shared/Date/excelToDateTimeObject, when calendar-1900 is in use,
returns an unexpected result if its argument is between 0 and 1,
which is nominally invalid for that calendar.
It uses a base-1970 calendar in that instance. That check is not justifiable
for calendar-1904, where values in that range are legal,
so I made the check specific to calendar-1900,
and adjusted 3 1904 unit test results accordingly. However, I have to admit that
I don't understand why that check should be made even for calendar-1900.
It certainly doesn't match anything that Excel does.
I would recommend scrapping that code altogether.
If agreed, I would do this as part of the break-up into smaller modules.
Another note -
more controversially, it is clear that PhpSpreadsheet needs to support
the Excel and PHP date formats. Although it requires further study,
I am not convinced that it needs to support Unix timestamp format.
Since that is a potential source of Y2038 problems on 32-bit systems,
I would like to open a PR to deprecate the use of that format.
Please let me know if you are aware of a valid reason to continue to support it.
- Refactoring of the Statistical Conditional functions (`AVERAGEIF()`, `AVERAGEIFS()`, `COUNTIF()`, `COUNTIFS()`, `MAXIFS()` and `MINIFS()` to use the new Database functions codebase.
- Extended unit testing
- Fix handling for null values
- Fixes to wildcard text searches
There's still scope for further improvements to memory usage and performance; but for now the code is stable with all unit tests passing
This issue arose while researching issue #1823. The issue was not a bug;
it just required clarification to the author of how to use the software.
But, while researching, I discovered that loading html into 2
sheets of a spreadsheet has a problem if the html title tag is the same
for the 2 sheets. PhpSpreadsheet would be able to save the resulting file,
but Excel would not be able to read it properly because of the duplicate title.
The worksheet setTitle method allows for disambiguation is such a circumstance.
The html reader passed a parameter indicating "don't disambiguate", but I can't
see any harm in changing that to "disambiguate". An extremely simple fix,
with tests to back it up.
* Enable support for dates and percentages in Excel Database functions, and CountIf/AverageIf/etc
* Enable support for booleans in Excel Database functions
* Refactor the Excel Database functions; and rewrite the query building to fix a bug with complex multi-criteria queries that involve both AND and OR conditions
* Fix handling for empty cells and NULL values in searches
* Expand unit tests; and add TODOs for dates, percentages, and wildcard text comparisons
Advanced Value Binder
- Improved format checking/setting for fractions;
- Better percentage checking;
- Some minor refactoring;
- Improved unit testing
I made a documentation change and noticed that the result when browsed
locally did not quite match what is seen when browsing from the web.
After some research, I found https://github.com/mkdocs/mkdocs/issues/2028
That described my situation well and suggested adding an extra
javascript script to the configuration. This worked exactly as desired
on my local machine. This accounts for the presence of extrajs.js
and mkdocs.yml in this request.
In addition to the display problem, "mkdocs build" generates the
documentation into a directory which is not ignored by git. I added
that directory to .gitignore as part of this request.
Finally, since I don't know how exactly the documentation makes it
to production, I made an insignificant change to one doc file as
a sanity check.
Mark Baker
oleibman
Simon Podlipsky
christof-b
Patrick Brouwers
Ivan Stanojevic
adancabanas