Implementing MySQLi

Getting My Feet Wet

To prepare for what was to be my big project, I rewrote the ini parser to better match Zend. This was released in HHVM 2.4.0. Interestingly, the parser was going to be shipped with HHVM 2.3.0; but when we tested the RC (release candidate), we noticed that we had a bunch of ini files at Facebook that the parser failed upon. So, I had to fix those files before we could release the new one.

MySQLi

After warming up with the parser, I was ready to start my big project: implement MySQLi . This has been a long requested feature for HHVM. And, this extension is required to help meet our compatibility goals .

Preparation

MySQLi is not a trivial extension. As a PHP developer, I am so glad that SaraG has created HNI (HHVM Native Interface), which makes it possible to write extensions in PHP directly instead of writing them in C++. Also, SaraG created a tool called docskel.php , which uses the raw XML that forms the php.net documentation to stub out the appropriate HNI-based signatures for extensions. While the documentation can be wrong sometimes when it comes to return types and parameter types, the stub generation saves a ton of time. After the stub generation, essentially the only work left is to implement all the required functions.

Implementation

For many of the MySQLi methods, it was simply a matter of essentially passing through calls to the older MySQL equivalent implementations. For example, mysqli::real_escape_string() just calls the implementation for mysql_real_escape_string() .

MySQLi has both a procedural and an object-oriented interface. However, because we can write part of the extension in PHP, I could implement almost all of the procedural interface in PHP by just having it call the object-oriented interface. For example, the procedurual function mysqli_real_escape_string() just calls the object-oriented function mysqli::real_escape_string() .

Testing

To test the correctness of my implementation, I used the Zend test suite . There are about 300 tests in this test suite. However, these tests are not meant to be run in parallel. Many tests uses the same database table which, when running in parallel, will randomly make the test fail because some other test touched the same table at the same time. Running the tests in serial wasn’t a great option because that would make development of MySQLi a lot slower. So, instead, I updated our Zend import script to fix the tests to use different tables, thus avoiding the conflicts when running in parallel.

Another testing issue came with what we call ZendParamMode . ZendParamMode is our flag to indicate the calling convention that Zend uses. In short, when using ZendParamMode , if a function is called with a parameter of the wrong type, a warning is logged and the function returns null . The problem is that functions in HNI that are implemented in PHP do not currently support ZendParamMode . So even if the function was doing everything correct, except how it handled ZendParamMode , the test would fail. This made it very hard to see what was implemented correctly and what was incorrect or missing. But, since making the function actually have correct behavior is more important than supporting ZendParamMode , I just changed the import script to remove most of the cases where ZendParamMode was tested.

After fixing the above issues, I was able to perform the normal “where does the test fail and fix it” methodology. This often included reading Zend’s MySQLi source code to determine what the right behavior should be.

Overcoming Roadblocks

The most difficult part of the MySQLi implementation was mysqli_stmt::bind_param() and mysqli_stmt::bind_result() . These methods take a variable number of arguments that are passed by reference. And, at the time, HNI did not support either. First SaraG added support for variable number of arguments , and then I added support for variables as references .

mysqli_multi_query() (and its friends mysqli_more_results() and mysqli_next_result() ) were initially going to be a bit tricky since they were not part of Zend. However, just the opposite occurred. The reason for that is that HHVM already had implementations for mysql_multi_query() , mysql_more_results() and mysql_next_result() , so all I had to do was essentially do a pass through call to those functions.

Current Status

We currently pass 182 of the Zend MySQLi tests. We fail 114. However, many of those 114 tests are failures are what I call “technicality” failures. For example, when we var_dump() a double , we print more digits which makes our output different from Zend.

For real-world compatibility, we pass 100% of doctrine/dbal unit tests and we pass 100% of the CodeIgniter unit tests when running on MySQL.

Missing Pieces

The biggest missing pieces when it comes to passing all Zend tests are:

Bugs that comes from the interaction between MySQLi and php.ini . Reading default values from php.ini is currently missing (but is being developed).

Sometimes we output different error messages which confuses the test runner when it compares output.

Zend has two different MySQL drivers. One is the normal MySQL C API and one is their own MySQL Native Driver. Because HHVM uses the MySQL C API we, don’t support the functions that require the MySQL Native Driver. We may be able to work around this issue for some functions, but, for now, we don’t support any of the functions that require MySQL Native Driver.

What happens when something goes wrong may be different between HHVM and Zend.

What’s next?

Support ZendParamMode for PHP functions in HNI.

Make more of the 114 failing Zend tests pass.

The plan is to release MySQLi in HHVM 2.5.0 . Please test the implementation as much as possible before the 2.5.0 release (you can use a nightly ), so we can continue to fix bugs.

Chase Christian : Thanks WizKid! I was one of the people waiting on the MySQLi and implemented it as soon as your commits were pushed to the nightly build. It has been working great! I haven't had any issues across multiple sites using MySQLi. Thanks again!

WizKid : Awesome to hear.

Marco Pivetta : Awesome! I have just enabled HHVM+MySQLi for Doctrine DBAL on Travis-CI as a reference :-) Hope to see green there soon!

Fred Emmott : This is currently only in the nightly builds, and master, which aren't supported by Travis; these tests will (hopefully :p ) start passing once 2.5.0 is released and Travis upgrade.

Daniel : I'm looking forward to this!

Ulf Wendel : Glad to hear that you found the php.net ext/mysqli/tests to be that strict and picky that they make you scratch your head.... Only picky tests can detect changes made without documenting them.

ms : Any plans for mysqlnd modules like mysqlnd_ms?

WizKid : Didn't even know that there existed other modules for mysqlnd. Currently there is no plan from our side but we would be happy to take pull requests.

CJ : "Zend" is a company, not an implementation of PHP.

Or is this some sort of twisted way to rebrand PHP as Zend, and this hvvm abomination PHP?

Paul Tarjan : Zend is the engine that powered PHP4 and Zend 2 powers PHP5 (but most people just call it zend now since PHP4 is retired). https://en.wikipedia.org/wiki/Zend_Engine

Paul Tarjan : No twisting intended. What would you rather we call things? We need a name for the language (PHP) and the runtime available at php.net. Because the underlying engine is named zend, that is term we usually use: https://en.wikipedia.org/wiki/Zend_Engine Also, why do you think HHVM is an abomination? Are you just trolling or do you have some qualms with our project and there is something we should change?

Johannes : For nitpicking: There is a company Zend Technologies Ltd., anengine called Zend Engine in different versions and a PHP runtie using it. The Zend Engine can be used for very few things in its own, but you need the PHP parts, like HTTP request handling or different PHP modules. But as said this is nitpicking, inpractical terms the software is called PHP, is licensed under PHP Liense (Zend Engine License allows such relicensing) and is developed by a large comunity where even the company Zend Technologies Ltd. is just one between others. Historially the split was more relevant as there was a clear separation between both things and it was more possible to switch out the engine, nowadays, especially with PHP 5 OO APIs this isn't the case anymore. Also historically Zend (the company) was by far the largest contributor to the engine, that has changed and some independent contributors see it as unfair to be pulled in that way. I myself don't have thosestrictfeelings, but some do ...

Schien : In Zend PHP there are benefits of using MySQLi over MySQL. Is the MySqli extension for HHVM just for parity I suppose? We changed the default db wrapper in our frameworks to MySQL so that they work with HHVM out of the box. There's no need to switch back to MySqli is there? Thanks.

Paul Tarjan : We don't mean to offend anyone, we just need names for things. What should we call the binary available on php.net? "The reference PHP implementation" is a bit long, "PHP" is confusing to the language, and "php-src" is a bit awkward as a noun.

Marco Pivetta : I already enforced DBAL to build on nightly ones - loads of tests are still failing though ;-)