Wednesday, December 12, 2018

PyMysql and the strange ubyte format error at connection

I didn't write much lately, mainly because I had to leave Java aside and turn to Python in my new work. However, interesting problems spring not only in Java, so this will probably become a not Java only blog.

The problem I met recently is linked to the pymysql library, that we use to connect to our SQL databases. Connection to the databases was always working fine, and then we started using dbdeployer to run a light SQL database for our unit tests. Running the tests on local machines would always work, but some tests would just fail on our Jenkins machine, with this strange error message:

struct.error: ubyte format requires 0 <= number <= 255
The good thing is that the python runner would just pinpoint the code in error:
data += struct.pack('B', len(connect_attrs)) + connect_attrs
The problem here is that we are trying to fit the length of a string into a byte. And the string is probably longer than 255 characters. A good thing with Pyhton being interpreted is that you have access to all the library's code. So let's have a look...
So what is this connect_attrs variable? It is a concatenation of keys and values from the _connect_attrs attribute of the Connection class. But this map is supposed to be quite small:
self._connect_attrs = { '_client_name': 'pymysql', '_pid': str(os.getpid()), '_client_version': VERSION_STRING, }
 Looking a little bit further shows an update to the map:
if program_name: self._connect_attrs["program_name"] = program_name elif sys.argv: self._connect_attrs["program_name"] = sys.argv[0]
If we do not define a program name, it retrieves it from sys.argv[0], which is the name of the file, including its complete path! That is why it failed sometimes in Jenkins, the workspace being quite deep under the root folder.
So the workaround becomes quite simple:
pymysql.connect( host=DB_HOST, port=DB_PORT, user=DB_USER, password=DB_PASSWORD, db=DB_NAME, program_name="my_program_name_that_fixes_the_bug" )

2 comments:

  1. Thank you thank you thank you!! Finally I came across an answer! Everybody keeps saying to change the connection settings, files input, etc - but this was what I was looking for. The reason it happened and HOW exactly would I be able to solve it. Thank you!

    ReplyDelete
  2. The automatic program name feature was removed in version 0.9.3, which fixes this bug.

    ReplyDelete