Just another day using Python 3

I was writing a RestructuredText document with bunch of code. Having the document doctested seemed like a good idea.

Tox and pytest were the natural choices here as I don't have to futz around with bootstrapping and testing scripts.

So I make this tox.ini configuration:

[tox]
envlist = py34

# because I only have bunch of documents for now ...
skipsdist = true

[testenv]
deps =
    pytest
    process-tests
    mock
    tornado
    aspectlib
commands = {posargs:py.test -vv}
[pytest]
norecursedirs =
    .git
    .tox
addopts =
    -rxEfs
    --strict
    --doctest-modules
    --doctest-glob *.rst
    --tb short

So I was using Python 3.4 because I want to include some asyncio examples. Unfortunately horror followed:

py34 runtests: PYTHONHASHSEED='2663730781'
py34 runtests: commands[0] | py.test -vv
============================= test session starts =============================
platform win32 -- Python 3.4.0 -- py-1.4.20 -- pytest-2.5.2 -- C:\docs\.tox\py34\Scripts\python.exe
collected 1 items

document.rst: [doctest] document.rst ERROR: InvocationError: 'C:\\docs\\.tox\\py34\\Scripts\\py.test.EXE -vv'
________________________________________________________________________________________________________ summary _________________________________________________________________
ERROR:   py34: commands failed

Now at this point I'm wondering what the hell is going on. Silent failure? Trying to run py.test manually:

C:\docs>.tox\py34\Scripts\py.test -vv
============================= test session starts =============================
platform win32 -- Python 3.4.0 -- py-1.4.20 -- pytest-2.5.2 -- C:\docs\.tox\py34\Scripts\python.exe
collected 1 items

document.rst: [doctest] document.rst

Gah!

Now at this point I remember that py.test captures output by default. Could that cause problems? Seems so:

C:\docs>.tox\py34\Scripts\py.test -vv --capture no
============================= test session starts =============================
platform win32 -- Python 3.4.0 -- py-1.4.20 -- pytest-2.5.2 -- C:\docs\.tox\py34\Scripts\python.exe
collected 1 items

document.rst: [doctest] document.rst
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "C:\docs\.tox\py34\lib\site-packages\_pytest\main.py", line 81, in wrap_session
INTERNALERROR>     doit(config, session)
INTERNALERROR>   File "C:\docs\.tox\py34\lib\site-packages\_pytest\main.py", line 118, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "C:\docs\.tox\py34\lib\site-packages\_pytest\core.py", line 377, in __call__
INTERNALERROR>     return self._docall(methods, kwargs)
INTERNALERROR>   File "C:\docs\.tox\py34\lib\site-packages\_pytest\core.py", line 388, in _docall
INTERNALERROR>     res = mc.execute()
INTERNALERROR>   File "C:\docs\.tox\py34\lib\site-packages\_pytest\core.py", line 289, in execute
INTERNALERROR>     res = method(**kwargs)
INTERNALERROR>   File "C:\docs\.tox\py34\lib\site-packages\_pytest\main.py", line 138, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "C:\docs\.tox\py34\lib\site-packages\_pytest\core.py", line 377, in __call__
INTERNALERROR>     return self._docall(methods, kwargs)
INTERNALERROR>   File "C:\docs\.tox\py34\lib\site-packages\_pytest\core.py", line 388, in _docall
INTERNALERROR>     res = mc.execute()
INTERNALERROR>   File "C:\docs\.tox\py34\lib\site-packages\_pytest\core.py", line 289, in execute
INTERNALERROR>     res = method(**kwargs)
INTERNALERROR>   File "C:\docs\.tox\py34\lib\site-packages\_pytest\runner.py", line 64, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "C:\docs\.tox\py34\lib\site-packages\_pytest\runner.py", line 74, in runtestprotocol
INTERNALERROR>     reports.append(call_and_report(item, "call", log))
INTERNALERROR>   File "C:\docs\.tox\py34\lib\site-packages\_pytest\runner.py", line 110, in call_and_report
INTERNALERROR>     report = hook.pytest_runtest_makereport(item=item, call=call)
INTERNALERROR>   File "C:\docs\.tox\py34\lib\site-packages\_pytest\main.py", line 162, in call_matching_hooks
INTERNALERROR>     return hookmethod.pcall(plugins, **kwargs)
INTERNALERROR>   File "C:\docs\.tox\py34\lib\site-packages\_pytest\core.py", line 381, in pcall
INTERNALERROR>     return self._docall(methods, kwargs)
INTERNALERROR>   File "C:\docs\.tox\py34\lib\site-packages\_pytest\core.py", line 388, in _docall
INTERNALERROR>     res = mc.execute()
INTERNALERROR>   File "C:\docs\.tox\py34\lib\site-packages\_pytest\core.py", line 289, in execute
INTERNALERROR>     res = method(**kwargs)
INTERNALERROR>   File "C:\docs\.tox\py34\lib\site-packages\_pytest\capture.py", line 246, in pytest_runtest_makereport
INTERNALERROR>     rep = __multicall__.execute()
INTERNALERROR>   File "C:\docs\.tox\py34\lib\site-packages\_pytest\core.py", line 289, in execute
INTERNALERROR>     res = method(**kwargs)
INTERNALERROR>   File "C:\docs\.tox\py34\lib\site-packages\_pytest\runner.py", line 208, in pytest_runtest_makereport
INTERNALERROR>     longrepr = item.repr_failure(excinfo)
INTERNALERROR>   File "C:\docs\.tox\py34\lib\site-packages\_pytest\doctest.py", line 61, in repr_failure
INTERNALERROR>     filelines = py.path.local(filename).readlines(cr=0)
INTERNALERROR>   File "C:\docs\.tox\py34\lib\site-packages\py\_path\common.py", line 134, in readlines
INTERNALERROR>     content = self.read('rU')
INTERNALERROR>   File "C:\docs\.tox\py34\lib\site-packages\py\_path\common.py", line 126, in read
INTERNALERROR>     return f.read()
INTERNALERROR>   File "C:\docs\.tox\py34\lib\encodings\cp1250.py", line 23, in decode
INTERNALERROR>     return codecs.charmap_decode(input,self.errors,decoding_table)[0]
INTERNALERROR> UnicodeDecodeError: 'charmap' codec can't decode byte 0x83 in position 309: character maps to <undefined>

==============================  in 0.20 seconds ===============================

It would appear I had some non-ascii characters in there. It's as unavoidable as my own surname (Mărieș - with a ș, not ş) - the document is in Română (Romanian). But cp1250?! My document is in utf-8 - what's going on here?

Turns out Python 3 will use GetACP to get the preferred encoding - which never seems to be the utf-8 code page (cp65001).

At this point I'm very intrigued and want to know if I could change that preferred encoding. But I can't:

C:\docs>py -3 -c "import locale; print(locale.setlocale(locale.LC_ALL, 'eng_US.65001'))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python34\lib\locale.py", line 592, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting

Note

I've tried other stuff. Still could change the preferred encoding ...

Was the argument format incorrect? I think not. With cp1252 it works fine:

C:\docs>py -3 -c "import locale; print(locale.setlocale(locale.LC_ALL, 'eng_US.1252'))"
English_United States.1252

I just give up and just monkey patch that open builtin. Because I still want pytest's test discovery and I don't want to write a script reading up all the documents myself. I should make a pytest bug report at least ...

Made a conftest.py and threw this in:

Note

You think this is ugly? Try patching a builtin yourself.

import aspectlib
aspectlib.weave(
    open,
    lambda open:
        lambda name, mode='r', *args, encoding='utf8', **kwargs:
            open(name, mode, *args, **kwargs)
            if 'b' in mode else
            open(name, mode, *args, encoding=encoding, **kwargs)
)

You can argue that's like killing mosquitoes with a bazooka, but I can get back to my doctests:

py34 runtests: PYTHONHASHSEED='1866354792'
py34 runtests: commands[0] | py.test -vv
============================= test session starts =============================
platform win32 -- Python 3.4.0 -- py-1.4.20 -- pytest-2.5.2 -- C:\docs\.tox\py34\Scripts\python.exe
collected 1 items

document.rst: [doctest] document.rst FAILED

================================== FAILURES ===================================
__________________________ [doctest] document.rst ___________________________
001 Silly stuff::
002
003     >>> print(1)
Expected:
    2
Got:
    1

C:\docs\document.rst:3: DocTestFailure
=========================== short test summary info ===========================
FAIL document.rst
========================== 1 failed in 0.08 seconds ===========================
ERROR: InvocationError: 'C:\\docs\\.tox\\py34\\Scripts\\py.test.EXE -vv'
________________________________________________________________________________________________________ summary _____________________________
ERROR:   py34: commands failed

Yay!

Now before making a patch for these two problems in pytest I wonder if Python 3 should have an escape hatch - let users set the preferred encoding (via a fictive PYTHONPREFERREDENCODING environment variable). At least on Windows where changing the system's locale is just hopeless. Because not all the applications and tools will get support for choosing encodings overnight. What do you think?