I find the python ecosystem deeply frustrating. On some level, they fixed some issues in previous languages. But at the same time, they chose to completely flout long-standing conventions, and have rewritten the world in ways that are different for no good reason. And when you rewrite the world, you always end up rewriting only the parts you care about, so your new implementation lacks pieces that other people find very important.
Today's annoyance: manpages. I have some tools written in python that I'm going
to distribute, and since this isn't intended to be user-hostile, I want to ship
manpages. The documentation already exists in the form of docstrings and
argparse option decriptions, so I don't want to write the manpages, but they
should be generated for me. I looked around and I can't find anything anywhere.
There're some hoaky hacks people have come up with to do some of this, but I
don't see any sort of "standard" way to do this at all. Even when I reduce the
requirements to almost nothing, I still can't find anything. Please tell me if
I missed something.
Anyway, I came up with yet another hoaky hack. It works for me. Sample project:
This has a python tool called
frobnicate, and a script to generate its manpage
Makefile has rules to make this
.pod and to convert it into
a manpage. It works by running
frobnicate --help, and parsing that output.
--help message is set up to be maximally useful by including the main
docstring into the message. This is good not just for the manpages, but to make
--help. And this has a nice consequence in that the manpage
generator only needs to look at the
It is assumed that the main docstring (and thus the
--help message) is
formatted like a manpage would be, beginning with a synopsis. This isn't usually
done in python, but it should be; they just like being contrarian for no good
With those assumptions I can parse the
--help output, and produce a reasonable
manpage. Converted to html, it looks like this. Not the most exciting thing in
the world, but that's the point.
This could all be cleaned up, and made less brittle. That would be great. In the meantime, it solves my use case. I'm releasing this into the public domain. Please use it and hack it up. If you fix it and give the changes back, I wouldn't complain. And if there're better ways to have done this, somebody please tell me.
A few people reached out to me with suggestions of tools they have found and/or used for this purpose. A survey:
This lives here and here. I actually found this thing in my search, but it didn't work at all, and I didn't want to go into the rabbithole of debugging it. Well, I debugged it now and know what it needed. Issues
- First off, there're 3 binaries that could be executed:
It appears that only the first one is meant to be functional. The rest throw errors that require the user to debug this thing
- To make it more exciting,
bin/argparse-manpagedoesn't work out of the box on Debian-based boxes: it begins with
Which isn't something that exists there. Changing this to
makes it actually runnable. Once you know that this is the thing that needs running, of course.
- All right, now that it can run, we need to figure out more of this. The
"right" invocation for the example project is this:
$ argparse-manpage/bin/argparse-manpage --pyfile python-argparse-generate-manpages-example/frobnicate --function parse_args > frobnicator.1
Unsurprisingly it doesn't work
- argparse-manpage wants a function that returns an
ArgumentParserobject. This went against what the example program did: return the already-parsed options.
- Once this is done, it still doesn't work. Apparently it attempts to actually run the program, and the program barfs that it wasn't given enough arguments. So for manpage-creation purposes I disable the actual option parsing, and make the program do nothing
- And it still doesn't work. Apparently the function that parses the arguments
doesn't see the
argparseimport when argparse-manpage works with it (even though it works fine when you just run the program). Moving that import into the function makes it work finally. A patch to the test program combining all of the workarounds together:
diff --git a/frobnicate b/frobnicate index 89da764..475ac63 100755 --- a/frobnicate +++ b/frobnicate @@ -16,6 +16,7 @@ import argparse def parse_args(): + import argparse parser = \ argparse.ArgumentParser(description = __doc__, formatter_class=argparse.RawDescriptionHelpFormatter) @@ -29,8 +30,8 @@ def parse_args(): nargs='+', help='''Inputs to process''') - return parser.parse_args() + return parser -args = parse_args() +#args = parse_args().parse_args()
Now that we actually get output, let's look at it. Converted to html, looks like
this. Some chunks are missing because I didn't pass them on the commandline
(author, email, etc). But it also generated information about the arguments
only. I.e. the description in the main docstring is missing even though
argparse was given it, and it's reported with
--help. And the synopsis
section contains the short options instead of an example, like it should.
This is in Debian in the
help2man package. It works out of the box at least.
help2man python-argparse-generate-manpages-example/frobnicate --no-discard-stderr > frobnicator.1
In html looks like this. Better. At least the description made it in. The usage
message is mistakenly in the
NAME section, and the
SYNOPSIS section is
missing with the synopsis ending up in the
DESCRIPTION section. The tool was
not given enough information to do this correctly. It could potentially do
something POD-like where the user is responsible for actually writing out all
the sections, and
help2man would just pick out and reformat the option-parsing
stuff. This would be very heuristic, and impossible to do "right".
There's apparently some way to do this with sphinx. I generally avoid the python-specific reimaginings of the world, so I haven't touched those. Besides, the "python" part is a detail in this project. The sphinx thing is here. And apparently you can invoke it like this:
python3 -m sphinx -b man ...
Not sure what a better solution to all this would be. Maybe some sort of
docstring2man tool that works like
pod2man combined with
docopt instead of