Generating manpages from python and argparse

I find the python ecosystem deeply frustrating. On some level, they fixed some issues in previous languages. But at the same time, they chose to completely flout long-standing conventions, and have rewritten the world in ways that are different for no good reason. And when you rewrite the world, you always end up rewriting only the parts you care about, so your new implementation lacks pieces that other people find very important.

Today's annoyance: manpages. I have some tools written in python that I'm going to distribute, and since this isn't intended to be user-hostile, I want to ship manpages. The documentation already exists in the form of docstrings and argparse option decriptions, so I don't want to write the manpages, but they should be generated for me. I looked around and I can't find anything anywhere. There're some hoaky hacks people have come up with to do some of this, but I don't see any sort of "standard" way to do this at all. Even when I reduce the requirements to almost nothing, I still can't find anything. Please tell me if I missed something.

Anyway, I came up with yet another hoaky hack. It works for me. Sample project:

https://github.com/dkogan/python-argparse-generate-manpages-example

This has a python tool called frobnicate, and a script to generate its manpage as a .pod. The Makefile has rules to make this .pod and to convert it into a manpage. It works by running frobnicate --help, and parsing that output.

The --help message is set up to be maximally useful by including the main docstring into the message. This is good not just for the manpages, but to make an informative --help. And this has a nice consequence in that the manpage generator only needs to look at the --help output.

It is assumed that the main docstring (and thus the --help message) is formatted like a manpage would be, beginning with a synopsis. This isn't usually done in python, but it should be; they just like being contrarian for no good reason.

With those assumptions I can parse the --help output, and produce a reasonable manpage. Converted to html, it looks like this. Not the most exciting thing in the world, but that's the point.

This could all be cleaned up, and made less brittle. That would be great. In the meantime, it solves my use case. I'm releasing this into the public domain. Please use it and hack it up. If you fix it and give the changes back, I wouldn't complain. And if there're better ways to have done this, somebody please tell me.

Update

A few people reached out to me with suggestions of tools they have found and/or used for this purpose. A survey:

argparse-manpage

This lives here and here. I actually found this thing in my search, but it didn't work at all, and I didn't want to go into the rabbithole of debugging it. Well, I debugged it now and know what it needed. Issues

First off, there're 3 binaries that could be executed:
- argparse-manpage
- wrap
- bin/argparse-manpage
It appears that only the first one is meant to be functional. The rest throw errors that require the user to debug this thing
To make it more exciting, bin/argparse-manpage doesn't work out of the box on Debian-based boxes: it begins with
```
#!/bin/python
```
Which isn't something that exists there. Changing this to
```
#!/usr/bin/python
```
makes it actually runnable. Once you know that this is the thing that needs running, of course.
All right, now that it can run, we need to figure out more of this. The "right" invocation for the example project is this:
```
$ argparse-manpage/bin/argparse-manpage --pyfile python-argparse-generate-manpages-example/frobnicate  --function parse_args > frobnicator.1
```
Unsurprisingly it doesn't work
argparse-manpage wants a function that returns an ArgumentParser object. This went against what the example program did: return the already-parsed options.
Once this is done, it still doesn't work. Apparently it attempts to actually run the program, and the program barfs that it wasn't given enough arguments. So for manpage-creation purposes I disable the actual option parsing, and make the program do nothing
And it still doesn't work. Apparently the function that parses the arguments doesn't see the argparse import when argparse-manpage works with it (even though it works fine when you just run the program). Moving that import into the function makes it work finally. A patch to the test program combining all of the workarounds together:

diff --git a/frobnicate b/frobnicate
index 89da764..475ac63 100755
--- a/frobnicate
+++ b/frobnicate
@@ -16,6 +16,7 @@ import argparse


 def parse_args():
+    import argparse
     parser = \
         argparse.ArgumentParser(description = __doc__,
                                 formatter_class=argparse.RawDescriptionHelpFormatter)
@@ -29,8 +30,8 @@ def parse_args():
                         nargs='+',
                         help='''Inputs to process''')

-    return parser.parse_args()
+    return parser



-args = parse_args()
+#args = parse_args().parse_args()

Now that we actually get output, let's look at it. Converted to html, looks like this. Some chunks are missing because I didn't pass them on the commandline (author, email, etc). But it also generated information about the arguments only. I.e. the description in the main docstring is missing even though argparse was given it, and it's reported with --help. And the synopsis section contains the short options instead of an example, like it should.

help2man

This is in Debian in the help2man package. It works out of the box at least. Invocation:

help2man python-argparse-generate-manpages-example/frobnicate --no-discard-stderr > frobnicator.1

In html looks like this. Better. At least the description made it in. The usage message is mistakenly in the NAME section, and the SYNOPSIS section is missing with the synopsis ending up in the DESCRIPTION section. The tool was not given enough information to do this correctly. It could potentially do something POD-like where the user is responsible for actually writing out all the sections, and help2man would just pick out and reformat the option-parsing stuff. This would be very heuristic, and impossible to do "right".

Others

There's apparently some way to do this with sphinx. I generally avoid the python-specific reimaginings of the world, so I haven't touched those. Besides, the "python" part is a detail in this project. The sphinx thing is here. And apparently you can invoke it like this:

python3 -m sphinx -b man ...

Thoughts

Not sure what a better solution to all this would be. Maybe some sort of docstring2man tool that works like pod2man combined with docopt instead of argparse.