The Penn Treebank tokenizer is fairly widely used. It is a solid tokenizer, that is based only on regex. For example, it is a word tokenizer in NLTK.
NLTK does not make mention of it in their LICENSE.txt. But they don't mention much there, so they may have attribution somewhere else in their documentation.
The original source for the tokeniser is on the waybackmachine
It includes the text:
by Robert MacIntyre, University of Pennsylvania, late 1995.
If this wasn't such a trivial program, I'd include all that stuff about no warrantee, free use, etc. from the GNU General Public License. If you want to be picky, assume that all of its terms apply. Okay?
This is really problematic, AFAICT.
As I see it, one has two choices:
- Do not be picky: then this has no license and so can not be incorporated into any open source project. All rights remain with Robert MacIntyre.
- Be picky: this is effectively under some version of the GPL, it can be incorporated only into things that can incorporate GPL works. Since this work is from 1995, it means it is either GPL 1, or GPL 2, is intended.
Q1. Is this right?
Q2 If so, how is NLTK using it? NLTK is Apache 2. Neither GPL2 nor GPL3 can be incorporated into an Apache 2 project. I'm not sure about GPL1,
Q3. What do I have to do to incorporate it into my own open-source current project? I'd prefer not to change my own license to GPL, but I think I might have to if I want to use this tokenizer.