10

The Penn Treebank tokenizer is fairly widely used. It is a solid tokenizer, that is based only on regex. For example, it is a word tokenizer in NLTK.

NLTK does not make mention of it in their LICENSE.txt. But they don't mention much there, so they may have attribution somewhere else in their documentation.

The original source for the tokeniser is on the waybackmachine

It includes the text:

by Robert MacIntyre, University of Pennsylvania, late 1995.
If this wasn't such a trivial program, I'd include all that stuff about no warrantee, free use, etc. from the GNU General Public License. If you want to be picky, assume that all of its terms apply. Okay?

This is really problematic, AFAICT.

As I see it, one has two choices:

  • Do not be picky: then this has no license and so can not be incorporated into any open source project. All rights remain with Robert MacIntyre.
  • Be picky: this is effectively under some version of the GPL, it can be incorporated only into things that can incorporate GPL works. Since this work is from 1995, it means it is either GPL 1, or GPL 2, is intended.

This seems counter to intuition: being not picky puts you under even more restrictions than being picky would.

Q1. Is this right?

Q2 If so, how is NLTK using it? NLTK is Apache 2. Neither GPL2 nor GPL3 can be incorporated into an Apache 2 project. I'm not sure about GPL1,

Q3. What do I have to do to incorporate it into my own open-source current project? I'd prefer not to change my own license to GPL, but I think I might have to if I want to use this tokenizer.

Improve this question
2
  • 4
    Either (1) Robert MacIntyre later license the tokenizer under Apache 2 (either generally to NLTK specifically) or (2) the NLTK authors ignored the GPL entirely and began distributing a derivative without GPL terms (either because they felt the work was too mechanical to be eligible for copyright, or because they read the flippant language of MacIntyre's license grant as implicit permission to do whatever they wanted, or some other reason). I do not know which option applies in reality.
    – apsillers ♦
    Commented Jun 14, 2017 at 16:23
  • 1
    To clarify your confusion about which version of the GPL applies: if the author doesn't specify a GPL version, you may choose any published version of the GPL.
    – apsillers ♦
    Commented Jun 14, 2017 at 16:26

1 Answer 1

Reset to default
3

This is really problematic, AFAICT.

I think you nailed it. You should file a ticket with NLTK to discuss this further. The key question is whether there is any code of the original sed-based tokenizer that was used there or not.

Improve this answer
2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.