Towards Text Generation With Adversarially Learned Neural Outlines.pdf

The key idea is to generate a high-level "neural outline" first, represented as sentence embeddings, which then guides the sequential generation of words.

Method:

1. Adversarial Model for Sentence Embeddings: A Generative Adversarial Network (GAN) learns to approximate the distribution of fixed-length sentence vectors induced by general-purpose sentence encoders. These embeddings serve as outlines for text generation.

2. Autoregressive Generation: A conditional GRU-based language model (decoder) generates words sequentially, conditioned on both the outline and previous outputs.

3. Conditional Text Generation: An extension of the model learns to transform a given hypothesis representation into a premise embedding that satisfies a specified entailment relationship.

4. Interpolation Technique: A gradient-based method is introduced to generate meaningful interpolations between two sentence embeddings, navigating high-density areas of the data manifold.

Results:

- Qualitative results demonstrate natural-looking sentences and interpolations.
- Quantitative evaluations show that conditioning information from generated outlines guides the autoregressive model to produce realistic samples comparable to maximum-likelihood trained language models.

Contributions:

- Leveraging pre-trained sentence representations for text generation.
- Extending GANs to conditional text generation.
- A technique for generating meaningful interpolations between sentence embeddings.