Synthetic Faces: Training Sets and the Diversity Trap

TL;DR

• StyleGAN and FFHQ create synthetic faces trained on 70,000 AI-generated portraits, eliminating real-person likeness and copyright concerns when building consistent LoRA character models
• The "diversity trap" reveals that synthetic faces inherit demographic biases from their training data sources, offering statistical but not true human representation
• Techniques like LoRA fine-tuning on GAN outputs provide ethical alternatives for character consistency, especially for therapeutic and creative applications

Why Synthetic Faces Are Changing Character Creation

The rise of generative AI has created a new paradigm for character design. Instead of recruiting real models or scanning photographs, artists turn to AI-generated portraits as training material for "consistent character" LoRA (Low-Rank Adaptation) models. This approach promises two advantages: it eliminates legal and ethical complications of using real people's likenesses without consent, and provides variation while maintaining core facial features. But beneath this solution lies a critical issue: the "diversity trap." AI-generated faces appear diverse but are statistically bounded by their training datasets' inherent biases.

Technical Foundation: From FFHQ to StyleGAN

StyleGAN traces back to Nvidia's architecture, which fundamentally changed AI image generation. The system uses the FFHQ dataset as its training ground — 70,000 high-quality PNG images at 1024×1024 resolution with diverse human faces including variation in age, ethnicity, and background.

FFHQ stands out for its ethical stance. As the documentation states: "Please note that this dataset is not intended for, and should not be used for, development or improvement of facial recognition technologies." Though FFHQ's images were crawled from Flickr without individual consent, inheriting "all the biases of that website," this early awareness about privacy and consent implications of facial datasets set a precedent.

The breakthrough lies in how StyleGAN learns to separate high-level attributes (like pose and identity) from fine-grained variations (freckles, hair color). The original paper explains this as "an automatically learned, unsupervised separation of high-level attributes (e.g., pose and identity when trained on human faces) and stochastic variation in the generated images (e.g., freckles, hair)." This disentanglement became the foundation for creating consistent yet diverse synthetic faces.

Wang's prediction that AI would become much better at synthesizing images proved accurate, leading to commercially viable services like Generated Photos, which published 100,000 AI faces as stock imagery by September 2019.

How LoRA Training Works with Synthetic Faces

The breakthrough connecting synthetic faces to character consistency comes from the LoRA technique. When artists train a LoRA (Low-Rank Adaptation) on a set of AI-generated faces from the same latent identity, they create a model that can generate that specific character in any context. The Jenni Green LoRA on Civitai exemplifies this approach: "Model created by training on different person with similarly looking photos." This LoRA has accumulated 5,340+ downloads and 34 positive reviews, demonstrating practical demand.

For Stable Diffusion users, LoRAs represent a significant advancement over traditional fine-tuning methods. Instead of requiring hundreds of real photos, modern LoRA training typically uses just 15-30 images — making AI-generated portraits an attractive alternative, particularly when those images originate from the same underlying AI model and share consistent visual characteristics.

The Ethical Shield: Consent Without Compromise

The most compelling argument for synthetic face training centers on consent. As Caroline Sinders, a machine learning designer and fellow at Harvard Kennedy School, explained:

"'On one hand, it is better to have a database full of images that are not real people—that doesn't violate anyone's consent. But what are those data sets used for? And who benefits from those systems?'"

This ethical positioning resonates strongly with communities who need character alternatives for therapeutic reasons. The Civitai source article explicitly addresses this: "LORAS don't have to be 'Accommodating' or for Alter reasons, it's not just for a mental health reason - if you need consistent characters that's what this list is also for." This includes both therapeutic use (for people with dissociative identity disorder) and general creative purposes.

The result is what Civitai calls "nobody" LoRAs — characters whose faces belong to no real person. These avoid the legal minefield of intellectual property rights and personality protection laws that complicate real-person likeness usage. Artists can create diverse character portfolios without worrying about copyright infringement or violating someone's right to their likeness.

The Diversity Trap: Statistical Representation

However, the apparent solution introduces its own problems. The "diversity trap" refers to the misconception that synthetic faces offer meaningful human variety when, in fact, their diversity is bounded by their training data. As FFHQ's documentation acknowledges, its images were crawled from Flickr and "inheriting all the biases of that website." The faces StyleGAN generates reflect Flickr's demographic skew — predominantly tech-savvy, younger demographics — rather than representing humanity more broadly.

Caroline Sinders, a machine learning designer specializing in AI ethics, challenged Generated Photos' "infinite diversity" claims during their 2019 push for corporate adoption:

"'Generally, I raise my eyebrows at anything as "infinite diversity,"' machine learning designer Caroline Sinders ... told Motherboard. 'Who is determining or defining that? Are their definition or parameters open to audit? Can they be changed? What is their process for ensuring infinite diversity?'"

Sinders' critique highlights the fundamental flaw in treating synthetic diversity as human diversity. The AI now merely redistributes existing social biases in digital form. When corporations champion synthetic faces as "diverse alternatives to traditional stock photos," they're often importing the same demographic blind spots under a tech-friendly guise.

This problem traces back to historical representation imbalances. As the NYTimes documented, even camera technology has struggled with darker skin tones since the Kodak Shirley cards era — proof that removing human control doesn't eliminate bias, it merely shifts its source.

Technical Trade-Offs and Alternatives

Understanding these limitations requires recognizing the trade-offs between synthetic and real-world training sets. StyleGAN2-ADA addresses small-dataset training through adaptive data augmentation, but fundamentally all GAN models reflect their training data's characteristics. The key is understanding what diversity means for their specific use case: geographic variety, age representation, or gender balance?

The Future Beyond Binary Choices

The conversation around synthetic faces needs to move beyond simple consent versus diversity trade-offs. As the technology evolves, hybrid approaches combine AI generation with human curation. Artists can "fine-tune" synthetic faces, adding personal artistic direction that neither pure real-world nor pure synthetic datasets can provide alone. As Sinders noted, the question isn't just about consent, but about benefits and control: Who defines the parameters for diversity? Can those parameters be auditable? Who benefits from these systems?

Common Errors and Fixes

FAQ

Q: What are "nobody" LoRAs and how do they differ from regular LoRAs? A: "nobody" LoRAs are characters whose faces belong to no real person, avoiding likeness and consent issues. Regular LoRAs use real-person photos. The term on Civitai marks these ethically unambiguous character packs.

Q: Can I use a LoRA trained on synthetic faces commercially? A: Most synthetic face datasets like FFHQ carry CC BY 2.0 or similar licenses allowing commercial use. Check the specific model's license. The Jenni Green LoRA demonstrates commercial viability with 5,340+ downloads.

Q: Will training LoRAs on synthetic faces help with character consistency? A: Yes. Synthetic faces from the same StyleGAN identity share underlying facial structure that's difficult to achieve with diverse real-world photos. This makes them valuable for animation and consistent world-building.

Q: Are synthetic face LoRAs more diverse than traditional stock photos? A: Not necessarily. Synthetic faces often reflect FFHQ's Flickr demographic biases (younger, tech-oriented users). Real-world stock photos might offer better geographic and age diversity, though at the cost of consent issues.

Q: What's the "diversity trap"? A: The false assumption that AI-generated faces offer meaningful human variety when they're actually bounded by training data biases. Generated Photos' "infinite diversity" claims were critiqued as marketing hype.

Q: Do synthetic face LoRAs work with specific Stable Diffusion checkpoints? A: Most "nobody" LoRAs support SD1.5 and SDXL checkpoints, with some extending to PDXL. The Jenni Green LoRA specifically targets SDXL.

Sources

1. https://github.com/NVlabs/ffhq-dataset — FFHQ dataset documentation (Nvidia)
2. https://arxiv.org/abs/1812.04948 — StyleGAN paper (Karras et al., 2018)
3. https://www.theverge.com/tldr/2019/2/15/18226005/ai-generated-fake-people-portraits-thispersondoesnotexist-stylegan — The Verge article on ThisPersonDoesNotExist
4. https://en.wikipedia.org/wiki/StyleGAN — Wikipedia StyleGAN article
5. https://www.vice.com/en/article/generated-photos-thinks-it-can-solve-diversity-with-100000-fake-ai-faces/ — Vice article on Generated Photos diversity claims
6. https://www.nytimes.com/interactive/2020/11/21/science/artificial-intelligence-fake-people-faces.html — NYTimes interactive on fake faces
7. https://civitai.com/models/120480/sdxl-jenni-green-this-person-does-not-exist — Civitai model page for Jenni Green LoRA
8. https://generated.photos/ — Generated Photos homepage (Icons8)
9. https://arxiv.org/abs/1703.06868 — AdaIN paper (Huang & Belongie, 2017) — foundational mechanism

graph TD
    A[FFHQ Dataset<br/>70,000 Faces at 1024×1024] --> B[StyleGAN Training<br/>Disentangled Latent Space]
    B --> C[AdaIN Mechanism<br/>Style Transfer]
    C --> D[ThisPersonDoesNotExist.com<br/>Real-Time Generation]
    D --> E[Generated Photos<br/>100,000 Stock Faces]
    A --> F[Training Data Biases<br/>Flickr Demographics]
    E --> G[LoRA Training<br/>5-30 Images per Character]
    G --> H[Synthetic Face LoRAs<br/>Consistent Characters]
    H --> I[Nobody Tag<br/>Therapeutic Use]
    H --> J[Commercial Use<br/>Jenni Green LoRA]
    F --> K[Diversity Trap<br/>Statistical Limitation]
    K --> L[Critical Analysis<br/>AI Ethics]
    style I fill:#e8f5e8,stroke:#4caf50
    style J fill:#fff3e0,stroke:#ff9800
    style L fill:#ffebee,stroke:#f44336