ControlNet (Vibe Transfer)

ControlNet allows you to control the composition, pose, and style of generated images using reference images.

Basic Usage

import { NovelAI } from 'novelai-sdk-unofficial';
import { readFileSync, writeFileSync } from 'fs';

const client = new NovelAI({ apiKey: 'your-api-key' });

const poseReference = readFileSync('pose.png');

const images = await client.image.generate({
  prompt: '1girl, standing',
  model: 'nai-diffusion-4-5-full',
  controlnet: {
    images: [
      {
        image: poseReference,
        infoExtracted: 0.7,
        strength: 0.6,
      },
    ],
    strength: 1.0,
  },
});

writeFileSync('output.png', images[0]);

Parameters

ControlNet Object

Parameter	Type	Default	Description
images	ControlNetImage[]	required	Array of ControlNet images
strength	number	1.0	Overall ControlNet strength (0-1)

ControlNetImage Object

Parameter	Type	Default	Description
image	ImageInput	required	Reference image
infoExtracted	number	0.7	Information extraction level (0.01-1)
strength	number	0.6	Individual image strength (0.01-1)
controlnetModel	ImageModel	current model	Model for encoding

Understanding Parameters

Info Extracted

Controls how much structural information is extracted from the reference:

• Lower values (0.1-0.3): Loose interpretation, more creative freedom
• Medium values (0.4-0.7): Balanced extraction
• Higher values (0.8-1.0): Strict structural adherence

Strength

Controls how strongly the extracted information influences the output:

• Lower values (0.1-0.3): Subtle influence
• Medium values (0.4-0.6): Moderate influence
• Higher values (0.7-1.0): Strong influence

Examples

Pose Transfer

const images = await client.image.generate({
  prompt: '1girl, dancing',
  controlnet: {
    images: [
      {
        image: poseImage,
        infoExtracted: 0.8,
        strength: 0.7,
      },
    ],
  },
});

Style/Composition Transfer

const images = await client.image.generate({
  prompt: 'landscape, mountains',
  controlnet: {
    images: [
      {
        image: compositionReference,
        infoExtracted: 0.5,
        strength: 0.5,
      },
    ],
  },
});

Multiple References

const images = await client.image.generate({
  prompt: '1girl, detailed background',
  controlnet: {
    images: [
      {
        image: poseReference,
        infoExtracted: 0.8,
        strength: 0.6,
      },
      {
        image: styleReference,
        infoExtracted: 0.4,
        strength: 0.4,
      },
    ],
    strength: 1.0,
  },
});

Tips

1. Start with defaults - Begin with infoExtracted: 0.7 and strength: 0.6, then adjust.

2. Use clear references - Images with clear composition and structure work best.

3. Balance with prompt - Your prompt should complement, not contradict, the reference.

4. Experiment - Different combinations of infoExtracted and strength produce varied results.

Combining with Other Features

With Character Reference (V4.5)

const images = await client.image.generate({
  prompt: '1girl, standing',
  model: 'nai-diffusion-4-5-full',
  characterReferences: [
    {
      image: characterRef,
      fidelity: 0.75,
    },
  ],
  controlnet: {
    images: [
      {
        image: poseRef,
        strength: 0.6,
      },
    ],
  },
});

With Image-to-Image

const images = await client.image.generate({
  prompt: 'enhanced version',
  i2i: {
    image: baseImage,
    strength: 0.5,
  },
  controlnet: {
    images: [
      {
        image: styleRef,
        strength: 0.4,
      },
    ],
  },
});