AI Animation / Field notes
New tools. New possibilities.
An evolving practice, one experiment at a time.
A geological cross section of creative AI: the tools that turned still images into performances, prompts into worlds, and experiments into films. Follow what each made possible, through work from my own practice.
Explore the films →33 documented milestones
Artbreeder let you breed and blend portraits by adjusting visual traits. WellSaid turned written lines into spoken narration, while TokkingHeads brought a still face to life. Together, these specialized tools offered a practical route to synthetic characters: design a face, give it a voice, and animate a performance without building a conventional 3D character.
Watch the experiment on InstagramVocaloid synthesized singing from a melody and lyrics, giving a musician control over the notes and phrasing. TokkingHeads supplied the moving face. This was an early way to assemble a virtual performer from separate parts: the song could be deliberately composed, while portrait animation supplied expression and presence.
Watch the experiment on InstagramAIVA generated musical compositions, while Vocaloid performed lyrics and melodies as synthesized singing. Artbreeder supplied an editable visual identity and TokkingHeads animated the portrait. Each tool handled a different part of a miniature production—character, performance, and score—making a complete audiovisual experiment possible without a single all-purpose generator.
Watch the experiment on InstagramVQGAN learned a compact visual vocabulary from images and used it to reconstruct or generate new ones. It became a building block of early generative-art notebooks, especially when paired with text guidance. For animation, the appeal was its ability to keep reinventing texture and form: images could seem to grow, melt, and discover new objects as they evolved.
Watch the experiment on InstagramVQGAN supplied the image-making machinery; CLIP judged how closely an image matched a text description. Repeated adjustments pushed the picture toward the prompt. This pairing made language a surprisingly flexible art direction tool, particularly for elaborate imaginary landscapes. Movement still had to be constructed around the image generator, rather than requested as a finished video.
Watch the experiment on InstagramIn this VQGAN-era approach, animation emerged from repeatedly transforming and regenerating an image. Rather than preserving a rigid scene, shapes could become other shapes as the sequence developed. That quality suited fractals, cosmic imagery, and dreamlike journeys: visual instability could become the subject of the animation, instead of simply an error to eliminate.
Watch the experiment on InstagramPYTTI organized text-guided image generation into an animation workflow, with controls for changing scenes, moving the image, and stabilizing successive frames. It offered something closer to directing a sequence than generating isolated pictures. Its evolving textures and continuous transformations were especially useful for music visuals, animated poetry, and journeys through impossible spaces.
Watch the experiment on InstagramPYTTI’s animation workflow combined scene prompts with controlled movement through the evolving image. That made it possible to explore a recognizable visual theme across time—here, a collision of science fiction and steampunk—while allowing unexpected forms to emerge. The creative task was to balance a planned direction with the generator’s tendency to reinterpret the scene.
Watch the experiment on InstagramDisco Diffusion built images through repeated denoising guided by text. Its Turbo workflow reused and warped imagery between fully generated frames, making flowing camera journeys more practical. Cocreator.ai supported back-and-forth writing, while AIVA supplied musical composition. Together, these tools extended generative filmmaking beyond the picture to the words and score surrounding it.
Watch the experiment on InstagramStyleGAN-XL expanded GAN image generation to a wider range of subjects. With CLIP guidance, text could steer a search through its learned visual space. Moving through that space offered fluid changes in appearance and form, making the combination useful for surreal transformations. It was an image-generation route to motion, before dedicated video models became commonplace.
Watch the experiment on InstagramDALL·E made it possible to describe unlikely combinations of objects, settings, and styles and receive a composed image. Its value for animation began before anything moved: it could supply concept art, imagined props, and visual starting points. A phrase such as a Renaissance inventor designing future infrastructure became something an artist could see, select, and develop.
Watch the experiment on InstagramDALL·E 2 brought sharper image generation, variations, and edits to selected regions of a picture. Those controls made it useful for developing an idea through alternatives rather than starting over with every prompt. For animation, it provided a way to establish characters, environments, and key images that could then be animated with other tools.
Watch the experiment on InstagramEarly Midjourney turned short text prompts into richly styled images, making it particularly useful for exploring atmosphere, composition, and an imaginary world’s visual identity. It was an image generator at this stage, not a video model. Animation began by choosing compelling stills and finding ways to move through, transform, or connect them.
Watch the experiment on InstagramDALL·E 2 supplied still images; depth maps estimated which parts were near or far, allowing apparent camera movement through a flat picture. Google’s FILM generated intermediate frames to smooth transitions. Visions of Chaos brought these kinds of experiments into a desktop workflow. The combination helped bridge the gap between a beautiful generated image and a moving sequence.
Watch the experiment on InstagramDisco Diffusion could reimagine footage with prompted textures and imagery, but independently changing frames could flicker. WarpFusion used optical flow—an estimate of how pixels move—to carry the previous result into the next frame. This helped a generated treatment follow an existing performance, making dance and other physical movement useful foundations for AI animation.
Watch the experiment on InstagramStable Diffusion generated images in a compressed visual space, reducing the computation needed compared with working directly at full image resolution. Its later release of model weights let artists run and adapt it on their own hardware. That opened the way to custom tools and animation pipelines, with much more access to the machinery behind each image.
Watch the experiment on InstagramStable Diffusion supplied generated imagery, while Mug Life turned a face in a photograph into an expressive animated character using inferred 3D structure. AIVA handled musical composition and DaVinci Resolve brought the pieces into an edit. This combination separated character design from facial performance and final timing, giving the maker different controls at each stage.
Watch the experiment on InstagramDeforum wrapped Stable Diffusion in an animation system with scheduled prompts and camera-like movement. A frame could be transformed, then regenerated into the next, producing continuous zooms, journeys, and changes of scene. It was especially suited to music-driven visual exploration, where the transformation itself could carry the sequence rather than a consistently acting character.
Watch the experiment on InstagramStable WarpFusion combined Stable Diffusion’s image transformation with techniques for carrying motion and visual information between frames. It helped a generated surface stay attached to an underlying movement, reducing the sense that every frame had been reinvented independently. That made it useful for stylized performances and music videos where recognizable action needed to survive a radical change of appearance.
Watch the experiment on InstagramDALL·E offered a way to develop a visual idea through generated stills, while Splash’s Beatbot turned a short description into a musical sketch. The pairing made it possible to explore picture and soundtrack together very quickly. Its strength was rapid experimentation: finding a mood, a hook, and a visual direction before committing to a longer piece.
Watch the experiment on InstagramRokoko’s motion-capture tools translated human performance into animation data; Stable WarpFusion transformed imagery while carrying motion between frames. These addressed complementary problems: giving a figure intentional movement and giving that movement a new visual treatment. The combination points toward an animation process grounded in performance, with generative tools shaping the final appearance.
Watch the experiment on InstagramRunway Gen-1 transformed existing video using a text prompt or reference image while drawing on the original footage’s structure. A filmed performance could become a different material, character treatment, or world. Its strength was keeping motion as a starting point. Here, I slowed the input before generation and sped it up again in the edit to shape the result.
Watch the experiment on InstagramRunway Gen-2 could generate a short shot from words or an image, reducing the need to film a source performance first. Stable Diffusion, Deforum, and Stable Warp offered other routes through still-image generation and transformation, while Beatbot supplied musical sketches. The toolkit now supported a short imagined story assembled from several kinds of generated material.
Watch the experiment on InstagramGen-2 supplied generated shots; Deforum and Stable WarpFusion offered more constructed animation workflows. MusicLM translated descriptions of mood and instrumentation into music, ChatGPT supported writing, and ElevenLabs synthesized narration. A small production could now span picture, words, and sound. Selection still mattered enormously: I estimated roughly 200 generations to find the basis for this film’s 20 shots.
Watch the experiment on InstagramAnimateDiff added learned motion to compatible Stable Diffusion image models, letting their visual styles carry into short animations. ComfyUI exposed the workflow as connected, reusable processing steps. Midjourney and DALL·E 3 expanded image development, with DALL·E 3 improving attention to detailed prompts. The advance was both motion and control over how different tools were assembled.
Watch the experiment on InstagramPika offered a relatively direct route from a prompt or still image to a short moving shot. DALL·E could establish the imagery, ElevenLabs could give the piece a spoken voice, and AIVA could supply a score. This made illustrated storytelling more approachable: develop a visual scene, animate it, then shape its emotional rhythm with narration and music.
Watch the experiment on InstagramAnimateDiff brought movement to the visual styles of compatible diffusion models; Suno generated music, including songs with synthesized vocals. This pairing made it easier to explore sound and image as one piece rather than treating music as an afterthought. Surreal animation could acquire a musical identity, with editing providing the final relationship between a beat and a visual change.
Watch the experiment on InstagramStability AI’s audio tools brought diffusion-based generation to music and sound: describe a mood, instrumentation, or texture and receive an audio starting point. For filmmakers, that offered another way to explore a soundtrack alongside the images. The useful shift was from searching only through existing recordings to generating material around the atmosphere a sequence needed.
Watch the experiment on InstagramFreepik’s Mystic supplied detailed generated imagery, while Luma offered a route from a still image into a moving shot. Udio generated music and ElevenLabs supplied synthetic audio. The creative opportunity was to treat image, movement, and sound as separate choices, combining specialized tools to develop a more complete audiovisual world.
Watch the experiment on InstagramThis workflow gave each tool a specific job: Midjourney for images, Leonardo for upscaling, Runway and Luma for animation, Suno for music, and ElevenLabs for sound effects. CapCut brought the pieces together. The important advance was a practical production chain—generate, refine, animate, and edit—with the final pacing and relationships still shaped by the filmmaker.
Watch the experiment on InstagramVideo-to-video generation lets an existing animation supply timing and movement while a model reinterprets its appearance. Runway made that a useful bridge between authored motion and generated imagery; ElevenLabs supplied sound effects from text descriptions. Starting with traditional animation gave the experiment a deliberate movement pattern instead of asking the model to invent every part of the shot.
Watch the experiment on InstagramMidjourney’s image-to-video model extended its image-making workflow into short moving scenes. A selected still established the starting composition, and motion prompts helped direct what happened next. For an imagined world, that meant the visual development could lead directly into animation: choose the frame, explore its movement, and assemble the resulting shots into a sequence.
Watch the experiment on InstagramSeedance 2.5, available through Runway, accepts text alongside image, video, and audio references to guide a generated shot. Those references give a maker more ways to specify appearance, action, and sound than a text prompt alone. For short-form filmmaking, the opportunity is to bring a character, performance idea, and visual treatment into the same generation process.
Watch the experiment on Instagram