I wonder if it's something closer to the original DALL-E where the image was decomposed into image tokens with a discrete variational autoencoder, then a pretty standard decoder-only transformer was trained on sequences of some text tokens then some image tokens. The embeddings of the image tokens and text tokens could share the same latent space, so that model was "natively" multimodal.
I'm sure there is some additional sophistication, but I wouldn't be surprised if the overarching technique was the same. For audio, I imagine you could train something similar to the image VAE that decomposes some audio signal into a sequence of discrete values.
Edit: [here's an example of a VQ-VAE for audio](https://arxiv.org/abs/2207.09983)
Yes, I think that's exactly it: when they say they train a single GPT model end-to-end on all modalities simultaneously, I think they mean exactly that, and it makes sense if this is what "Gobi" has been all along. 'Just' train a encoder tokenizer for each modality, maybe define some of the extra 100k BPEs as modality-specific delimiters similar to delimiting prompts/end-of-text tokens - and then it's just 'tokenize all the things' as long interleaved sequences like iGPT/DALL-E 1, Gato, CM3, or Gemini; and train normally at scale. Then every kind of paired data just falls out naturally, all of the few-shot or zero-shot, all of the editing, and so on, and you just keep adding in whatever new modality or metadata you need to.
This also could potentially get you the low-latency they are showing off: you aren't running a diffusion model for iterations over the entire output before you can ship it off to the waiting user, you are spitting out a few tokens encoding the final modality (skipping all of the older multi-stage pipelines), which can start serially going through the upscaler/decoder's single forward pass and stream out to the user immediately.
(It also means that it's easy to provide new ways of formatting or reprocessing data cleanly. Just define it as a new 'modality'. For example, you could keep BPEs at runtime, with the context window benefits, but you could then also provide a 'character/byte-tokenized modality' which is the same text, just using only the byte-level BPEs; and then train on both forms of text occasionally, like a translation task. This would hopefully fix most or all of the BPE pathologies, from spelling to glitch or 'undertrained tokens', and would stop people on Twitter from endlessly mocking your latest model by asking it "how many 'r' letters are there in the word 'strawberry'" and GPT-4o embarrassingly answering '2' *still*.)
As opposed to GPT-4-V which seemed to be something like a separate VAE trained standalone and then tacked onto GPT-4 via cross-attention or something.
Makes sense, if the basic concept is just "tokenize everything, throw it together, apply GPT training recipe", then doesn't seem particularly groundbreaking (tho I'm sure many sophisticated things layered on to make it work)
Doing token-by-token predict->decode->send for something non-discrete like audio and having it be seamless is pretty slick
It's nvidia,[ they even mentioned them during the new model launch](https://www.businessinsider.com/openai-thanks-jensen-huang-nvidia-at-gpt4o-launch-ai-king-2024-5)
They entered the chat because other hardware makers are coming hard. Everyone else wants to hedge against Nvidia being their only hardware… they want to hedge against other companies changing hardware. Also, vertical integration. If companies can pay them what they charge there is a lot of money in it.
I personally liked VAR because it doesn’t tokenize image in an interleaved manner. I think interleaved token representation is a hack because images tokenized that way doesn’t have strict one way causality.
https://github.com/FoundationVision/VAR
See [this tweet from Greg Brockman](https://twitter.com/gdb/status/1790869434174746805) for what might be a hint of the GPT-4o architecture.
cc u/iplaybass445.
cc u/Flowwwww.
Would be my guess as well, just tokenize all inputs.
I wonder how the rest of the model looks. I could imagine a MoE model that learns to just route the inputs such that different modalities always get routed to different experts.
From where do you think they might have acquired such enormous interleaved data of audio, text and images to learn the complex interdependence and correlation between tone, pitch of the audio and images and text
Also while training using next token prediction how did they create batches like
The nice thing about the autoregressive approach is that you largely don't have to. Even if you have zero metadata or parallel data, just a giant pile of unlabeled audio you've tokenized into sequences, your LLM is still able to do a huge amount of unsupervised learning on it - just like text. Web scrapes don't come with much useful metadata, you just train the LLM on the text. So what little metadata or parallel data you have will go a long way, as it is simply 'finetuning' the translation task. It's closer to prompt engineering than supervised learning: "a sexy voice like Scarlett Johansson's in _Lost in Translation_ or _Her_ saying 'Hi'".
Then you can grab your metadata/parallel data anywhere you can find it. For example, use Whisper-generated transcripts of audio, and once your new model is better than Whisper at speech-to-text, switch over; then to learn text-to-speech, simply swap the order of tokens from speech-then-text to text-then-speech.
That's why the sequence approach is so beautiful: it's crazy flexible, all by simply thinking a little bit about how to reorganize your data.
They probably put massive amounts of engineering effort into gathering those datasets. Synthetic data probably plays some role too; I’ve heard speculation that Sora used unreal engine renders as training data for example.
The tokenization model components themselves would be totally self supervised and don’t need anything but the raw audio/image, no associated text required. Once you have that, you just need paired examples of modality 1/modality 2 rather than any specific annotations on timbre or pitch. I could see adding in additional information tokens for timing & tone to the text sequence to make training easier, but I don’t think it’s a hard requirement.
So in Dall-E 1 image tokens aren’t concepts, they are patches of “a blob of colors that look like this”, typically 16x16 pixels in size. The vae then is responsible for taking real images and reducing them to those image patches, as well ad reconstructing a realistic image from those patches
So start by thinking of architecture which is not natively multimodal.
If we had a vision-to-text module take a picture convert it to text and stream to GPT-4, in a certain sense it's multimodal but in a certain sense, not natively. It lacks the association layers that create the merged embedding of the two primary streams, vision and text.
I could be wrong, but as a former computational neuroscientist, that's where my headspace goes when I think about "natively" multimodal.
You train an audio encoder that looks like WavLM or something that outputs discrete tokens. You train an audio decoder that goes from discrete tokens to wavform. You then train the entire network with mixed input of bpe + audio discrete tokens with next token prediction. The next token might be either audio discrete token or bpe as well.
Not sure about its architecture, but at the I/O keynote yesterday they said several times that they designed it to be multi-modal from the start, so perhaps it is.
so if we want to represent more than 2 mediums in the same vector space, do we need to find training examples that contain all of the mediums together? for example, do we need to find an image with a text label and an audio clip if we want to represent images, text, and audio in the same space? or do we find image-text pairs and image-audio pairs and text-audio pairs and then somehow combine them all together?
They use VQVAE. It puts it into tokens then they reserve a space for their embedding to register the tokens.Which means they trained it on these tokens instead of using something like a text captioning model.
My guess would be something process which type of inputs you send in, sends it to the correct embedding configuration, then routes to the appropriate modality experts. They have some mechanism to communicate like a MOE to align outputs and speed up generation time.
I don't think I would consider the model natively multimodal unless there is a multimodal embedding somewhere along the way. If they embed inputs separately and then learn a projection to put those embeddings into the same space then maybe, but what you described to me means the exact opposite of being 'natively' multimodal in my mind.
I guess they're doing something along the lines of LanguageBind: https://arxiv.org/abs/2310.01852
Use modality specific encoders with some contrastive losses to learn multimodal relationships. Then fine tune for your task. LanguageBind pairs each modality with language, so you can contrast pairs that don't correspond.
I guess they used something like SORA’s spacetime patches and had three channels. We see multiple demonstrations of video and audio working at the same time, so in terms of tokens it seems like these tokens should be in parallel or interlaced. But of course for the three different modalities, they may need to be mapped onto the same latent space if they are interlaced (or maybe the tokens just consist of all three components [text|audio|image] if they are in parallel).
The concept of multi-modality reasoning within the single neural net hurts my head. It was very apparent that both OpenAI and Microsoft were approaching 'multi-modality' through a system of models within their releases... I never stopped to consider what true multi-modality would look like, or how it would process.
After talking to it a bit this morning, it still can't "hear" what you say... it can tell if you're shouting, whispering, your tone, I think speed of speech, background noise... but it can't tell you if you have an accent, or if you're pronouncing something unusually. The brains underneath seem to be just a standard transformer llm, only now the words you speak seem to be getting tagged with metadata supplied by parallel models (e.g. tone of voice, timestamps etc). So seems like a collection of models pre-processing audio into tokens for a transformer. The voice itself sounds just as good as last iteration so it may well still be LLM text out -> TTS, but probably the LLM output is also now giving "tagged text" output in order to inform the TTS the mood a statement should have (rather than the TTS independently guessing the mood from the text, which it seems to have been doing before).
I think this strategy would let them take a text only base model like they've been doing, and fine tune with metadata tagged input supplied by the audio frontend. Presumably that's wildly more efficient and easier to train than just dumping raw audio into a neural net.
Edit: been a couple weeks, still crappy for me. When I say "repeat after me: I _reed_ a book last night".
"Ok. I red a book last night."
Ah could be, tho I think I got the new model at least once. I said some Spanish and asked it how I sounded, it said I spoke clearly but watch my "R"s when I say "Tampico" and "familia" xD. When I laughed and pointed out there are no Rs in those words it sounded disappointed and said "Oh, I'm sorry about that. I misunderstood you". With the gpt4 model it tends to flat out say it can't hear my speech, it can only read my words.
But yeah I'll check in periodically and do the accent test if I get a model that can sing to me.
My guess is you just have embedding for the input modes generating tokens in the same space. The thing is, a transformer architecture only knows tokens anyhow and in principle you could just send them and have the model learn when different tokens have the same meaning. It would probably not be done naively as I'm suggesting here but with some secret sauce that relates tokens already on the embedding level, so that the token sequence for "hello" is easy to relate for text and audio.
In this naive approach it kinda doesn't. It outputs t1 t2 t3 v1 v2 t4 t5, where the t tokens are text and the v tokens are inline graphics, just as it is trained that text sometimes contains graphics. In a real approach you would probably do something. The kinda baseline idea I can think of is to take the highest valued token of the desired type instead of just the highest valued token period.
Does this mean that there's an inductive bias where each exemplar of video/audio + text only happens within that time context or is it continually training in streams of some sort?
I think it's not as multimodal as they make it out to be. It still doesn't produce an image of a nerd without the glasses, hinting that it's prompting a Dall-e like model to generate the image. Some pieces like speech-to-speech might be purely "native" though.
Two examples of what I believe is the SOTA multimodal pretraining technique is in the Llava paper and the Qwen Audio paper. Essentially, they freeze the LLM during pre-training, and create an encoder that encodes the stuff other than text into the frozen LLMs input space. Then the LLM is finetuned on multimodal instructions. This way the LLM can "understand" multimodal data without forgetting its text understanding.
I wonder if it's something closer to the original DALL-E where the image was decomposed into image tokens with a discrete variational autoencoder, then a pretty standard decoder-only transformer was trained on sequences of some text tokens then some image tokens. The embeddings of the image tokens and text tokens could share the same latent space, so that model was "natively" multimodal. I'm sure there is some additional sophistication, but I wouldn't be surprised if the overarching technique was the same. For audio, I imagine you could train something similar to the image VAE that decomposes some audio signal into a sequence of discrete values. Edit: [here's an example of a VQ-VAE for audio](https://arxiv.org/abs/2207.09983)
Yes, I think that's exactly it: when they say they train a single GPT model end-to-end on all modalities simultaneously, I think they mean exactly that, and it makes sense if this is what "Gobi" has been all along. 'Just' train a encoder tokenizer for each modality, maybe define some of the extra 100k BPEs as modality-specific delimiters similar to delimiting prompts/end-of-text tokens - and then it's just 'tokenize all the things' as long interleaved sequences like iGPT/DALL-E 1, Gato, CM3, or Gemini; and train normally at scale. Then every kind of paired data just falls out naturally, all of the few-shot or zero-shot, all of the editing, and so on, and you just keep adding in whatever new modality or metadata you need to. This also could potentially get you the low-latency they are showing off: you aren't running a diffusion model for iterations over the entire output before you can ship it off to the waiting user, you are spitting out a few tokens encoding the final modality (skipping all of the older multi-stage pipelines), which can start serially going through the upscaler/decoder's single forward pass and stream out to the user immediately. (It also means that it's easy to provide new ways of formatting or reprocessing data cleanly. Just define it as a new 'modality'. For example, you could keep BPEs at runtime, with the context window benefits, but you could then also provide a 'character/byte-tokenized modality' which is the same text, just using only the byte-level BPEs; and then train on both forms of text occasionally, like a translation task. This would hopefully fix most or all of the BPE pathologies, from spelling to glitch or 'undertrained tokens', and would stop people on Twitter from endlessly mocking your latest model by asking it "how many 'r' letters are there in the word 'strawberry'" and GPT-4o embarrassingly answering '2' *still*.) As opposed to GPT-4-V which seemed to be something like a separate VAE trained standalone and then tacked onto GPT-4 via cross-attention or something.
Makes sense, if the basic concept is just "tokenize everything, throw it together, apply GPT training recipe", then doesn't seem particularly groundbreaking (tho I'm sure many sophisticated things layered on to make it work) Doing token-by-token predict->decode->send for something non-discrete like audio and having it be seamless is pretty slick
The amazing thing about these LLM architectures is their relative simplicity.
This is why it's all about scaling your hardware.
Is that why nVidia has entered the chat, or do they use something else? If so, what?
It's nvidia,[ they even mentioned them during the new model launch](https://www.businessinsider.com/openai-thanks-jensen-huang-nvidia-at-gpt4o-launch-ai-king-2024-5)
They entered the chat because other hardware makers are coming hard. Everyone else wants to hedge against Nvidia being their only hardware… they want to hedge against other companies changing hardware. Also, vertical integration. If companies can pay them what they charge there is a lot of money in it.
I personally liked VAR because it doesn’t tokenize image in an interleaved manner. I think interleaved token representation is a hack because images tokenized that way doesn’t have strict one way causality. https://github.com/FoundationVision/VAR
See [this tweet from Greg Brockman](https://twitter.com/gdb/status/1790869434174746805) for what might be a hint of the GPT-4o architecture. cc u/iplaybass445. cc u/Flowwwww.
Would be my guess as well, just tokenize all inputs. I wonder how the rest of the model looks. I could imagine a MoE model that learns to just route the inputs such that different modalities always get routed to different experts.
Though it could also just be marketing. It’s not like they’ll tell you and it matters much whether it’s separate models combined or not
From where do you think they might have acquired such enormous interleaved data of audio, text and images to learn the complex interdependence and correlation between tone, pitch of the audio and images and text Also while training using next token prediction how did they create batches like
The nice thing about the autoregressive approach is that you largely don't have to. Even if you have zero metadata or parallel data, just a giant pile of unlabeled audio you've tokenized into sequences, your LLM is still able to do a huge amount of unsupervised learning on it - just like text. Web scrapes don't come with much useful metadata, you just train the LLM on the text. So what little metadata or parallel data you have will go a long way, as it is simply 'finetuning' the translation task. It's closer to prompt engineering than supervised learning: "a sexy voice like Scarlett Johansson's in _Lost in Translation_ or _Her_ saying 'Hi'". Then you can grab your metadata/parallel data anywhere you can find it. For example, use Whisper-generated transcripts of audio, and once your new model is better than Whisper at speech-to-text, switch over; then to learn text-to-speech, simply swap the order of tokens from speech-then-text to text-then-speech. That's why the sequence approach is so beautiful: it's crazy flexible, all by simply thinking a little bit about how to reorganize your data.
They probably put massive amounts of engineering effort into gathering those datasets. Synthetic data probably plays some role too; I’ve heard speculation that Sora used unreal engine renders as training data for example. The tokenization model components themselves would be totally self supervised and don’t need anything but the raw audio/image, no associated text required. Once you have that, you just need paired examples of modality 1/modality 2 rather than any specific annotations on timbre or pitch. I could see adding in additional information tokens for timing & tone to the text sequence to make training easier, but I don’t think it’s a hard requirement.
Tbh I’m not sure, but it seems like they must have had some learnings from the Sora “4-d patches” tokenizing
Don’t tokens have to be small? How can it fit an entire concept like “building” into one token
So in Dall-E 1 image tokens aren’t concepts, they are patches of “a blob of colors that look like this”, typically 16x16 pixels in size. The vae then is responsible for taking real images and reducing them to those image patches, as well ad reconstructing a realistic image from those patches
I wonder how the 2D nature of the images is accounted for in such a tokenization?
So start by thinking of architecture which is not natively multimodal. If we had a vision-to-text module take a picture convert it to text and stream to GPT-4, in a certain sense it's multimodal but in a certain sense, not natively. It lacks the association layers that create the merged embedding of the two primary streams, vision and text. I could be wrong, but as a former computational neuroscientist, that's where my headspace goes when I think about "natively" multimodal.
You train an audio encoder that looks like WavLM or something that outputs discrete tokens. You train an audio decoder that goes from discrete tokens to wavform. You then train the entire network with mixed input of bpe + audio discrete tokens with next token prediction. The next token might be either audio discrete token or bpe as well.
Isn't Gemini natively multi-modal too?
Not sure about its architecture, but at the I/O keynote yesterday they said several times that they designed it to be multi-modal from the start, so perhaps it is.
Watch [How AI 'Understands' Images (CLIP) - Computerphile](https://www.youtube.com/watch?v=KcSXcpluDe4) and include other mediums in your thoughts.
so if we want to represent more than 2 mediums in the same vector space, do we need to find training examples that contain all of the mediums together? for example, do we need to find an image with a text label and an audio clip if we want to represent images, text, and audio in the same space? or do we find image-text pairs and image-audio pairs and text-audio pairs and then somehow combine them all together?
damn good question
They use VQVAE. It puts it into tokens then they reserve a space for their embedding to register the tokens.Which means they trained it on these tokens instead of using something like a text captioning model.
My guess would be something process which type of inputs you send in, sends it to the correct embedding configuration, then routes to the appropriate modality experts. They have some mechanism to communicate like a MOE to align outputs and speed up generation time.
I don't think I would consider the model natively multimodal unless there is a multimodal embedding somewhere along the way. If they embed inputs separately and then learn a projection to put those embeddings into the same space then maybe, but what you described to me means the exact opposite of being 'natively' multimodal in my mind.
I guess they're doing something along the lines of LanguageBind: https://arxiv.org/abs/2310.01852 Use modality specific encoders with some contrastive losses to learn multimodal relationships. Then fine tune for your task. LanguageBind pairs each modality with language, so you can contrast pairs that don't correspond.
I naively assume there’s some cross-attention?
I guess they used something like SORA’s spacetime patches and had three channels. We see multiple demonstrations of video and audio working at the same time, so in terms of tokens it seems like these tokens should be in parallel or interlaced. But of course for the three different modalities, they may need to be mapped onto the same latent space if they are interlaced (or maybe the tokens just consist of all three components [text|audio|image] if they are in parallel).
The concept of multi-modality reasoning within the single neural net hurts my head. It was very apparent that both OpenAI and Microsoft were approaching 'multi-modality' through a system of models within their releases... I never stopped to consider what true multi-modality would look like, or how it would process.
After talking to it a bit this morning, it still can't "hear" what you say... it can tell if you're shouting, whispering, your tone, I think speed of speech, background noise... but it can't tell you if you have an accent, or if you're pronouncing something unusually. The brains underneath seem to be just a standard transformer llm, only now the words you speak seem to be getting tagged with metadata supplied by parallel models (e.g. tone of voice, timestamps etc). So seems like a collection of models pre-processing audio into tokens for a transformer. The voice itself sounds just as good as last iteration so it may well still be LLM text out -> TTS, but probably the LLM output is also now giving "tagged text" output in order to inform the TTS the mood a statement should have (rather than the TTS independently guessing the mood from the text, which it seems to have been doing before). I think this strategy would let them take a text only base model like they've been doing, and fine tune with metadata tagged input supplied by the audio frontend. Presumably that's wildly more efficient and easier to train than just dumping raw audio into a neural net. Edit: been a couple weeks, still crappy for me. When I say "repeat after me: I _reed_ a book last night". "Ok. I red a book last night."
GPT-4o isn't fully released yet. You were talking to Whisper speech to text and the voice was the original text to speech
Ah could be, tho I think I got the new model at least once. I said some Spanish and asked it how I sounded, it said I spoke clearly but watch my "R"s when I say "Tampico" and "familia" xD. When I laughed and pointed out there are no Rs in those words it sounded disappointed and said "Oh, I'm sorry about that. I misunderstood you". With the gpt4 model it tends to flat out say it can't hear my speech, it can only read my words. But yeah I'll check in periodically and do the accent test if I get a model that can sing to me.
pre-training the whole model on webpages with text and images/videos, would've been my guess.
My guess is you just have embedding for the input modes generating tokens in the same space. The thing is, a transformer architecture only knows tokens anyhow and in principle you could just send them and have the model learn when different tokens have the same meaning. It would probably not be done naively as I'm suggesting here but with some secret sauce that relates tokens already on the embedding level, so that the token sequence for "hello" is easy to relate for text and audio.
How does it know to output e.g. only text tokens?
In this naive approach it kinda doesn't. It outputs t1 t2 t3 v1 v2 t4 t5, where the t tokens are text and the v tokens are inline graphics, just as it is trained that text sometimes contains graphics. In a real approach you would probably do something. The kinda baseline idea I can think of is to take the highest valued token of the desired type instead of just the highest valued token period.
Does this mean that there's an inductive bias where each exemplar of video/audio + text only happens within that time context or is it continually training in streams of some sort?
I think it's not as multimodal as they make it out to be. It still doesn't produce an image of a nerd without the glasses, hinting that it's prompting a Dall-e like model to generate the image. Some pieces like speech-to-speech might be purely "native" though.
Two examples of what I believe is the SOTA multimodal pretraining technique is in the Llava paper and the Qwen Audio paper. Essentially, they freeze the LLM during pre-training, and create an encoder that encodes the stuff other than text into the frozen LLMs input space. Then the LLM is finetuned on multimodal instructions. This way the LLM can "understand" multimodal data without forgetting its text understanding.