Technical Review: CLIP Visual Embeddings

OpenAI CLIP: Technical Review and Current Source Assessment

Review date:July 1, 2026

Scope:The original OpenAI CLIP research project, its released OpenAI checkpoints, the current official repository, and the official model card. This review does not treat every later model described as "CLIP" or every OpenCLIP checkpoint as behavior of the original OpenAI release.

Executive assessment

CLIP's central contribution was to replace a permanently fixed image-classification head with a shared image-text representation. An image encoder and a text encoder map their inputs into a common space, and a contrastive objective makes matched image-text pairs more similar than mismatched pairs. A new classification task can then be expressed through text prompts rather than by retraining the visual encoder for a new fixed label vocabulary.[^1]

That core interpretation in the original review is correct. The most useful output of CLIP is often a reusable embedding, and natural language can be used to construct a zero-shot classifier, a retrieval query, or a supervised feature representation. However, several distinctions are necessary:

-model.encode_image()andmodel.encode_text()return unnormalized projected features in the released code.

-model(image, text)normalizes those features and returns scaled similarity logits, not embeddings or calibrated class probabilities.

-The scale is a learned checkpoint parameter. The paper describes clipping it during training, but the released inferenceforward()method does not visibly apply that clamp.

-The text architecture, embedding dimension, input resolution, and preprocessing depend on the checkpoint.

-Prompt wording and the candidate taxonomy are part of the classifier.

-The official model card describes any deployed use as out of scope without careful task-specific evaluation, and it restricts the released model to English-language use.[^2][^3][^4]

CLIP should therefore be understood as a powerful, historically important image-text representation system, not as a universal semantic oracle, a calibrated recognition service, a general language embedding model, or a native object-detection and segmentation system.

1. What CLIP actually produces

The original OpenAI release exposes three related interfaces:

image_features = model.encode_image(image)

text_features = model.encode_text(text)

logits_per_image, logits_per_text = model(image, text)

The first two operations return projected feature vectors. In the released source they are not normalized before being returned. The complete modelforward()method performs L2 normalization, exponentiates the learnedlogit_scale, and multiplies the cosine-similarity matrix by that scale.[^3]

This distinction matters because the following two operations are not equivalent:

raw_feature_dot_product = image_features @ text_features.T

and:

image_features = image_features / image_features.norm(dim=-1, keepdim=True)

text_features = text_features / text_features.norm(dim=-1, keepdim=True)

cosine_similarity = image_features @ text_features.T

The official zero-shot example manually normalizes the outputs ofencode_image()andencode_text()before computing similarities.[^2]A vector database that stores raw features from one code path and normalized features from another can therefore produce inconsistent ranking behavior.

The embedding is also not a direct explanation of an image. Individual coordinates generally have no stable human-readable meaning. The useful information lies in relative geometry under a specific checkpoint, preprocessing pipeline, and normalization convention. Even two checkpoints with the same output dimension occupy independently learned coordinate systems and should not be mixed in one index.

2. Training data and the contrastive objective

The original paper describes a training set of approximately 400 million image-text pairs, called WebImageText or WIT. The authors constructed a query vocabulary of about 500,000 terms and collected up to 20,000 examples per query to reduce extreme imbalance.[^1]

For a batch ofNmatched image-text pairs, CLIP forms anN x Nsimilarity matrix. The diagonal contains the supplied positive pairs. The off-diagonal entries are treated as negatives by the symmetric image-to-text and text-to-image cross-entropy losses. In the original large-scale training, the batch size was 32,768, providing many in-batch alternatives for each positive pair.[^1]

A practical implication is that an off-diagonal pair can be treated as a negative even when it is semantically compatible. Two captions may describe similar concepts, or two images may legitimately fit the same phrase. This is an inference from the batch construction rather than a separately reported CLIP benchmark result, but it is important when interpreting the learned geometry: the objective learns relative matching under noisy web supervision, not a logically complete semantic ontology.

The authors did not begin with the final contrastive design. In their experiments, predicting exact captions was much less efficient for learning zero-shot classifiers. A bag-of-words objective improved efficiency, and replacing that objective with contrastive image-text matching improved ImageNet zero-shot learning efficiency by roughly another factor of four in the reported comparison.[^1]

The training recipe was computationally substantial. The paper reports 32 training epochs, mixed-precision optimization, a batch size of 32,768, and very large GPU allocations for the largest checkpoints. The high-resolution ViT-L/14@336px model was produced by continuing the lower-resolution ViT-L/14 model for one additional epoch at 336-pixel resolution rather than training a completely unrelated architecture from scratch.[^1]

The 400-million-pair training dataset was not released, and the official repository does not provide a complete end-to-end reproduction system for the original pretraining run. The project released inference and evaluation code plus checkpoints, but exact pretraining reproduction and independent auditing of the full data mixture are not possible from the public artifacts alone.[^4]

3. Similarity, temperature, and probability

CLIP applies linear projections to the visual and textual representations, normalizes both projected vectors, and computes a cosine-similarity matrix. A learned scalar then controls the sharpness of the resulting logits.[^1][^3]

The paper describes the temperature as initialized to 0.07 and optimized during training, with the corresponding logit scale clipped to a maximum of 100 for stability.[^1]The current released source initializes:

self.logit_scale = nn.Parameter(torch.ones([]) * np.log(1/0.07))

and later computes:

logit_scale =self.logit_scale.exp()

logits_per_image = logit_scale * image_features @ text_features.t()

No clamp is visible in the released inferenceforward()path.[^3]This is a paper-versus-source distinction: the paper describes a training safeguard, while the distributed inference class uses the learned value stored in the checkpoint.

The README says that the returned logits are cosine similarities multiplied by 100, and its standalone zero-shot example also uses the constant100.0.[^2]That is a convenient approximation to the trained maximum scale, but the source implementation is more exact:model(image, text)usesexp(model.logit_scale). Reproducible work should report whether it used the model's learned scale, a fixed value such as 100, or unscaled cosine similarity.

A softmax over CLIP logits is conditional on the candidate set. Adding or removing labels can change every resulting value. The highest softmax output therefore does not mean that the chosen class is correct, and it is not a candidate-independent probability that the concept is present. If the available labels are all wrong, softmax still distributes essentially all mass among them.

For practical classification, an application may need anunknownor rejection mechanism, domain-calibrated thresholds, and explicit testing of alternative taxonomies. For retrieval, nearest neighbor similarly means "closest among the indexed candidates," not necessarily "correct."

4. Image encoders and released checkpoints

The original project evaluated modified ResNet and Vision Transformer image encoders.[^1]

The modified ResNet differs from a standard torchvision ResNet in several ways. The released implementation uses a three-convolution stem, average pooling rather than the usual initial max-pool design, anti-aliased downsampling, and attention pooling instead of ordinary global average pooling.[^3]

The Vision Transformer path uses image patches and a transformer encoder. The original paper reports ViT-B/32, ViT-B/16, ViT-L/14, and a 336-pixel ViT-L/14 variant. The current official loader lists nine released checkpoints in total:

RN50

RN101

RN50x4

RN50x16

RN50x64

ViT-B/32

ViT-B/16

ViT-L/14

ViT-L/14@336px

The model list is encoded directly in the official loader, along with OpenAI-hosted checkpoint URLs and expected SHA-256 digests.[^5]

Embedding dimension is checkpoint-specific. For example, the OpenAI ViT-B checkpoints use a 512-dimensional shared representation, while ViT-L/14 uses 768 dimensions. Several ResNet variants use other dimensions.[^1]Dimension alone is not enough to establish compatibility: two independently trained 512-dimensional checkpoints do not share a coordinate system.

A durable embedding index should therefore store at least the exact checkpoint, checkpoint hash, embedding dimension, input resolution, preprocessing configuration, feature-normalization convention, and software revision.

5. Text encoder and tokenizer

The paper's base text encoder is a 12-layer causal Transformer with width 512 and eight attention heads. That description should not be generalized to every released checkpoint. The paper reports increasing text-transformer width for larger ResNet configurations while keeping the depth fixed.[^1]

The text representation is taken from the highest-layer activation at the end-of-text position, then layer-normalized and linearly projected into the shared space. In the released source, the end-of-text position is found withtext.argmax(dim=-1)because the end-of-text token has the highest token ID in the sequence.[^3]

The tokenizer is a lowercasing, byte-level BPE implementation. It first applies text repair and repeated HTML unescaping, normalizes whitespace, converts the text to lowercase, encodes bytes through a reversible Unicode mapping, and applies BPE merges.[^6]

There is an official paper/source inconsistency in the vocabulary count:

-The paper reports a vocabulary of 49,152 tokens.[^1]

-The released tokenizer source constructs 256 byte symbols, 256 end-of-word forms, 48,894 merge-derived entries, and two special tokens. By direct calculation from the visible slice and construction code, this yields 49,408 entries.[^6]

The 49,408 figure is a source-code derivation, not a number explicitly printed by the tokenizer file. Researchers should prefer the loaded checkpoint'stoken_embedding.weight.shape[0]when documenting the actual vocabulary size used by a released model.

All official released models use a context length of 77 token positions. The tokenizer adds both start-of-text and end-of-text tokens, leaving at most 75 ordinary BPE positions when the complete text fits. The defaulttruncate=Falsebehavior raises an error when an input is too long; truncation occurs only when explicitly requested.[^5]

CLIP's text encoder should be described as a visual-semantic text encoder. It was not trained as a general-purpose long-document or sentence-embedding model, and the official model card says the released model was not intentionally trained or evaluated for languages other than English.[^4]

6. Image preprocessing is part of the checkpoint

The official inference preprocessing performs bicubic resizing, center cropping, RGB conversion, tensor conversion, and normalization with fixed channel means and standard deviations.[^5]

This means CLIP represents the processed image, not every pixel of the original file. Center cropping can remove peripheral objects, captions, or contextual details. A classification or retrieval result may change if a different crop, resize rule, color conversion, or normalization is used.

Training and inference preprocessing should also be kept separate. The paper describes using a random square crop from resized images during original training. The released inference transform is deterministic center cropping.[^1][^5]

For reproducibility, the preprocessing function and its dependency versions should be treated as part of the model. The current requirements file lists packages such astorchandtorchvisionwithout exact version pins, while the README still uses an older PyTorch 1.7.1-era installation example.[^2][^7]A reproducible environment should therefore record the actual PyTorch, torchvision, Pillow, and device configuration rather than relying only on the name "OpenAI CLIP."

7. Zero-shot classification and prompt design

For zero-shot classification, each class is converted into one or more textual prompts. The text embeddings act as the directions of an L2-normalized linear classifier, and the image embedding is compared against those directions.[^1]

Prompt engineering is not cosmetic. In the original experiments, replacing a bare class name with the contexta photo of a {label}improved ImageNet top-1 accuracy by 1.3 percentage points. The authors also used task-specific context to reduce ambiguity and better match dataset domains.[^1]

Prompt ensembling further improved performance. For ImageNet, the paper used 80 templates and reported a 3.5-point improvement beyond the default prompt, for an overall gain of almost five points from prompt engineering plus ensembling.[^1]

These results show that a zero-shot classifier is jointly defined by:

-the checkpoint;

-the image preprocessing;

-the class names;

-the prompt templates;

-any class descriptions or disambiguation;

-the method used to combine prompt embeddings;

-the candidate class set;

-the logit scale and decision rule.

A methods section that says only "we used CLIP zero-shot classification" is incomplete. Prompt text should be treated like executable model configuration and preserved exactly.

The candidate taxonomy can also alter bias and error behavior. The original paper and model card report that harmful and disparate classifications can change depending on which categories are included or excluded.[^1][^4]Open vocabulary therefore transfers part of the responsibility for classifier construction from training time to application design time.

8. Transfer, ImageNet, and robustness

The best model in the original study reached 76.2 percent zero-shot ImageNet top-1 accuracy and approximately matched the original supervised ResNet-50 without using ImageNet's labeled training set for task-specific fitting.[^1]This remains a historically important result, but it is a 2021 result for the original OpenAI checkpoint family, not a statement of current state-of-the-art vision performance.

The paper evaluated transfer across more than 30 datasets spanning object recognition, scenes, textures, OCR, actions, geolocation, counting, distance estimation, and natural distribution shifts.[^1]The breadth of this evaluation is one of the strongest parts of the work.

CLIP's robustness result also needs precise wording. The paper studies "effective robustness" relative to models with comparable ImageNet accuracy on a specified set of natural ImageNet distribution shifts. Zero-shot CLIP often preserved more of its performance under those shifts than conventionally supervised ImageNet models. This is not proof of universal robustness to every corruption, sensor, scientific domain, adversarial input, or cultural distribution.[^1]

The paper includes an especially useful specialization result. Training a supervised logistic-regression probe on ImageNet features raised ImageNet accuracy by 9.2 points to 85.4 percent, but average accuracy on the selected natural distribution shifts did not improve and fell slightly overall.[^1]This demonstrates that stronger in-distribution specialization can reduce the relative robustness advantage of the zero-shot classifier.

Zero-shot performance and representation quality are not identical. A poor prompt may fail to expose information that remains present in the image representation. Conversely, a good linear probe on frozen features does not prove that the natural-language classifier is reliable. The model card also notes that linear probes can underestimate representation performance in some settings.[^4]

9. Retrieval and embedding applications

The shared image-text space naturally supports text-to-image retrieval, image-to-text ranking, and image-to-image similarity. The official code makes these applications easy because embeddings can be computed once, stored, and compared later.[^2]

These are derived uses of the representation, not guarantees that a raw nearest-neighbor system is production-ready. A robust retrieval application should evaluate:

-recall and precision at multiple ranks;

-query classes for which no valid result exists;

-similarity distributions in the target domain;

-rejection thresholds;

-prompt sensitivity;

-crop and image-transformation sensitivity;

-subgroup and content-category behavior;

-embedding drift after software or checkpoint changes.

A second-stage reranker or domain-specific verifier may be appropriate for high-precision applications. The nearest vector is always the nearest vector, even when every indexed item is irrelevant.

CLIP image embeddings are also useful as frozen features for supervised models. The official repository includes a logistic-regression example on frozen image features.[^2]This should be distinguished from zero-shot inference because it uses labeled target-domain examples and can change the balance between in-domain accuracy and distribution robustness.

10. Limitations, bias, and intended use

The released CLIP model produces one global representation per image. It does not natively return bounding boxes, instance masks, semantic-segmentation maps, or an explicit object-relation graph. Later systems can use CLIP features inside localized architectures, but those capabilities should not be attributed to the vanilla OpenAI model.

The original model and model card identify limitations in fine-grained classification and counting. A global semantic vector can emphasize that an image contains dogs while representing exact cardinality or detailed spatial relations less reliably.[^4]

Bias is not restricted to a final classifier head. The embedding is learned from internet image-text data and can encode social, cultural, demographic, and platform-specific associations. The model card notes that the data disproportionately reflect people and societies with greater internet presence and skew toward developed countries and younger, male users.[^4]

The official use restrictions are stronger than a generic recommendation to "test before deployment":

-The model card says any deployed use, commercial or otherwise, is out of scope for the released model.

-Even constrained non-deployed image search is not recommended without thorough in-domain testing using a fixed taxonomy.

-Surveillance and facial-recognition uses are always out of scope.

-Use should be limited to English because the model was not intentionally trained or evaluated for other languages.[^4]

These statements describe the intended scope of the original OpenAI release. They should be included in a current technical review, especially when the review discusses production search, moderation, people-related classification, or multilingual retrieval.

11. Current official software state

The official repository remains deliberately small. It provides model loading, tokenization, embedding generation, zero-shot examples, linear-probe examples, and nine checkpoint URLs. It does not provide a vector database, production retrieval service, complete pretraining stack, or a modern deployment framework.[^2][^5]

Checkpoint downloads are verified with SHA-256 digests embedded in the model URLs. This is a useful integrity feature.[^5]

The model-building path converts applicable convolution, linear, attention, and projection parameters to FP16. When a non-JIT model is loaded on CPU, the loader converts it to float32.[^3][^5]This supports efficient accelerator inference, but it should not be described as a guarantee that every operation or every device uses identical precision.

The repository is MIT licensed.[^8]It has no formal GitHub Releases, and the official installation instructions continue to install directly from the Git repository.[^2][^9]

There was a real packaging failure in early 2026 after newer setuptools versions removedpkg_resources. The currentsetup.pyno longer importspkg_resources; pull request 529 updated the file in February 2026.[^10]The historical issue should therefore be described as a compatibility problem that prompted a source fix, not as an unchanged defect in currentmain.

Packaging remains lightweight. The current setup file reports package version1.0, has nopyproject.toml-style locked build configuration, and reads largely unpinned requirements such astorchandtorchvisionfromrequirements.txt.[^7][^11]This is functional for research use but weaker provenance than a versioned release with pinned compatibility ranges.

12. Reproducible reporting checklist

A reproducible CLIP analysis should preserve at least:

Model identity

-OpenAI CLIP versus another CLIP-style implementation;

-exact checkpoint name;

-checkpoint file hash;

-embedding dimension;

-input resolution;

-repository commit;

-JIT or non-JIT loading path;

-device and dtype.

Software environment

-Python version;

-PyTorch version;

-torchvision version;

-Pillow version;

-CUDA and driver versions when applicable;

-dependency lock file or complete environment export.

Image processing

-source-image identity or hash;

-resize and crop procedure;

-RGB conversion;

-channel-normalization values;

-any additional image augmentation, tiling, masking, or multi-crop aggregation.

Text processing

-exact prompt text;

-class-name source;

-class descriptions and disambiguation;

-prompt templates;

-prompt-ensemble rule;

-tokenizer implementation;

-context length;

-truncation behavior;

-language.

Similarity and decision rule

-raw or L2-normalized features;

-cosine, dot product, or another distance;

-learned checkpoint scale, fixed scale, or no scale;

-full candidate taxonomy;

-softmax usage;

-threshold or rejection rule;

-calibration dataset;

-nearest-neighbor index and search parameters.

Evaluation

-target-domain dataset and sampling;

-class and subgroup metrics;

-prompt and taxonomy sensitivity;

-out-of-domain rejection;

-robustness to crop, compression, color, and resolution changes;

-manual review or second-stage verification;

-known limitations and excluded uses.

Overall assessment

CLIP is one of the most influential representation-learning architectures of its generation. Its major contribution is the construction of a language-addressable visual space: text can synthesize classifier directions, query images, or define task concepts without changing the visual encoder.

The architecture is elegant because it is simple at the interface. Images and texts are encoded, projected, normalized, and compared. At the scale of 400 million noisy pairs and 32,768-example batches, that simple objective produced broad transfer and historically strong zero-shot classification.[^1]

The same design creates the major limitations. The representation is global and lossy. Prompt wording and category selection change the classifier. Similarity is not a universal probability. Internet supervision imports social and statistical bias. The released model is English-centered, not natively localized, and its own model card places untested deployment outside the intended scope.[^4]

The most defensible conclusion is therefore:

>OpenAI CLIP provides a strong and unusually reusable image-text representation. Its embeddings should be treated as checkpoint-specific learned measurements whose meaning must be calibrated for the application, not as neutral explanations or universally valid semantic coordinates.