Kushal Kafle

Senior Research Scientist, Adobe Research

kushalkafle@gmail.comkushalkafle.comGoogle Scholar/kushalkafle/kushalkafle

Download CV (PDF)

Research Interests

Multimodal LLMs / vision-language models (VLMs), representation learning, foundation model training on web-scale data, benchmarks, evaluation and model probing, robustness and fairness

Selected Technology Transfers

I lead Adobe's vision-language foundation models: image-text embedding models and, recently, multimodal LLMs / VLMs. Our embedding model matches or beats the best open-weight models (e.g., Perception Encoder, SigLIP 2) on text-to-image retrieval benchmarks, by a wider margin on Adobe content, and since 2023 has powered features across Adobe products:

  • Adobe Express and Adobe Stock2023 – present

    Natural-language and image search for templates in Adobe Express (+46% click-through) and for assets in Adobe Stock, plus "More like this" template similarity in Express. Search has since grown to multilingual, behavior-tuned, and steerable, and now runs on a new generation of our foundation model.

  • Adobe Lightroom2025 – present

    Natural-language search of photo libraries with full phrases like "photos of dog taken at the beach", no keywords needed, plus assisted culling and Auto Stack via a compact on-device model.

  • Adobe Firefly Image Generation2023 – 2025

    Firefly Image 1, 2, and 3 used our text and image encoders for their reference features, including Style Reference uploads, generating similar images, and using images as references. Our models also power Firefly's NSFW safety filter, used for both image and video generation, and the classifiers that filter its training data.

  • Premiere Pro Media Intelligence2025

    Visual search in Premiere Pro's Media Intelligence and Search panel, built with the video team on our image-text foundation model.

  • Other transfers: Firefly image-to-video (prompt coherence and safety); AEM Smart Tags; Photoshop prompt-free object compositing.

Research Experience

Adobe Research, San Jose, CAMar 2020 – present

Senior Research Scientist 2 (Feb 2026 – present); Senior Research Scientist (Jan 2024 – Feb 2026); Research Scientist 2 (Feb 2022 – Jan 2024); Research Scientist (Mar 2020 – Feb 2022). Manager: Scott Cohen.

  • Lead Adobe's vision-language foundation models across the full lifecycle (data, legal, training infrastructure, novel algorithms, codebase development, and tech transfer): image-text embedding models that power features in Firefly, Express, Stock, Lightroom, and Premiere Pro, and, in the last few years, multimodal LLMs / VLMs.
  • Mentored 28 research interns (2020–2026), mostly PhD students; many internships led to publications (full list).
Research Intern, Microsoft Research2019

With Dinei Florêncio: scene-text entity recognition.

Research Intern, Adobe Research2017 – 2018

With Scott Cohen and Brian Price: DVQA and PReFIL.

Education

Ph.D. in Imaging Science, Rochester Institute of TechnologyAug 2014 – Feb 2020

Chester F. Carlson Center for Imaging Science. Advisor: Christopher Kanan (klab).

Thesis: Advancing Multi-modal Deep Learning: Towards Language-grounded Visual Understanding

Professional Services

  • Area Chair: WACV 2024, WACV 2025, ICCV 2025 (Lead AC), WACV 2026, CVPR 2026 (Lead AC), CVPR 2027
  • Senior Program Committee: AAAI 2027
  • Workshop Organization: Shortcomings in Vision and Language (SiVL) at ECCV 2018 and NAACL 2019; Drawings and Abstract Imagery: Representation and Analysis (DIRA) at ECCV 2022
  • Journal Editing: Special Topic on Identifying, Analyzing, and Overcoming Challenges in Vision and Language Research, cross-posted to: Frontiers in Big Data, Frontiers in Robotics and AI, and Frontiers in Artificial Intelligence. Associate Editor.
  • Journal Reviewing: TPAMI (2019), TIP (2019), CVIU (2017–18), ACM-MM (2018–2019), TMI (2020), TDKE (2019), PR (2021), TMLR (2022–2024)
  • Conference Reviewing: CVPR (2017–2025), NeurIPS (2016–2025), AAAI (2017–2020), ICCV (2019, 2023, 2024), ICML (2020–2023), EMNLP (2019–2021), ACL (2020), NAACL (2021), ICLR (2022, 2024), ECCV (2026)
  • Outstanding Reviewer Awards: CVPR 2018, 2019, 2020, and 2021; NeurIPS 2021; ECCV 2026

Publications

Peer Reviewed International Conferences

  1. (C1)Fioresi, J., Heilbron, F. C., Nathani, P., Shah, M., & Kafle, K. (2026). Controlling Embedding Spaces with Text-Conditioned Transformations. ECCV 2026.
  2. (C2)Sethuraman, T. V., Khosla, S., Susladkar, O. K., Tiwari, A., Oh, S. W., Kafle, K., Lee, J.-Y., Hoiem, D., & Jenni, S. (2026). ViTAL-X: Video-Text Alignment with Cross-Modal Temporal Edits. ECCV 2026.
  3. (C3)Akdemir, K., Shi, J., Kafle, K., Price, B. L., & Yanardag, P. (2026). Plot’n polish: Zero-shot story visualization and disentangled editing with text-to-image diffusion models. AAAI 2026.
  4. (C4)Hernandez, J., Shi, J., Jenni, S., Ordonez, V., & Kafle, K. (2025). Improving Large Vision and Language Models by Learning from a Panel of Peers. ICCV 2025.
  5. (C5)Slyman, E., Tanjim, M., Kafle, K., & Lee, S. (2025). Calibrating MLLM-as-a-Judge via Multimodal Bayesian Prompt Ensembles. ICCV 2025.
  6. (C6)Khosla, S., Tiwari, A., Kafle, K., Jenni, S., Zhao, H., Collomosse, J., & Shi, J. (2025). MAGNET: Augmenting Generative Decoders with Representation Learning and Infilling Capabilities. ACL 2025.
  7. (C7)Hua, H., Shi, J., Kafle, K., Jenni, S., Zhang, D., Collomosse, J., Cohen, S., & Luo, J. (2024). FineMatch: Aspect-Based Fine-Grained Image and Text Mismatch Detection and Correction. ECCV 2024.
  8. (C8)Slyman, E., Lee, S., Cohen, S., & Kafle, K. (2024). FairDeDup: Detecting and Mitigating Vision-Language Fairness Disparities in Semantic Dataset Deduplication. CVPR 2024.
  9. (C9)Sameni, S., Kafle, K., Tan, H., & Jenni, S. (2024). Building Vision-Language Models on Solid Foundations with Masked Distillation. CVPR 2024.
  10. (C10)Yang, Z., Kafle, K., Lin, Z., Cohen, S., Ding, Z., & Ordonez, V. (2024). SCoRD: Subject-Conditional Relation Detection With Text-Augmented Data. WACV 2024.
  11. (C11)Tanjim, M. M., Singh, K. K., Kafle, K., Sinha, R., & Cottrell, G. W. (2024). Discovering and Mitigating Biases in CLIP-Based Image Editing. WACV 2024. Oral Presentation (2.6% of submissions)
  12. (C12)Yang, Z., Kafle, K., Dernoncourt, F., & Ordonez, V. (2023). Improving Visual Grounding by Encouraging Consistent Gradient-Based Explanations. CVPR 2023.
  13. (C13)Tanjim, M. M., Singh, K. K., Kafle, K., Sinha, R., & Cottrell, G. W. (2022). Debiasing Image-to-Image Translation Models. BMVC 2022.
  14. (C14)Pham, K., Kafle, K., Lin, Z., Ding, Z., Cohen, S., Tran, Q., & Shrivastava, A. (2022). Improving Closed and Open-Vocabulary Attribute Prediction Using Transformers. ECCV 2022.
  15. (C15)Shrestha, R., Kafle, K., & Kanan, C. (2022). OccamNets: Mitigating Dataset Bias by Favoring Simpler Hypotheses. ECCV 2022. Oral Presentation (2.7% of submissions)
  16. (C16)Shrestha, R., Kafle, K., & Kanan, C. (2022). An Investigation of Critical Issues in Bias Mitigation Techniques. WACV 2022.
  17. (C17)Ravi, H., Kafle, K., Cohen, S., Brandt, J., & Kapadia, M. (2021). AESOP: Abstract Encoding of Stories, Objects, and Pictures. ICCV 2021.
  18. (C18)Pham, K., Kafle, K., Lin, Z., Ding, Z., Cohen, S., Tran, Q., & Shrivastava, A. (2021). Learning To Predict Visual Attributes in the Wild. CVPR 2021.
  19. (C19)Teney, D., Abbasnejad, E., Kafle, K., Shrestha, R., Kanan, C., & van den Hengel, A. (2020). On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law. NeurIPS 2020.
  20. (C20)Shrestha, R., Kafle, K., & Kanan, C. (2020). A negative case analysis of visual grounding methods for VQA. ACL 2020.
  21. (C21)Hayes, T. L.*, Kafle, K.*, Shrestha, R.*, Acharya, M., & Kanan, C. (2020). REMIND Your Neural Network to Prevent Catastrophic Forgetting. ECCV 2020. *equal contribution
  22. (C22)Kafle, K., Shrestha, R., Cohen, S., Price, B., & Kanan, C. (2020). Answering Questions about Data Visualizations using Efficient Bimodal Fusion. WACV 2020.
  23. (C23)Shrestha, R., Kafle, K., & Kanan, C. (2019). Answer Them All! Toward Universal Visual Question Answering Models. CVPR 2019.
  24. (C24)Acharya, M., Kafle, K., & Kanan, C. (2019). TallyQA: Answering Complex Counting Questions. AAAI 2019.
  25. (C25)Kafle, K., Price, B., Cohen, S., & Kanan, C. (2018). DVQA: Understanding Data Visualizations via Question Answering. CVPR 2018.
  26. (C26)Kafle, K. & Kanan, C. (2017). An Analysis of Visual Question Answering Algorithms. ICCV 2017.
  27. (C27)Kafle, K., Yousefhussien, M., & Kanan, C. (2017). Data Augmentation for Visual Question Answering. INLG 2017.
  28. (C28)Kafle, K. & Kanan, C. (2016). Answer-Type Prediction for Visual Question Answering. CVPR 2016.

Peer Reviewed Journals

  1. (J1)Kafle, K., Shrestha, R., & Kanan, C. (2019). Challenges and Prospects in Vision and Language Research. Frontiers in AI 2019.
  2. (J2)Kafle, K. & Kanan, C. (2017). Visual question answering: Datasets, algorithms, and future challenges. CVIU 2017.

Workshop Papers

  1. (W1)Lu, J., Srivastava, S., Chen, J., Shrestha, R., Acharya, M., Kafle, K., & Kanan, C. (2025). Revisiting Multi-Modal LLM Evaluation. CVPRW 2025.

Preprints

  1. (P1)Abdel Magid, S., Chun, S., Slyman, E., Kafle, K., & Russakovsky, O. (2026). Copying As Defense: Protecting Images from Editing By Eliciting Model Copying. arXiv preprint.
  2. (P2)Lu, J., Jenni, S., Kafle, K., Shi, J., Zhao, H., & Fu, Y. (2026). Seeing Through Words: Controlling Visual Retrieval Quality with Language Models. arXiv preprint.
  3. (P3)Wu, Q., Shi, J., Jenni, S., Kafle, K., Wang, T., Chang, S., & Zhao, H. (2026). RetouchIQ: MLLM Agents for Instruction-Based Image Retouching with Generalist Reward. arXiv preprint.
  4. (P4)Dong, Q., Figueroa, L., Zhao, H., Kafle, K., Kuen, J., Ding, Z., Cohen, S., & Fu, Y. (2026). CoT Referring: Improving Referring Expression Tasks with Grounded Reasoning. arXiv preprint.
  5. (P5)Just, H. A., Fan, Y., Zhao, H., Gu, J., Zhang, R., Jenni, S., Kafle, K., Jia, R., & Shi, J. (2025). More Than the Final Answer: Improving Visual Extraction and Logical Consistency in Vision-Language Models. arXiv preprint.
  6. (P6)Zang, Y., Tan, H., Yoon, S., Dernoncourt, F., Gu, J., Kafle, K., Sun, C., & Bui, T. (2025). MS4UI: A Dataset for Multi-modal Summarization of User Interface Instructional Videos. arXiv preprint.
  7. (P7)Abdel Magid, S., Wang, J.-H., Kafle, K., & Pfister, H. (2024). They're All Doctors: Synthesizing Diverse Counterfactuals to Mitigate Associative Bias. arXiv preprint.
  8. (P8)Qian, Z., Hayes, T. L., Kafle, K., & Kanan, C. (2020). Do We Need Fully Connected Output Layers in Convolutional Networks? arXiv preprint.

Invited Talks and Guest Lectures

  • Understanding and mitigating model bias in the age of large models. Keynote, BISCUIT workshop at ICCV 2025 (Oct 2025)
  • Vision-language models and VQA. Invited lecture, 3rd Nepal Winter School in AI (NAAMII) (Dec 2021)
  • Lessons learned from creating and solving the first ChartQA datasets. Invited talk, Chart Question Answering (CQA) workshop at CVPR 2021 (video) (Jun 2021)
  • Why bias affects machine learning and what can we do about it? Invited talk, SIGIR eCom 2021 workshop (Jul 2021)
  • Bias in vision and language: visual question answering. Guest lectures, Vision and Language courses at the University of Virginia and RIT (Sep 2020)
  • Language-grounded visual understanding. Invited talks at Microsoft and Adobe Research (Nov 2019)

Skills

  • Technical: PyTorch, Python, large-scale distributed training, billion-scale data pipelines and filtering, Git, LaTeX, Bash, AWS
  • Data: Scraping and curating web-scale data, data licensing and legal review, vendor and crowdsourced annotation (MTurk, Appen, LabelBox, Hive, iMerit)