Multimodal image attachment is now available for Gemini in Android Studio

Posted by Paris Hsu – Product Manager, Android Studio

At every stage of the development lifecycle, Gemini in Android Studio has become your AI-powered companion, making it easier to build high quality apps. We are excited to announce a significant expansion: Gemini in Android Studio now supports multimodal inputs, which lets you attach images directly to your prompts! This unlocks a wealth of new possibilities that improve team collaboration and UI development workflows.

You can try out this new feature by downloading the latest Android Studio canary. We’ve outlined a few use cases to try, but we’d love to hear what you think as we work through bringing this feature into future stable releases. Check it out:

Image attachment – a new dimension of interaction

We first previewed Gemini’s multimodal capabilities at Google I/O 2024. This technology allows Gemini in Android Studio to understand simple wireframes, and transform them into working Jetpack Compose code.

You’ll now find an image attachment icon in the Gemini chat window. Simply attach JPEG or PNG files to your prompts and watch Gemini understand and respond to visual information. We’ve observed that images with strong color contrasts yield the best results.

1.1 New “Attach Image File” icon in chat window

1.2 Example multimodal response in chat

We encourage you to experiment with various prompts and images. Here are a few compelling use cases to get you started:

• Rapid UI prototyping and iteration: Convert a simple wireframe or high-fidelity mock of your app’s UI into working code.
• Diagram explanation and documentation: Gain deeper insights into complex architecture or data flow diagrams by having Gemini explain their components and relationships.
• UI troubleshooting: Capture screenshots of UI bugs and ask Gemini for solutions.

Rapid UI prototyping and iteration

Gemini’s multimodal support lets you convert visual designs into functional UI code. Simply upload your image and use a clear prompt. It works whether you’re working from your own sketches or from a designer mockup.

Here’s an example prompt: “For this image provided, write Android Jetpack Compose code to make a screen that’s as close to this image as possible. Make sure to include imports, use Material3, and document the code.” And then you can append any specific or additional instructions related to the image.

2. Example of generating Compose code from high-fidelity mock using Gemini in Android Studio (code output)

For more complex UIs, refine your prompts to capture specific functionality. For instance, when converting a calculator mockup, adding “make the interactions and calculations work as you’d expect” results in a fully functional calculator:

3. Example of generating Compose code from wireframe via Gemini in Android Studio (code output)

Note: this feature provides an initial design scaffold. It’s a good “first draft” and your edits and adjustments will be needed. Common refinements include ensuring correct drawable imports and importing icons. Consider the generated code a highly efficient starting point, accelerating your UI development workflow.

Diagram explanation and documentation

With Gemini’s multimodal capabilities, you can also try uploading an image of your diagram and ask for explanations or documentation.

Example prompt: Upload the Now in Android architecture diagram and say “Explain the components and data flow in this diagram” or “Write documentation about this diagram”.

4. Example of asking Gemini to help document the NowInAndroid architecture diagram

UI troubleshooting

Leverage Gemini’s visual analysis to identify and resolve bugs quickly. Upload a screenshot of the problematic UI, and Gemini will analyze the image and suggest potential solutions. You can also include relevant code snippets for more precise assistance.

In the example below, we used Compose UI check and found that the button is stretched too wide in tablet screens, so we took a screenshot and asked Gemini for solutions – it was able to leverage the window size classes to provide the right fix.

5. Example of fixing UI bugs using Image Attachment (code output)

Download Android Studio today

Download the latest Android Studio canary today to try the new multimodal features!

As always, Google is committed to the responsible use of AI. Android Studio won’t send any of your source code to servers without your consent. You can read more on Gemini in Android Studio’s commitment to privacy.

We appreciate any feedback on things you like or features you would like to see. If you find a bug, please report the issue and also check out known issues. Remember to also follow us on X, Medium, or YouTube for more Android development updates!