Automation Skill
Once you’ve enabled automation, your AI bot gains the ability to control your phone’s UI using Python and uiautomator2. This page explains how to use this skill.
What It Does
Section titled “What It Does”The automation skill allows your bot to:
- Open and navigate apps
- Tap buttons and menu items
- Input text (including Chinese/CJK)
- Read screen content
- Perform complex multi-step workflows
Prerequisites
Section titled “Prerequisites”Before using automation commands, ensure:
- Shizuku is running
- u2 service is running
Basic Usage
Section titled “Basic Usage”Simply tell your bot what you want to do on your phone. For example:
- “Open WeChat and send ‘Hello’ to John”
- “Take a screenshot of my current screen”
- “Open Settings and check my battery status”
- “Post a photo to Instagram with caption ‘Beautiful sunset’”
The bot will break down complex tasks into step-by-step actions, observing the screen after each step.
How It Works
Section titled “How It Works”The automation follows an observe → act → observe pattern:
- Observe: Read current screen state (which app, what’s visible)
- Act: Perform one action (tap, type, swipe)
- Observe: Check the result before next action
This step-by-step approach ensures reliability, especially for dynamic app UIs.
Supported Operations
Section titled “Supported Operations”Navigation
Section titled “Navigation”- Open apps by name
- Tap buttons, menu items, icons
- Swipe and scroll
- Press back/home buttons
Text Input
Section titled “Text Input”- Type in search boxes, chat inputs, forms
- Supports Chinese and other languages via FastInputIME
- Clear and replace text
Reading Screen
Section titled “Reading Screen”- Get current app and activity
- Check if elements exist
- Read text content from screen
Examples
Section titled “Examples”Open an App
Section titled “Open an App”"Open Twitter"The bot will find and launch the Twitter app.
Send a Message
Section titled “Send a Message”"Open WhatsApp, find the chat with Mom, and send 'I'll be home for dinner'"The bot will:
- Open WhatsApp
- Search for Mom’s chat
- Tap to open the conversation
- Type the message
- Tap send
Check Information
Section titled “Check Information”"Open Settings and tell me my phone's Android version"The bot will navigate to About Phone and read the version info.
Tips for Best Results
Section titled “Tips for Best Results”Be Specific
Section titled “Be Specific”Instead of “post something on social media”, say “open Instagram, create a new post with the most recent photo, and add the caption ‘Sunday vibes’”.
Allow Time
Section titled “Allow Time”Complex flows take time. The bot waits for screens to load and verifies each step.
Handle Errors Gracefully
Section titled “Handle Errors Gracefully”If something goes wrong, the bot will try to recover or ask for guidance.
Limitations
Section titled “Limitations”- No password entry: For security, avoid asking the bot to enter passwords
- App-specific quirks: Some apps have custom UIs that may need special handling
- Network-dependent: App actions require working internet connection
- Screen must be on: Automation requires an active display
Troubleshooting
Section titled “Troubleshooting””u2 service not running”
Section titled “”u2 service not running””Go to Automation Panel and tap “Start u2 Service”.
Actions don’t seem to work
Section titled “Actions don’t seem to work”- Make sure the target app is fully loaded
- Try being more specific about what to tap
- Check if the app needs to be updated
Text input fails
Section titled “Text input fails”- The bot will automatically try different input methods
- For Chinese text, FastInputIME is used for reliability
Bot seems stuck
Section titled “Bot seems stuck”- Say “stop” or “cancel” to interrupt
- Check if the app is showing a popup or dialog