I'm using Apache TVM do benchmark DL models - this is long story, but I want to preview GPU usage of TVM. On Android device I have runing APK, which spawn 1-2 processes, which use GPU. I have turned on "GPU Profiling", and I dumped logs via adb shell dumpsys gfxinfo. But I'm not sure how to properly read it? Example of TVM process is below.
My questions:
- do you know any tutorial/instruction for reading and understanding following stats? how to interpret it?
- I do not care about jank frames, I need information about GPU usage, GPU memory usage
** Graphics info for pid 14811 [org.apache.tvm.tvmrpc:RPCProcess] **
Stats since: 5644930627488ns
Total frames rendered: 1
Janky frames: 1 (100.00%)
50th percentile: 97ms
90th percentile: 97ms
95th percentile: 97ms
99th percentile: 97ms
Number Missed Vsync: 0
Number High input latency: 0
Number Slow UI thread: 1
Number Slow bitmap uploads: 0
Number Slow issue draw commands: 1
Number Frame deadline missed: 1
HISTOGRAM: 5ms=0 6ms=0 7ms=0 8ms=0 9ms=0 10ms=0 11ms=0 12ms=0 13ms=0 14ms=0 15ms=0 16ms=0 17ms=0 18ms=0 19ms=0 20ms=0 21ms=0 22ms=0 23ms=0 24ms=0 25ms=0 26ms=0 27ms=0 28ms=0 29ms=0 30ms=0 31ms=0 32ms=0 34ms=0 36ms=0 38ms=0 40ms=0 42ms=0 44ms=0 46ms=0 48ms=0 53ms=0 57ms=0 61ms=0 65ms=0 69ms=0 73ms=0 77ms=0 81ms=0 85ms=0 89ms=0 93ms=0 97ms=1 101ms=0 105ms=0 109ms=0 113ms=0 117ms=0 121ms=0 125ms=0 129ms=0 133ms=0 150ms=0 200ms=0 250ms=0 300ms=0 350ms=0 400ms=0 450ms=0 500ms=0 550ms=0 600ms=0 650ms=0 700ms=0 750ms=0 800ms=0 850ms=0 900ms=0 950ms=0 1000ms=0 1050ms=0 1100ms=0 1150ms=0 1200ms=0 1250ms=0 1300ms=0 1350ms=0 1400ms=0 1450ms=0 1500ms=0 1550ms=0 1600ms=0 1650ms=0 1700ms=0 1750ms=0 1800ms=0 1850ms=0 1900ms=0 1950ms=0 2000ms=0 2050ms=0 2100ms=0 2150ms=0 2200ms=0 2250ms=0 2300ms=0 2350ms=0 2400ms=0 2450ms=0 2500ms=0 2550ms=0 2600ms=0 2650ms=0 2700ms=0 2750ms=0 2800ms=0 2850ms=0 2900ms=0 2950ms=0 3000ms=0 3050ms=0 3100ms=0 3150ms=0 3200ms=0 3250ms=0 3300ms=0 3350ms=0 3400ms=0 3450ms=0 3500ms=0 3550ms=0 3600ms=0 3650ms=0 3700ms=0 3750ms=0 3800ms=0 3850ms=0 3900ms=0 3950ms=0 4000ms=0 4050ms=0 4100ms=0 4150ms=0 4200ms=0 4250ms=0 4300ms=0 4350ms=0 4400ms=0 4450ms=0 4500ms=0 4550ms=0 4600ms=0 4650ms=0 4700ms=0 4750ms=0 4800ms=0 4850ms=0 4900ms=0 4950ms=0
50th gpu percentile: 4950ms
90th gpu percentile: 4950ms
95th gpu percentile: 4950ms
99th gpu percentile: 4950ms
GPU HISTOGRAM: 1ms=0 2ms=0 3ms=0 4ms=0 5ms=0 6ms=0 7ms=0 8ms=0 9ms=0 10ms=0 11ms=0 12ms=0 13ms=0 14ms=0 15ms=0 16ms=0 17ms=0 18ms=0 19ms=0 20ms=0 21ms=0 22ms=0 23ms=0 24ms=0 25ms=0 4950ms=0
Font Cache (CPU):
Size: 2.66 kB
Glyph Count: 2
CPU Caches:
GPU Caches:
Other:
Other: 988.00 bytes (1 entry)
Shadow Gaussian Falloff:
Texture: 128.00 bytes (1 entry)
Scratch:
Buffer Object: 48.00 KB (1 entry)
Texture: 2.00 MB (1 entry)
Other Caches:
Current / Maximum
Layers Total 0.00 KB (numLayers = 0)
Total GPU memory usage:
2147420 bytes, 2.05 MB (128.00 bytes is purgeable)
Pipeline=Skia (OpenGL)
Profile data in ms:
org.apache.tvm.tvmrpc/org.apache.tvm.tvmrpc.RPCActivity/android.view.ViewRootImpl@a9f6eba (visibility=0)
Draw Prepare Process Execute
Window: org.apache.tvm.tvmrpc/org.apache.tvm.tvmrpc.RPCActivity
Stats since: 5644998711873ns
Total frames rendered: 1
Janky frames: 1 (100.00%)
50th percentile: 97ms
90th percentile: 97ms
95th percentile: 97ms
99th percentile: 97ms
Number Missed Vsync: 0
Number High input latency: 0
Number Slow UI thread: 1
Number Slow bitmap uploads: 0
Number Slow issue draw commands: 1
Number Frame deadline missed: 1
HISTOGRAM: 5ms=0 6ms=0 7ms=0 8ms=0 9ms=0 10ms=0 11ms=0 12ms=0 13ms=0 14ms=0 15ms=0 16ms=0 17ms=0 18ms=0 19ms=0 20ms=0 21ms=0 22ms=0 23ms=0 24ms=0 25ms=0 26ms=0 27ms=0 28ms=0 29ms=0 30ms=0 31ms=0 32ms=0 34ms=0 36ms=0 38ms=0 40ms=0 42ms=0 44ms=0 46ms=0 48ms=0 53ms=0 57ms=0 61ms=0 65ms=0 69ms=0 73ms=0 77ms=0 81ms=0 85ms=0 89ms=0 93ms=0 97ms=1 101ms=0 105ms=0 109ms=0 113ms=0 117ms=0 121ms=0 125ms=0 129ms=0 133ms=0 150ms=0 200ms=0 250ms=0 300ms=0 350ms=0 400ms=0 450ms=0 500ms=0 550ms=0 600ms=0 650ms=0 700ms=0 750ms=0 800ms=0 850ms=0 900ms=0 950ms=0 1000ms=0 1050ms=0 1100ms=0 1150ms=0 1200ms=0 1250ms=0 1300ms=0 1350ms=0 1400ms=0 1450ms=0 1500ms=0 1550ms=0 1600ms=0 1650ms=0 1700ms=0 1750ms=0 1800ms=0 1850ms=0 1900ms=0 1950ms=0 2000ms=0 2050ms=0 2100ms=0 2150ms=0 2200ms=0 2250ms=0 2300ms=0 2350ms=0 2400ms=0 2450ms=0 2500ms=0 2550ms=0 2600ms=0 2650ms=0 2700ms=0 2750ms=0 2800ms=0 2850ms=0 2900ms=0 2950ms=0 3000ms=0 3050ms=0 3100ms=0 3150ms=0 3200ms=0 3250ms=0 3300ms=0 3350ms=0 3400ms=0 3450ms=0 3500ms=0 3550ms=0 3600ms=0 3650ms=0 3700ms=0 3750ms=0 3800ms=0 3850ms=0 3900ms=0 3950ms=0 4000ms=0 4050ms=0 4100ms=0 4150ms=0 4200ms=0 4250ms=0 4300ms=0 4350ms=0 4400ms=0 4450ms=0 4500ms=0 4550ms=0 4600ms=0 4650ms=0 4700ms=0 4750ms=0 4800ms=0 4850ms=0 4900ms=0 4950ms=0
50th gpu percentile: 4950ms
90th gpu percentile: 4950ms
95th gpu percentile: 4950ms
99th gpu percentile: 4950ms
GPU HISTOGRAM: 1ms=0 2ms=0 3ms=0 4ms=0 5ms=0 6ms=0 7ms=0 8ms=0 9ms=0 10ms=0 11ms=0 12ms=0 13ms=0 14ms=0 15ms=0 16ms=0 17ms=0 18ms=0 19ms=0 20ms=0 21ms=0 22ms=0 23ms=0 24ms=0 25ms=0 4950ms=0
---PROFILEDATA---
Flags,IntendedVsync,Vsync,OldestInputEvent,NewestInputEvent,HandleInputStart,AnimationStart,PerformTraversalsStart,DrawStart,SyncQueued,SyncStart,IssueDrawCommandsStart,SwapBuffers,FrameCompleted,DequeueBufferDuration,QueueBufferDuration,GpuCompleted,
1,5645013569772,5645013569772,9223372036854775807,0,5645014148949,5645014150180,5645014151103,5645081986026,5645084159488,5645092585719,5645092703796,5645118001873,5645120370719,0,1326769,0,
8,5645030205523,5645113538858,9223372036854775807,0,5645125786949,5645125790103,5645125954565,5645126136103,5645126148565,5645126504257,0,0,0,0,0,0,
---PROFILEDATA---
View hierarchy:
org.apache.tvm.tvmrpc/org.apache.tvm.tvmrpc.RPCActivity/android.view.ViewRootImpl@a9f6eba
14 views, 26.56 kB of render nodes
Total ViewRootImpl : 1
Total attached Views : 14
Total RenderNode : 26.56 kB (used) / 97.82 kB (capacity)
And this is log only for Apk:
Applications Graphics Acceleration Info:
Uptime: 6901736 Realtime: 6901736
** Graphics info for pid 12574 [org.apache.tvm.tvmrpc] **
Stats since: 500523944950ns
Total frames rendered: 439
Janky frames: 23 (5.24%)
50th percentile: 6ms
90th percentile: 10ms
95th percentile: 19ms
99th percentile: 61ms
Number Missed Vsync: 1
Number High input latency: 69
Number Slow UI thread: 12
Number Slow bitmap uploads: 0
Number Slow issue draw commands: 5
Number Frame deadline missed: 13
HISTOGRAM: 5ms=199 6ms=39 7ms=45 8ms=72 9ms=21 10ms=20 11ms=7 12ms=5 13ms=1 14ms=3 15ms=4 16ms=0 17ms=1 18ms=0 19ms=1 20ms=0 21ms=0 22ms=0 23ms=0 24ms=3 25ms=1 26ms=0 27ms=1 28ms=1 29ms=3 30ms=0 31ms=1 32ms=3 34ms=0 36ms=0 38ms=0 40ms=1 42ms=2 44ms=0 46ms=0 48ms=0 53ms=0 57ms=0 61ms=1 65ms=0 69ms=0 73ms=1 77ms=0 81ms=0 85ms=1 89ms=0 93ms=0 97ms=0 101ms=0 105ms=0 109ms=0 113ms=0 117ms=0 121ms=0 125ms=0 129ms=0 133ms=0 150ms=0 200ms=1 250ms=1 300ms=0 350ms=0 400ms=0 450ms=0 500ms=0 550ms=0 600ms=0 650ms=0 700ms=0 750ms=0 800ms=0 850ms=0 900ms=0 950ms=0 1000ms=0 1050ms=0 1100ms=0 1150ms=0 1200ms=0 1250ms=0 1300ms=0 1350ms=0 1400ms=0 1450ms=0 1500ms=0 1550ms=0 1600ms=0 1650ms=0 1700ms=0 1750ms=0 1800ms=0 1850ms=0 1900ms=0 1950ms=0 2000ms=0 2050ms=0 2100ms=0 2150ms=0 2200ms=0 2250ms=0 2300ms=0 2350ms=0 2400ms=0 2450ms=0 2500ms=0 2550ms=0 2600ms=0 2650ms=0 2700ms=0 2750ms=0 2800ms=0 2850ms=0 2900ms=0 2950ms=0 3000ms=0 3050ms=0 3100ms=0 3150ms=0 3200ms=0 3250ms=0 3300ms=0 3350ms=0 3400ms=0 3450ms=0 3500ms=0 3550ms=0 3600ms=0 3650ms=0 3700ms=0 3750ms=0 3800ms=0 3850ms=0 3900ms=0 3950ms=0 4000ms=0 4050ms=0 4100ms=0 4150ms=0 4200ms=0 4250ms=0 4300ms=0 4350ms=0 4400ms=0 4450ms=0 4500ms=0 4550ms=0 4600ms=0 4650ms=0 4700ms=0 4750ms=0 4800ms=0 4850ms=0 4900ms=0 4950ms=0
50th gpu percentile: 3ms
90th gpu percentile: 6ms
95th gpu percentile: 11ms
99th gpu percentile: 24ms
GPU HISTOGRAM: 1ms=0 2ms=56 3ms=192 4ms=100 5ms=16 6ms=17 7ms=6 8ms=0 9ms=1 10ms=2 11ms=10 12ms=6 13ms=4 14ms=1 15ms=0 16ms=0 17ms=0 18ms=0 19ms=0 20ms=0 21ms=1 22ms=0 23ms=2 24ms=5 25ms=0 4950ms=0
Font Cache (CPU):
Size: 17.38 kB
Glyph Count: 5
CPU Caches:
GPU Caches:
Other:
Other: 0.00 bytes (1 entry)
Shadow Gaussian Falloff:
Texture: 128.00 bytes (1 entry)
Image:
Texture: 54.45 KB (3 entries)
Scratch:
Buffer Object: 48.00 KB (1 entry)
Texture: 2.00 MB (1 entry)
Other Caches:
Current / Maximum
Layers Total 0.00 KB (numLayers = 0)
Total GPU memory usage:
2202192 bytes, 2.10 MB (54.58 KB is purgeable)
Pipeline=Skia (OpenGL)
Profile data in ms:
org.apache.tvm.tvmrpc/org.apache.tvm.tvmrpc.MainActivity/android.view.ViewRootImpl@1038509 (visibility=8)
Draw Prepare Process Execute
View hierarchy:
org.apache.tvm.tvmrpc/org.apache.tvm.tvmrpc.MainActivity/android.view.ViewRootImpl@1038509
27 views, 38.25 kB of render nodes
Total ViewRootImpl : 1
Total attached Views : 27
Total RenderNode : 38.25 kB (used) / 38.25 kB (capacity)
My thoughts about it:
Total frames rendered - Number of frames created to render
Janky Frames - Number of dropped frames
50th percentile: 6ms - 50% of frames was rendered in or under 6ms
90th percentile: 10ms - 90% of frames was rendered in or under 10ms
95th percentile: 19ms - 95% of frames was rendered in or under 19ms
99th percentile: 61ms - 99% of frames was rendered in or under 61ms
Number Missed Vsync - vsync is complex mechnism for syncronizing frames from GPU and frames displayed
Number High input latency - Input latency or lag refers to the amount of time it takes from when it receives a signal (when you press a button on your controller or move your mouse) to when that action occurs on the screen.
Number Slow UI thread: ???
Number Slow bitmap uploads: ???
Number Slow issue draw commands: ???
Number Frame deadline missed: ???
HISTOGRAM - histogram of what? ms is time from this moment?
GPU HISTOGRAM - histogram of GPU usage? in what units?
--edit--
I was trying to look for gfxinfo and dumpsys in AOSP, but I haven't found anything interesting. There is parser of gfxinfo written in java, just parses few things. In dumpsys I cannot find reference to gfxinfo service (https://cs.android.com/android/platform/superproject/+/master:frameworks/native/cmds/dumpsys/dumpsys.cpp)
