I have an Android application that is logging several Android sensors at approximately 100Hz. So if i am logging 10 sensors, I am writing about 3000 data points per second to a file (each sensor typically has 3 entries). Now the problem is that i want to minimize the effect of this writing on the rest of the app. Specifically, I do not want the logging to slow down event delivery... i want to make sure that i am getting events as soon as they happen, and not with a delay (i know there will always be some delay because Android is not realtime and because of the "pull" nature of the Sensor event framework).
Next, I will outline my approach, which does not appear to be working well. I would like suggestions for how to improve.
My current procedure is this...
For each sensor, i create a separate thread with a BlockingQueue of events to log. Inside the thread, I have a while loop that pulls from the queue and does the file writing using a buffered writer. When the sensor manager delivers a new SensorEvent, the event is put in the appropriate queue (thereby triggering the file IO on the other thread) so as not to delay the main thread on which SensorEvents are delivered.
I want to be getting the events as soon as they occur, so it is important that i do not introduce any delays in the Sensor framework. If, for instance, I did the file IO directly in the onEvent callback, then i am worried that events could start piling up in the pipeline, and that they would then be out of date by the time they are finally delivered. The above approach mitigates these worries.
But there is another issue...
Despite the fact that the file IO occurs off of the sensor event delivery thread, at times the application still feels sluggish. That is, sometimes i will see events occur in rapid succession (e.g. 5 events are delivered within 1 ms of each other). This indicates that although the IO is not happening on the sensor delivery thread, the delivery thread is still getting delayed. A few reasons have been suggested to me:
I am creating too many IO threads. Perhaps if i pushed all of the writing to a single thread I would increase the likelihood that the sensor delivery thread is alive when a new event comes in. In the current setup, it could be that all of the active threads are being used for file IO when an Event comes in, resulting in the events backing up until one of the writing events finishes.
Currently, i am using flat file output, not a database. The benefits of using a database for retrieval are clear to me. What is not clear to me is whether i should also expect a database to be faster if I am only appending data to a file.... that is, i never need to read from the file or insert the data into a random place , I just literally append to the end of the file. It seems to me that a database cannot be any faster than standard file IO in that case. Or am I wrong?
Others have suggested that the garbage collector is probably interfering with my threads and that the likely source of the problem is memory thrashing due to the large number of events that are being created.
From which angle should i approach this?
3 Answers
Answers 1
It's all guesswork until you attach a profiler and see what really happens. From general experience I'd say that points 1 and 3 are definitely valid. Point 2 less so; if a database is faster than appending to a file I think that's likely down to implementation quality and/or using C with native API's and you should be able to overcome the difference.
With regards to GC load, take a look at Looper (api) and Handler (api). Using these you can replace your BlockingQueue-based approach by one that generates no GC load at all. I wrote a blog post exploring this in some detail.
Furthermore, if you're using any/all of the Clean Code practices, it may be time for some judicious use of dirty code (field access, mutability) to lighten the memory churn.
With regards to IO threads: I'd say definitely scale down to a single one, and try to write the lines to file in batches, not one by one. A buffered Stream or Writer would probably do the trick on its own, so long as you avoid explicitly calling flush().
Answers 2
Yours is a subject that can hardly be evaluated if it is not tested, but I make some suggestions:
To use a single thread you could use a Handler and a Looper as proposed by Barend and when you have a new sensor reading a message is send to the this Handler, so there is the possibility of deleting old messages when you have several pending readings of the same sensor, that add some sort of anti-overflow protection, if the system can't process all readings some will be lost.
As you say Android is not real time, if the sensors are not read at regular times maybe you should attach a timestamp to each reading.
To avoid allocating & GC'ing variables or objects for each sensor reading you could use a pool (array) of variables previously allocated and each time you have a reading you use the following index. This would be fine for the case of sending messages because it would only have to send the index of the new reading in the message arguments.
I hope it helps.
Answers 3
https://docs.oracle.com/javase/7/docs/api/java/math/BigInteger.html#toByteArray()
Using above will save enough on IO to make your sensor code work. I've seen android do higher bit rates recording video. There shouldn't be a problem writing a file. 3000 elements per second Generous 20 bytes per row 480kbs.
Put your code into a service it will stay running when the app goes to the background while user checks instagram or goes looking for pokemon or the user presses the home button.
GC doesn't GC avoid final objects? IDK. Maybe I'm wrong but you should only be creating complex objects once. The service stays in memory running. not a problem.
five in one ms. This is normal for asycronous code. It's the nature of multitasking. Program had five events queued up and it processed them. You have to prioritize the tasks but don't put too much into your prioritizing or it will cause problems. For example if the write to file doesn't have enough priority the file write may never happen. for now keep all tasks at the same priority until you understand the profiler output.
Write to file. Maybe its going through encyption and onto the SDCard that would be bad and would take all kinds of CPU and cause even more problems.
0 comments:
Post a Comment