I have an Android app that was modeled after the Tensorflow Android demo for classifying images,
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples/android
The original app uses a tensorflow graph (.pb) file to classify a generic set of images from Inception v3 (I think)
I then trained my own graph for my own images following the instruction in Tensorflow for Poets blog,
https://petewarden.com/2016/02/28/tensorflow-for-poets/
and this worked in the Android app very well, after changing the settings in,
ClassifierActivity
private static final int INPUT_SIZE = 299; private static final int IMAGE_MEAN = 128; private static final float IMAGE_STD = 128.0f; private static final String INPUT_NAME = "Mul"; private static final String OUTPUT_NAME = "final_result"; private static final String MODEL_FILE = "file:///android_asset/optimized_graph.pb"; private static final String LABEL_FILE = "file:///android_asset/retrained_labels.txt"; To port the app to iOS, I then used the iOS camera demo, https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples/ios/camera
and used the same graph file and changed the settings in,
CameraExampleViewController.mm
// If you have your own model, modify this to the file name, and make sure // you've added the file to your app resources too. static NSString* model_file_name = @"tensorflow_inception_graph"; static NSString* model_file_type = @"pb"; // This controls whether we'll be loading a plain GraphDef proto, or a // file created by the convert_graphdef_memmapped_format utility that wraps a // GraphDef and parameter file that can be mapped into memory from file to // reduce overall memory usage. const bool model_uses_memory_mapping = false; // If you have your own model, point this to the labels file. static NSString* labels_file_name = @"imagenet_comp_graph_label_strings"; static NSString* labels_file_type = @"txt"; // These dimensions need to match those the model was trained with. const int wanted_input_width = 299; const int wanted_input_height = 299; const int wanted_input_channels = 3; const float input_mean = 128f; const float input_std = 128.0f; const std::string input_layer_name = "Mul"; const std::string output_layer_name = "final_result"; After this the app is working on iOS, however...
The app on Android performs much better than iOS in detecting classified images. If I fill the camera's view port with the image, both perform similar. But normally the image to detect is only part of the camera view port, on Android this doesn't seem to impact much, but on iOS it impacts a lot, so iOS cannot classify the image.
My guess is that Android is cropping if camera view port to the central 299x299 area, where as iOS is scaling its camera view port to the central 299x299 area.
Can anyone confirm this? and does anyone know how to fix the iOS demo to better detect focused images? (make it crop)
In the demo Android class,
ClassifierActivity.onPreviewSizeChosen()
rgbFrameBitmap = Bitmap.createBitmap(previewWidth, previewHeight, Config.ARGB_8888); croppedBitmap = Bitmap.createBitmap(INPUT_SIZE, INPUT_SIZE, Config.ARGB_8888); frameToCropTransform = ImageUtils.getTransformationMatrix( previewWidth, previewHeight, INPUT_SIZE, INPUT_SIZE, sensorOrientation, MAINTAIN_ASPECT); cropToFrameTransform = new Matrix(); frameToCropTransform.invert(cropToFrameTransform); and on iOS is has,
CameraExampleViewController.runCNNOnFrame()
const int sourceRowBytes = (int)CVPixelBufferGetBytesPerRow(pixelBuffer); const int image_width = (int)CVPixelBufferGetWidth(pixelBuffer); const int fullHeight = (int)CVPixelBufferGetHeight(pixelBuffer); CVPixelBufferLockFlags unlockFlags = kNilOptions; CVPixelBufferLockBaseAddress(pixelBuffer, unlockFlags); unsigned char *sourceBaseAddr = (unsigned char *)(CVPixelBufferGetBaseAddress(pixelBuffer)); int image_height; unsigned char *sourceStartAddr; if (fullHeight <= image_width) { image_height = fullHeight; sourceStartAddr = sourceBaseAddr; } else { image_height = image_width; const int marginY = ((fullHeight - image_width) / 2); sourceStartAddr = (sourceBaseAddr + (marginY * sourceRowBytes)); } const int image_channels = 4; assert(image_channels >= wanted_input_channels); tensorflow::Tensor image_tensor( tensorflow::DT_FLOAT, tensorflow::TensorShape( {1, wanted_input_height, wanted_input_width, wanted_input_channels})); auto image_tensor_mapped = image_tensor.tensor<float, 4>(); tensorflow::uint8 *in = sourceStartAddr; float *out = image_tensor_mapped.data(); for (int y = 0; y < wanted_input_height; ++y) { float *out_row = out + (y * wanted_input_width * wanted_input_channels); for (int x = 0; x < wanted_input_width; ++x) { const int in_x = (y * image_width) / wanted_input_width; const int in_y = (x * image_height) / wanted_input_height; tensorflow::uint8 *in_pixel = in + (in_y * image_width * image_channels) + (in_x * image_channels); float *out_pixel = out_row + (x * wanted_input_channels); for (int c = 0; c < wanted_input_channels; ++c) { out_pixel[c] = (in_pixel[c] - input_mean) / input_std; } } } CVPixelBufferUnlockBaseAddress(pixelBuffer, unlockFlags); 1 Answers
Answers 1
Since you are not using YOLO Detector the MAINTAIN_ASPECT flag is set to false. Hence the image on Android app is not getting cropped, but it's scaled. However, in the code snippet provided I don't see the actual initialisation of the flag. Confirm that the value of the flag is actually false in your app.
I know this isn't a complete solution but hope this helps you in debugging the issue.
0 comments:
Post a Comment