What is a JSON feed? Learn more

JSON Feed Viewer

Browse through the showcased feeds, or enter a feed URL below.

Now supporting RSS and Atom feeds thanks to Andrew Chilton's feed2json.org service

CURRENT FEED

Stories by Ravindu Senaratne on Medium

Stories by Ravindu Senaratne on Medium

A feed by yourfriends@medium.com

XML


Implementing Real-Time OCR Using Flutter

Permalink - Posted on 2021-06-14 13:40

Using Flutter Mobile Vision package to implement a real-time OCR on Mobile

Photo by Dries Augustyns on Unsplash

In the previous articles of this series on developing Flutter application with TensorFlow Lite, we looked at how we can develop Digit Recognizer with Flutter and TensorFlow Lite, Image Classification with Flutter and TensorFlow Lite, Object Detection with Flutter and TensorFlow Lite, Turning the Mobile Camera into a Real-Time Object Detector with Flutter and TensorFlow Lite and Implementing Real-Time Pose Estimation on Mobile Using Flutter.

In this article, we will look into developing a real-time OCR detection without using TensorFlow Lite. The application we are going to build will be able to detect characters presented from a live feed provided by the mobile camera.

Application and Use Cases

OCR or Optical Character Recognition is the technology used to distinguish printed or handwritten text within digital images or physical documents. OCR is commonly used to turn hardcopy, legal, or historic documents into PDFs. Simply put, an OCR system analyses an image and identifies dark areas as the characters that need to be recognized, enabling digital archiving, editing, and document searching with common programs such as Microsoft Word or Google Docs. Characters are recognized using one of two algorithms, Pattern Recognition or Feature Detection.

Required Packages

  • Flutter Mobile Vision

flutter_mobile_vision | Flutter Package

Use the above package as a library, add the dependencies to pubspec.yaml file and run flutter pub get command in the terminal.

dependencies:
flutter_mobile_vision: ^0.1.4+1

Import it into your preferred page using

import 'package:flutter_mobile_vision/flutter_mobile_vision.dart';

Flutter Application

Now that we have set up the required library, we can move on to the development of the application that does optical character recognition.

First, create a new dart file named ocr_detection.dart within the lib folder. Inside this page, we will put a button that will initiate the scanning and capture the words from the live feed.

You can access the camera through the flutter_mobile_vision library using the below code:

int OCR_CAM = FlutterMobileVision.CAMERA_BACK;
To get the most out of your mobile ML models, you’ll need to monitor and mange them in production. Our new free ebook explores best practices for this and other stages of the project lifecycle.

Now, let us look at how to open the camera and capture the text.

Future<Null> _read() async {
List<OcrText> words = [];
try {
words = await FlutterMobileVision.read(
camera: OCR_CAM,
waitTap: true,
);

setState(() {
word = words[0].value;
});
} on Exception {
words.add( OcrText('Unable to recognize the word'));
}
}

The above read function will get the texts that appear in the live feed and show them inside a rectangular box that has the text inside.

Full Code:

import 'dart:async';
import 'package:flutter/cupertino.dart';
import 'package:flutter/material.dart';
import 'package:flutter_mobile_vision/flutter_mobile_vision.dart';


class OCRPage extends StatefulWidget {
@override
_OCRPageState createState() => _OCRPageState();
}

class _OCRPageState extends State<OCRPage> {

int OCR_CAM = FlutterMobileVision.CAMERA_BACK;
String word = "TEXT";

@override
Widget build(BuildContext context) {
return SafeArea(
child: Scaffold(
backgroundColor: Colors.white70,
appBar: AppBar(
automaticallyImplyLeading: false,
title: Text('Real time OCR'),
centerTitle: true,
),
body: Container(
child: Column(
mainAxisAlignment: MainAxisAlignment.center,
crossAxisAlignment: CrossAxisAlignment.center,
children: [
Center(
child: RaisedButton(
onPressed: _read,
color: Colors.red,
child: Text('Start Scanning',
style: TextStyle(fontSize: 16),
),
),
),
],
),
),
),
);
}

Future<Null> _read() async {
List<OcrText> words = [];
try {
words = await FlutterMobileVision.read(
camera: OCR_CAM,
waitTap: true,
);

setState(() {
word = words[0].value;
});
} on Exception {
words.add( OcrText('Unable to recognize the word'));
}
}
}

As you can see from the above code snippet, the application will load onto a screen that contains a button Start Scanning that will run the _read() future function. It will then capture the live feed and show all the texts available on-screen within the rectangular box.

Result

Source Code

ravindu9701/Flutter-OCR

Conclusion

If you have read my previous articles about developing applications with TensorFlow Lite, you will find this tutorial to be easy to implement because this library does most of the calculation by itself. If you refer to the documentation of Flutter Mobile Vision, you can see that it can also be used to implement a Face Detection application. Give it a try!

Editor’s Note: Heartbeat is a contributor-driven online publication and community dedicated to exploring the emerging intersection of mobile app development and machine learning. We’re committed to supporting and inspiring developers and engineers from all walks of life.

Editorially independent, Heartbeat is sponsored and published by Fritz AI, the machine learning platform that helps developers teach devices to see, hear, sense, and think. We pay our contributors, and we don’t sell ads.

If you’d like to contribute, head on over to our call for contributors. You can also sign up to receive our weekly newsletter (Deep Learning Weekly), join us on Slack, and follow Fritz AI on Twitter for all the latest in mobile machine learning.


Implementing Real-Time OCR Using Flutter was originally published in Heartbeat on Medium, where people are continuing the conversation by highlighting and responding to this story.


Implementing Real-Time Pose Estimation on Mobile Using Flutter

Permalink - Posted on 2021-03-10 14:42

Integrating Mobile Camera for Real-Time Pose Estimation with Flutter, TensorFlow Lite, and PoseNet

Photo by Katie Bush on Unsplash

In the previous articles of this series on developing Flutter application with TensorFlow Lite, we looked at how we can develop Digit Recognizer with Flutter and TensorFlow Lite, Image Classification with Flutter and TensorFlow Lite, Object Detection with Flutter and TensorFlow Lite, and Turning the Mobile Camera into a Real-Time Object Detector with Flutter and TensorFlow Lite.

In the fifth article of this series, we’ll continue working with TensorFlow Lite, and this time focus on implementing Real-Time Pose Detection through the mobile camera. The application we are going to build will be able to estimate the pose of a person from a live feed provided by the camera. By making a few adjustments and refactoring to my Real-Time Object Detection codebase, I was able to come up with a working solution. In this article, I will guide you through my solution.

Application and Use Cases

TensorFlow Lite gives us pre-trained and optimized models to identify hundreds of classes of objects including people, activities, animals, plants, and places. Using the PoseNet MobileNet V1 model and Flutter Camera Plugin, we can develop a real-time pose estimation application.

Required Packages

  • TensorFlow Lite

tflite | Flutter Package

  • Camera Plugin

Using this plugin, we can capture the live feed provided by the mobile camera frame-by-frame. Then we can provide these frames as images to estimate the pose of a human.

camera | Flutter Package

  • PoseNet MobileNet V1

PoseNet is a deep learning model that allows us to detect the pose of a person based on where their joints are. The PoseNet model takes an image as an input and outputs an array of information about key points with a confidence score. An example of various body joints detected by the PoseNet model are tabulated below:

The high-level architecture of pose estimations is as follows:

https://www.tensorflow.org/lite/models/images/output_stride.png

Pose estimation | TensorFlow Lite

To start, download the required model from the above link and copy the extracted files to the assets folder of the Flutter project.

Flutter Application

Now that we have set up the initial configurations, we need to install the required packages. Then we can develop our Flutter application to turn the mobile camera into a pose detector.

This project will contain three different main classes:

  • Camera — This class contains the implementation of the camera plugin to receive the live feed.
  • BindBox — This class contains the key points of the body with their names.
  • Home — This class contains the error handling, loading of the model, and passing the data through the above-mentioned classes.

Now I will further explain the required code snippet for the above-mentioned classes.

First, we need to load the downloaded PoseNet model since we are running the application offline, and the loaded model needs to be run through the mentioned classes in order to detect the pose estimation of a body.

As you can see from the above code snippet, the loadModel() method will be called through a button. After loading the model, it will be passed through the Camera class. The Camera class will provide a live feed frame by frame to detect the pose estimation presented in the feed. Inside the Camera class, we first check whether permission to use the camera is granted or not. If it is given, then we will use the camera to gather the live feed frame-by-frame and run it through the Tflite.runPoseNetOnFrame() to detect the pose key points using the model.

Feeling inspired? Fritz AI Studio has the tools to build, test, and improve mobile machine learning models. Start building and teach your devices to see, hear, sense, and think.

Now that I have explained how to use the camera plugin with TensorFlow Lite, let’s look at how to detect key points of the body from the live feed provided by the Camera class. Using the BindBox class, we can render key points as below.

Result

Now that we’ve implemented the Flutter application code, let’s look at the output of the application when it’s up and running:

Example

Source Code

ravindu9701/Flutter-Pose-Estimation-

Conclusion

You can use other models that are compatible with TensorFlow Lite such as MPII Human Pose, COCO (Microsoft Common Object in Context), and Penn Action to develop pose estimation systems with different accuracies.

That’s all for this article, but in the next article, I’ll try to integrate OCR using the mobile camera.

Editor’s Note: Heartbeat is a contributor-driven online publication and community dedicated to exploring the emerging intersection of mobile app development and machine learning. We’re committed to supporting and inspiring developers and engineers from all walks of life.

Editorially independent, Heartbeat is sponsored and published by Fritz AI, the machine learning platform that helps developers teach devices to see, hear, sense, and think. We pay our contributors, and we don’t sell ads.

If you’d like to contribute, head on over to our call for contributors. You can also sign up to receive our weekly newsletters (Deep Learning Weekly and the Fritz AI Newsletter), join us on Slack, and follow Fritz AI on Twitter for all the latest in mobile machine learning.


Implementing Real-Time Pose Estimation on Mobile Using Flutter was originally published in Heartbeat on Medium, where people are continuing the conversation by highlighting and responding to this story.


Turning the Mobile Camera into a Real-Time Object Detector with Flutter and TensorFlow Lite

Permalink - Posted on 2020-11-25 13:33

Integrating Mobile Camera for Real-Time Object Detection with Flutter, TensorFlow Lite, and COCO SSD MobileNet

Photo by Dũng Trần Việt on Unsplash

In the previous article of this series on developing Flutter applications with TensorFlow Lite, we looked at how we can develop Digit Recognizer with Flutter and TensorFlow Lite, Image Classification with Flutter and TensorFlow Lite, and Object Detection with Flutter and TensorFlow Lite.

In the fourth article of this series, we’ll keep working with TensorFlow Lite, this time focusing on implementing Real-Time Object Detection by integrating the mobile camera. The application we are going to build will be able to recognize the objects from the live feed provided by the camera. After a month of researching and reading blogs, I was able to come with a solution to integrate the mobile camera as a real-time image provider and do the object detection through it. In this article, I will show you how to do the same.

Application and Use cases

TensorFlow Lite gives us pre-trained and optimized models to identify hundreds of classes of objects, including people, activities, animals, plants, and places. Using the COCO SSD MobileNet v1 model and Camera Plugin from Flutter, we will be able to develop a real-time object detector application.

Required Packages

  • TensorFlow Lite

tflite | Flutter Package

  • Camera Plugin

Using this plugin we can display live camera preview as a widget. Using CameraImage class we can capture the live feed frame-by-frame where we can provide these frames as images to do object detection.

camera | Flutter Package

  • COCO SSD MobileNet v1 Model

The SSD MobileNet model is a single shot multibox detection (SSD) network intended to perform object detection. This model can detect up to 10 objects in a frame. It is trained to recognize 80 classes of objects.

Object detection | TensorFlow Lite

Download the required model from the above link and copy the extracted files to the assets folder of the Flutter project.

Flutter Application

Now that we have put the model inside the project folder, we need to install the required packages. Then we can develop our Flutter application to turn the mobile camera into an object detector.

This project will be divided into 3 classes.

  • BindBox — Which contains the border dimensions of the detected objects with their name and confidence level as a percentage.
  • Camera — This contains the implementation of the camera plugin with the detection of objects using TFLite.
  • Home — This contains the loading of the model and passing the data through the BindBox and Camera classes.
Embedding ML models inside mobile apps can reduce latency, increase data security, and decrease costs. Subscribe to the Fritz AI Newsletter to learn more about the benefits of on-device ML.

Now I will explain further the required code snippet for the above classes. First, we need to load the model since we are running it offline, and the loaded model will be used to detect objects in the presented frame. To do that, we have to pass the data to BindBox class and Camera class.

After passing the model name, camera widget, and set recognition (which contains a dynamic list for store results with image height and width) to the Camera class. The Camera class will use these data to run the live feed (which is provided from the camera widget) frame-by-frame to detect objects presented in the feed. But first, we’ll check whether the camera is present or not. If it is, (having permission to use) we will pass the ImageStream through the TFlite.detectObjectOnFrame() to detect the objects using the model. This class also contains the dimensions of the camera view area. In this example, I have given the camera dimension to be full screen.

From the Home class, we have passed the data to both the Camera class and BindBox class. Now that I have explained why we need a Camera class, we can move on to the BindBox class which contains the dimension calculation needed to wrap the detected object with a box. This box will contain the object’s name alongside the confidence level as a percentage.

Result

Now that we’ve implemented the Flutter application code, let’s look at the output of the application when it’s up and running:

Example

Source Code:

ravindu9701/Real-Time-Object-Detection-Mobile

Conclusion

You can use other models that are compatible with TensorFlow Lite and develop your own real-time object detection mobile application using the above code. Different models have their own perks and accuracy levels which would be beneficial to the goal you’re trying to achieve.

That’s all for this article, but in the next article, I’ll try to integrate real-time pose detection using the mobile camera.

Editor’s Note: Heartbeat is a contributor-driven online publication and community dedicated to exploring the emerging intersection of mobile app development and machine learning. We’re committed to supporting and inspiring developers and engineers from all walks of life.

Editorially independent, Heartbeat is sponsored and published by Fritz AI, the machine learning platform that helps developers teach devices to see, hear, sense, and think. We pay our contributors, and we don’t sell ads.

If you’d like to contribute, head on over to our call for contributors. You can also sign up to receive our weekly newsletters (Deep Learning Weekly and the Fritz AI Newsletter), join us on Slack, and follow Fritz AI on Twitter for all the latest in mobile machine learning.


Turning the Mobile Camera into a Real-Time Object Detector with Flutter and TensorFlow Lite was originally published in Heartbeat on Medium, where people are continuing the conversation by highlighting and responding to this story.


Object Detection with Flutter and TensorFlow Lite

Permalink - Posted on 2020-09-22 13:17

How to develop an object detector mobile application with Flutter, using TensorFlow Lite and SSD MobileNet

Photo by Debby Hudson on Unsplash

In the previous article of this series on developing Flutter applications with TensorFlow Lite, we looked at how we can develop digit recognizer with Flutter and TensorFlow Lite and image classification with Flutter and TensorFlow Lite.

In the third article of this series, we’ll keep working with TensorFlow Lite, this time focusing on implementing object detection. The application we are going to build will be able to recognize objects presented in an image.

Application and Use cases

TensorFlow Lite gives us pre-trained and optimized models to identify hundreds of classes of objects including people, activities, animals, plants, and places. Using the SSD MobileNet model we can develop an object detection application.

Required Packages

  • TensorFlow Lite

tflite | Flutter Package

  • Image Picker

image_picker | Flutter Package

  • SSD MoblieNet (tflite) Model

The SSD MobileNet model is a single shot multibox detection (SSD) network intended to perform object detection.

ravindu9701/tflite-models

Download the tflite folder from the above link, and put it inside the assets folder of the Flutter project.

Kickstart your mobile machine learning projects — subscribe to the Fritz AI Newsletter for looks at common mobile ML problems, helpful resources, and much more.

Flutter Application

Now that we have put the model inside the project folder, we can develop our Flutter application to detect objects. Let’s get started.

We’ll need to initiate five variables to carry out the application properly. To receive the output, we’ll need a List variable, to get the uploaded image we’ll need a File variable, to get the image’s height and width we’ll need two double variables, and to manage errors we’ll need a Boolean variable:

File _image;

double _imgWidth;
double _imgHeight;
bool _busy = false;

List _recognitions;

Since the model is working offline, we need to load the model when the application is launched.

loadModel() async {
Tflite.close();
try {
await Tflite.loadModel(
model: "assets/tflite/ssd_mobilenet.tflite",
labels: "assets/tflite/ssd_mobilenet.txt",
);
} on PlatformException {
print("Failed to load the model");
}
}

Using the above function we can load the model. First, we need to close any other running models using Tflite.close(). Then we can load our model inside a try-catch.

selectFromImagePicker() async {
var image = await ImagePicker.pickImage(source: ImageSource.gallery);
if (image == null) return;
setState(() {
_busy = true;
});
predictImage(image);
}

In the above function, we have used ImagePicker to pick an image from the phone’s gallery. After receiving an image we need to pass that image through the model. In order to do that, I have created another function called predictImage(image) and passed the selected image into it.

predictImage(File image) async {
if (image == null) return;
await ssdMobileNet(image);

FileImage(image)
.resolve(ImageConfiguration())
.addListener((ImageStreamListener((ImageInfo info, bool _) {
setState(() {
_imgWidth = info.image.width.toDouble();
_imgHeight = info.image.height.toDouble();
});
})));

setState(() {
_image = image;
_busy = false;
});
}

Using the above function’s FileImage we can get the image’s dimensions using listeners. The image will be passed on to the ssdMobileNet() function to run the image through the model.

ssdMobileNet(File image) async {
var recognitions = await Tflite.detectObjectOnImage(
path: image.path, numResultsPerClass: 1);

setState(() {
_recognitions = recognitions;
});
}

When we use the SSD MobileNet model, we don’t need to give much information because it already has them as its default settings. We need to give the image path and the number of results per class.

Now we need to create a function to get detection boxes (to draw the boxes on the top of the image) which will get the size of the screen as an input.

List<Widget> renderBoxes(Size screen) {
if (_recognitions == null) return [];
if (_imgWidth == null || _imgHeight == null) return [];

double factorX = screen.width;
double factorY = _imgHeight / _imgHeight * screen.width;

return _recognitions.map((re) {
return Positioned(
left: re["rect"]["x"] * factorX,
top: re["rect"]["y"] * factorY,
width: re["rect"]["w"] * factorX,
height: re["rect"]["h"] * factorY,
child: Container(
decoration: BoxDecoration(
border: Border.all(
color: Colors.red,
width: 3,
)),
child: Text(
"${re["detectedClass"]} ${(re["confidenceInClass"] * 100).toStringAsFixed(0)}%",
style: TextStyle(
background: Paint()..color = blue,
color: Colors.white,
fontSize: 15,
),
),
),
);
}).toList();
}

First, we need to check that our _recognitions is not null. Secondly, we need to check the image width and height to see whether they are not null. Then we need to find the factorX and factorY using the above formula. Finally, we need to map the recognitions using interpolation. Now inside the child, we can define the name of the object and the confidence percentage.

Now that we’ve identified the required functions, we can develop the UI so that our app can actually surface these results to the user:

@override
Widget build(BuildContext context) {
Size size = MediaQuery.of(context).size;

List<Widget> stackChildren = [];

stackChildren.add(Positioned(
top: 0.0,
left: 0.0,
width: size.width,
child: _image == null ? Text("No Image Selected") : Image.file(_image),
));

stackChildren.addAll(renderBoxes(size));

if (_busy) {
stackChildren.add(Center(
child: CircularProgressIndicator(),
));
}

return Scaffold(
appBar: AppBar(
title: Text("Object detetction"),
backgroundColor: Colors.red,
),
floatingActionButton: FloatingActionButton(
child: Icon(Icons.image),
backgroundColor: Colors.red,
tooltip: "Pick an Image",
onPressed: selectFromImagePicker,
),
body: Stack(
children: stackChildren,
),
);
}

Results

Now that we’ve implemented the Flutter application code, let’s look at the output of the application when it’s up and running:

Example

Full code:

Source Code:

ravindu9701/Obj-detect-flutter-tflite

Conclusion

You can use other models that are compatible with TensorFlow Lite such as YOLOv2, v3, v4, Pose_net, and Deeplabv2 to develop your own object detection mobile application. These models have their own perks and accuracy levels which would be beneficial to the goal you're trying to achieve.

That’s all for this article, but in the next article, I’ll try to integrate real-time object detection using the mobile camera.

Editor’s Note: Heartbeat is a contributor-driven online publication and community dedicated to exploring the emerging intersection of mobile app development and machine learning. We’re committed to supporting and inspiring developers and engineers from all walks of life.

Editorially independent, Heartbeat is sponsored and published by Fritz AI, the machine learning platform that helps developers teach devices to see, hear, sense, and think. We pay our contributors, and we don’t sell ads.

If you’d like to contribute, head on over to our call for contributors. You can also sign up to receive our weekly newsletters (Deep Learning Weekly and the Fritz AI Newsletter), join us on Slack, and follow Fritz AI on Twitter for all the latest in mobile machine learning.


Object Detection with Flutter and TensorFlow Lite was originally published in Heartbeat on Medium, where people are continuing the conversation by highlighting and responding to this story.


Image Classification on Mobile with Flutter, TensorFlow Lite, and Teachable Machine

Permalink - Posted on 2020-09-14 13:26

Develop an image classifier mobile application with Flutter, using TensorFlow Lite and Google’s Teachable Machine

In the previous article of this series on developing Flutter applications with TensorFlow Lite, we looked at how we can develop a Digit Recognizer using TensorFlow Lite.

In the second article of the series, we’ll keep working with TensorFlow Lite, this time focusing on implementing image classification to classify images between two classes. The application we are going to build will be able to classify whether an input image contains a horse or a human.

Application and Use Cases

TensorFlow Lite gives us pre-trained and optimized models to identify hundreds of classes of objects, including people, activities, animals, plants, and places. Using Teachable Machine from Google, we can develop our own custom model using some of our own images. Teachable Machine 2.0 allows you to train machine learning models in the browser, without any ML code. You can train models on images, sounds, and poses, and then you can save the models you trained and use them on your own projects.

Required Packages

  • TensorFlow Lite

tflite | Flutter Package

  • Image Picker

image_picker | Flutter Package

  • Horse or Human Dataset

The set contains 500 rendered images of various species of horses, in various poses, and in various locations. It also contains 527 rendered images of humans in various poses and locations:

Horses Or Humans Dataset

This dataset will be used to create the model using the Teachable Machine. First, download the dataset from the above link.

Training a model with Teachable Machine

Before we train our data using the Teachable Machine, we need to gather the required data for the machine to learn. The data collection process hinges on the problem you're trying to solve. For image classification problems, after this data is collected, it needs to be arranged into relevant classes (i.e. labeled). This is where the Teachable Machine comes in.

After we’ve downloaded the required data through the Kaggle, we need to train a TensorFlow Lite model on it.

For this application, we need to divide our dataset into 2 classes. (Horses and Humans). The downloaded dataset from the Kaggle contains two different folders, depending on the class. We need to provide these images folders to the Teachable Machine and label them as follows:

The second step is to train the model, which you can do by clicking the Train Model button. It will take a few minutes to train the model. You can try Advanced settings and give a custom number of epochs (basically, the time to train the model will depend on the number of epochs you define).

After the training is complete, you will be able to click the Export Model button.

Download the model for TensorFlow Lite with the model conversion point Floating Point.

A quick look at all of your options here:

  • Floating Point — Qualified for CPU and GPU (2x smaller, GPU acceleration)
  • Quantized — Qualified for CPU (4x smaller, 2x-3x speedup)
  • EdgeTPU — Qualified for EdgeTPU and Microcontrollers (4x smaller, 3x speedup)

Flutter Application

Now that we have the trained model, we can develop our Flutter application to classify an image as including either a horse or a human. Let’s get started.

First, create an assets folder inside the project folder and copy the downloaded model into that folder.

Since this is a beginner-level application, we can write the code inside the main.dart file without creating another page.

We’ll need to initiate three variables to carry out the application properly. To receive the output, we’ll need a List variable; to get the uploaded image we’ll need a File variable, and to manage errors we’ll need a Boolean variable:

List _outputs;
File _image;
bool _loading = false;

Since the model will also be working offline (one of the benefits of on-device machine learning), we need to load the model when the app is launched:

loadModel() async {
await Tflite.loadModel(
model: "assets/model_unquant.tflite",
labels: "assets/labels.txt",
);
}

Using the function above, we can load the model. To run this function when the application is getting started, we can call this function inside the initState() method:

pickImage() async {
var image = await ImagePicker.pickImage(source: ImageSource.gallery);
if (image == null) return null;
setState(() {
_loading = true;
_image = image;
});
classifyImage(image);
}

With the above function, we can use the ImagePicker to pick an image from the gallery. We can then pass this image on to the classifyImage() function to run it through the classification model:

classifyImage(File image) async {
var output = await Tflite.runModelOnImage(
path: image.path,
numResults: 2,
threshold: 0.5,
imageMean: 127.5,
imageStd: 127.5,
);
setState(() {
_loading = false;
_outputs = output;
});
}

Now that we’ve identified the required functions, we can develop the UI so that our app can actually surface these results to the user:

@override
Widget build(BuildContext context) {
return Scaffold(
appBar: AppBar(
backgroundColor: Colors.red,
title: Text('Horse/Human Detector'),
),
body: _loading
? Container(
alignment: Alignment.center,
child: CircularProgressIndicator(),
)
: Container(
width: MediaQuery.of(context).size.width,
child: Column(
crossAxisAlignment: CrossAxisAlignment.center,
mainAxisAlignment: MainAxisAlignment.center,
children: [
_image == null ? Container() : Image.file(_image),
SizedBox(
height: 20,
),
_outputs != null
? Text(
"${_outputs[0]["label"]}",
style: TextStyle(
color: Colors.black,
fontSize: 20.0,
background: Paint()..color = Colors.white,
),
)
: Container()
],
),
),
floatingActionButton: FloatingActionButton(
onPressed: pickImage,
backgroundColor: Colors.red,
child: Icon(Icons.image),
),
);
}

Results

Now that we’ve implemented the Flutter application code, let’s look at the output of the application when it’s up and running:

Example

Full code:

Source code:

ravindu9701/Image-Classification-Flutter-TensorFlow-Lite

Conclusion

Once you get the hang of it, you can see how easy is it to use TensorFlow Lite with Flutter to develop proof-of-concept machine learning-based mobile applications. Teachable Machine from Google Creative Labs helps us create custom models pretty easily and without much ML expertise. Moreover, you can visit Kaggle and download various datasets to develop different classification models.

Editor’s Note: Heartbeat is a contributor-driven online publication and community dedicated to exploring the emerging intersection of mobile app development and machine learning. We’re committed to supporting and inspiring developers and engineers from all walks of life.

Editorially independent, Heartbeat is sponsored and published by Fritz AI, the machine learning platform that helps developers teach devices to see, hear, sense, and think. We pay our contributors, and we don’t sell ads.

If you’d like to contribute, head on over to our call for contributors. You can also sign up to receive our weekly newsletters (Deep Learning Weekly and the Fritz AI Newsletter), join us on Slack, and follow Fritz AI on Twitter for all the latest in mobile machine learning.


Image Classification on Mobile with Flutter, TensorFlow Lite, and Teachable Machine was originally published in Heartbeat on Medium, where people are continuing the conversation by highlighting and responding to this story.


Digit Recognizer with Flutter and TensorFlow Lite

Permalink - Posted on 2020-08-20 13:44

How to develop a digit recognizer mobile application with Flutter, using TensorFlow Lite and the MNIST dataset

Photo by Andrew Buchanan on Unsplash

Machine learning and AI are taking mobile application development to a new level. Apps that utilizing machine learning can recognize speech, images, and gestures. This gives us new and compelling ways to engage and interact with people in the world around us. But how do we integrate machine learning into our mobile apps?

Developing mobile applications that incorporate machine learning has long been a difficult task. But with the help of platforms and dev tools such as Fritz AI, Firebase’s ML, and TensorFlow Lite, it’s getting easier to do so. These tools provide us with pre-trained machine learning models as well as tools to train and import our own custom models. But how do we actually develop a compelling experience on top of those machine learning models? That’s where Flutter comes in.

The Flutter SDK is a portable UI toolkit built by Google and its open-source community to develop applications for Android, IOS, Web, and Desktop. At its core, Flutter combines a high-performance graphic engine with the Dart programming language. Dart provides both robust type safety and stateful hot reload, which helps developers build reliable apps quickly. Using Flutter, we can build mobile apps with machine learning capabilities like image classification and object detection, for both Android and iOS platforms.

Editor’s note: For a deeper dive into what’s possible with machine learning on mobile, check out our free ebook exploring 14 real-world use cases.

In this article, we’ll combine the power of Flutter and on-device ML to develop a Flutter application that can recognize handwritten digits, using TensorFlow Lite and the famous MNIST dataset.

Required Packages

  • TensorFlow Lite

tflite | Flutter Package

In the android/app/build.gradle file, add the following setting in the android block and change the miniSdkVersion to 19. The installation walkthrough is given in the above Flutter package.

aaptOptions {
noCompress 'tflite'
noCompress 'lite'
}
  • MNIST (Modified National Institute of Standards and Technology database) dataset: One of the largest databases of handwritten digits, commonly used to train image processing systems.

Introduction to CNN Keras - 0.997 (top 6%)

Download the mnist.tflite file from the above link, and put it inside the assets folder of the Flutter project.

Create a mnist.txt file inside the assets folder and add the appropriate Model labels. (0 to 9 numbers)

Now that we’ve taken a quick look at how to set up your Flutter app to run TensorFlow Lite, let’s look at how to get it up and running.

Before we get started, we need to create 4 packages, models, screens, services, and utils.

folder structure

I’ll show the required code snippets for the above files.

In the above code snippet, We use the MNIST dataset of handwritten digits and create an application with a canvas where the user draws the digits.Widget_drawCanvasWidget() is used to determine the size of the drawing canvas. After defining the size of the draft we need to pass the data to using theCustomPaint(painter: DrawingPainter). Through drawing_painter.dart we can get the canvas details which contains properties for the canvas (brush size, color, background-color, etc.).

After the drawing (i.e. after stopping the drawing) the onPanEnd attribute calls the recognize() class and passes the image data. This will be used inside the service -> recognizer.dart.

The Recognizer class will load the mnist.tflite model and the mnist.txt label file. This will then be run through the image data we pass when we draw on the canvas. It’ll check for the similarity of the digit the dataset contains and pass the input through the prediction.dart class to convert the index, confidence, and label to JSON format. Then, we pass this to the prediction_widget.dart to display the most accurate digit.

prediction_widget.dart contains the styles and numbers required to show the predicted digit for the one we drafted as input.

Now that we’ve taken a look inside the Flutter application code, let’s look at the output of the application when it’s up and running.

Example

Source Code:

ravindu9701/digit_recognizer

Conclusion

Once you get the hang of it, you can see how easy is it to use TensorFlow Lite with Flutter to develop proof-of-concept machine learning mobile applications. To improve your knowledge, you can visit the Kaggle site and download various datasets to develop different classification models.

Editor’s Note: Heartbeat is a contributor-driven online publication and community dedicated to exploring the emerging intersection of mobile app development and machine learning. We’re committed to supporting and inspiring developers and engineers from all walks of life.

Editorially independent, Heartbeat is sponsored and published by Fritz AI, the machine learning platform that helps developers teach devices to see, hear, sense, and think. We pay our contributors, and we don’t sell ads.

If you’d like to contribute, head on over to our call for contributors. You can also sign up to receive our weekly newsletters (Deep Learning Weekly and the Fritz AI Newsletter), join us on Slack, and follow Fritz AI on Twitter for all the latest in mobile machine learning.


Digit Recognizer with Flutter and TensorFlow Lite was originally published in Heartbeat on Medium, where people are continuing the conversation by highlighting and responding to this story.


How to Augmentate Data Using Keras

Permalink - Posted on 2020-07-21 16:17

Data augmentation using Python and Keras


Make Your Own Deepfake Video in a Few Easy Steps

Permalink - Posted on 2020-07-13 13:41

A simple guide for creating your own Deepfake video

Photo by Christian Gertenbach on Unsplash

If you’re reading this article, then you’ve likely seen or heard about a particular AI algorithm called deepfakes. Much of this hype has comes as a result of major apps like Snapchat, FaceApp, or Instagram using this algorithm to create entertaining content.

In this article, I’ll show you how to quickly and easily create your own deepfake video. To do this, I’ll be using the implementation for the research effort First Order Motion Model for Image Animation, developed by Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe.

I'll show you how to use their code to create deepfakes using your own pictures and videos.

What is Deepfake?

A deepfake video is a fake photo, video, or story generated by neural networks. Deepfake developers use artificial intelligence and machine learning algorithms to imitate the facial features and body postures of real humans.

These deepfakes diverge from other common forms of fake media by being extremely hard to identify. In addition to their impressive technological underpinnings, deepfakes have the potential to cause enormous damage.

In April 2018, BuzzFeed showcased how far deepfake video technology has come by combining Barack Obama’s face with Jordan Peele’s convincing voice impersonation.

Deepfakes depend on autoencoder neural network type. It contains an encoder, which decreases an image to a lower-dimensional latent space, and a decoder, which reconstructs the image from the latent representation. Deepfakes use this structure by having a universal encoder which encodes a character into the latent space. The latent representation includes key characteristics of their facial features and body posture.

Now that we’ve taken a quick look at deepfakes and some of the controversy surrounding them, let’s look at how to implement our own.

Don’t worry about missing out on the latest ML news. Subscribe and let us find it and share it with you and 14,000 others every week.

For setting up our deepfake, I’ll be using Aliaksandr Siarohin’s Google Colab code.

First, make sure you copy his code to your Google Drive since we don’t want to alter his code:

Google Colaboratory

For a faster compile time go to “Runtime -> Change runtime type -> Select GPU

Before we get started, we need to download a few required files from the Drive link below. You just need to download one .jpeg image, one .mp4 video, a vox-cpk.pth.tar file, and a vox-adv-cpk.pth.tar file. But if you need to check with the images and videos they provided, download everything.

first-order-motion-model - Google Drive

After downloading the files, create a folder named first-order-motion-model in your Google Drive and upload the downloaded files.

To run with your own data…

  • If you want your face to be added to a video, then take a selfie and change its size to 256 X 256 pixels (you can simply use Paint to edit the size). This is the input size that the model accepts.

Upload all your data to the first-order-motion-model folder on your Drive.

To run your face on the videos they’ve provided, change the following code to your file name. Then follow the rest of the steps as presented.

source_image = imageio.imread('/content/gdrive/My Drive/first-order-motion-model/*File Name.png*')
  • If you want someone else’s face to be added to your video file, that file also needs to be on the size of 256 X 256 pixels.

To convert your video into the above size, scroll down to the part of the code labeled “Run on your data”. Remove the boilerplate code provided and add the following code:

!ffmpeg -i /content/gdrive/My\ Drive/first-order-motion-model/*File Name.mp4* -s 256x256 -c:a copy result.mp4

This will create a converted video result.mp4

Copy the result.mp4 file into thefirst-order-motion-model folder and change the following code from “Load driving video and source image”

driving_video = imageio.mimread('/content/first-order-model/result.mp4')

And that’s it. Here is an example I created using Nicolas Cage’s face and the TV show Friends.

Example

I hope this quick start guide to setting up your very own deepfake was useful. Have fun making some funny content (nothing dangerous!). Thank you, and stay safe!

Resources

AliaksandrSiarohin/first-order-model

Editor’s Note: Heartbeat is a contributor-driven online publication and community dedicated to exploring the emerging intersection of mobile app development and machine learning. We’re committed to supporting and inspiring developers and engineers from all walks of life.

Editorially independent, Heartbeat is sponsored and published by Fritz AI, the machine learning platform that helps developers teach devices to see, hear, sense, and think. We pay our contributors, and we don’t sell ads.

If you’d like to contribute, head on over to our call for contributors. You can also sign up to receive our weekly newsletters (Deep Learning Weekly and the Fritz AI Newsletter), join us on Slack, and follow Fritz AI on Twitter for all the latest in mobile machine learning.


Make Your Own Deepfake Video in a Few Easy Steps was originally published in Heartbeat on Medium, where people are continuing the conversation by highlighting and responding to this story.


Using DeOldify to Colorize and Restore Grayscale Images and Videos

Permalink - Posted on 2020-07-06 13:26

Bring back missing colors using DeOldify

Photo by Daniel Curran on Unsplash

Image colorization is an engaging topic in the field of image-to-image translation. Even though color photography was invented in 1907, It didn’t become popular for the average person until the 1960s because of its expensiveness and inaccessibility. All the photography and videography up until then was done on Black & White. Colorizing these images was impossible—until the DeOldify deep learning model came to life.

What is DeOldify?

DeOldify is an open-source deep learning model created by Jason Antic, used to add high-quality colorization to grayscale images with amazing results. Simply put, the goal of this deep learning model is to colorize, restore, and give new life to old images and film footage.

Before we get started on the installation, let’s see an example of how it might look when we use DeOldify to colorize old film footage.

The old version of DeOldify used GANs (Generative Adversarial Networks), a network architecture that includes repetitive interaction between two complementary neural networks: a generator and discriminator. The pre-trained generator model does the task of adding colors to the recognized objects, and the discriminator does the task of “criticizing” the generator’s choice of color.

After the introduction of NoGAN training (covered in more details below) and combining it with GAN training, DeOldify was able to produce more stable and colorful images.

What is NoGAN?

To address stabilization problems in the previous DeOldify model, Jason Antic developed this new type of GAN training, which requires less time on GAN training and more time on pre-training the generator and discriminator separately with uncomplicated, fast, and reliable methods.

Effect of using NoGAN

Now that we’ve taken a quick look at how the inside of DeOldify works, let’s look at how to get it up and running.

Don’t worry about missing out on the latest ML news. Subscribe and let us find it and share it with you and 14,000 others every week.

Setting Up DeOldify

Method 1: Using Google Colab

The easiest and simplest way of trying out DeOldify is to use the Google Colab versions.

  • Image (Artistic)

Google Colaboratory

  • Video (Artistic)

Google Colaboratory

If you’re having trouble with the artistic image colorizer, try the “stable” version below. It usually won’t present colors that are as attractive as on the Artistic version, but the glitches are noticeably reduced.
  • Image (Stable)

Google Colaboratory

Method 2: Using the Jupyter Notebook

To use a Jupyter Notebook, first, you need to clone the DeOldify repository.

git clone https://github.com/jantic/DeOldify.git DeOldify

Next, create a folder called models inside the DeOldify folder and download the pre-trained weights into that folder.

cd DeOldify
mkdir models
  • For Image Colorizer (Artistic):
!wget https://www.dropbox.com/s/zkehq1uwahhbc2o/ColorizeArtistic_gen.pth?dl=0 -O ./models/ColorizeArtistic_gen.pth
  • For Video Colorizer (Artistic):
!wget https://www.dropbox.com/s/336vn9y4qwyg9yz/ColorizeVideo_gen.pth?dl=0 -O ./models/ColorizeVideo_gen.pth
  • For Image Colorizer (Stable):
!wget https://www.dropbox.com/s/mwjep3vyqk5mkjc/ColorizeStable_gen.pth?dl=0 -O ./models/ColorizeStable_gen.pth
If an error throws (!wget), manually download the pre-trained weights using the above links.

After configuring the pre-trained weights, we need to install the required dependencies:

pip install -r requirements.txt

Once the installation is done, open the project using the Jupyter Notebook. All the necessary steps are commented on the .ipynb file and you can continue from there onwards.

Conclusion

I hope this guide to setting up DeOldify on your system was useful. Below is an example of DeOldify that I particularly like. Have fun colorizing your favorite grayscale content. Thank you and stay safe!

The Kid 1921 (Charlie Chaplin Movie) — Colorized

Resources

jantic/DeOldify

Editor’s Note: Heartbeat is a contributor-driven online publication and community dedicated to exploring the emerging intersection of mobile app development and machine learning. We’re committed to supporting and inspiring developers and engineers from all walks of life.

Editorially independent, Heartbeat is sponsored and published by Fritz AI, the machine learning platform that helps developers teach devices to see, hear, sense, and think. We pay our contributors, and we don’t sell ads.

If you’d like to contribute, head on over to our call for contributors. You can also sign up to receive our weekly newsletters (Deep Learning Weekly and the Fritz AI Newsletter), join us on Slack, and follow Fritz AI on Twitter for all the latest in mobile machine learning.


Using DeOldify to Colorize and Restore Grayscale Images and Videos was originally published in Heartbeat on Medium, where people are continuing the conversation by highlighting and responding to this story.


CLAHE and Thresholding in Python

Permalink - Posted on 2020-07-03 16:36

Contrast Limited Adaptive Histogram Equalization and Thresholding using OpenCV in Python