Use a Java client

Summary

Java applications can directly access the model-loading services provided by TensorFlow Serving. We need to compile the Java gRPC client code.

Complete example

Here is an example of exporting a model by using Java to access the model: https://github.com/tobegit3hub/deep_recommend_system/tree/master/java_predict_client</a>.

When compiling different models with Maven, only a single Java file needs to be modified. Other external dependencies are already managed and recommended for modification in the project.

Java client Implementation principles

Whether for a server or a client, Java is implemented in projects that are independent of gRPC. The code is at https://github.com/grpc/grpc-java. When Java is used, a gRPC class must be implemented. It is recommended to use Maven to manage dependencies and to add the following dependencies in pom.xml.

<dependency>
  <groupId>io.grpc</groupId>
  <artifactId>grpc-netty</artifactId>
  <version>1.0.0</version>
</dependency>
<dependency>
  <groupId>io.grpc</groupId>
  <artifactId>grpc-protobuf</artifactId>
  <version>1.0.0</version>
</dependency>
<dependency>
  <groupId>io.grpc</groupId>
  <artifactId>grpc-stub</artifactId>
  <version>1.0.0</version>
</dependency>

Given that the use of gRPC also requires the use of Protobuf-generated Java code, if the command-generated Jar file is difficult to manage, a Maven plugin can be used to copy the Proto file to a designated directory, and a Java file will be automatically generated in the Target directory during compilation.

<build>
  <extensions>
    <extension>
      <groupId>kr.motd.maven</groupId>
      <artifactId>os-maven-plugin</artifactId>
      <version>1.4.1.Final</version>
    </extension>
  </extensions>
  <plugins>
    <plugin>
      <groupId>org.xolstice.maven.plugins</groupId>
      <artifactId>protobuf-maven-plugin</artifactId>
      <version>0.5.0</version>
      <configuration>
        <!--
          The version of protoc must match protobuf-java. If you don't depend on
          protobuf-java directly, you will be transitively depending on the
          protobuf-java version that grpc depends on.
        -->
        <protocArtifact>com.google.protobuf:protoc:3.0.0:exe:${os.detected.classifier}</protocArtifact>
        <pluginId>grpc-java</pluginId>
        <pluginArtifact>io.grpc:protoc-gen-grpc-java:1.0.0:exe:${os.detected.classifier}</pluginArtifact>
      </configuration>
      <executions>
        <execution>
          <goals>
            <goal>compile</goal>
            <goal>compile-custom</goal>
          </goals>
        </execution>
      </executions>
    </plugin>
  </plugins>
</build>

Note that we need to add the Proto files for the TensorFlow Serving and TensorFlow project. Since we do not use Bazel compiling, the dependency path of the Proto file must be modified. Please refer to the complete project above.

Build a TensorProto object

The requested interface is defined using Protobuf, but we also need to build the TensorProto object in the Protobuf generation code, which is essentially multidimensional data whose functions can be generated directly in C++ and Python.

Java can define the multidimensional data, after which you can proceed to build by referring to the Stackoverflow answer here: http://stackoverflow.com/questions/39443019/how-can-i-create-tensorproto-for-tensorflow-in-java. The following is a code for building two TensorProtos:

// Generate features TensorProto
float[][] featuresTensorData = new float[][]{
    {10f, 10f, 10f, 8f, 6f, 1f, 8f, 9f, 1f},
    {10f, 10f, 10f, 8f, 6f, 1f, 8f, 9f, 1f},
};
TensorProto.Builder featuresTensorBuilder = TensorProto.newBuilder();
        featuresTensorBuilder.setDtype(org.tensorflow.framework.DataType.DT_FLOAT);

for (int i = 0; i < featuresTensorData.length; ++i) {
    for (int j = 0; j < featuresTensorData[i].length; ++j) {
        featuresTensorBuilder.addFloatVal(featuresTensorData[i][j]);
    }
}

TensorShapeProto.Dim dim1 = TensorShapeProto.Dim.newBuilder().setSize(2).build();
TensorShapeProto.Dim dim2 = TensorShapeProto.Dim.newBuilder().setSize(9).build();
TensorShapeProto shape = TensorShapeProto.newBuilder().addDim(dim1).addDim(dim2).build();
featuresTensorBuilder.setTensorShape(shape);
TensorProto featuresTensorProto = featuresTensorBuilder.build();

Note that Shape and Dtype, in addition to Data, must also be set manually. Otherwise, the server is unable to parse TensorProto into tensor objects.

Read an image file to generate a TensorProto

In scenarios such as image classification, we need to read an image file to generate a TensorProto object before we can request TensorFlow Serving service through gRPC. Here is a Java example where the test supports the JPG and PNG image formats.

Here is a complete example of using the CNN training model and inference where the Java client can directly read local files to request prediction and classification services: https://github.com/tobegit3hub/deep_cnn/tree/master/java_predict_client< /0>.</p>

// Generate image file to array
int[][][][] featuresTensorData = new int[2][32][32][3];

String[] imageFilenames = new String[]{"../data/inference/Mew.png", "../data/inference/Pikachu.png"};

for (int i = 0; i < imageFilenames.length; i++) {

    // Convert image file to multi-dimension array
    File imageFile = new File(imageFilenames[i]);
    try {
        BufferedImage image = ImageIO.read(imageFile);
        logger.info("Start to convert the image: " + imageFile.getPath());

        int imageWidth = 32;
        int imageHeight = 32;
        int[][] imageArray = new int[imageHeight][imageWidth];

        for (int row = 0; row < imageHeight; row++) {
            for (int column = 0; column < imageWidth; column++) {
                imageArray[row][column] = image.getRGB(column, row);

                int pixel = image.getRGB(column, row);
                int red = (pixel >> 16) & 0xff;
                int green = (pixel >> 8) & 0xff;
                int blue = pixel & 0xff;

                featuresTensorData[i][row][column][0] = red;
                featuresTensorData[i][row][column][1] = green;
                featuresTensorData[i][row][column][2] = blue;
            }
        }
    } catch (IOException e) {
        logger.log(Level.WARNING, e.getMessage());
        System.exit(1);
    }
}

// Generate features TensorProto
TensorProto.Builder featuresTensorBuilder = TensorProto.newBuilder();

for (int i = 0; i < featuresTensorData.length; ++i) {
    for (int j = 0; j < featuresTensorData[i].length; ++j) {
        for (int k = 0; k < featuresTensorData[i][j].length; ++k) {
            for (int l = 0; l < featuresTensorData[i][j][k].length; ++l) {
                featuresTensorBuilder.addFloatVal(featuresTensorData[i][j][k][l]);
            }
        }
    }
}

TensorShapeProto.Dim featuresDim1 = TensorShapeProto.Dim.newBuilder().setSize(2).build();
TensorShapeProto.Dim featuresDim2 = TensorShapeProto.Dim.newBuilder().setSize(32).build();
TensorShapeProto.Dim featuresDim3 = TensorShapeProto.Dim.newBuilder().setSize(32).build();
TensorShapeProto.Dim featuresDim4 = TensorShapeProto.Dim.newBuilder().setSize(3).build();

TensorShapeProto featuresShape = TensorShapeProto.newBuilder().addDim(featuresDim1).addDim(featuresDim2).addDim(featuresDim3).addDim(featuresDim4).build();
featuresTensorBuilder.setDtype(org.tensorflow.framework.DataType.DT_FLOAT).setTensorShape(featuresShape);
TensorProto featuresTensorProto = featuresTensorBuilder.build();