Doing asynchronous work and continuing after getting the result

from the CommonsWare Community archives

At June 6, 2019, 11:11am, Raptor asked:

Hello,

I have a problem that is a little complicated.

I am trying to use an image-processing library called Scanbot to take a picture and from the Bitmap created in onPictureTaken() callback (created from the image byteArray) I want it to send that to make some processing (extract the info in a QR code, process some MRZ information etc) and if the results of that processing are not null, send the picture and the parsed information to the backend.

There are two problems with this:

  1. Once you send the image to the library and then try to use it somewhere else, it throws an exception, that the picture has been recycled (probably the library does this after it processes it). I just had an idea that maybe I should just create a brand new Bitmap from the same byte array that I don’t send to the library, so that it isn’t recycled, and use that one (since it’s the same picture, basically) for my API call.

  2. The other problem is that I am sending the Bitmap to the library for the library to extract stuff from it. The library does this on another thread. The problem is that my method keeps on going, checking if the result is not null, and of course it is null - the other thread hasn’t finished with the processing, yet.

What is a good way to “wait” for the other thread to finish and then check the result? I am working in Kotlin right now, I’ve tried with a @Synchronized method but with no success (not sure exactly how this works in Kotlin and haven’t used synchronized blocks in java until now since I was never in such a scenario, so far).

Here’s how the method looks like:

override fun onPictureTaken(image: ByteArray?, imageOrientation: Int) {

    //obtain the original picture
    var originalBitmap: Bitmap = BitmapFactory.decodeByteArray(image, 0, image!!.size)

    //rotate the image if required
    if (imageOrientation > 0) {
        val matrix: Matrix = Matrix()
        matrix.setRotate(imageOrientation.toFloat(), originalBitmap.width / 2f, originalBitmap.height / 2f)
        originalBitmap = Bitmap.createBitmap(originalBitmap, 0, 0, originalBitmap.width, originalBitmap.height, matrix, false)
    }

    //run contour detection on original image
    val detector: ContourDetector = ContourDetector()
    detector.detect(originalBitmap)

    //crop using the detected polygon in the final document image
    val documentImage: Bitmap = detector.processImageAndRelease(originalBitmap, detector.polygonF, ContourDetector.IMAGE_FILTER_NONE)

    //get back the byte array from the document image
    val size = documentImage.rowBytes * documentImage.height
    val byteBuffer = ByteBuffer.allocate(size)
    documentImage.copyPixelsToBuffer(byteBuffer)
    val documentByteArray = byteBuffer.array()

    val barcodeDetector = scanbotSDK.barcodeDetector()
    val detectedBarcode = barcodeDetector.decodeWithState(documentByteArray, documentImage.width, documentImage.height, 0)
    if (currentDetectionType == DetectionType.DOCUMENT && !isQrObtained) {
        runOnUiThread {
            val gson = Gson()
            try {
                qrDetectionResult = gson.fromJson(detectedBarcode?.text, QrDetectionResult::class.java)
                if (qrDetectionResult == null) {
                    showInvalidQR()
                    restartScan(false)
                    return@runOnUiThread
                }
                if (qrDetectionResult!!.documentTypeCode != null && qrDetectionResult!!.creditApplicationDocId != null &&
                        qrDetectionResult!!.creditApplicationNo != null && qrDetectionResult!!.cnp != null && qrDetectionResult!!.firstName != null && qrDetectionResult!!.lastName != null) {
                        //take snapshot when ready, now that we have the information in the QR
                        isQrObtained = true
                    } else {
                        showInvalidQR()
                    }
                } catch (e: IllegalStateException) {
                    e.printStackTrace()
                    showInvalidQR()
                }
            }
        }
}

The asynchronous call happens here:

val detectedBarcode = barcodeDetector.decodeWithState(documentByteArray, documentImage.width, documentImage.height, 0)

Thank you!

At June 6, 2019, 12:36pm, mmurphy replied:

I am assuming that you are referring to this class. If so… you are sure that’s asynchronous? If so, they do not seem to have any sort of callback or other notification mechanism to let you know when that work is done. That is grounds for termination.

What is a good way to “wait” for the other thread to finish and then check the result?

Use a semi-infinite loop with Thread.sleep() calls. I am not even certain how you can tell when the result is completed, as I cannot see from BarcodeScanningResult how you can distinguish between a failed scan and a scan that is not yet completed. But, let’s pretend that you wrote an extension function called isItSoupYet() that returns true when the scan is completed and false when it is not. You could use something like:

  suspend fun waitForBarcode(detectedBarcode: BarcodeScanningResult): Boolean = withContext(Dispatchers.Default) {
    for (i in 1..300) {
      if (detectedBarcode.isItSoupYet()) { break } else { Thread.sleep(33) }
    }
    
    detectedBarcode.isItSoupYet()
  }

It is possible that there is a more elegant Kotlin construction instead of the for loop, but it will need to amount to the same thing: checking every so often and sleeping for a bit if we are still not done. Eventually (~3 seconds in the code snippet), you just stop checking, as a way of implementing a timeout.

The pseudo-Kotlin shown above uses coroutines. If you are not yet using coroutines, you will need your own “run this on a background thread, then get the results over to this other thread” logic. Frankly, I would recommend just investing in a bit of coroutines rather than trying to roll your own solution.

At June 6, 2019, 1:42pm, Raptor replied:

I actually ended up using LiveData to do it. I then observe the LiveData in my Activity and act accordingly (it still returns null, so that’s a problem):

public void processQrCode(ScanbotBarcodeDetector barcodeDetector, byte[] documentByteArray, Bitmap documentImage, int width, int height, int imageRotation) {
    qrScanLiveData.postValue(Resource.loading());
    try {
        qrScanLiveData.postValue(Resource.success(new Pair<>(barcodeDetector.decodeWithState(documentByteArray, width, height, imageRotation), documentImage)));
    } catch (Exception e) {
        qrScanLiveData.postValue(Resource.error("Invalid QR"));
    }
}

And in my activity I have:

private fun observeQrParsed() {
    //observe the parsing of the QR
    documentDetectorViewModel!!.qrScanLiveData.observe(this, Observer { qrScanResource ->
        if (qrScanResource.state == Resource.State.LOADING) {
            //show a loading progress
            startProgress()
        } else if (qrScanResource.state == Resource.State.ERROR) {
            //hide the loading progress
            stopProgress()
            val snackBarInfo = SnackBarInfo().setMessage("Invalid QR").setType(SnackBarInfo.Type.ERROR)
            SnackBarUtils.showToast(snackBarInfo, rootView!!)
            restartScan(false)
        } else if (qrScanResource.state == Resource.State.SUCCESS) {
            //hide the loading progress
            stopProgress()
            if (currentDetectionType == DetectionType.DOCUMENT && !isQrObtained) {
                runOnUiThread {
                    val gson = Gson()
                    try {
                        qrDetectionResult = gson.fromJson(qrScanResource.data?.first?.text, QrDetectionResult::class.java)
                        if (qrDetectionResult == null) {
                            showInvalidQR()
                            restartScan(false)
                            return@runOnUiThread
                        }
                        if (qrDetectionResult!!.documentTypeCode != null && qrDetectionResult!!.creditApplicationDocId != null &&
                                qrDetectionResult!!.creditApplicationNo != null && qrDetectionResult!!.cnp != null && qrDetectionResult!!.firstName != null && qrDetectionResult!!.lastName != null) {
                            isQrObtained = true
                            documentDetectorViewModel?.sendDocument(DetectionType.DOCUMENT, cnp, qrDetectionResult, qrScanResource.data?.second)
                        } else {
                            showInvalidQR()
                        }
                    } catch (e: IllegalStateException) {
                        e.printStackTrace()
                        showInvalidQR()
                    }
                }
            }
            restartScan(true)
        }
    })
}

It seems like a sensible way to do it (it works with some other similar data).

At June 6, 2019, 1:55pm, mmurphy replied:

That code would only seem to work if decodeWithState() is synchronous. Otherwise, your Resource.success() will be posted before the decode work is completed.

At June 6, 2019, 2:09pm, Raptor replied:

Are you sure? Shouldn’t it wait for the result of the processing?

By the way, I put a breakpoint there and used Evaluate Expression and it returned null. Apparently, it can’t parse the QR for whatever reason.

At June 6, 2019, 2:28pm, mmurphy replied:

I don’t know what success() does, but it is unlikely to have some sort of automatic “hey, figure out when this background thread that we can’t see is completed” logic. If decodeWithState() is synchronous, though, then what you have should be OK.

At June 6, 2019, 2:39pm, Raptor replied:

You’re right, I have put a breakpoint where the evaluation is done and it does return the QR result (with Google Vision, this time) when you do the evaluation, but the posted result in the LiveData is null. Basically, when you evaluate, it does the evaluation and shows you the result.

The question remains - how do I post the success value only when the QR is actually parsed in this context? How do I determine some sort of “hey, the parsing is completed” response?

At June 6, 2019, 2:46pm, Raptor replied:

Looking again at your method (I’m in a Java environment where I do the processing, since I moved the processing to the ViewModel) I think one way I could do it is indeed to make a TimerTask or something that checks every say 200 ms if the barcode is not null, for 3 seconds, say.

If it’s not null, it posts the LiveData success. If it’s still null after the three seconds, it posts an error.

At June 6, 2019, 2:55pm, mmurphy replied:

I don’t know how you are supposed to tell when the result is complete. This is where I have some serious issues with this SDK that you are using. You will need to contact the developers and ask them:

They should be providing you with a callback interface, where you could supply decodeWithState() an implementation of that callback. Or, they could return a Future. Or, they could integrate with RxJava. They seem to be doing none of this, and frankly, that’s awful. So, I would ask them what to do.

Your TimerTask approach could work, as an alternative to my polling loop. Just make sure that you do eventually give up and don’t try polling forever.

At June 6, 2019, 3:11pm, Raptor replied:

I made it work with TimerTask. Yeah, it’s really weird that it doesn’t have a callback, maybe I missed it? I tried to see what methods it has but … maybe there’s some other way to do it, I don’t know.

Here’s what I did to make it work:

qrScanLiveData.postValue(Resource.loading());
    long currentTime = System.currentTimeMillis();
    SparseArray<Barcode> barcodes = barcodeDetector.detect(frame);
    Timer timer = new Timer();
    timer.scheduleAtFixedRate(new TimerTask() {
        @Override
        public void run() {
            if (System.currentTimeMillis() - currentTime > 3000) {
                qrScanLiveData.postValue(Resource.error("Invalid QR"));
                this.cancel();
            }
            if (barcodes.size() > 0 && barcodes.valueAt(0) != null && !barcodes.valueAt(0).rawValue.equals("{}")) {
                qrScanLiveData.postValue(Resource.success(new Pair<>(barcodes.valueAt(0).rawValue, documentImage)));
                this.cancel();
            }
        }
    }, 200, 200);

At June 6, 2019, 3:18pm, Raptor replied:

In their solution they have a BarcodeDetectorFrameHandler which does indeed allow you to receive the result of the processing, but on a live frame (it attaches to the camera).

But in my case, I am processing this in onPictureTaken() so… it’s different.

It looks like this:

cameraView = (ScanbotCameraView) findViewById(R.id.camera);
    cameraView.setCameraOpenCallback(new CameraOpenCallback() {
        @Override
        public void onCameraOpened() {
            cameraView.postDelayed(new Runnable() {
                @Override
                public void run() {
                    cameraView.continuousFocus();
                    cameraView.useFlash(flashEnabled);
                }
            }, 700);
        }
    });

    BarcodeDetectorFrameHandler barcodeDetectorFrameHandler = BarcodeDetectorFrameHandler.attach(cameraView, new ScanbotSDK(this));

    // Default detection interval is 10000 ms
    barcodeDetectorFrameHandler.setDetectionInterval(2000);

    barcodeDetectorFrameHandler.addResultHandler(this);

    findViewById(R.id.flash).setOnClickListener(new View.OnClickListener() {
        @Override
        public void onClick(View v) {
            flashEnabled = !flashEnabled;
            cameraView.useFlash(flashEnabled);
        }
    });

And then, for result handling, you have:

@Override
public boolean handleResult(final BarcodeScanningResult detectedBarcode) {
    runOnUiThread(new Runnable() {
        @Override
        public void run() {
            if (detectedBarcode != null) {

                cameraView.stopPreview();

                final AlertDialog.Builder builder = new AlertDialog.Builder(BarcodeScannerActivity.this);

                builder.setTitle("Result")
                        .setMessage(detectedBarcode.getBarcodeFormat().toString() + "\n\n" + detectedBarcode.getText());

                builder.setPositiveButton("OK", new DialogInterface.OnClickListener() {
                    public void onClick(DialogInterface dialog, int id) {
                        cameraView.continuousFocus();
                        cameraView.startPreview();
                    }
                });

                final AlertDialog dialog = builder.create();
                dialog.show();