Comparing Language Understanding on Dialogflow and NeuralSpace
Continuing from our previous comparison of Indian languages on Dialogflow ES, we at NeuralSpace are here with an expansion to more languages.
In this article, we evaluate the Language Understanding services of Google Cloud’s Dialogflow ES and NeuralSpace, for all languages that had an overlap between Dialogflow’s language support and the Amazon MASSIVE dataset, apart from the Indian languages that we benchmarked on in the previous comparison.
We also do a comparison of both platforms’ language support, feature set, UI and UX. (This section remains the same as the previous benchmarking blog except for the language support).
We wouldn’t like to keep you waiting for the results so let’s get right into it! The table below shows the intent accuracy of Google’s Dialogflow ES and NeuralSpace’s Language Understanding app, on the Amazon MASSIVE dataset. We compare the languages: Afrikaans, Armenian, Azerbaijani, Chinese, Danish, Dutch, English, Finnish, French, German, Greek, Hungarian, Indonesian, Italian, Japanese, Javanese, Korean, Latvian, Norwegian, Polish, Portuguese, Romanian, Russian, Slovenian, Spanish, Swahili, Swedish, Turkish and Vietnamese.
As we had mentioned in the previous post, since the release of the Amazon MASSIVE blog post, NeuralSpace has worked on improving the architecture of the Language Understanding app. This has led to a further improvement in language-wise performance. Exactly as the last time, for our results, we tried to emulate how a customer would use our platform. So, we trained 5 models for each language and picked the best one.
A quick note on intent accuracy for the uninitiated:
The higher the accuracy, the better!
As can be seen above, NeuralSpace performs better for every single language, and on average 6.4% better than Dialogflow at identifying user intents across the languages that we have compared above.
Just like last time, these results highlight only surface-level differences between the two platforms, both feature-wise and usage-wise. Setting up and then using Dialogflow was not super intuitive and there were quite some hoops that we had to pass through.
Let’s take a detailed look at Dialogflow’s features (or a lack thereof) and compare them with NeuralSpace. Apart from the Language Support section, the other feature comparisons are the same as the previous blog in which we had compared the two platforms for Indian Languages.
Right off the bat, you must have noticed that we did not have strict and partial F1 scores in our comparison table above. The reason for that is, entity tagging (where we can input an example sentence, and get the sentence with the entities tagged automatically as output) is not available on Dialogflow. When providing the training examples, the entities can be annotated. But while querying for a test example, only the intent can be obtained via their APIs and the UI.
We, at NeuralSpace though, have support for entity tagging and it can be achieved rather simply by just training the data with the entities annotated.
We suspect that the entities on Dialogflow are not trainable, and use only regex and synonym-based lookups. There are a set of pre-trained entities available though, but that is about it. But due to a lack of explanation in the documentation, this inference is inconclusive.
The NeuralSpace platform supports proper entity recognition including training as well as tagging of entities.
One of NeuralSpace’s highlight features that we would like to keep reminding you of is ‘Click, Train, Chill’! In achieving this, our Import Dataset feature plays a key role.
The Import feature allows you to directly import popular datasets into your projects using a no-code way. Without having to convert the data yourself into the required formats, you can easily import data at the click of a single button! The Amazon MASSIVE dataset, on which we have run this benchmarking comparison, has been available to import to NeuralSpace’s Data Studio since almost as soon as it became public. That’s one extra step less for you! A similar feature is not available on Dialogflow, though.
But even if you have to manually upload a dataset to the NeuralSpace Platform, doing that is not very difficult either. You could upload in multiple file formats and we have even provided examples for all of them that you could refer to!
Supporting Docs and other materials
NeuralSpace has easy to understand documentation that is also very beginner friendly. We have provided quickstarts to get you started in just a few minutes with usage examples. We have also provided Colab Notebooks and Postman Collection where you can try things out without even having to do anything on your local machine. On top of that, we have made a lot of quick and handy tutorial videos to help you navigate around and get things done! We believe that all of this makes the NeuralSpace Platform a lot easier to use compared to Dialogflow.
While NeuralSpace as an NLP platform provides state-of-the-art NLU services which would be the brains of your chatbot (or any other app that you so desire), you would have to use a framework like Rasa (which is open-source) that integrates seamlessly with our APIs. Dialogflow on the other hand though, is the complete solution to build a chatbot, even if with not so smart a brain. 😉 Though, the training time on Dialogflow for NLU is very low, at just a few seconds.
Dialogflow UI and UX hurdles
We have borrowed the best (or in this case the worst?) from the previous blog. We would not shy away from saying that Dialogflow does not really have the most intuitive or the most beginner-friendly UI. The UX story is also similar, with any developer requiring quite some experience with the platform to be able to use it. We have categorized all the pains of using Dialogflow into the following points (there are quite a few, which is why we have had to delay this blog by so much):
Let's start with the docs, which any developer would first start with as well. To put it simply, the docs are complicated. There are 3 sections in the docs, one that is common for both Dialogflow ES and Dialogflow CX, one only for ES, and one only for CX. But there is quite some overlap between all these three sections. There are multiple tabs in each, and multiple sub-sections and pages that one can navigate to from the sidebar. To get a hold of even the structure of the documentation requires deciphering and is a task in itself. If that wasn’t all, weirdly enough, navigating to certain tabs makes the docs switch sections automatically!
After having all of that figured out and going through the setup steps, the quickstarts page shows what the different elements (like entities, intents, etc.) are, and how to use the UI to add examples. Thus, we tried using the UI. But of course, manually creating each example was not a viable option. In the UI, there was an option to export the current training data and to import from a zip file as well. We converted all the required data in a format similar to the export. And then imported the zip with all the training examples, intents and entities. Unfortunately, we were only able to get the intents and the examples for them but not the entities using this method.
There was an entity file upload option on a different page which wanted a separate csv each for all the entity types. But even after uploading all of them we realised that the annotations were missing in the training examples, and the only way to fix them was doing them manually.
Using the APIs
After all this trial and error, we had to reach back again to the docs for a proper solution and a way to upload all the training examples in the correct format. Under the ‘Reference’ tab was the documentation for all the APIs, which we then tried to implement next. But these were no less complex either. For some APIs, to even figure out the required format of the request body json, we had to go into upto 6 interlinked pages for each API! After collecting information from all of those pages, we could finally create our own sample example of the API request body, since none were provided.
We then converted all of the data in these required formats, which were a bit complex and quite unnecessarily so in our opinion.
Using the API was not so straightforward as having the POST/GET request and having created the request body. There were quite a few steps after that:
The APIs first had to be enabled on a project basis from the APIs library page.
Then a key had to be generated from the APIs and Services page.
A service account that had the proper permissions to use the APIs had to be created.
Its keys had to be generated, downloaded and set as environment variable from the IAM and Admin page.
Now, all of these happen from different parts of the Google Cloud console which gets quite messy and you are very likely to get lost unless you are already familiar with it.
A lack of examples caused more trouble here as we had a hard time figuring out how to pass authentication steps while calling the API. It turned out that there was an API explorer on the API page's right-hand side, which only when made full screen, displayed a cURL request. Using this we were finally able to figure things out and get the API running.
All of these things above make Dialogflow as a platform almost unusable for a new user unless they are ready to invest quite some time and effort into it. The only redeeming part is that everything is in some way or the other available in the docs. But you need to be very persistent and keep doing a lot of trial and error, to eventually get things running.
To conclude the comparison between NeuralSpace and Dialogflow, we believe our no-code solution to Language Understanding, as well as our features, are super easy to use and provide a lot of value. On top of that, our supporting documentation is relatively easy to understand.
Combined with our superior model performance across all languages that we tested, which is on average 6.4% better for non-Indian languages and 9.2% better for Indian languages, we believe that we propose a very competitive alternative to Dialogflow, for all your Language Understanding needs.
We hope that this blog has provided you with insights and will help you to choose the best language-agnostic NLU service for your specific use case.
At NeuralSpace, we will be happy to connect with you if you would like a demo or have any questions or feedback to help us improve our product. We aim to provide a powerful resource to accelerate your pipelines and empower the next billion users to use the Internet in the language and mode of their choice. Together we can contribute to the engineering and research community.
Try out NeuralSpace’s Language Understanding service now.
Check out our Documentation to read more about the NeuralSpace Platform and its different services.
Join the NeuralSpace Slack Community to connect with us. Also, receive updates and discuss topics in NLP for low-resource languages with fellow developers and researchers.